WO2024041479A1 - 一种数据处理方法及其装置 - Google Patents

一种数据处理方法及其装置 Download PDF

Info

Publication number
WO2024041479A1
WO2024041479A1 PCT/CN2023/114002 CN2023114002W WO2024041479A1 WO 2024041479 A1 WO2024041479 A1 WO 2024041479A1 CN 2023114002 W CN2023114002 W CN 2023114002W WO 2024041479 A1 WO2024041479 A1 WO 2024041479A1
Authority
WO
WIPO (PCT)
Prior art keywords
text
image
feature
sub
feature representation
Prior art date
Application number
PCT/CN2023/114002
Other languages
English (en)
French (fr)
Inventor
刘志广
柏昊立
孟笑君
李文涛
谢念
王靓伟
侯璐
蒋欣
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Publication of WO2024041479A1 publication Critical patent/WO2024041479A1/zh

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T11/00—2D [Two Dimensional] image generation
    • G06T11/60—Editing figures and text; Combining figures or text
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T9/00—Image coding
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
    • G06V10/761—Proximity, similarity or dissimilarity measures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
    • G06V30/10—Character recognition
    • G06V30/14—Image acquisition
    • G06V30/148—Segmentation of character regions
    • G06V30/153—Segmentation of character regions using recognition of characters or words
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
    • G06V30/10—Character recognition
    • G06V30/19—Recognition using electronic means
    • G06V30/19007—Matching; Proximity measures
    • G06V30/19093—Proximity measures, i.e. similarity or distance measures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]

Definitions

  • This application relates to the field of artificial intelligence, and in particular, to a data processing method and its device.
  • Artificial Intelligence is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.
  • artificial intelligence is a branch of computer science that attempts to understand the nature of intelligence and produce a new class of intelligent machines that can respond in a manner similar to human intelligence.
  • Artificial intelligence is the study of the design principles and implementation methods of various intelligent machines, so that the machines have the functions of perception, reasoning and decision-making.
  • the existing cross-modal visual language models (or can be called multi-modal models or multi-modal language models) have weak text modeling capabilities in the alignment method.
  • TIA simply determines whether a certain character is occluded, and does not take into account the semantic information between the occluded characters, resulting in the model not truly understanding the content of the document. .
  • the current multi-modal model has weak image and text alignment capabilities, that is, it does not have the ability to align text and image elements.
  • TIM text-image matching
  • the training goal is to determine whether the global text matches the global scanned document image. This task is difficult to learn, so the trained model does not have the alignment of text and image elements. ability.
  • This application provides a data processing method that can make the features extracted by the image encoder and text encoder contain more accurate semantic information of the text in the image, thereby enhancing the image-text matching ability of the subsequent network.
  • the tower method can improve the alignment ability between images and text, and then use a single tower structure to further enhance the interactive learning ability of features.
  • this application provides a data processing method, including: obtaining a first feature representation and a second feature representation.
  • the first feature representation is an image feature obtained by processing an image according to an image encoder.
  • the second feature representation is the text feature of the first text; the first text is the text content included in the image; according to the first feature representation and the second feature representation, the third feature representation is obtained through the target encoder; the target encoder is used to implement the feature representation between interaction; execute the downstream task according to the third feature representation and obtain the execution result; the similarity between the execution result and the corresponding annotation and the similarity between the first feature representation and the second feature representation are used to update the image encoder.
  • text-image contrastive learning is used to enhance the matching ability between pictures and texts. Specifically, based on the image features of the image and the text features of the text, it is constructed through similarity comparison. Loss to update the graphics encoder (if the text features are obtained through the text encoder, the text encoder can also be updated). Although the image features of the image and the text features of the text in the image are not features of the same modality, since the text is the text in the image, the image features also contain certain semantic information of the text. Therefore, image features and text features (belonging to the same image) will be related in the semantic dimension (or other information dimensions). The above similarity can include the similarity between the semantic information in the image and the semantic information of the text.
  • the architecture of the cross-modal language model includes a graphic encoder, a text encoder, and a target encoder used to provide interactive information between features.
  • the intermediate output of the model that is, the graphic encoder and the output of the text encoder
  • the feature representation based on the output of the graphic encoder and the text encoder can achieve downstream tasks (the accuracy is not based on the target encoder
  • the output feature representation is used for downstream tasks (high), and in some scenarios, due to the large amount of data to be processed by the downstream tasks, the feature representation output by the graphics encoder and text encoder can be used for rough sorting. Fine ranking is performed using the feature representation output by the target encoder to improve recall.
  • the second feature representation is a text feature obtained by processing the first text through a text encoder; the similarity between the first feature representation and the second feature representation is also used to update the text encoder.
  • the similarity is related to the similarity between the text semantic information contained in the image and the semantic information of the text itself.
  • the similarity between the execution result and the corresponding annotation is also used to update the target encoder.
  • the first text is all text contained in the image; or,
  • the first text is part of the total text contained in the image.
  • the image is a partial image area extracted from the original input image, and the text included in the image is part of the text included in the input image; or,
  • the image is the original input image.
  • the image may be the original input image (or a partial image of the original input image, but the partial image contains the entire text of the input image).
  • the first text can be all text contained in the image.
  • the image may be the original input image (or a partial image of the original input image, but the partial image contains the entire text of the input image).
  • the first text may be part of the text contained in the image.
  • the image may be a partial image area obtained by extracting the original input image.
  • the partial image area includes partial text of all the texts in the original input image, and the first text may be a portion of the text included in the image. All text.
  • the image may be a partial image area obtained by extracting the original input image.
  • the partial image area includes part of all text in the original input image, and the first text may be a part of the text contained in the image. Part of the text within the text.
  • the input original image may include one or more lines of text, or the input original image may include one or more columns of text.
  • the image may include a row or column of text of the original image, and the first text may be one or more text units.
  • the loss used to update the graphic encoder and the text encoder can be constructed based on the similarity between the image features of the image and the text features of the first text.
  • the similarity is the same as that of the image. It is related to the similarity between the text semantic information contained in the text and the semantic information of the text itself.
  • the loss used to update the graphics encoder as well as the text encoder can also be constructed based on the similarity between the image features of the image and the text features of the text not included in the image.
  • the difference is that the image features of the image and the image Chinese
  • the text features of this image are positive examples (that is, the corresponding annotation similarity is high), and the image features of the image and the text features of the text not included in the image are negative examples (that is, the corresponding annotation similarity is low).
  • the image features corresponding to the entire input image can be compared with the text features of all texts included in the input image, or partial images in the input image can be compared.
  • the image features of the region are compared with the text features of part of the text included in the input image, or the image features corresponding to the entire input image can be compared with the text features of part of the text included in the input image, or one of the above methods The combination.
  • the text features corresponding to all the texts included in the input image can be compared with the image features of the partial image areas in the input image, or the input
  • the text features of the partial text included in the image are compared with the image features of the partial image area in the input image.
  • the text features corresponding to the entire text included in the input image can also be compared with the image features corresponding to the entire input image.
  • the text features corresponding to the entire text included in a part of the input image can be compared with the image features of part of the image area in the input image, or a part of the input image can be compared Text features of part of the text included in the input image are compared with image features of part of the image area in the input image, or text features corresponding to the entire text included in part of the input image can be compared with image features corresponding to the entire input image. Comparison, or a combination of the above methods.
  • all text included in a part of the input image can be The corresponding text features are compared with the image features of the partial image area in the input image, or the text features of the partial text included in a part of the input image are compared with the image features of the partial image area in the input image, or Text features corresponding to all texts included in a part of the input image are compared with image features corresponding to the entire input image.
  • the first text includes a first sub-text and a second sub-text
  • the second feature representation includes a first sub-feature corresponding to the first sub-text, and a second sub-feature corresponding to the second sub-text
  • the first sub-feature does not contain the position of the first sub-text in the image
  • the second sub-feature contains the position of the second sub-text in the image
  • perform downstream tasks including: predict according to the third feature representation
  • the first position of the first sub-text in the image; the similarity between the first position and the corresponding annotation is used to update the image encoder and the target encoder.
  • the image includes multiple image blocks; the first position is the image block where the first sub-text is predicted to be located; and the image block is marked as the actual location of the first sub-text.
  • This application proposes BGM in the single tower part to improve spatial layout perception, which helps improve the performance of the model in position-sensitive tasks. For example, in information extraction, key-value pairs are usually adjacent pairs of data. Because the model has learned the ability to perceive position and layout information in the pre-training stage, the model will have good performance in tasks such as information extraction.
  • the first feature representation includes a third sub-feature and a fourth sub-feature; the method further includes: based on the fourth sub-feature, the second position of the fourth sub-feature in the first feature representation, and the third sub-feature.
  • the third position of the three sub-features in the first feature representation is used to obtain the feature prediction value at the third position; the similarity between the feature prediction value and the third sub-feature is used to update the image encoder.
  • the third position is obtained
  • the feature prediction value includes: based on the fourth sub-feature, the second position of the fourth sub-feature in the first feature representation, and the third position of the third sub-feature in the first feature representation, through the self-attention network, The fourth feature representation is obtained; according to the fourth feature representation, through the prediction network, the feature prediction value at the third position is obtained.
  • a loss can also be constructed based on image features. Specifically, parts of the image can be masked, and the results obtained by the image encoder can be used to mask the area of the image. Image reconstruction is performed and the network is updated based on the difference between the reconstruction result and the pixel values of the real image in the masked area.
  • the image representation learning ability of existing methods is weak. Specifically, in order to reduce the processing overhead of the model, the image needs to be compressed to a smaller resolution (such as 224*224), and the image contains dense text. In this case, the pixels in the image are difficult to express accurate text information, and the image representation learning ability of the trained model is limited.
  • the image features of the image may not be masked, and the image features of the masked area may be reconstructed based on the reconstruction results and the image of the masked area.
  • the difference between features is used to update the network. Since image features can carry richer image information and text information than pixel values themselves, it can improve the learning ability of image representation of the trained network.
  • the above method can be called masking.
  • Image modeling mask image modeling, MIM).
  • this application provides a data processing method, which includes:
  • the image is processed to obtain the first feature representation
  • the text contained in the image is processed to obtain the second feature representation
  • the third feature representation is obtained through the target encoder; the target encoder is used to realize the interaction between feature representations;
  • the downstream task is executed and the execution result is obtained.
  • this application provides a data processing device, including:
  • the acquisition module is used to obtain a first feature representation and a second feature representation.
  • the first feature representation is an image feature obtained by processing an image according to an image encoder.
  • the second feature representation is a text feature of the first text; the first text is the text content included in the image; the similarity between the first feature representation and the second feature representation is used to update the image encoder;
  • the encoding module is used to obtain the third feature representation through the target encoder according to the first feature representation and the second feature representation; the target encoder is used to realize the interaction between the feature representations;
  • the task execution module is used to execute downstream tasks according to the third feature representation and obtain execution results.
  • the second feature representation is a text feature obtained by processing the first text through a text encoder; the similarity between the first feature representation and the second feature representation is also used to update the text encoder.
  • the similarity is related to the similarity between the text semantic information contained in the image and the semantic information of the text itself.
  • the first text is all text contained in the image; or,
  • the first text is part of the total text contained in the image.
  • the image is a partial image area extracted from the original input image, and the text included in the image is part of the text included in the input image; or,
  • the image is the original input image.
  • the first text includes a first sub-text and a second sub-text
  • the second feature representation includes a first sub-feature corresponding to the first sub-text, and a second sub-feature corresponding to the second sub-text
  • the first sub-feature does not include the position of the first sub-text in the image
  • the second sub-feature includes the position of the second sub-text in the image
  • Task execution module specifically used for:
  • the first position of the first sub-text in the image is predicted; the similarity between the first position and the corresponding annotation is used to update the image encoder and the target encoder.
  • the image includes multiple image blocks; the first position is the image block where the first sub-text is predicted to be located; and the image block is marked as the actual location of the first sub-text.
  • the first feature representation includes a third sub-feature and a fourth sub-feature; the method further includes:
  • a prediction module configured to obtain a feature prediction at the third position based on the fourth sub-feature, the second position of the fourth sub-feature in the first feature representation, and the third position of the third sub-feature in the first feature representation. value; the similarity between the feature prediction value and the third sub-feature is used to update the image encoder.
  • the prediction module is specifically used to:
  • the fourth feature representation is obtained through the self-attention network
  • the feature prediction value at the third position is obtained.
  • this application provides a data processing device, which includes:
  • Acquisition module used to obtain images
  • the encoding module is used to process the image through the image encoder to obtain the first feature representation
  • the text contained in the image is processed to obtain the second feature representation
  • the third feature representation is obtained through the target encoder; the target encoder is used to realize the interaction between feature representations;
  • the task execution module is used to execute downstream tasks according to the third feature representation and obtain execution results.
  • a training device which may include a memory, a processor, and a bus system.
  • the memory is used to store programs
  • the processor is used to execute programs in the memory to perform the above-mentioned first aspect and any of its optional methods.
  • embodiments of the present application provide an execution device, which may include a memory, a processor, and a bus system, wherein the memory is used to store programs, and the processor is used to execute programs in the memory to perform the above-mentioned second aspect and any of its optional methods.
  • embodiments of the present application provide a computer-readable storage medium that stores a computer program that, when run on a computer, causes the computer to execute the first aspect and any one of the above-mentioned aspects.
  • embodiments of the present application provide a computer program that, when run on a computer, causes the computer to execute the above-mentioned first aspect and any optional method thereof, as well as the above-mentioned second aspect and any optional method thereof. Methods.
  • the present application provides a chip system, which includes a processor for supporting the execution of a data processing device to implement the functions involved in the above aspects, for example, sending or processing data involved in the above methods; or ,information.
  • the chip system further includes a memory, and the memory is used to store necessary program instructions and data for executing the device or training the device.
  • the chip system may be composed of chips, or may include chips and other discrete devices.
  • Figure 1A is a schematic structural diagram of the main framework of artificial intelligence
  • FIGS. 1B and Figure 1C are schematic diagrams of the application system framework of the present invention.
  • Figure 1D is a schematic diagram of an optional hardware structure of the terminal
  • Figure 2 is a schematic structural diagram of a server
  • Figure 3 is a schematic diagram of a system architecture of this application.
  • Figure 4 shows the process of a cloud service
  • Figure 5 shows the process of a cloud service
  • Figure 6 is a schematic diagram of a system architecture of this application.
  • Figure 7 is a schematic flowchart of a data processing method provided by an embodiment of the present application.
  • FIGS. 8 to 10 are flow diagrams of a data processing method provided by embodiments of the present application.
  • Figure 11 is a schematic diagram of a system architecture of this application.
  • Figure 12 is a schematic structural diagram of a data processing device provided by an embodiment of the present application.
  • Figure 13 is a schematic structural diagram of an execution device provided by an embodiment of the present application.
  • Figure 14 is a schematic structural diagram of the training equipment provided by the embodiment of the present application.
  • Figure 15 is a schematic structural diagram of a chip provided by an embodiment of the present application.
  • the terms “substantially”, “about” and similar terms are used as terms of approximation, not as terms of degree, and are intended to take into account measurements or values that would be known to one of ordinary skill in the art. The inherent bias in calculated values. Furthermore, in elaborating The use of “may” when referring to the embodiments of the present invention refers to “one or more possible embodiments”. As used herein, the terms “use”, “using”, and “used” may be deemed to be the same as the terms “utilize”, “utilizing”, and “utilize”, respectively. Synonymous with “utilized”. Additionally, the term “exemplary” is intended to refer to an example or illustration.
  • Figure 1A shows a structural schematic diagram of the artificial intelligence main framework.
  • the following is from the “intelligent information chain” (horizontal axis) and “IT value chain” ( The above artificial intelligence theme framework is elaborated on the two dimensions of vertical axis).
  • the "intelligent information chain” reflects a series of processes from data acquisition to processing. For example, it can be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, the data has gone through the condensation process of "data-information-knowledge-wisdom".
  • the "IT value chain” reflects the value that artificial intelligence brings to the information technology industry, from the underlying infrastructure of human intelligence and information (providing and processing technology implementation) to the systematic industrial ecological process.
  • Infrastructure provides computing power support for artificial intelligence systems, enables communication with the external world, and supports it through basic platforms.
  • computing power is provided by smart chips (hardware acceleration chips such as CPU, NPU, GPU, ASIC, FPGA, etc.);
  • the basic platform includes distributed computing framework and network and other related platform guarantees and support, which can include cloud storage and Computing, interconnection networks, etc.
  • sensors communicate with the outside world to obtain data, which are provided to smart chips in the distributed computing system provided by the basic platform for calculation.
  • Data from the upper layer of the infrastructure is used to represent data sources in the field of artificial intelligence.
  • the data involves graphics, images, voice, and text, as well as IoT data of traditional devices, including business data of existing systems and sensory data such as force, displacement, liquid level, temperature, and humidity.
  • Data processing usually includes data training, machine learning, deep learning, search, reasoning, decision-making and other methods.
  • machine learning and deep learning can perform symbolic and formal intelligent information modeling, extraction, preprocessing, training, etc. on data.
  • Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formal information to perform machine thinking and problem solving based on reasoning control strategies. Typical functions are search and matching.
  • Decision-making refers to the process of decision-making after intelligent information is reasoned, and usually provides functions such as classification, sorting, and prediction.
  • some general capabilities can be formed based on the results of further data processing, such as algorithms or a general system, such as translation, text analysis, computer vision processing, speech recognition, and image processing. identification, etc.
  • Intelligent products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of overall artificial intelligence solutions, productizing intelligent information decision-making and realizing practical applications. Its application fields mainly include: intelligent terminals, intelligent transportation, Smart healthcare, autonomous driving, smart cities, etc.
  • This application can be applied to the field of natural language processing in the field of artificial intelligence. Taking natural language processing as an example, multiple application scenarios that have been implemented into products will be introduced below.
  • This application can be, but is not limited to, applied to applications with text processing functions for images (hereinafter referred to as cross-modal language processing applications) or cloud services provided by cloud-side servers, etc. , which will be introduced separately below:
  • the product form of the embodiment of the present application may be a cross-modal language processing application.
  • Cross-modal language processing applications can run on terminal devices or cloud-side servers.
  • a cross-modal language processing application can implement text-related processing tasks in the image based on the input image and obtain the processing results.
  • the processing tasks related to the text in the above images may include information extraction, contract review, retrieval and question answering, etc. in information
  • users can customize the fields of interest in the image, such as Party A, Party B, contract number, etc.
  • relevant personnel can confirm whether different versions of the contract are consistent, such as whether a change in Party A's name will introduce risk clauses, etc.
  • retrieval and question-and-answer scenarios it can help users quickly retrieve relevant text in images by asking questions or keywords, and find possible answers in the text.
  • the aforementioned images are images collected by devices such as cameras, printers, and scanners.
  • devices such as cameras, printers, and scanners.
  • companies need to scan documents such as receipts or invoices to obtain image files, and recognize the text in the image files to extract text information.
  • This enables functions such as digital file archiving, rapid file indexing, or file analysis.
  • the user needs to input the information on the ID card, driver's license, driving license or passport. The user can use the camera to capture the image of the aforementioned document, and recognize the text in the image to extract Key information, etc.
  • the user can open a cross-modal language processing application installed on the terminal device and input an image.
  • the cross-modal language processing application can be trained through the method provided by the embodiments of this application.
  • the cross-modal language model processes the image and presents the processing results to the user (the presentation method can be but is not limited to display, save, upload to the cloud, etc.).
  • the user can open a cross-modal language processing application installed on the terminal device and input an image.
  • the cross-modal language processing application can send the image to a server on the cloud side.
  • the cloud The server on the side processes the image through the cross-modal language model trained by the method provided by the embodiment of the application, and transmits the processing result back to the terminal device.
  • the terminal device can present the processing result to the user (the presentation method can be but not limited to Display, save, upload to the cloud, etc.).
  • Figure 1B is a schematic diagram of the functional architecture of a cross-modal language processing application in an embodiment of the present application:
  • the cross-modal language processing application 102 can receive input parameters 101 (for example, including images) and generate processing results 103 .
  • the cross-modal language processing application 102 may execute, for example, on at least one computer system and includes computer code that, when executed by one or more computers, causes the computers to perform execution The cross-modal language model trained by the method provided in the embodiment of this application.
  • Figure 1C is a schematic diagram of the entity architecture for running a cross-modal language processing application in an embodiment of the present application:
  • FIG. 1C shows a schematic diagram of a system architecture.
  • the system may include a terminal 100 and a server 200.
  • the server 200 may include one or more servers (one server is used as an example in FIG. 1C for illustration), and the server 200 may provide cross-modal language processing function services for one or more terminals.
  • a cross-modal language processing application can be installed on the terminal 100, or a web page related to the cross-modal language processing function can be opened.
  • the above-mentioned application program and web page can provide an interface, and the terminal 100 can receive the user's cross-modal language processing function.
  • the relevant parameters entered on the state language processing function interface are sent to the server 200.
  • the server 200 can obtain the processing results based on the received parameters and return the processing results to the terminal 100.
  • the terminal 100 can also complete the action of obtaining the processing result based on the received parameters by itself without requiring the cooperation of the server, which is not limited by the embodiments of this application.
  • the terminal 100 in the embodiment of the present application can be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR)/virtual reality (VR) device, a notebook computer, or an ultra mobile personal computer (ultra mobile personal computer).
  • - mobile personal computer UMPC
  • netbook personal digital assistant
  • PDA personal digital assistant
  • FIG. 1D shows an optional hardware structure diagram of the terminal 100.
  • the terminal 100 may include a radio frequency unit 110, a memory 120, an input unit 130, a display unit 140, a camera 150 (optional), an audio circuit 160 (optional), a speaker 161 (optional), Microphone 162 (optional), processor 170, external interface 180, power supply 190 and other components.
  • a radio frequency unit 110 may include a radio frequency unit 110, a memory 120, an input unit 130, a display unit 140, a camera 150 (optional), an audio circuit 160 (optional), a speaker 161 (optional), Microphone 162 (optional), processor 170, external interface 180, power supply 190 and other components.
  • FIG. 1D is only an example of a terminal or a multi-function device, and It is not intended to limit the terminal or multi-function device, and may include more or fewer components than shown in the figures, or combine certain components, or use different components.
  • the input unit 130 may be used to receive input numeric or character information and generate key signal input related to user settings and function control of the portable multi-function device.
  • the input unit 130 may include a touch screen 131 (optional) and/or other input devices 132.
  • the touch screen 131 can collect the user's touch operations on or near it (such as the user's operations on or near the touch screen using fingers, knuckles, stylus, or any other suitable objects), and drive the corresponding according to the preset program. Connect the device.
  • the touch screen can detect the user's touch action on the touch screen, convert the touch action into a touch signal and send it to the processor 170, and can receive and execute commands from the processor 170; the touch signal at least includes contact point coordinate information.
  • the touch screen 131 can provide an input interface and an output interface between the terminal 100 and the user.
  • touch screens can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic wave.
  • the input unit 130 may also include other input devices.
  • other input devices 132 may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys 132, switch keys 133, etc.), trackball, mouse, joystick, etc.
  • the input device 132 may receive input images and the like.
  • the display unit 140 may be used to display information input by the user or information provided to the user, various menus of the terminal 100, interactive interfaces, file display, and/or playback of any kind of multimedia files.
  • the display unit 140 may be used to display the interface, processing results, etc. of a cross-modal language processing application program.
  • the memory 120 can be used to store instructions and data.
  • the memory 120 can mainly include a storage instruction area and a storage data area.
  • the storage data area can store various data, such as multimedia files, text, etc.;
  • the storage instruction area can store operating systems, applications, at least Software units such as instructions required for a function, or their subsets or extensions.
  • Non-volatile random access memory may also be included; providing the processor 170 with management of hardware, software and data resources in the computing processing device and supporting control software and applications. It is also used for storage of multimedia files, as well as storage of running programs and applications.
  • the processor 170 is the control center of the terminal 100. It uses various interfaces and lines to connect various parts of the entire terminal 100, and executes various functions of the terminal 100 by running or executing instructions stored in the memory 120 and calling data stored in the memory 120. functions and process data to provide overall control of the terminal device.
  • the processor 170 may include one or more processing units; preferably, the processor 170 may integrate an application processor and a modem processor, where the application processor mainly processes operating systems, user interfaces, application programs, etc. , the modem processor mainly handles wireless communications. It can be understood that the above-mentioned modem processor may not be integrated into the processor 170 .
  • the processor and memory can be implemented on a single chip, and in some embodiments, they can also be implemented on separate chips.
  • the processor 170 can also be used to generate corresponding operation control signals, send them to corresponding components of the computing processing device, read and process data in the software, especially read and process the data and programs in the memory 120, so that the Each functional module performs a corresponding function, thereby controlling the corresponding components to act according to the instructions.
  • the memory 120 can be used to store software codes related to the data processing method, and the processor 170 can execute the steps of the data processing method of the chip, and can also schedule other units (such as the above-mentioned input unit 130 and the display unit 140) to implement corresponding functions. .
  • the radio frequency unit 110 (optional) can be used to send and receive information or receive and send signals during calls. For example, after receiving downlink information from the base station, it is processed by the processor 170; in addition, the designed uplink data is sent to the base station.
  • RF circuits include but are not limited to antennas, at least one amplifier, transceivers, couplers, low noise amplifiers (LNA), duplexers, etc.
  • the radio frequency unit 110 can also communicate with network devices and other devices through wireless communication.
  • the wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile communication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (Code Division) Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), etc.
  • GSM Global System of Mobile communication
  • GPRS General Packet Radio Service
  • CDMA Code Division Multiple Access
  • WCDMA Wideband Code Division Multiple Access
  • LTE Long Term Evolution
  • SMS Short Messaging Service
  • the radio frequency unit 110 can send the image to the server 200 and receive the processing result sent by the server 200.
  • radio frequency unit 110 is optional and can be replaced by other communication interfaces, such as a network port.
  • the terminal 100 also includes a power supply 190 (such as a battery) that supplies power to various components.
  • a power supply 190 such as a battery
  • the power supply can be logically connected to the processor 170 through a power management system, so that functions such as charging, discharging, and power consumption management can be implemented through the power management system.
  • the terminal 100 also includes an external interface 180, which can be a standard Micro USB interface or a multi-pin connector, which can be used to connect the terminal 100 to communicate with other devices, or can be used to connect a charger to charge the terminal 100. .
  • an external interface 180 can be a standard Micro USB interface or a multi-pin connector, which can be used to connect the terminal 100 to communicate with other devices, or can be used to connect a charger to charge the terminal 100.
  • the terminal 100 may also include a flash light, a wireless fidelity (WiFi) module, a Bluetooth module, sensors with different functions, etc., which will not be described again here. Some or all of the methods described below may be applied in the terminal 100 as shown in FIG. 1D.
  • WiFi wireless fidelity
  • Bluetooth Bluetooth
  • FIG. 2 provides a schematic structural diagram of a server 200.
  • the server 200 includes a bus 201, a processor 202, a communication interface 203 and a memory 204.
  • the processor 202, the memory 204 and the communication interface 203 communicate through the bus 201.
  • the bus 201 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc.
  • PCI peripheral component interconnect
  • EISA extended industry standard architecture
  • the bus can be divided into address bus, data bus, control bus, etc. For ease of presentation, only one thick line is used in Figure 2, but it does not mean that there is only one bus or one type of bus.
  • the processor 202 may be a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP) or a digital signal processor (DSP). any one or more of them.
  • CPU central processing unit
  • GPU graphics processing unit
  • MP microprocessor
  • DSP digital signal processor
  • Memory 204 may include volatile memory, such as random access memory (RAM).
  • RAM random access memory
  • the memory 204 may also include non-volatile memory (non-volatile memory), such as read-only memory (ROM), flash memory, mechanical hard drive (hard drive drive, HDD) or solid state drive (solid state drive). , SSD).
  • ROM read-only memory
  • HDD hard drive drive
  • SSD solid state drive
  • the memory 204 can be used to store software codes related to the data processing method, and the processor 202 can execute the steps of the data processing method of the chip, and can also schedule other units to implement corresponding functions.
  • the terminal 100 and the server 200 may be centralized or distributed devices, and the processors (such as the processor 170 and the processor 202) in the terminal 100 and the server 200 may be hardware circuits (such as application specific integrated circuits) application specific integrated circuit (ASIC), field-programmable gate array (FPGA), general-purpose processor, digital signal processing (DSP), microprocessor or microcontroller, etc.), Or a combination of these hardware circuits.
  • the processor can be a hardware system with the function of executing instructions, such as CPU, DSP, etc., or a hardware system without the function of executing instructions, such as ASIC, FPGA, etc., or the above-mentioned processor without the function of executing instructions.
  • the steps related to the model reasoning process in the embodiments of the present application involve AI-related operations.
  • the instruction execution architecture of the terminal device and server is not limited to the processor-memory architecture introduced above.
  • the system architecture provided by the embodiment of the present application will be introduced in detail below with reference to Figure 3 .
  • FIG 3 is a schematic diagram of the system architecture provided by an embodiment of the present application.
  • the system architecture 500 includes an execution device 510, a training device 520, a database 530, a client device 540, a data storage system 550 and a data collection system 560.
  • the execution device 510 includes a computing module 511, an I/O interface 512, a preprocessing module 513 and a preprocessing module 514.
  • the target model/rule 501 may be included in the calculation module 511, and the preprocessing module 513 and the preprocessing module 514 are optional.
  • the execution device 510 may be the above-mentioned terminal device or server that runs the cross-modal language processing application program.
  • Data collection device 560 is used to collect training samples. Training samples can be multiple images, etc. After collecting the training samples, the data collection device 560 stores the training samples into the database 530 .
  • the training device 520 can maintain the training samples in the database 530 and the neural network to be trained (such as the cross-modal language model in the embodiment of the present application (for example, including a text encoder, an image encoder, a target encoder, etc.)) to obtain Target Model/Rule 501.
  • the neural network to be trained such as the cross-modal language model in the embodiment of the present application (for example, including a text encoder, an image encoder, a target encoder, etc.) to obtain Target Model/Rule 501.
  • the training device 520 can maintain training samples in the database 530, perform a pre-training process on the neural network to be trained, or perform fine-tuning of the model based on the pre-training.
  • the training samples maintained in the database 530 are not necessarily collected from the data collection device 560, and may also be received from other devices.
  • the training device 520 may not necessarily train the target model/rules 501 based entirely on the training samples maintained by the database 530. It may also obtain training samples from the cloud or other places to perform the model. Training, the above description should not be used as a limitation on the embodiments of this application.
  • the target model/rules 501 trained according to the training device 520 can be applied to different systems or devices, such as the execution device 510 shown in Figure 3.
  • the execution device 510 can be a terminal, such as a mobile phone terminal, a tablet computer, and a notebook.
  • AR augmented reality
  • VR virtual reality
  • the training device 520 can transfer the trained model to the execution device 510 .
  • the execution device 510 is configured with an input/output (I/O) interface 512 for data interaction with external devices.
  • the user can input data to the I/O interface 512 through the client device 540 (such as this Images in application examples, etc.).
  • the preprocessing module 513 and the preprocessing module 514 are used to perform preprocessing according to the input data received by the I/O interface 512. It should be understood that there may be no preprocessing module 513 and 514 or only one preprocessing module. When the preprocessing module 513 and the preprocessing module 514 do not exist, the computing module 511 can be directly used to process the input data.
  • the execution device 510 When the execution device 510 preprocesses input data, or when the calculation module 511 of the execution device 510 performs calculations and other related processes, the execution device 510 can call data, codes, etc. in the data storage system 550 for corresponding processing. , the data, instructions, etc. obtained by corresponding processing can also be stored in the data storage system 550.
  • the I/O interface 512 provides the processing results to the client device 540 and thereby to the user.
  • the user can manually set input data, and the "manually given input data" can be operated through the interface provided by the I/O interface 512 .
  • the client device 540 can automatically send input data to the I/O interface 512. If requiring the client device 540 to automatically send the input data requires the user's authorization, the user can set corresponding permissions in the client device 540. The user can view the results output by the execution device 510 on the client device 540, and the specific presentation form may be display, sound, action, etc.
  • the client device 540 can also be used as a data collection terminal to collect the input data of the input I/O interface 512 and the output results of the output I/O interface 512 as new sample data, and store them in the database 530.
  • the I/O interface 512 directly uses the input data input to the I/O interface 512 and the output result of the output I/O interface 512 as a new sample as shown in the figure.
  • the data is stored in database 530.
  • Figure 3 is only a schematic diagram of a system architecture provided by an embodiment of the present application.
  • the positional relationship between the devices, devices, modules, etc. shown in the figure does not constitute any limitation.
  • the data The storage system 550 is an external memory relative to the execution device 510. In other cases, the data storage system 550 can also be placed in the execution device 510. It should be understood that the above execution device 510 may be deployed in the client device 540.
  • the computing module 511 of the above-mentioned execution device 520 can obtain the code stored in the data storage system 550 to implement the steps related to the model inference process in the embodiment of this application.
  • the computing module 511 of the execution device 520 may include hardware circuits (such as application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), general-purpose processors, digital signal processing (DSP), microprocessor or microcontroller, etc.), or a combination of these hardware circuits.
  • the training device 520 can be a hardware system with the function of executing instructions, such as a CPU, DSP, etc. , or it is a hardware system that does not have the function of executing instructions, such as ASIC, FPGA, etc., or it is a combination of the above-mentioned hardware system that does not have the function of executing instructions and a hardware system that has the function of executing instructions.
  • the calculation module 511 of the execution device 520 can be a hardware system with the function of executing instructions.
  • the steps related to the model reasoning process provided by the embodiment of the present application can be software codes stored in the memory.
  • the calculation module 511 of the execution device 520 The software code can be obtained from the memory, and the obtained software code can be executed to implement the steps related to the model inference process provided by the embodiments of the present application.
  • the computing module 511 of the execution device 520 can be a combination of a hardware system that does not have the function of executing instructions and a hardware system that has the function of executing instructions. Some of the steps related to the model inference process provided by the embodiments of the present application can also be implemented by The computing module 511 of the execution device 520 is implemented by a hardware system that does not have the function of executing instructions, which is not limited here.
  • the above-mentioned training device 520 can obtain the code stored in the memory (not shown in Figure 3, which can be integrated with the training device 520 or deployed separately from the training device 520) to implement the model training in the embodiment of the present application. Related steps.
  • the training device 520 may include hardware circuits (such as application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), general-purpose processors, digital signal processors (digital signal processing, DSP, microprocessor or microcontroller, etc.), or a combination of these hardware circuits.
  • ASICs application specific integrated circuits
  • FPGAs field-programmable gate arrays
  • DSP digital signal processors
  • the training device 520 can be a hardware system with the function of executing instructions, such as a CPU, DSP, etc., or a combination of other hardware circuits.
  • a hardware system with the function of executing instructions such as ASIC, FPGA, etc., or a combination of the above-mentioned hardware systems without the function of executing instructions and a hardware system with the function of executing instructions.
  • the training device 520 can be a combination of a hardware system that does not have the function of executing instructions and a hardware system that has the function of executing instructions. Some of the steps related to model training provided by the embodiments of the present application can also be implemented by the training device 520 that does not have the function of executing instructions. It is implemented by the hardware system that executes the instruction function, which is not limited here.
  • the server can provide cross-modal language processing function services for the client side through an application programming interface (API).
  • API application programming interface
  • the terminal device can send relevant parameters (such as images) to the server through the API provided by the cloud.
  • the server can obtain processing results based on the received parameters, etc.) and return the processing results to the terminal.
  • Figure 4 shows the process of using a cross-modal language processing functional cloud service provided by a cloud platform.
  • SDK software development kit
  • the cloud platform provides multiple development versions of SDK for users to choose according to the needs of the development environment, such as JAVA version of SDK and python version. SDK, PHP version SDK, Android version SDK, etc.
  • the local development environment can also develop other functions, forming a collection of cross- Application of modal language processing functional class capabilities.
  • API calls for cross-modal language processing functions can be triggered.
  • an API request is initiated to the running instance of the cross-modal language processing function service in the cloud environment.
  • the API request carries an image, which is processed by the running instance in the cloud environment. The image is processed and the processing result is obtained.
  • the cloud environment returns the processing results to the application, thus completing a cross-modal language processing function service call.
  • the server can pre-train a model with cross-modal language processing capabilities based on training data provided by the client (which may include images, for example).
  • the server can provide model training services to the client side through an application programming interface (API).
  • API application programming interface
  • the terminal device can send relevant parameters to the server through the API provided by the cloud.
  • the server can obtain the processing results based on the received parameters and return the processing results (such as a model with cross-modal language processing functions, etc.) to terminal.
  • Figure 5 shows the process of using a model training cloud service provided by a cloud platform.
  • Federated learning is divided into two steps: model delivery and model upload.
  • the central node delivers the model to the terminal device through the network; each terminal device uses local data to calculate the gradient of the model; each distributed node encrypts the gradient and uploads it to the central node;
  • the central node summarizes the gradients of each terminal distributed node and uses the parameter average algorithm to update the parameters of the central node model.
  • FIG. 6 is a schematic diagram of the architecture of a model training method provided by an embodiment of the present application.
  • the architecture provided by an embodiment of the present application includes: a cloud-side central node, which may be a cloud-side server, for example.
  • A1, A2, ... are distributed nodes of type A, such as mobile phone products held by users.
  • B1, B2, ... are distributed nodes of type B, such as personal computers held by users.
  • the administrators (such as mobile phone and computer users) agree, the administrators of the distributed nodes voluntarily share the data generated during their daily use of the equipment while protecting privacy, and join the model training plan, and the equipment becomes part of the architecture. distributed nodes.
  • the system in this embodiment can also include more types of distributed nodes, such as smart watches and so on.
  • distributed nodes do not upload data to the central node and only save data locally.
  • Distributed nodes are connected to cloud servers through communication networks.
  • the central node on the cloud side can run large models, while each distributed node can only run small models due to hardware capacity limitations, and A and B can have different data processing capabilities.
  • the neural network can be composed of neural units.
  • the neural unit can refer to an arithmetic unit that takes xs (ie, input data) and intercept 1 as input.
  • the output of the arithmetic unit can be:
  • s 1, 2,...n, n is a natural number greater than 1
  • Ws is the weight of xs
  • b is the bias of the neural unit.
  • f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into an output signal.
  • the output signal of this activation function can be used as the input of the next convolutional layer, and the activation function can be a sigmoid function.
  • a neural network is a network formed by connecting multiple above-mentioned single neural units together, that is, the output of one neural unit can be the input of another neural unit.
  • the input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field.
  • the local receptive field can be an area composed of several neural units.
  • the neural network includes an embedding layer and at least one transformer layer.
  • At least one transformer layer can be N transformer layers (N is an integer greater than 0), where each transformer layer includes successively adjacent attention layers, summation and normalization. (add&norm) layer, feed forward layer and summation and normalization layer.
  • the current input is embedded to obtain multiple embedding vectors; in the attention layer, P input vectors are obtained from the upper layer of the first transformer layer, and any of the P input vectors are The first input vector is the center. Based on the correlation between each input vector within the preset attention window range and the first input vector, the intermediate vector corresponding to the first input vector is obtained. In this way, P input vectors are determined.
  • Corresponding P intermediate vectors; in the pooling layer the P intermediate vectors are merged into Q output vectors, where the multiple output vectors obtained by the last transformer layer in the transformer layer are used as features of the current input express.
  • the attention mechanism imitates the internal process of biological observation behavior, that is, a mechanism that aligns internal experience and external sensation to increase the precision of observation in some areas, and can use limited attention resources to quickly filter out high-value information from a large amount of information. .
  • the attention mechanism can quickly extract important features of sparse data and is therefore widely used in natural language processing tasks, especially machine translation.
  • the self-attention mechanism is an improvement of the attention mechanism, which reduces the dependence on external information and is better at capturing the internal correlation of data or features.
  • the essential idea of the attention mechanism can be rewritten as the following formula:
  • Lx
  • represents the length of Source.
  • the meaning of the formula is to imagine that the constituent elements in Source are composed of a series of data pairs. At this time, given a certain element Query in the target Target, by calculating the Query and Based on the similarity or correlation of each Key, the weight coefficient of each Key's corresponding Value is obtained, and then the Value is weighted and summed to obtain the final Attention value. So essentially the Attention mechanism is a weighted summation of the Value values of the elements in the Source, and Query and Key are used to calculate the weight coefficient of the corresponding Value.
  • Attention can be understood as selectively filtering out a small amount of important information from a large amount of information and focusing on this important information, while ignoring most of the unimportant information.
  • the process of focusing is reflected in the calculation of the weight coefficient.
  • the self-attention mechanism can be understood as internal Attention (intra attention).
  • the Attention mechanism occurs between the Target element Query and all elements in the Source.
  • the self-attention mechanism refers to between the internal elements of the Source or between the internal elements of the Target.
  • NLP Natural language processing
  • Natural language is human language
  • natural language processing is the processing of human language.
  • Natural language processing is the process of systematically analyzing, understanding and extracting information from text data in an intelligent and efficient way.
  • automatic summarization automatic summarization
  • MT machine translation
  • NER Named entity recognition
  • RE relationship extraction
  • IE information extraction
  • sentiment analysis sentiment analysis
  • speech recognition speech recognition
  • question answering system question answering
  • topic segmentation etc.
  • the pre-trained language model is a natural language sequence encoder that encodes each word in the natural language sequence into a vector representation to perform prediction tasks. Its training consists of two stages. In the pre-training stage, the model performs language model training on large-scale unsupervised text to learn a word representation. In the finetuning stage, the model uses the parameters learned in the pre-training stage for initialization, and performs fewer steps of training on downstream tasks such as text classification and sequence labeling. The semantic information obtained by pre-training can be successfully transferred to downstream tasks.
  • the convolutional neural network can use the error back propagation (BP) algorithm to modify the size of the parameters in the initial super-resolution model during the training process, so that the reconstruction error loss of the super-resolution model becomes smaller and smaller.
  • BP error back propagation
  • forward propagation of the input signal until the output will produce an error loss
  • the parameters in the initial super-resolution model are updated by back-propagating the error loss information, so that the error loss converges.
  • the backpropagation algorithm is a backpropagation movement dominated by error loss, aiming to obtain the optimal parameters of the super-resolution model, such as the weight matrix.
  • the existing cross-modal visual language models (or can be called multi-modal models or multi-modal language models) have weak text modeling capabilities in the alignment method. Take TIA as an example. TIA simply determines whether a certain character is occluded, and does not take into account the semantic information between the occluded characters, causing the model to not truly understand the content of the document.
  • the current multi-modal model has weak image and text alignment capabilities, that is, it does not have the ability to align text and image elements.
  • TIM text-image matching
  • the training goal is to determine whether the global text matches the global scanned document image. This task is difficult to learn, so the trained model does not have the alignment of text and image elements. ability.
  • Figure 7 is a flow diagram of a data processing method provided by an embodiment of the present application.
  • a data processing method provided by an embodiment of the present application may include steps 701 to 703. The following are respectively These steps are described in detail.
  • the first feature representation is an image feature obtained by processing an image according to an image encoder.
  • the second feature representation is a text feature of the first text;
  • the first text is text content included in the image.
  • the original input image can be obtained, and the original input image can be encoded through an image encoder to obtain an encoding result of the input image.
  • the encoding result can include an image feature representation of the input image.
  • the original input image in this application may refer to an image input into the extended-modality language model, or an image pre-processed on an image in the extended-modality language model.
  • the image encoder can perform feature extraction on the input image.
  • the image encoder can be embodied as a convolutional neural network, a histogram of oriented gradient (HOG), or a local binary pattern. (local binary pattern, LBP) or other neural networks used to extract features from images, etc.
  • HOG histogram of oriented gradient
  • LBP local binary pattern
  • the input image can be processed through an image encoder to obtain an image feature representation corresponding to the input image.
  • text contained in the image can be identified.
  • the text contained in the image can be recognized through optical character recognition (OCR).
  • OCR optical character recognition
  • the position information corresponding to each text unit in the text can also be identified.
  • the position information can indicate the position of the area where the text unit is located in the image.
  • the area where the text unit is located can be represented by a rectangular box, and the position information can be are the coordinates of the upper left point and the lower right point of the rectangular frame.
  • the text feature representation corresponding to the text can be obtained.
  • the text can be embedded through the embedding layer to obtain an embedding vector, which can be a text feature representation corresponding to the text.
  • the embedding layer can be called the input embedding layer.
  • the current input can be every text cell in the text.
  • the embedding layer can embed each text unit in the current input and obtain the embedding vector corresponding to each text unit.
  • the position vector of each text unit can also be obtained.
  • the position vector is used to indicate the position of the text unit; where the position is used to represent the position of the text unit in the text.
  • the position is Indicates the relative position of a text unit to other text units in the text.
  • the embedding layer may include an input embedding layer and a positional encoding layer.
  • word embedding processing can be performed on each text unit in the current input to obtain the embedding vector of each text unit.
  • the position coding layer the position of each text unit in the current input can be obtained, and then a position vector is generated for the position of each text unit.
  • the position of each text unit in the text may be the absolute position of each text unit in the text. Taking the current input as "What number should I pay back the Huabei?" for example, the position of "number” can be represented as the first digit, the position of "number” can be represented as the second digit,... In some examples, the position of each text unit in the text may be the relative position of each text unit in the text. Still taking the current input as "what number should I pay back Huabei" as an example, the position of "what number” can be expressed as before “number”, and the position of "number” can be expressed as after "what number” and before “should",... ....
  • the position vector of each text unit and the corresponding embedding vector can be fused to obtain the embedding vector of each text unit, that is, multiple embeddings corresponding to the current input are obtained vector.
  • the fusion method can be to add the embedding vector and the position vector, or to use other operations so that the embedding vector can carry information about a text unit in the text and the position of the one text unit in the text.
  • the specific fusion method is not limited here.
  • fusion methods include but are not limited to concatenation (conact), addition (add), fusion (fusion), multiplication, etc.
  • the text can be embedded through the embedding layer to obtain the embedding vector, and the embedding vector can be processed through the text encoder to obtain the text feature representation corresponding to the text.
  • text-image contrastive learning is used to enhance the matching ability between images and texts, that is, to improve the model's information processing of cross-modal data such as image information and text information. ability.
  • the graphics encoder is updated through the loss constructed by similarity comparison (if the text features are obtained through the text encoder, the text encoder can also be updated).
  • the image features of the image and the text features of the text in the image are not features of the same modality, since the text is the text in the image, the image features also contain certain semantic information of the text. Therefore, image features and text features (belonging to the same image) will be related in the semantic dimension (or other information dimensions).
  • the above similarity can include the similarity between the semantic information in the image and the semantic information of the text. Therefore, updating the image encoder and the text encoder (if there is a text encoder in the architecture) based on the loss constructed by this similarity can Make image encoder and text encoder
  • the features extracted by the coder contain more accurate semantic information of the text in the image, thereby enhancing the image-text matching ability of the subsequent network.
  • the architecture of the cross-modal language model includes a graphic encoder, a text encoder, and a target encoder used to provide interactive information between features.
  • the intermediate output of the model that is, the graphic encoder and the output of the text encoder
  • the feature representation based on the output of the graphic encoder and the text encoder can achieve downstream tasks (the accuracy is not based on the target encoder
  • the output feature representation performs high downstream tasks), and in some scenarios, due to the large amount of data to be processed by the downstream tasks, the feature representation output by the graphics encoder and text encoder can be used for rough sorting, and target encoding is used
  • the feature representation output by the processor is refined to improve the recall rate.
  • the image may be the original input image (or a partial image of the original input image, but the partial image contains the entire text of the input image).
  • the first text can be all text contained in the image.
  • the image may be the original input image (or a partial image of the original input image, but the partial image contains the entire text of the input image).
  • the first text may be part of the text contained in the image.
  • the image may be a partial image area obtained by extracting the original input image.
  • the partial image area includes partial text of all the texts in the original input image, and the first text may be a portion of the text included in the image. All text.
  • the image may be a partial image area obtained by extracting the original input image.
  • the partial image area includes part of all text in the original input image, and the first text may be a part of the text contained in the image. Part of the text within the text.
  • the input original image may include one or more lines of text, or the input original image may include one or more columns of text.
  • the image may include a row or column of text of the original image, and the first text may be one or more text units.
  • the loss used to update the graphic encoder and the text encoder can be constructed based on the similarity between the image features of the image and the text features of the first text.
  • the similarity is the same as that of the image. It is related to the similarity between the text semantic information contained in the text and the semantic information of the text itself.
  • the loss used to update the graphics encoder as well as the text encoder can also be constructed based on the similarity between the image features of the image and the text features of the text not included in the image.
  • the difference is that the image features of the image and the image Chinese
  • the text features of this image are positive examples (that is, the corresponding annotation similarity is high), and the image features of the image and the text features of the text not included in the image are negative examples (that is, the corresponding annotation similarity is low).
  • the above similarity can be: the similarity between text features and image features, or the similarity between image features and text features.
  • the image features corresponding to the entire input image can be compared with the text features of all texts included in the input image, or partial images in the input image can be compared.
  • the image features of the region are compared with the text features of part of the text included in the input image, or the image features corresponding to the entire input image can be compared with the text features of part of the text included in the input image, or one of the above methods The combination.
  • the text features corresponding to all the texts included in the input image can be compared with the image features of the partial image areas in the input image, or the input
  • the text features of the partial text included in the image are compared with the image features of the partial image area in the input image.
  • the text features corresponding to the entire text included in the input image can also be compared with the image features corresponding to the entire input image.
  • the text features corresponding to the entire text included in a part of the input image can be compared with the image features of part of the image area in the input image, or a part of the input image can be compared Text features of part of the text included in the input image are compared with image features of part of the image area in the input image, or text features corresponding to the entire text included in part of the input image can be compared with image features corresponding to the entire input image. Comparison, or a combination of the above methods.
  • the text features corresponding to the entire text included in a part of the input image can be compared with the image features of part of the image area in the input image, or a part of the input image can be compared Text features of part of the text included in the input image are compared with image features of part of the image area in the input image, or text features corresponding to the entire text included in part of the input image can be compared with image features corresponding to the entire input image. Comparison.
  • the image region feature extraction module (for example, can be implemented through the region of interest (ROI) head) can be used to determine the partial image area in the image.
  • Image features for example, the image region feature extraction module can perform bilinear interpolation on the image features of the image region (output by the image encoder) based on the position information of the image region to be extracted, to obtain the image features of part of the image region.
  • coarse-grained and fine-grained contrastive learning can be used to enhance the overall image-text matching and region-text line matching capabilities, thereby improving the performance of the model on downstream tasks such as information extraction tasks.
  • the loss function of image-text contrastive learning can be defined as:
  • the loss function of text-image comparison learning can be defined as:
  • the total contrastive learning loss function of text line-image area can be defined as:
  • the target encoder is used to implement the interaction between feature representations.
  • this application can improve multi-modal alignment capabilities by first using a double-tower approach, and then using a multi-modal single-tower structure to further enhance the interactive learning capabilities of features, which can be beneficial to downstream tasks such as efficient document retrieval (rough sorting first and then fine sorting). Row).
  • a third feature representation can be obtained through a target encoder according to the first feature representation and the second feature representation.
  • the third feature representation can include text features corresponding to the first text, and The image features corresponding to the image are different from the first feature representation and the second feature representation in that the features output by the target encoder are learned based on the interaction between the first feature representation and the second feature representation.
  • the third feature representation execute the downstream task and obtain the execution result; the similarity between the execution result and the corresponding annotation and the similarity between the first feature representation and the second feature representation. Used to update the image encoder.
  • the target encoder can obtain a third feature representation, and the third feature representation can be used to perform downstream tasks to obtain execution results.
  • the execution results can be combined with corresponding annotations to construct a loss to update the graph encoder. , text encoder (if any) and target encoder.
  • different losses can be constructed by setting up different downstream tasks.
  • downstream tasks may include bounding-box grid matching (BGM).
  • BGM bounding-box grid matching
  • the first text may include a first sub-text and a second sub-text
  • the second feature representation includes a first sub-feature corresponding to the first sub-text, and the second sub-text.
  • the second sub-feature corresponding to the sub-text; the first sub-feature does not include the position of the first sub-text in the image; the second sub-feature includes the position of the second sub-text in the image Position; that is to say, when generating text features of the first text, for part of the text in the first text (such as the second sub-text), the text and position information can be used to generate corresponding text features (such as the second sub-feature) ), for part of the text in the first text (for example, the first sub-text), the text and position information can be used to generate corresponding text features (for example, the first sub-feature), which is equivalent to the fact that the first sub-feature does not include the The position of the first sub-text in the image; the second sub-feature includes the position of the second sub-text in the image.
  • optical character recognition can be used to identify the text contained in the image and the corresponding position information.
  • the position information can indicate the position of the area where the text is located in the image.
  • the area where the text is located can be represented by a rectangular box, and the position information can be the coordinates of the upper left point and the lower right point of the rectangular box.
  • the third feature representation may include the position information of the second sub-text but not the position information of the first sub-text (the non-inclusion here can be understood as being caused by the input, but it can be included implicitly) according to the second sub-text.
  • Three features represent predictions. Specifically, the first position of the first sub-text in the image can be predicted according to the third feature representation; the first position and the corresponding annotation (that is, the true position of the first sub-text in the image The similarity between positions) is used to update the image encoder and the target encoder.
  • the image includes multiple image blocks; the first position is the image block where the first sub-text is predicted to be located; and the annotation is the image where the first sub-text is actually located. piece.
  • this embodiment can be divided into the following steps:
  • E word and E layout represent the word embedding and layout information of the i-th word respectively.
  • the BGM loss function can be defined as:
  • y ij represents the true label of the i-th word
  • p ij represents the probability that the i-th word belongs to the j-th image area.
  • This application proposes BGM in the single tower part to improve spatial layout perception, which helps improve the performance of the model in position-sensitive tasks. For example, in information extraction, key-value pairs are usually adjacent pairs of data. Because the model has learned the ability to perceive position and layout information in the pre-training stage, the model will have good performance in tasks such as information extraction.
  • downstream tasks can also include MLM.
  • text units in the text can be masked (for example, they can be randomly masked), and the masked text units can be used as predicted text.
  • the unmasked text units can be used as known text, and the model can be based on the unmasked text units (or the predicted masked text units), and sequentially perform the text positions where the masked text units are located.
  • Text prediction for example, the text can be "the cat sat on the mat”. After masking the text, you can get "the_sat_the mat", where the symbol "_” here means masking, not referring to the underline in .
  • the text at the text position where one of the "_" is located can be predicted through the initial identification bit and the unmasked text unit.
  • a loss can also be constructed based on image features. Specifically, parts of the image can be masked, and the results obtained by the image encoder can be used to mask the area of the image. Image reconstruction is performed and the network is updated based on the difference between the reconstruction result and the pixel values of the real image in the masked area.
  • the image representation learning ability of existing methods is weak. Specifically, in order to reduce the processing overhead of the model, the image needs to be compressed to a smaller resolution (such as 224*224), and the image contains dense text. In this case, the pixels in the image are difficult to express accurate text information, and the image representation learning ability of the trained model is limited.
  • the image features of the image may not be masked, and the image features of the masked area may be reconstructed based on the reconstruction results and the image of the masked area.
  • the difference between features is used to update the network. Since image features can carry richer image information and text information than pixel values themselves, it can improve the learning ability of image representation of the trained network.
  • the above method can be called masking.
  • Image modeling mask image modeling, MIM).
  • the first feature representation includes a third sub-feature and a fourth sub-feature; the fourth sub-feature and the fourth sub-feature can be included in the first feature representation according to the fourth sub-feature and the fourth sub-feature.
  • the second position and the third position of the third sub-feature in the first feature representation are used to obtain the feature prediction value at the third position; the difference between the feature prediction value and the third sub-feature is The similarity is used to update the image encoder.
  • the method can be based on the fourth sub-feature, the second position of the fourth sub-feature in the first feature representation, and the position of the third sub-feature in the first feature representation.
  • the fourth feature representation is obtained through the self-attention network; according to the fourth feature representation, through the prediction network, the feature prediction value at the third position is obtained.
  • the masked object can be obtained by the image encoder, or can be obtained by performing certain processing on the image features obtained by the image encoder, for example, through an image area feature extraction module (for example, Image features that can be determined for part of the image area in the image through the region of interest (ROI) head.
  • image area feature extraction module for example, Image features that can be determined for part of the image area in the image through the region of interest (ROI) head.
  • this embodiment can be divided into the following steps:
  • W and b are network parameters.
  • H is the latent space representation of ROI features
  • Atten is the self-attention module
  • ⁇ is the network parameter.
  • the visual mask feature reconstruction loss function for the text line area in the image can be expressed as:
  • H i represents The corresponding groundtruth labels.
  • This embodiment uses higher resolution and MIM (Masked Image Modeling) to improve visual representation learning of text line areas.
  • the masked visual feature map is reconstructed based on the feature map angle, and the ROI features of adjacent unmasked text lines are used to help reconstruct the visual features of the masked text lines.
  • MIM Mask Image Modeling
  • the present application provides a data processing method, which includes: obtaining a first feature representation and a second feature representation.
  • the first feature representation is an image feature obtained by processing an image according to an image encoder.
  • the second feature representation is the text feature of the first text; the first text is the text content included in the image; the similarity between the first feature representation and the second feature representation is used to update the image encoder;
  • a third feature representation is obtained through a target encoder; the target encoder is used to realize the interaction between feature representations; according to the third feature representation, perform downstream Task and get execution results.
  • text-image contrastive learning is used to enhance the matching ability between images and texts.
  • similarity comparison is used Construct the loss to update the graph encoder (if the text features are obtained through the text encoder, the text encoder can also be updated).
  • image features of the image and the text features of the text in the image are not features of the same modality, since the text is the text in the image, the image features also contain certain semantic information of the text. Therefore, image features and text features (belonging to the same image) will be related in the semantic dimension (or other information dimensions).
  • the above similarity can include the similarity between the semantic information in the image and the semantic information of the text.
  • the architecture of the cross-modal language model includes a graphic encoder, a text encoder, and a target encoder used to provide interactive information between features.
  • the intermediate output of the model that is, the graphic encoder and the output of the text encoder
  • the feature representation based on the output of the graphic encoder and the text encoder can achieve downstream tasks (the accuracy is not based on the target encoder
  • the output feature representation performs high downstream tasks), and in some scenarios, due to the large amount of data to be processed by the downstream tasks, the feature representation output by the graphics encoder and text encoder can be used for rough sorting, and target encoding is used
  • the feature representation output by the processor is refined to improve the recall rate.
  • FIG 11 is a schematic diagram of the architecture of the model in this application, in which two towers are first used to align graphic and text features, and then a single tower is connected to build a graphic and text multi-modal interaction module.
  • Twin Towers part features are first extracted from image and text encoding respectively, and then coarse-grained and fine-grained contrastive learning is used to enhance the overall image-text matching and region-text line matching capabilities.
  • BGM Box Grid Matching
  • MIM Mask Image Modeling
  • this application also provides a data processing method, which method includes:
  • a third feature representation is obtained through a target encoder; the target encoder is used to realize the interaction between feature representations;
  • the downstream task is executed and the execution result is obtained.
  • Figure 12 is a schematic structural diagram of a data processing device provided by an embodiment of the present application.
  • a data processing device 1200 provided by an embodiment of the present application includes:
  • the acquisition module 1201 is used to obtain a first feature representation and a second feature representation.
  • the first feature representation is obtained by processing the image according to the image encoder.
  • the second feature representation is the text of the first text.
  • the first text is the text content included in the image;
  • step 701 For a specific description of the acquisition module 1201, reference may be made to the description of step 701 in the above embodiment, which will not be described again here.
  • the encoding module 1202 is used to obtain a third feature representation through a target encoder according to the first feature representation and the second feature representation; the target encoder is used to implement interaction between feature representations;
  • step 702 For a specific description of the encoding module 1202, reference may be made to the description of step 702 in the above embodiment, which will not be described again here.
  • Task execution module 1203 configured to execute downstream tasks according to the third feature representation and obtain execution results; the similarity between the execution results and the corresponding annotations, as well as the first feature representation and the second feature representation The similarity between them is used to update the image encoder.
  • step 703 For a specific description of the task execution module 1203, reference may be made to the description of step 703 in the above embodiment, which will not be described again here.
  • the second feature representation is a text feature obtained by processing the first text through a text encoder; the similarity between the first feature representation and the second feature representation Also used to update the text encoder.
  • the similarity between the first feature representation and the second feature representation is related to the similarity between the text semantic information contained in the image and the semantic information of the text itself.
  • the first text is all text contained in the image; or,
  • the first text is part of the total text contained in the image.
  • the image is a partial image area extracted from the original input image, and the text included in the image is part of the text included in the input image; or,
  • the image is the original input image.
  • the first text includes a first sub-text and a second sub-text
  • the second feature representation includes a first sub-feature corresponding to the first sub-text, and the second sub-text.
  • the second sub-feature corresponding to the text; the first sub-feature does not include the position of the first sub-text in the image; the second sub-feature includes the position of the second sub-text in the image ;
  • the task execution module is specifically used for:
  • the third feature representation predict the first position of the first sub-text in the image; the similarity between the first position and the corresponding annotation is used to update the image encoder and the target encoder.
  • the image includes multiple image blocks; the first position is the image block where the first sub-text is predicted to be located; and the annotation is the image where the first sub-text is actually located. piece.
  • the first feature representation includes a third sub-feature and a fourth sub-feature; the method further includes:
  • a prediction module configured to predict the fourth sub-feature based on the fourth sub-feature, the second position of the fourth sub-feature in the first feature representation, and the third position of the third sub-feature in the first feature representation. position to obtain the feature prediction value at the third position; the similarity between the feature prediction value and the third sub-feature is used to update the image encoder.
  • the prediction module is specifically used to:
  • the fourth sub-feature the second position of the fourth sub-feature in the first feature representation, and the third position of the third sub-feature in the first feature representation, through self-attention Force network to obtain the fourth feature representation;
  • a feature prediction value at the third position is obtained through a prediction network.
  • this application also provides a data processing device, which includes:
  • Acquisition module used to obtain images
  • An encoding module used to process the image through an image encoder to obtain a first feature representation
  • a third feature representation is obtained through a target encoder; the target encoder is used to realize the interaction between feature representations;
  • a task execution module configured to execute downstream tasks according to the third feature representation and obtain execution results.
  • FIG. 13 is a schematic structural diagram of an execution device provided by an embodiment of the present application.
  • the execution device 1300 can be embodied as a virtual reality VR device, a mobile phone, Tablet, laptop, smart phone Wearable devices, monitoring data processing equipment or servers, etc. are not limited here.
  • the execution device 1300 includes: a receiver 1301, a transmitter 1302, a processor 1303 and a memory 1304 (the number of processors 1303 in the execution device 1300 can be one or more, one processor is taken as an example in Figure 13) , wherein the processor 1303 may include an application processor 13031 and a communication processor 13032.
  • the receiver 1301, the transmitter 1302, the processor 1303, and the memory 1304 may be connected through a bus or other means.
  • Memory 1304 may include read-only memory and random access memory and provides instructions and data to processor 1303 .
  • a portion of memory 1304 may also include non-volatile random access memory (NVRAM).
  • NVRAM non-volatile random access memory
  • the memory 1304 stores processor and operating instructions, executable modules or data structures, or a subset thereof, or an extended set thereof, where the operating instructions may include various operating instructions for implementing various operations.
  • the processor 1303 controls the execution of operations of the device.
  • various components of the execution device are coupled together through a bus system.
  • the bus system may also include a power bus, a control bus, a status signal bus, etc.
  • various buses are called bus systems in the figure.
  • the methods disclosed in the above embodiments of the present application can be applied to the processor 1303 or implemented by the processor 1303.
  • the processor 1303 may be an integrated circuit chip with signal processing capabilities. During the implementation process, each step of the above method can be completed by instructions in the form of hardware integrated logic circuits or software in the processor 1303 .
  • the above-mentioned processor 1303 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor or a microcontroller, and can further include an application specific integrated circuit (ASIC), a field programmable Gate array (field-programmable gate array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
  • DSP digital signal processor
  • ASIC application specific integrated circuit
  • FPGA field-programmable gate array
  • the processor 1303 can implement or execute each method, step and logical block diagram disclosed in the embodiment of this application.
  • a general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.
  • the steps of the method disclosed in conjunction with the embodiments of the present application can be directly implemented by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor.
  • the software module can be located in random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers and other mature storage media in this field.
  • the storage medium is located in the memory 1304.
  • the processor 1303 reads the information in the memory 1304 and completes the steps related to the model inference process in the above method in combination with its hardware.
  • the receiver 1301 may be configured to receive input numeric or character information and generate signal inputs related to performing relevant settings and functional controls of the device.
  • the transmitter 1302 can be used to output numeric or character information through the first interface; the transmitter 1302 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; the transmitter 1302 can also include a display device such as a display screen .
  • FIG. 14 is a schematic structural diagram of the training device provided by the embodiment of the present application.
  • the training device 1400 is implemented by one or more servers.
  • the training device 1400 There may be relatively large differences due to different configurations or performance, which may include one or more central processing units (CPU) 1414 (for example, one or more processors) and memory 1432, one or more storage applications Storage medium 1430 for program 1442 or data 1444 (eg, one or more mass storage devices).
  • the memory 1432 and the storage medium 1430 may be short-term storage or persistent storage.
  • the program stored in the storage medium 1430 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the training device. Furthermore, the central processor 1414 may be configured to communicate with the storage medium 1430 and execute a series of instruction operations in the storage medium 1430 on the training device 1400 .
  • the training device 1400 may also include one or more power supplies 1426, one or more wired or wireless network interfaces 1450, one or more input and output interfaces 1458; or, one or more operating systems 1441, such as Windows ServerTM, Mac OS XTM , UnixTM, LinuxTM, FreeBSDTM and so on.
  • operating systems 1441 such as Windows ServerTM, Mac OS XTM , UnixTM, LinuxTM, FreeBSDTM and so on.
  • the central processor 1414 is used to perform actions related to model training in the above embodiment.
  • An embodiment of the present application also provides a computer program product that, when run on a computer, causes the computer to perform the steps performed by the foregoing execution device, or causes the computer to perform the steps performed by the foregoing training device.
  • Embodiments of the present application also provide a computer-readable storage medium.
  • the computer-readable storage medium stores a program for performing signal processing.
  • the program When the program is run on a computer, it causes the computer to perform the steps performed by the aforementioned execution device. , or, make the computer Perform the same steps as previously performed for the training device.
  • the execution device, training device or terminal device provided by the embodiment of the present application may specifically be a chip.
  • the chip includes: a processing unit and a communication unit.
  • the processing unit may be, for example, a processor.
  • the communication unit may be, for example, an input/output interface. Pins or circuits, etc.
  • the processing unit can execute the computer execution instructions stored in the storage unit, so that the chip in the execution device executes the data processing method described in the above embodiment, or so that the chip in the training device executes the data processing method described in the above embodiment.
  • the storage unit is a storage unit within the chip, such as a register, cache, etc.
  • the storage unit may also be a storage unit located outside the chip in the wireless access device, such as Read-only memory (ROM) or other types of static storage devices that can store static information and instructions, random access memory (random access memory, RAM), etc.
  • ROM Read-only memory
  • RAM random access memory
  • Figure 15 is a structural schematic diagram of a chip provided by an embodiment of the present application.
  • the chip can be represented as a neural network processor NPU 1500.
  • the NPU 1500 serves as a co-processor and is mounted to the main CPU (Host). CPU), tasks are allocated by the Host CPU.
  • the core part of the NPU is the arithmetic circuit 1503.
  • the arithmetic circuit 1503 is controlled by the controller 1504 to extract the matrix data in the memory and perform multiplication operations.
  • the computing circuit 1503 internally includes multiple processing units (Process Engine, PE).
  • arithmetic circuit 1503 is a two-dimensional systolic array.
  • the arithmetic circuit 1503 may also be a one-dimensional systolic array or other electronic circuit capable of performing mathematical operations such as multiplication and addition.
  • arithmetic circuit 1503 is a general-purpose matrix processor.
  • the arithmetic circuit obtains the corresponding data of matrix B from the weight memory 1502 and caches it on each PE in the arithmetic circuit.
  • the operation circuit takes matrix A data and matrix B from the input memory 1501 to perform matrix operations, and the partial result or final result of the obtained matrix is stored in an accumulator (accumulator) 1508 .
  • the unified memory 1506 is used to store input data and output data.
  • the weight data directly passes through the storage unit access controller (Direct Memory Access Controller, DMAC) 1505, and the DMAC is transferred to the weight memory 1502.
  • Input data is also transferred to unified memory 1506 via DMAC.
  • DMAC Direct Memory Access Controller
  • BIU is the Bus Interface Unit, that is, the bus interface unit 1510, which is used for the interaction between the AXI bus and the DMAC and the Instruction Fetch Buffer (IFB) 1509.
  • IFB Instruction Fetch Buffer
  • the bus interface unit 1510 (Bus Interface Unit, BIU for short) is used to fetch the memory 1509 to obtain instructions from the external memory, and is also used for the storage unit access controller 1505 to obtain the original data of the input matrix A or the weight matrix B from the external memory.
  • BIU Bus Interface Unit
  • DMAC is mainly used to transfer the input data in the external memory DDR to the unified memory 1506 or the weight data to the weight memory 1502 or the input data to the input memory 1501 .
  • the vector calculation unit 1507 includes multiple arithmetic processing units, and if necessary, further processes the output of the arithmetic circuit 1503, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc.
  • vector calculation unit 1507 can store the processed output vectors to unified memory 1506 .
  • the vector calculation unit 1507 can apply a linear function; or a nonlinear function to the output of the operation circuit 1503, such as linear interpolation on the feature plane extracted by the convolution layer, or a vector of accumulated values, to generate an activation value.
  • vector calculation unit 1507 generates normalized values, pixel-wise summed values, or both.
  • the processed output vector can be used as an activation input to the arithmetic circuit 1503, such as for use in a subsequent layer in a neural network.
  • the instruction fetch buffer 1509 connected to the controller 1504 is used to store instructions used by the controller 1504;
  • the unified memory 1506, the input memory 1501, the weight memory 1502 and the fetch memory 1509 are all On-Chip memories. External memory is private to the NPU hardware architecture.
  • the processor mentioned in any of the above places can be a general central processing unit, a microprocessor, an ASIC, or one or more integrated circuits used to control the execution of the above programs.
  • the device embodiments described above are only illustrative.
  • the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physically separate.
  • the physical unit can be located in one place, or it can be distributed across multiple network units. You can select some or all of the modules according to actual needs to implement The purpose of this embodiment is achieved.
  • the connection relationship between modules indicates that there are communication connections between them, which can be specifically implemented as one or more communication buses or signal lines.
  • the present application can be implemented by software plus necessary general hardware. Of course, it can also be implemented by dedicated hardware including dedicated integrated circuits, dedicated CPUs, dedicated memories, Special components, etc. to achieve. In general, all functions performed by computer programs can be easily implemented with corresponding hardware. Moreover, the specific hardware structures used to implement the same function can also be diverse, such as analog circuits, digital circuits or special-purpose circuits. circuit etc. However, for this application, software program implementation is a better implementation in most cases. Based on this understanding, the technical solution of the present application can be embodied in the form of a software product in essence or that contributes to the existing technology.
  • the computer software product is stored in a readable storage medium, such as a computer floppy disk. , U disk, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., including several instructions to cause a computer device (which can be a personal computer, training device, or network device, etc.) to execute the steps described in various embodiments of this application. method.
  • a computer device which can be a personal computer, training device, or network device, etc.
  • the computer program product includes one or more computer instructions.
  • the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device.
  • the computer instructions may be stored in or transmitted from one computer-readable storage medium to another, for example, the computer instructions may be transferred from a website, computer, training device, or data
  • the center transmits to another website site, computer, training equipment or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) means.
  • wired such as coaxial cable, optical fiber, digital subscriber line (DSL)
  • wireless such as infrared, wireless, microwave, etc.
  • the computer-readable storage medium may be any available medium that a computer can store, or a data storage device such as a training device or a data center integrated with one or more available media.
  • the available media may be magnetic media (eg, floppy disk, hard disk, magnetic tape), optical media (eg, DVD), or semiconductor media (eg, solid state disk (Solid State Disk, SSD)), etc.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Evolutionary Computation (AREA)
  • Health & Medical Sciences (AREA)
  • Computing Systems (AREA)
  • Databases & Information Systems (AREA)
  • Artificial Intelligence (AREA)
  • General Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Software Systems (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

一种数据处理方法，应用于包含文字的图像的处理，涉及人工智能领域，包括：获取第一特征表示以及第二特征表示，第二特征表示为第一文本的文本特征；第一文本为图像中包括的文本内容；根据第一特征表示以及第二特征表示，通过目标编码器，得到第三特征表示；第三特征表示用于执行下游任务；执行结果和对应的标注之间的相似度以及第一特征表示和第二特征表示之间的相似度用于更新图像编码器。本申请通过先双塔方式可以提升图文之间的对齐能力，再利用一个单塔结构进一步增强特征的交互学习能力。

Description

一种数据处理方法及其装置
本申请要求于2022年8月22日提交中国专利局、申请号为202211005409.3、发明名称为“一种数据处理方法及其装置”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及人工智能领域，尤其涉及一种数据处理方法及其装置。
背景技术
人工智能(Artificial Intelligence，AI)是利用数字计算机或者数字计算机控制的机器模拟、延伸和扩展人的智能，感知环境、获取知识并使用知识获得最佳结果的理论、方法、技术及应用***。换句话说，人工智能是计算机科学的一个分支，它企图了解智能的实质，并生产出一种新的能以人类智能相似的方式作出反应的智能机器。人工智能也就是研究各种智能机器的设计原理与实现方法，使机器具有感知、推理与决策的功能。
现有的跨模态视觉语言模型(或者可以称之为多模态模型或者多模态语言模型)所采用的对齐方式中的文本建模能力较弱。以图文对齐(text-image alignment，TIA)为例，TIA只是简单的判断某个字符是否被遮挡，并没有很好的考虑到遮挡字符之间的语义信息导致模型不是真正的理解文档的内容。
当前的多模态模型的图文对齐能力较弱，也就是不具备文本和图像元素的对齐能力。以图文匹配(text-image matching，TIM)为例，训练目标用来判断全局文字是否和全局的扫描文档图片是否匹配，该任务学习难度小，使得训练的模型不具备文本和图像元素的对齐能力。
发明内容
本申请提供了一种数据处理方法，可以使得图像编码器和文本编码器提取的特征中蕴含更准确的图像中文本的语义信息，进而增强后续网络的图文匹配能力，本申请中通过先双塔方式可以提升图文之间的对齐能力，再利用一个单塔结构进一步增强特征的交互学习能力。
第一方面，本申请提供了一种数据处理方法，包括：获取第一特征表示以及第二特征表示，第一特征表示为根据图像编码器对图像进行处理得到的图像特征得到，第二特征表示为第一文本的文本特征；第一文本为图像中包括的文本内容；根据第一特征表示以及第二特征表示，通过目标编码器，得到第三特征表示；目标编码器用于实现特征表示之间的交互；根据第三特征表示，执行下游任务，得到执行结果；所述执行结果和对应的标注之间的相似度以及所述第一特征表示和所述第二特征表示之间的相似度用于更新所述图像编码器。
本申请中，通过图文对比学习(text-image contrastive learning，TIC)来增强图文之间的匹配能力，具体的，针对于图像的图像特征和文本的文本特征，通过相似度比对构建的损失，来更新图形编码器(如果文本特征为通过文本编码器得到的，也可以更新文本编码器)。虽然图像的图像特征和图像中文本的文本特征不是同一个模态的特征，但是由于文本是图像中的文本，在图像特征中也会蕴含一定的文本的语义信息，因此，图像特征和文本特征(属于同一个图像)之间在语义维度(或者其他信息维度上)会存在关联。上述相似度可以包含图像中语义信息和文本的语义信息之间的相似度，因此，基于该相似度构建的损失来更新图像编码器以及文本编码器(如果架构中存在文本编码器的话)，能够使得图像编码器和文本编码器提取的特征中蕴含更准确的图像中文本的语义信息，进而增强后续网络的图文匹配能力。
此外，在跨模态的语言模型的架构中，包括图形编码器、文本编码器以及用于提供特征之间交互信息的目标编码器，本申请中，将模型的中间输出(也就是图形编码器和文本编码器的输出)之间的相似度构建的损失来更新图形编码器和文本编码器，使得基于图形编码器和文本编码器输出的特征表示就可以实现下游任务(精度没有基于目标编码器输出的特征表示进行的下游任务高)，而在一些场景中，由于下游任务要处理的数据的数量较大，因此可以使用图形编码器和文本编码器输出的特征表示进行粗排， 使用目标编码器输出的特征表示进行精排，以提高召回率。
在一种可能的实现中，第二特征表示为通过文本编码器对第一文本进行处理得到的文本特征；第一特征表示和第二特征表示之间的相似度还用于更新文本编码器。
在一种可能的实现中，相似度与图像中所蕴含的文本语义信息和文本自身的语义信息之间的相似度有关。
在一种可能的实现中所述执行结果和对应的标注之间的相似度还用于更新目标编码器。
在一种可能的实现中，
第一文本为图像中包含的全部文本；或者，
第一文本为图像中包含的全部文本中的部分。
在一种可能的实现中，
图像为从原始的输入图像中提取的部分图像区域，图像包括的文本为输入图像包含的文本的部分；或者，
图像为原始的输入图像。
在一种可能的实现中，图像可以为原始的输入图像(或者是原始的输入图像的部分图像，但该部分图像包含输入图像的全部文本)。第一文本可以为图像中包含的全部文本。
在一种可能的实现中，图像可以为原始的输入图像(或者是原始的输入图像的部分图像，但该部分图像包含输入图像的全部文本)。第一文本可以为图像中包含的文本中的部分文本。
在一种可能的实现中，图像可以为对原始的输入图像进行提取得到的部分图像区域，该部分图像区域内包括原始的输入图像中全部文本的部分文本，第一文本可以为图像中包括的全部文本。
在一种可能的实现中，图像可以为对原始的输入图像进行提取得到的部分图像区域，该部分图像区域内包括原始的输入图像中全部文本的部分文本，第一文本可以为图像中包含的文本中的部分文本。
在一种可能的实现中，输入的原始图像中可以包括一行或多行文本，或者，输入的原始图像中可以包括一列或多列文本。可选的，图像中可以包括原始图像的一行或者一列文本，第一文本可以为一个或多个文本单元。
在进行图文对比学习时，可以根据图像的图像特征和第一文本的文本特征之间的相似度来构建用于更新图形编码器以及文本编码器的损失，可选的，该相似度与图像中所蕴含的文本语义信息和文本自身的语义信息之间的相似度有关。
应理解，还可以根据图像的图像特征和图像中不包括的文本的文本特征之间的相似度来构建用于更新图形编码器以及文本编码器的损失，区别在于，图像的图像特征和图像中文本的文本特征为正例(也就是对应的标注相似度高)，图像的图像特征和图像中不包括的文本的文本特征为负例(也就是对应的标注相似度低)。
在一种可能的实现中，针对于同一张原始的输入图像，可以将输入图像整体对应的图像特征和输入图像中包括的全部文本的文本特征进行比对，也可以将输入图像中的部分图像区域的图像特征和输入图像中包括的部分文本的文本特征进行比对，也可以将输入图像整体对应的图像特征和输入图像中包括的部分文本的文本特征进行比对，或者是上述几种方式的组合。
在一种可能的实现中，针对于同一张原始的输入图像，可以将输入图像中包括的全部文本整体对应的文本特征和输入图像中的部分图像区域的图像特征进行比对，也可以将输入图像中包括的部分文本的文本特征和输入图像中的部分图像区域的图像特征进行比对，也可以将输入图像中包括的全部文本整体对应的文本特征和输入图像整体对应的图像特征进行比对。
在一种可能的实现中，针对于不同的原始输入图像，可以将一部分输入图像中包括的全部文本整体对应的文本特征和输入图像中的部分图像区域的图像特征进行比对，也可以将一部分输入图像中包括的部分文本的文本特征和输入图像中的部分图像区域的图像特征进行比对，也可以将一部分输入图像中包括的全部文本整体对应的文本特征和输入图像整体对应的图像特征进行比对，或者是上述几种方式的组合。
在一种可能的实现中，针对于不同的原始输入图像，可以将一部分输入图像中包括的全部文本整体 对应的文本特征和输入图像中的部分图像区域的图像特征进行比对，也可以将一部分输入图像中包括的部分文本的文本特征和输入图像中的部分图像区域的图像特征进行比对，也可以将一部分输入图像中包括的全部文本整体对应的文本特征和输入图像整体对应的图像特征进行比对。
在一种可能的实现中，第一文本包括第一子文本和第二子文本，第二特征表示包括第一子文本对应的第一子特征、以及第二子文本对应的第二子特征；第一子特征不包含第一子文本在图像中的位置；第二子特征包含第二子文本在图像中的位置；根据第三特征表示，执行下游任务，包括：根据第三特征表示，预测第一子文本在图像中的第一位置；第一位置和对应的标注之间的相似度用于更新图像编码器以及目标编码器。
在一种可能的实现中，图像包括多个图像块；第一位置为对第一子文本预测所在的图像块；标注为第一子文本真实所在的图像块。
本申请在单塔部分提出BGM来提升空间布局感知能力，有助于提升位置敏感的任务中模型的性能。例如在信息提取中，key-value对通常是相邻的成对数据。因为模型在预训练阶段已经学会了对位置以及布局信息的感知能力，所以在信息抽取这类任务中模型会有很好的性能。
在一种可能的实现中，第一特征表示包括第三子特征和第四子特征；方法还包括：根据第四子特征、第四子特征在第一特征表示中的第二位置、以及第三子特征在第一特征表示中的第三位置，得到第三位置处的特征预测值；特征预测值和第三子特征之间的相似度用于更新图像编码器。
在一种可能的实现中，根据第四子特征、第四子特征在第一特征表示中的第二位置、以及第三子特征在第一特征表示中的第三位置，得到第三位置处的特征预测值，包括：根据第四子特征、第四子特征在第一特征表示中的第二位置、以及第三子特征在第一特征表示中的第三位置，通过自注意力网络，得到第四特征表示；根据第四特征表示，通过预测网络，得到第三位置处的特征预测值。
在一种可能的实现中，为了提高图像表征的学习能力，还可以基于图像特征来构建损失，具体的，可以对图像中的部分进行掩码，通过图像编码器得到的结果对掩码区域图像进行图像重建，并基于重建结果和掩码区域的真实图像的像素值之间的差异来更新网络。然而，现有方法图像表征学习能力较弱，具体是由于，为了降低模型的处理算例开销，需要将图像压缩到一个较小的分辨率(例如224*224)，在图像中包含密集型文本的情况下，图像中的像素难以表达出准确的文字信息，训练后的模型的图像表征的学习能力有限。
本申请实施例中，为了提高图像表征的学习能力，可以不对图像而是对图像的图像特征进行掩码，并对掩码区域的图像特征进行特征重建，并基于重建结果和掩码区域的图像特征之间的差异来更新网络，由于图像特征相比像素值本身可以携带更丰富的图像信息以及文本信息，因此可以提高训练后的网络的图像表征的学习能力，上述方式可以称之为掩码图像建模(mask image modeling，MIM)。
第二方面，本申请提供了一种数据处理方法，方法包括：
获取图像；
通过图像编码器，对图像进行处理，得到第一特征表示；
通过文本编码器，对图像中包含的文本进行处理，得到第二特征表示；
根据第一特征表示以及第二特征表示，通过目标编码器，得到第三特征表示；目标编码器用于实现特征表示之间的交互；
根据第三特征表示，执行下游任务，得到执行结果。
第三方面，本申请提供了一种数据处理装置，包括：
获取模块，用于获取第一特征表示以及第二特征表示，第一特征表示为根据图像编码器对图像进行处理得到的图像特征得到，第二特征表示为第一文本的文本特征；第一文本为图像中包括的文本内容；第一特征表示和第二特征表示之间的相似度用于更新图像编码器；
编码模块，用于根据第一特征表示以及第二特征表示，通过目标编码器，得到第三特征表示；目标编码器用于实现特征表示之间的交互；
任务执行模块，用于根据第三特征表示，执行下游任务，得到执行结果。
在一种可能的实现中，第二特征表示为通过文本编码器对第一文本进行处理得到的文本特征；第一特征表示和第二特征表示之间的相似度还用于更新文本编码器。
在一种可能的实现中，相似度与图像中所蕴含的文本语义信息和文本自身的语义信息之间的相似度有关。
在一种可能的实现中，
第一文本为图像中包含的全部文本；或者，
第一文本为图像中包含的全部文本中的部分。
在一种可能的实现中，
图像为从原始的输入图像中提取的部分图像区域，图像包括的文本为输入图像包含的文本的部分；或者，
图像为原始的输入图像。
在一种可能的实现中，第一文本包括第一子文本和第二子文本，第二特征表示包括第一子文本对应的第一子特征、以及第二子文本对应的第二子特征；第一子特征不包含第一子文本在图像中的位置；第二子特征包含第二子文本在图像中的位置；
任务执行模块，具体用于：
根据第三特征表示，预测第一子文本在图像中的第一位置；第一位置和对应的标注之间的相似度用于更新图像编码器以及目标编码器。
在一种可能的实现中，图像包括多个图像块；第一位置为对第一子文本预测所在的图像块；标注为第一子文本真实所在的图像块。
在一种可能的实现中，第一特征表示包括第三子特征和第四子特征；方法还包括：
预测模块，用于根据第四子特征、第四子特征在第一特征表示中的第二位置、以及第三子特征在第一特征表示中的第三位置，得到第三位置处的特征预测值；特征预测值和第三子特征之间的相似度用于更新图像编码器。
在一种可能的实现中，预测模块，具体用于：
根据第四子特征、第四子特征在第一特征表示中的第二位置、以及第三子特征在第一特征表示中的第三位置，通过自注意力网络，得到第四特征表示；
根据第四特征表示，通过预测网络，得到第三位置处的特征预测值。
第四方面，本申请提供了一种数据处理装置，装置包括：
获取模块，用于获取图像；
编码模块，用于通过图像编码器，对图像进行处理，得到第一特征表示；
通过文本编码器，对图像中包含的文本进行处理，得到第二特征表示；
根据第一特征表示以及第二特征表示，通过目标编码器，得到第三特征表示；目标编码器用于实现特征表示之间的交互；
任务执行模块，用于根据第三特征表示，执行下游任务，得到执行结果。
第五方面，本申请实施例提供了一种训练装置，可以包括存储器、处理器以及总线***，其中，存储器用于存储程序，处理器用于执行存储器中的程序，以执行如上述第一方面及其任一可选的方法。
第六方面，本申请实施例提供了一种执行装置，可以包括存储器、处理器以及总线***，其中，存储器用于存储程序，处理器用于执行存储器中的程序，以执行如上述第二方面及其任一可选的方法。
第七方面，本申请实施例提供了一种计算机可读存储介质，所述计算机可读存储介质中存储有计算机程序，当其在计算机上运行时，使得计算机执行上述第一方面及其任一可选的方法、以及上述第二方面及其任一可选的方法。
第八方面，本申请实施例提供了一种计算机程序，当其在计算机上运行时，使得计算机执行上述第一方面及其任一可选的方法、以及上述第二方面及其任一可选的方法。
第九方面，本申请提供了一种芯片***，该芯片***包括处理器，用于支持执行数据处理装置实现上述方面中所涉及的功能，例如，发送或处理上述方法中所涉及的数据；或，信息。在一种可能的设计中，所述芯片***还包括存储器，所述存储器，用于保存执行设备或训练设备必要的程序指令和数据。该芯片***，可以由芯片构成，也可以包括芯片和其他分立器件。
附图说明
图1A为人工智能主体框架的一种结构示意图；
图1B和图1C为本发明的应用***框架示意；
图1D为终端的一种可选的硬件结构示意图；
图2为一种服务器的结构示意图；
图3为本申请的一种***架构示意；
图4为一种云服务的流程；
图5为一种云服务的流程；
图6为本申请的一种***架构示意；
图7为本申请实施例提供的一种数据处理方法的流程示意；
图8至图10为本申请实施例提供的一种数据处理方法的流程示意；
图11为本申请的一种***架构示意；
图12为本申请实施例提供的数据处理装置的一种结构示意图；
图13为本申请实施例提供的执行设备的一种结构示意图；
图14为本申请实施例提供的训练设备一种结构示意图；
图15为本申请实施例提供的芯片的一种结构示意图。
具体实施方式
下面结合本发明实施例中的附图对本发明实施例进行描述。本发明的实施方式部分使用的术语仅用于对本发明的具体实施例进行解释，而非旨在限定本发明。
下面结合附图，对本申请的实施例进行描述。本领域普通技术人员可知，随着技术的发展和新场景的出现，本申请实施例提供的技术方案对于类似的技术问题，同样适用。
本申请的说明书和权利要求书及上述附图中的术语“第一”、“第二”等是用于区别类似的对象，而不必用于描述特定的顺序或先后次序。应该理解这样使用的术语在适当情况下可以互换，这仅仅是描述本申请的实施例中对相同属性的对象在描述时所采用的区分方式。此外，术语“包括”和“具有”以及他们的任何变形，意图在于覆盖不排他的包含，以便包含一系列单元的过程、方法、***、产品或设备不必限于那些单元，而是可包括没有清楚地列出的或对于这些过程、方法、产品或设备固有的其它单元。
本文中所用用语“基本(substantially)”、“大约(about)”及类似用语用作近似用语、而并非用作程度用语，且旨在考虑到所属领域中的普通技术人员将知的测量值或计算值的固有偏差。此外，在阐述 本发明实施例时使用“可(may)”是指“可能的一个或多个实施例”。本文中所用用语“使用(use)”、“正使用(using)”、及“被使用(used)”可被视为分别与用语“利用(utilize)”、“正利用(utilizing)”、及“被利用(utilized)”同义。另外，用语“示例性(exemplary)”旨在指代实例或例示。
首先对人工智能***总体工作流程进行描述，请参见图1A，图1A示出的为人工智能主体框架的一种结构示意图，下面从“智能信息链”(水平轴)和“IT价值链”(垂直轴)两个维度对上述人工智能主题框架进行阐述。其中，“智能信息链”反映从数据的获取到处理的一列过程。举例来说，可以是智能信息感知、智能信息表示与形成、智能推理、智能决策、智能执行与输出的一般过程。在这个过程中，数据经历了“数据—信息—知识—智慧”的凝练过程。“IT价值链”从人智能的底层基础设施、信息(提供和处理技术实现)到***的产业生态过程，反映人工智能为信息技术产业带来的价值。
(1)基础设施
基础设施为人工智能***提供计算能力支持，实现与外部世界的沟通，并通过基础平台实现支撑。通过传感器与外部沟通；计算能力由智能芯片(CPU、NPU、GPU、ASIC、FPGA等硬件加速芯片)提供；基础平台包括分布式计算框架及网络等相关的平台保障和支持，可以包括云存储和计算、互联互通网络等。举例来说，传感器和外部沟通获取数据，这些数据提供给基础平台提供的分布式计算***中的智能芯片进行计算。
(2)数据
基础设施的上一层的数据用于表示人工智能领域的数据来源。数据涉及到图形、图像、语音、文本，还涉及到传统设备的物联网数据，包括已有***的业务数据以及力、位移、液位、温度、湿度等感知数据。
(3)数据处理
数据处理通常包括数据训练，机器学习，深度学习，搜索，推理，决策等方式。
其中，机器学习和深度学习可以对数据进行符号化和形式化的智能信息建模、抽取、预处理、训练等。
推理是指在计算机或智能***中，模拟人类的智能推理方式，依据推理控制策略，利用形式化的信息进行机器思维和求解问题的过程，典型的功能是搜索与匹配。
决策是指智能信息经过推理后进行决策的过程，通常提供分类、排序、预测等功能。
(4)通用能力
对数据经过上面提到的数据处理后，进一步基于数据处理的结果可以形成一些通用的能力，比如可以是算法或者一个通用***，例如，翻译，文本的分析，计算机视觉的处理，语音识别，图像的识别等等。
(5)智能产品及行业应用
智能产品及行业应用指人工智能***在各领域的产品和应用，是对人工智能整体解决方案的封装，将智能信息决策产品化、实现落地应用，其应用领域主要包括：智能终端、智能交通、智能医疗、自动驾驶、智慧城市等。
本申请可以应用于人工智能领域的自然语言处理领域中，下面以自然语言处理为例将对多个落地到产品的多个应用场景进行介绍。
首先介绍本申请的应用场景，本申请可以但不限于应用在具有对图像的文本处理功能的应用程序(以下可以简称为跨模态的语言处理类应用程序)或者云侧服务器提供的云服务等，接下来分别进行介绍：
一、跨模态的语言处理类应用程序
本申请实施例的产品形态可以为跨模态的语言处理类应用程序。跨模态的语言处理类应用程序可以运行在终端设备或者云侧的服务器上。
在一种可能的实现中，跨模态的语言处理类应用程序可以实现基于输入的图像进行图像中文本相关的处理任务，得到处理结果。
示例性的，上述图像中文本相关的处理任务可以包括信息提取、合同审阅和检索与问答等。在信息 提取场景中，用户可以自定义图像中所关心的字段，如甲方、乙方、合同编号等。在合同审阅场景中，相关人员可以确认不同版本的合同是否一致，如甲方的名称发生了变化是否会引入风险条款等。在检索与问答场景中，可以帮助用户通过提出问题或关键词快速检索出图像中的相关文本，并在文本中找出可能的答案。
示例性的，前述图像为通过摄像机、打印机、扫描机等设备采集得到的图像。作为示例，例如在一种应用场景中，在金融、会计和税务领域等领域中，企业需要对收据或***等文件进行扫描得到图像文件，并对图像文件中的文本进行识别以提取文本信息，进而能够实现文件数字化归档、文件快速索引或文件分析等功能。在另一种应用场景中，用户需要输入身份证、驾驶证、行驶证或护照等证件上的信息，则用户可以利用摄像机采集得到前述证件的图像，并对图像中的文本进行识别以提取出关键信息等。
应当理解，此处举例仅为方便对本申请实施例的应用场景进行理解，不对本申请实施例的应用场景进行穷举。
在一种可能的实现中，用户可以打开终端设备上安装的跨模态的语言处理类应用程序，并输入图像，跨模态的语言处理类应用程序可以通过本申请实施例提供的方法训练得到的跨模态语言模型对图像进行处理，并将处理结果呈现给用户(呈现方式可以但不限于是显示、保存、上传到云侧等)。
在一种可能的实现中，用户可以打开终端设备上安装的跨模态的语言处理类应用程序，并输入图像，跨模态的语言处理类应用程序可以将图像发送至云侧的服务器，云侧的服务器通过本申请实施例提供的方法训练得到的跨模态语言模型对图像进行处理，并将处理结果回传至终端设备，终端设备可以将处理结果呈现给用户(呈现方式可以但不限于是显示、保存、上传到云侧等)。
接下来分别从功能架构以及实现功能的产品架构介绍本申请实施例中的跨模态的语言处理类应用程序。
参照图1B，图1B为本申请实施例中跨模态的语言处理类应用程序的功能架构示意：
在一种可能的实现中，如图1B所示，跨模态的语言处理类应用程序102可接收输入的参数101(例如包含图像)且产生处理结果103。跨模态的语言处理类应用程序102可在(举例来说)至少一个计算机***上执行，且包括计算机代码，所述计算机代码在由一或多个计算机执行时致使所述计算机执行用于执行通过本申请实施例提供的方法训练得到的跨模态语言模型。
参照图1C，图1C为本申请实施例中运行跨模态的语言处理类应用程序的实体架构示意：
参见图1C，图1C示出了一种***架构示意图。该***可以包括终端100、以及服务器200。其中，服务器200可以包括一个或者多个服务器(图1C中以包括一个服务器作为示例进行说明)，服务器200可以为一个或者多个终端提供跨模态的语言处理功能服务。
其中，终端100上可以安装有跨模态的语言处理类应用程序，或者打开与跨模态的语言处理功能相关的网页，上述应用程序和网页可以提供一个界面，终端100可以接收用户在跨模态的语言处理功能界面上输入的相关参数，并将上述参数发送至服务器200，服务器200可以基于接收到的参数，得到处理结果，并将处理结果返回至至终端100。
应理解，在一些可选的实现中，终端100也可以由自身完成基于接收到的参数，得到处理结果的动作，而不需要服务器配合实现，本申请实施例并不限定。
接下来描述图1C中终端100的产品形态；
本申请实施例中的终端100可以为手机、平板电脑、可穿戴设备、车载设备、增强现实(augmented reality，AR)/虚拟现实(virtual reality，VR)设备、笔记本电脑、超级移动个人计算机(ultra-mobile personal computer，UMPC)、上网本、个人数字助理(personal digital assistant，PDA)等，本申请实施例对此不作任何限制。
图1D示出了终端100的一种可选的硬件结构示意图。
参考图1D所示，终端100可以包括射频单元110、存储器120、输入单元130、显示单元140、摄像头150(可选的)、音频电路160(可选的)、扬声器161(可选的)、麦克风162(可选的)、处理器170、外部接口180、电源190等部件。本领域技术人员可以理解，图1D仅仅是终端或多功能设备的举例，并 不构成对终端或多功能设备的限定，可以包括比图示更多或更少的部件，或者组合某些部件，或者不同的部件。
输入单元130可用于接收输入的数字或字符信息，以及产生与该便携式多功能装置的用户设置以及功能控制有关的键信号输入。具体地，输入单元130可包括触摸屏131(可选的)和/或其他输入设备132。该触摸屏131可收集用户在其上或附近的触摸操作(比如用户使用手指、关节、触笔等任何适合的物体在触摸屏上或在触摸屏附近的操作)，并根据预先设定的程序驱动相应的连接装置。触摸屏可以检测用户对触摸屏的触摸动作，将该触摸动作转换为触摸信号发送给该处理器170，并能接收该处理器170发来的命令并加以执行；该触摸信号至少包括触点坐标信息。该触摸屏131可以提供该终端100和用户之间的输入界面和输出界面。此外，可以采用电阻式、电容式、红外线以及表面声波等多种类型实现触摸屏。除了触摸屏131，输入单元130还可以包括其他输入设备。具体地，其他输入设备132可以包括但不限于物理键盘、功能键(比如音量控制按键132、开关按键133等)、轨迹球、鼠标、操作杆等中的一种或多种。
其中，输入设备132可以接收到输入的图像等等。
该显示单元140可用于显示由用户输入的信息或提供给用户的信息、终端100的各种菜单、交互界面、文件显示和/或任意一种多媒体文件的播放。在本申请实施例中，显示单元140可用于显示跨模态的语言处理类应用程序的界面、处理结果等。
该存储器120可用于存储指令和数据，存储器120可主要包括存储指令区和存储数据区，存储数据区可存储各种数据，如多媒体文件、文本等；存储指令区可存储操作***、应用、至少一个功能所需的指令等软件单元，或者他们的子集、扩展集。还可以包括非易失性随机存储器；向处理器170提供包括管理计算处理设备中的硬件、软件以及数据资源，支持控制软件和应用。还用于多媒体文件的存储，以及运行程序和应用的存储。
处理器170是终端100的控制中心，利用各种接口和线路连接整个终端100的各个部分，通过运行或执行存储在存储器120内的指令以及调用存储在存储器120内的数据，执行终端100的各种功能和处理数据，从而对终端设备进行整体控制。可选的，处理器170可包括一个或多个处理单元；优选的，处理器170可集成应用处理器和调制解调处理器，其中，应用处理器主要处理操作***、用户界面和应用程序等，调制解调处理器主要处理无线通信。可以理解的是，上述调制解调处理器也可以不集成到处理器170中。在一些实施例中，处理器、存储器、可以在单一芯片上实现，在一些实施例中，他们也可以在独立的芯片上分别实现。处理器170还可以用于产生相应的操作控制信号，发给计算处理设备相应的部件，读取以及处理软件中的数据，尤其是读取和处理存储器120中的数据和程序，以使其中的各个功能模块执行相应的功能，从而控制相应的部件按指令的要求进行动作。
其中，存储器120可以用于存储数据处理方法相关的软件代码，处理器170可以执行芯片的数据处理方法的步骤，也可以调度其他单元(例如上述输入单元130以及显示单元140)以实现相应的功能。
该射频单元110(可选的)可用于收发信息或通话过程中信号的接收和发送，例如，将基站的下行信息接收后，给处理器170处理；另外，将设计上行的数据发送给基站。通常，RF电路包括但不限于天线、至少一个放大器、收发信机、耦合器、低噪声放大器(Low Noise Amplifier，LNA)、双工器等。此外，射频单元110还可以通过无线通信与网络设备和其他设备通信。该无线通信可以使用任一通信标准或协议，包括但不限于全球移动通讯***(Global System of Mobile communication，GSM)、通用分组无线服务(General Packet Radio Service，GPRS)、码分多址(Code Division Multiple Access，CDMA)、宽带码分多址(Wideband Code Division Multiple Access，WCDMA)、长期演进(Long Term Evolution，LTE)、电子邮件、短消息服务(Short Messaging Service，SMS)等。
其中，在本申请实施例中，该射频单元110可以将图像发送至服务器200，并接收到服务器200发送的处理结果。
应理解，该射频单元110为可选的，其可以被替换为其他通信接口，例如可以是网口。
终端100还包括给各个部件供电的电源190(比如电池)，优选的，电源可以通过电源管理***与处理器170逻辑相连，从而通过电源管理***实现管理充电、放电、以及功耗管理等功能。
终端100还包括外部接口180，该外部接口可以是标准的Micro USB接口，也可以使多针连接器，可以用于连接终端100与其他装置进行通信，也可以用于连接充电器为终端100充电。
尽管未示出，终端100还可以包括闪光灯、无线保真(wireless fidelity，WiFi)模块、蓝牙模块、不同功能的传感器等，在此不再赘述。下文中描述的部分或全部方法均可以应用在如图1D所示的终端100中。
接下来描述图1C中服务器200的产品形态；
图2提供了一种服务器200的结构示意图，如图2所示，服务器200包括总线201、处理器202、通信接口203和存储器204。处理器202、存储器204和通信接口203之间通过总线201通信。
总线201可以是外设部件互连标准(peripheral component interconnect，PCI)总线或扩展工业标准结构(extended industry standard architecture，EISA)总线等。总线可以分为地址总线、数据总线、控制总线等。为便于表示，图2中仅用一条粗线表示，但并不表示仅有一根总线或一种类型的总线。
处理器202可以为中央处理器(central processing unit，CPU)、图形处理器(graphics processing unit，GPU)、微处理器(micro processor，MP)或者数字信号处理器(digital signal processor，DSP)等处理器中的任意一种或多种。
存储器204可以包括易失性存储器(volatile memory)，例如随机存取存储器(random access memory，RAM)。存储器204还可以包括非易失性存储器(non-volatile memory)，例如只读存储器(read-only memory，ROM)，快闪存储器，机械硬盘(hard drive drive，HDD)或固态硬盘(solid state drive，SSD)。
其中，存储器204可以用于存储数据处理方法相关的软件代码，处理器202可以执行芯片的数据处理方法的步骤，也可以调度其他单元以实现相应的功能。
应理解，上述终端100和服务器200可以为集中式或者是分布式的设备，上述终端100和服务器200中的处理器(例如处理器170以及处理器202)可以为硬件电路(如专用集成电路(application specific integrated circuit，ASIC)、现场可编程门阵列(field-programmable gate array，FPGA)、通用处理器、数字信号处理器(digital signal processing，DSP)、微处理器或微控制器等等)、或这些硬件电路的组合，例如，处理器可以为具有执行指令功能的硬件***，如CPU、DSP等，或者为不具有执行指令功能的硬件***，如ASIC、FPGA等，或者为上述不具有执行指令功能的硬件***以及具有执行指令功能的硬件***的组合。
应理解，本申请实施例中的和模型推理过程相关的步骤涉及AI相关的运算，在执行AI运算时，终端设备和服务器的指令执行架构不仅仅局限在上述介绍的处理器结合存储器的架构。下面结合图3对本申请实施例提供的***架构进行详细的介绍。
图3为本申请实施例提供的***架构示意图。如图3所示，***架构500包括执行设备510、训练设备520、数据库530、客户设备540、数据存储***550以及数据采集***560。
执行设备510包括计算模块511、I/O接口512、预处理模块513和预处理模块514。计算模块511中可以包括目标模型/规则501，预处理模块513和预处理模块514是可选的。
其中，执行设备510可以为上述运行跨模态的语言处理类应用程序的终端设备或者服务器。
数据采集设备560用于采集训练样本。训练样本可以为多个图像等。在采集到训练样本之后，数据采集设备560将这些训练样本存入数据库530。
训练设备520可以基于数据库530中维护训练样本，对待训练的神经网络(例如本申请实施例中的跨模态语言模型(例如包括文本编码器、图像编码器、目标编码器等))，以得到目标模型/规则501。
应理解，训练设备520可以基于数据库530中维护训练样本，对待训练的神经网络进行预训练过程，或者是在预训练的基础上进行模型的微调。
需要说明的是，在实际应用中，数据库530中维护的训练样本不一定都来自于数据采集设备560的采集，也有可能是从其他设备接收得到的。另外需要说明的是，训练设备520也不一定完全基于数据库530维护的训练样本进行目标模型/规则501的训练，也有可能从云端或其他地方获取训练样本进行模型 训练，上述描述不应该作为对本申请实施例的限定。
根据训练设备520训练得到的目标模型/规则501可以应用于不同的***或设备中，如应用于图3所示的执行设备510，该执行设备510可以是终端，如手机终端，平板电脑，笔记本电脑，增强现实(augmented reality，AR)/虚拟现实(virtual reality，VR)设备，车载终端等，还可以是服务器等。
具体的，训练设备520可以将训练后的模型传递至执行设备510。
在图3中，执行设备510配置输入/输出(input/output，I/O)接口512，用于与外部设备进行数据交互，用户可以通过客户设备540向I/O接口512输入数据(例如本申请实施例中的图像等)。
预处理模块513和预处理模块514用于根据I/O接口512接收到的输入数据进行预处理。应理解，可以没有预处理模块513和预处理模块514或者只有的一个预处理模块。当不存在预处理模块513和预处理模块514时，可以直接采用计算模块511对输入数据进行处理。
在执行设备510对输入数据进行预处理，或者在执行设备510的计算模块511执行计算等相关的处理过程中，执行设备510可以调用数据存储***550中的数据、代码等以用于相应的处理，也可以将相应处理得到的数据、指令等存入数据存储***550中。
最后，I/O接口512将处理结果提供给客户设备540，从而提供给用户。
在图3所示情况下，用户可以手动给定输入数据，该“手动给定输入数据”可以通过I/O接口512提供的界面进行操作。另一种情况下，客户设备540可以自动地向I/O接口512发送输入数据，如果要求客户设备540自动发送输入数据需要获得用户的授权，则用户可以在客户设备540中设置相应权限。用户可以在客户设备540查看执行设备510输出的结果，具体的呈现形式可以是显示、声音、动作等具体方式。客户设备540也可以作为数据采集端，采集如图所示输入I/O接口512的输入数据及输出I/O接口512的输出结果作为新的样本数据，并存入数据库530。当然，也可以不经过客户设备540进行采集，而是由I/O接口512直接将如图所示输入I/O接口512的输入数据及输出I/O接口512的输出结果，作为新的样本数据存入数据库530。
值得注意的是，图3仅是本申请实施例提供的一种***架构的示意图，图中所示设备、器件、模块等之间的位置关系不构成任何限制，例如，在图3中，数据存储***550相对执行设备510是外部存储器，在其它情况下，也可以将数据存储***550置于执行设备510中。应理解，上述执行设备510可以部署于客户设备540中。
从模型的推理侧来说：
本申请实施例中，上述执行设备520的计算模块511可以获取到数据存储***550中存储的代码来实现本申请实施例中的和模型推理过程相关的步骤。
本申请实施例中，执行设备520的计算模块511可以包括硬件电路(如专用集成电路(application specific integrated circuit，ASIC)、现场可编程门阵列(field-programmable gate array，FPGA)、通用处理器、数字信号处理器(digital signal processing，DSP)、微处理器或微控制器等等)、或这些硬件电路的组合，例如，训练设备520可以为具有执行指令功能的硬件***，如CPU、DSP等，或者为不具有执行指令功能的硬件***，如ASIC、FPGA等，或者为上述不具有执行指令功能的硬件***以及具有执行指令功能的硬件***的组合。
具体的，执行设备520的计算模块511可以为具有执行指令功能的硬件***，本申请实施例提供的和模型推理过程相关的步骤可以为存储在存储器中的软件代码，执行设备520的计算模块511可以从存储器中获取到软件代码，并执行获取到的软件代码来实现本申请实施例提供的和模型推理过程相关的步骤。
应理解，执行设备520的计算模块511可以为不具有执行指令功能的硬件***以及具有执行指令功能的硬件***的组合，本申请实施例提供的和模型推理过程相关的步骤的部分步骤还可以通过执行设备520的计算模块511中不具有执行指令功能的硬件***来实现，这里并不限定。
从模型的训练侧来说：
本申请实施例中，上述训练设备520可以获取到存储器(图3中未示出，可以集成于训练设备520或者与训练设备520分离部署)中存储的代码来实现本申请实施例中和模型训练相关的步骤。
本申请实施例中，训练设备520可以包括硬件电路(如专用集成电路(application specific integrated circuit，ASIC)、现场可编程门阵列(field-programmable gate array，FPGA)、通用处理器、数字信号处理器(digital signal processing，DSP)、微处理器或微控制器等等)、或这些硬件电路的组合，例如，训练设备520可以为具有执行指令功能的硬件***，如CPU、DSP等，或者为不具有执行指令功能的硬件***，如ASIC、FPGA等，或者为上述不具有执行指令功能的硬件***以及具有执行指令功能的硬件***的组合。
应理解，训练设备520可以为不具有执行指令功能的硬件***以及具有执行指令功能的硬件***的组合，本申请实施例提供的中和模型训练相关的部分步骤还可以通过训练设备520中不具有执行指令功能的硬件***来实现，这里并不限定。
二、服务器提供的跨模态的语言处理功能类云服务：
在一种可能的实现中，服务器可以通过应用程序编程接口(application programming interface，API)为端侧提供跨模态的语言处理功能的服务。
其中，终端设备可以通过云端提供的API，将相关参数(例如图像)发送至服务器，服务器可以基于接收到的参数，得到处理结果等)，并将处理结果返回至至终端。
关于终端以及服务器的描述可以上述实施例的描述，这里不再赘述。
如图4示出了使用一项云平台提供的跨模态的语言处理功能类云服务的流程。
1.开通并购买内容审核服务。
2.用户可以下载内容审核服务对应的软件开发工具包(software development kit，SDK)，通常云平台提供多个开发版本的SDK，供用户根据开发环境的需求选择，例如JAVA版本的SDK、python版本的SDK、PHP版本的SDK、Android版本的SDK等。
3.用户根据需求下载对应版本的SDK到本地后，将SDK工程导入至本地开发环境，在本地开发环境中进行配置和调试，本地开发环境还可以进行其他功能的开发，使得形成一个集合了跨模态的语言处理功能类能力的应用。
4.跨模态的语言处理功能类应用在被使用的过程中，当需要进行跨模态的语言处理功能时，可以触发跨模态的语言处理功能的API调用。当应用触发跨模态的语言处理功能功能时，发起API请求至云环境中的跨模态的语言处理功能类服务的运行实例，其中，API请求中携带图像，由云环境中的运行实例对图像进行处理，获得处理结果。
5.云环境将处理结果返回至应用，由此完成一次的跨模态的语言处理功能服务调用。
三、服务器提供的模型训练类云服务：
在一种可能的实现中，服务器可以基于客户提供的训练数据(例如可以包括图像)，来预训练一个具备跨模态的语言处理功能的模型。
在一种可能的实现中，服务器可以通过应用程序编程接口(application programming interface，API)为端侧提供模型训练的服务。
其中，终端设备可以通过云端提供的API，将相关参数发送至服务器，服务器可以基于接收到的参数，得到处理结果，并将处理结果(例如具备跨模态的语言处理功能的模型等)返回至终端。
如图5示出了使用一项云平台提供的模型训练类云服务的流程。
关于终端以及服务器的描述可以上述实施例的描述，这里不再赘述。
四、联邦训练
联邦学***均算法更新中心节点模型的参数。
参见图6，图6为本申请实施例提供的一种模型训练方法的架构示意，如图6所示，本申请实施例提供的架构包括：云侧中心节点，例如可以是云侧的服务器。A1、A2、…为类型为A的分布式节点，如用户持有的手机产品。B1、B2、…为类型为B的分布式节点，如用户持有的个人电脑。在经过分布式节点 的管理员(如手机、电脑的用户)同意后，分布式节点的管理员自愿在隐私得到保护的情况下共享其日常使用设备的过程中产生的数据，加入到模型训练计划，设备成为架构中的分布式节点。本实施例中的***也可以包含更多类型的分布式节点，如智能手表等等。为保护数据隐私，分布式节点不会将数据上传至中心节点，仅在本地保存数据。分布式节点通过通信网络与云服务器连接。云侧中心节点可以运行大模型，而各分布式节点受硬件能力限制只能运行小模型，且A和B可以拥有不同的数据处理能力。
由于本申请实施例涉及大量神经网络的应用，为了便于理解，下面先对本申请实施例涉及的相关术语及神经网络等相关概念进行介绍。
(1)神经网络
神经网络可以是由神经单元组成的，神经单元可以是指以xs(即输入数据)和截距1为输入的运算单元，该运算单元的输出可以为：
其中，s＝1、2、……n，n为大于1的自然数，Ws为xs的权重，b为神经单元的偏置。f为神经单元的激活函数(activation functions)，用于将非线性特性引入神经网络中，来将神经单元中的输入信号转换为输出信号。该激活函数的输出信号可以作为下一层卷积层的输入，激活函数可以是sigmoid函数。神经网络是将多个上述单一的神经单元联结在一起形成的网络，即一个神经单元的输出可以是另一个神经单元的输入。每个神经单元的输入可以与前一层的局部接受域相连，来提取局部接受域的特征，局部接受域可以是由若干个神经单元组成的区域。
(2)transformer层
神经网络包括嵌入层和至少一个transformer层，至少一个transformer层可以为N个transformer层(N大于0的整数)，其中，每个transformer层包括依次相邻的注意力层、加和与归一化(add&norm)层、前馈(feed forward)层和加和与归一化层。在嵌入层，对当前输入进行嵌入处理，得到多个嵌入向量；在所述注意力层，从所述第一transformer层的上一层获取P个输入向量，以P个输入向量中的任意的第一输入向量为中心，基于预设的注意力窗口范围内的各个输入向量与该第一输入向量之间的关联度，得到该第一输入向量对应的中间向量，如此确定出P个输入向量对应的P个中间向量；在所述池化层，将所述P个中间向量合并为Q个输出向量，其中transformer层中最后一个transformer层得到的多个输出向量用作所述当前输入的特征表示。
(3)注意力机制(attention mechanism)
注意力机制模仿了生物观察行为的内部过程，即一种将内部经验和外部感觉对齐从而增加部分区域的观察精细度的机制，能够利用有限的注意力资源从大量信息中快速筛选出高价值信息。注意力机制可以快速提取稀疏数据的重要特征，因而被广泛用于自然语言处理任务，特别是机器翻译。而自注意力机制(self-attention mechanism)是注意力机制的改进，其减少了对外部信息的依赖，更擅长捕捉数据或特征的内部相关性。注意力机制的本质思想可以改写为如下公式：
其中，Lx＝||Source||代表Source的长度，公式含义即将Source中的构成元素想象成是由一系列的数据对构成，此时给定目标Target中的某个元素Query，通过计算Query和各个Key的相似性或者相关性，得到每个Key对应Value的权重系数，然后对Value进行加权求和，即得到了最终的Attention数值。所以本质上Attention机制是对Source中元素的Value值进行加权求和，而Query和Key用来计算对应Value的权重系数。从概念上理解，把Attention可以理解为从大量信息中有选择地筛选出少量重要信息并聚焦到这些重要信息上，忽略大多不重要的信息。聚焦的过程体现在权重系数的计算上，权重越大越聚焦于其对应的Value值上，即权重代表了信息的重要性，而Value是其对应的信息。自注意力机制可以理解为内部Attention(intra attention)，Attention机制发生在Target的元素Query和Source中的所有元素之间，自注意力机制指的是在Source内部元素之间或者Target内部元素之间 发生的Attention机制，也可以理解为Target＝Source这种特殊情况下的注意力计算机制，其具体计算过程是一样的，只是计算对象发生了变化而已。
(4)自然语言处理(natural language processing，NLP)
自然语言(natural language)即人类语言，自然语言处理(NLP)就是对人类语言的处理。自然语言处理是以一种智能与高效的方式，对文本数据进行***化分析、理解与信息提取的过程。通过使用NLP及其组件，我们可以管理非常大块的文本数据，或者执行大量的自动化任务，并且解决各式各样的问题，如自动摘要(automatic summarization)，机器翻译(machine translation，MT)，命名实体识别(named entity recognition，NER)，关系提取(relation extraction，RE)，信息抽取(information extraction，IE)，情感分析，语音识别(speech recognition)，问答***(question answering)以及主题分割等等。
(5)预训练语言模型(pre-trained language model)
预训练语言模型是一个自然语言序列编码器，为自然语言序列中的每个词进行编码成为一个向量表示，从而进行预测任务。它的训练包含两个阶段。在预训练(pre-training)阶段，该模型在大规模无监督文本上进行语言模型任务的训练，从而学习到一个词表示。在微调(finetuning)阶段，该模型利用预训练阶段学到的参数做初始化，在文本分类(text classification)，序列标注(sequence labeling)等下游任务(downstream task)上进行较少步骤的训练，就可以成功把预训练得到的语义信息成功迁移到下游任务上来。
(6)反向传播算法
卷积神经网络可以采用误差反向传播(back propagation，BP)算法在训练过程中修正初始的超分辨率模型中参数的大小，使得超分辨率模型的重建误差损失越来越小。具体地，前向传递输入信号直至输出会产生误差损失，通过反向传播误差损失信息来更新初始的超分辨率模型中参数，从而使误差损失收敛。反向传播算法是以误差损失为主导的反向传播运动，旨在得到最优的超分辨率模型的参数，例如权重矩阵。
(7)损失函数
在训练深度神经网络的过程中，因为希望深度神经网络的输出尽可能的接近真正想要预测的值，所以可以通过比较当前网络的预测值和真正想要的目标值，再根据两者之间的差异情况来更新每一层神经网络的权重向量(当然，在第一次更新之前通常会有初始化的过程，即为深度神经网络中的各层预先配置参数)，比如，如果网络的预测值高了，就调整权重向量让它预测低一些，不断地调整，直到深度神经网络能够预测出真正想要的目标值或与真正想要的目标值非常接近的值。因此，就需要预先定义“如何比较预测值和目标值之间的差异”，这便是损失函数(loss function)或目标函数(objective function)，它们是用于衡量预测值和目标值的差异的重要方程。其中，以损失函数举例，损失函数的输出值(loss)越高表示差异越大，那么深度神经网络的训练就变成了尽可能缩小这个loss的过程。
现有的跨模态视觉语言模型(或者可以称之为多模态模型或者多模态语言模型)所采用的对齐方式中的文本建模能力较弱。以TIA为例，TIA只是简单的判断某个字符是否被遮挡，并没有很好的考虑到遮挡字符之间的语义信息导致模型不是真正的理解文档的内容。
当前的多模态模型的图文对齐能力较弱，也就是不具备文本和图像元素的对齐能力。以图文匹配(text-image matching，TIM)为例，训练目标用来判断全局文字是否和全局的扫描文档图片是否匹配，该任务学习难度小，使得训练的模型不具备文本和图像元素的对齐能力。
为了解决上述问题，本申请实施例提供了一种数据处理方法。下面结合附图对本申请实施例的数据处理方法进行详细的介绍。
参照图7，图7为本申请实施例提供的一种数据处理方法的流程示意，如图7所示，本申请实施例提供的一种数据处理方法，可以包括步骤701至703，下面分别对这些步骤进行详细的描述。
701、获取第一特征表示以及第二特征表示，所述第一特征表示为根据图像编码器对图像进行处理得到的图像特征得到，所述第二特征表示为第一文本的文本特征；所述第一文本为所述图像中包括的文本内容。
在一种可能的实现中，可以获取到原始的输入图像，并通过图像编码器对原始的输入图像进行编码，得到输入图像的编码结果，该编码结果可以包括输入图像的图像特征表示。应理解，本申请中的原始的输入图像可以指输入到扩模态的语言模型中的图像、或者是对扩模态的语言模型中的图像进行预处理后的图像。
在一种可能的实现中，图像编码器可以对输入图像进行特征提取，其中，图像编码器具体可以表现为卷积神经网络、方向梯度直方图(histogram of oriented gradient，HOG)、局部二值模式(local binarypattern，LBP)或其他用于对图像进行特征提取的神经网络等。
在一种可能的实现中，可以通过图像编码器对输入图像进行处理，得到输入图像对应的图像特征表示。
在一种可能的实现中，可以识别出图像中所包含的文本。例如可以通过光学字符识别(optical character recognition，OCR)来识别出图像中所包含的文本。可选的，还可以识别出文本中各个文本单元对应的位置信息，该位置信息可以指示文本单元所在的区域在图像中的位置，例如，文本单元所在的区域可以通过矩形框表示，位置信息可以为矩形框的左上点坐标和右下点坐标。
在一种可能的实现中，可以获取到文本对应的文本特征表示。
在一种可能的实现中，可以通过嵌入层对文本进行嵌入处理，以得到嵌入向量，该嵌入向量可以为文本对应的文本特征表示。
其中，嵌入层可以称为输入嵌入(input embedding)层。当前输入可以为文本中的每个文本单元。嵌入层在获取当前输入后，可以对该当前输入中各个文本单元进行嵌入处理，可得到各个文本单元对应的嵌入向量。
在一些实施例中，还可以获取每个文本单元的位置向量，所述位置向量用于指示文本单元的位置；其中，位置用于表示文本单元在文本中的位置，具体的，所述位置用于指示文本单元与文本中其他文本单元之间的相对位置关系。
在一种实现中，所述嵌入层可以包括输入嵌入层和位置编码(positional encoding)层。在输入嵌入层，可以对当前输入中的各个文本单元进行词嵌入处理，从而得到各个文本单元的嵌入向量。在位置编码层，可以获取各个文本单元在该当前输入中的位置，进而对各个文本单元的位置生成位置向量。
在一些示例中，各个文本单元在文本中的位置可以为各个文本单元在文本中的绝对位置。以当前输入为“几号应还花呗”为例，其中的“几”的位置可以表示为第一位，“号”的位置可以表示为第二位，……。在一些示例中，各个文本单元在文本中的位置可以为各个文本单元在文本中的相对位置。仍以当前输入为“几号应还花呗”为例，其中的“几”的位置可以表示为“号”之前，“号”的位置可以表示为“几”之后、“应”之前，……。当得到当前输入中各个文本单元的嵌入向量和位置向量时，可以将各个文本单元的位置向量和对应的嵌入向量进行融合，得到各个文本单元的嵌入向量，即得到该当前输入对应的多个嵌入向量。应理解，融合的方式可以是对嵌入向量和位置向量进行加法运算，或者是通过其他运算使得嵌入向量可以携带文本中的一个文本单元以及所述一个文本单元在所述文本中的位置的信息，这里并不限定具体的融合方式。例如，融合的方式包括但不限于拼接(conact)、相加(add)、融合(fusion)和相乘等。
在一种可能的实现中，可以通过嵌入层对文本进行嵌入处理，以得到嵌入向量，并通过文本编码器对嵌入向量进行处理，得到文本对应的文本特征表示。
本申请实施例中，通过图文对比学习(text-image contrastive learning，TIC)来增强图文之间的匹配能力，也就是提高模型对于图像信息和文本信息这种跨模态的数据的信息处理能力。具体的，针对于图像的图像特征和文本的文本特征，通过相似度比对构建的损失，来更新图形编码器(如果文本特征为通过文本编码器得到的，也可以更新文本编码器)。虽然图像的图像特征和图像中文本的文本特征不是同一个模态的特征，但是由于文本是图像中的文本，在图像特征中也会蕴含一定的文本的语义信息，因此，图像特征和文本特征(属于同一个图像)之间在语义维度(或者其他信息维度上)会存在关联。上述相似度可以包含图像中语义信息和文本的语义信息之间的相似度，因此，基于该相似度构建的损失来更新图像编码器以及文本编码器(如果架构中存在文本编码器的话)，能够使得图像编码器和文本编 码器提取的特征中蕴含更准确的图像中文本的语义信息，进而增强后续网络的图文匹配能力。
此外，在跨模态的语言模型的架构中，包括图形编码器、文本编码器以及用于提供特征之间交互信息的目标编码器，本申请中，将模型的中间输出(也就是图形编码器和文本编码器的输出)之间的相似度构建的损失来更新图形编码器和文本编码器，使得基于图形编码器和文本编码器输出的特征表示就可以实现下游任务(精度没有基于目标编码器输出的特征表示进行的下游任务高)，而在一些场景中，由于下游任务要处理的数据的数量较大，因此可以使用图形编码器和文本编码器输出的特征表示进行粗排，使用目标编码器输出的特征表示进行精排，以提高召回率。
以需要进行图文对比学习(也就是文本-图像对齐)的对象为图像以及第一文本为例进行说明：
在一种可能的实现中，图像可以为原始的输入图像(或者是原始的输入图像的部分图像，但该部分图像包含输入图像的全部文本)。第一文本可以为图像中包含的全部文本。
在一种可能的实现中，图像可以为原始的输入图像(或者是原始的输入图像的部分图像，但该部分图像包含输入图像的全部文本)。第一文本可以为图像中包含的文本中的部分文本。
在一种可能的实现中，图像可以为对原始的输入图像进行提取得到的部分图像区域，该部分图像区域内包括原始的输入图像中全部文本的部分文本，第一文本可以为图像中包括的全部文本。
在一种可能的实现中，图像可以为对原始的输入图像进行提取得到的部分图像区域，该部分图像区域内包括原始的输入图像中全部文本的部分文本，第一文本可以为图像中包含的文本中的部分文本。
在一种可能的实现中，输入的原始图像中可以包括一行或多行文本，或者，输入的原始图像中可以包括一列或多列文本。可选的，图像中可以包括原始图像的一行或者一列文本，第一文本可以为一个或多个文本单元。
在进行图文对比学习时，可以根据图像的图像特征和第一文本的文本特征之间的相似度来构建用于更新图形编码器以及文本编码器的损失，可选的，该相似度与图像中所蕴含的文本语义信息和文本自身的语义信息之间的相似度有关。
应理解，还可以根据图像的图像特征和图像中不包括的文本的文本特征之间的相似度来构建用于更新图形编码器以及文本编码器的损失，区别在于，图像的图像特征和图像中文本的文本特征为正例(也就是对应的标注相似度高)，图像的图像特征和图像中不包括的文本的文本特征为负例(也就是对应的标注相似度低)。
在一种可能的实现中，上述相似度可以为：文本特征和图像特征之间的相似度，也可以为图像特征和文本特征之间的相似度。
在一种可能的实现中，针对于同一张原始的输入图像，可以将输入图像整体对应的图像特征和输入图像中包括的全部文本的文本特征进行比对，也可以将输入图像中的部分图像区域的图像特征和输入图像中包括的部分文本的文本特征进行比对，也可以将输入图像整体对应的图像特征和输入图像中包括的部分文本的文本特征进行比对，或者是上述几种方式的组合。
在一种可能的实现中，针对于同一张原始的输入图像，可以将输入图像中包括的全部文本整体对应的文本特征和输入图像中的部分图像区域的图像特征进行比对，也可以将输入图像中包括的部分文本的文本特征和输入图像中的部分图像区域的图像特征进行比对，也可以将输入图像中包括的全部文本整体对应的文本特征和输入图像整体对应的图像特征进行比对。
在一种可能的实现中，针对于不同的原始输入图像，可以将一部分输入图像中包括的全部文本整体对应的文本特征和输入图像中的部分图像区域的图像特征进行比对，也可以将一部分输入图像中包括的部分文本的文本特征和输入图像中的部分图像区域的图像特征进行比对，也可以将一部分输入图像中包括的全部文本整体对应的文本特征和输入图像整体对应的图像特征进行比对，或者是上述几种方式的组合。
在一种可能的实现中，针对于不同的原始输入图像，可以将一部分输入图像中包括的全部文本整体对应的文本特征和输入图像中的部分图像区域的图像特征进行比对，也可以将一部分输入图像中包括的部分文本的文本特征和输入图像中的部分图像区域的图像特征进行比对，也可以将一部分输入图像中包括的全部文本整体对应的文本特征和输入图像整体对应的图像特征进行比对。
在一种可能的实现中，可以根据图像编码器得到的图像特征，通过图像区域特征提取模块(例如可以通过感兴趣区域(region of interest，ROI)head实现)来确定图像中的部分图像区域的图像特征，例如图像区域特征提取模块可以基于要去提取的图像区域的位置信息，对图像区域的图像特征(图像编码器输出的)进行双线性插值，得到部分图像区域的图像特征。
通过上述方式，利用粗粒度和细粒度对比学习，可以增强图文整体匹配和区域-文本行匹配能力，进而可以提升模型在下游任务如信息提取任务上的性能。
示例性的，在构建图文对比学习时，图像-文字对比学习的损失函数可以定义为：
其中，表示第k个图像和第j个文本之间的相似度。
示例性的，文字-图像对比学习的损失函数可以定义为：
其中，表示第k个文本和第j个图像之间的相似度。
示例性的，文本行-图像区域总的对比学习损失函数可以定义为：
为了建立细粒度的相似性矩阵，图像-文本行区域之间的双向相似度可以分别表述为：
其中，
接下来介绍一个图文对比学习(text-image contrastive learning，TIC)的应用实施例：
参照图8，该实施例可以分为以下几个步骤：
1)将图片缩放到某个分辨率(如448x448)；
2)利用OCR工具获取文档的文字行内容和位置(bounding box)；
3)利用MaskRCNN抽取文本行的视觉表征；
4)利用Roberta抽取文档文本行的文字表征；
5)构建视觉-文本匹配矩阵，选择视觉(文本)对应最匹配的文本(视觉)token；
6)优化图文对比学习TIC loss。
702、根据所述第一特征表示以及所述第二特征表示，通过目标编码器，得到第三特征表示；所述 目标编码器用于实现特征表示之间的交互。
其中，本申请通过先双塔方式可以提升多模态的对齐能力，再利用一个多模态的单塔结构进一步增强特征的交互学习能力，可以利于高效文档检索等下游任务(先粗排后精排)。
在一种可能的实现中，可以根据所述第一特征表示以及所述第二特征表示，通过目标编码器，得到第三特征表示，第三特征表示可以包括第一文本对应的文本特征、以及图像对应的图像特征，和第一特征表示以及所述第二特征表示不同的是，目标编码器输出的特征为根据第一特征表示以及所述第二特征表示之间的交互学习得到的。
703、根据所述第三特征表示，执行下游任务，得到执行结果；所述执行结果和对应的标注之间的相似度以及所述第一特征表示和所述第二特征表示之间的相似度用于更新所述图像编码器。
在一种可能的实现中，目标编码器可以得到第三特征表示，第三特征表示可以用于执行下游任务，以得到执行结果，该执行结果可以和对应的标注构建损失，以更新图形编码器、文本编码器(如果有的话)以及目标编码器。
在一种可能的实现中，可以通过设置不同的下游任务来构建不同的损失。
在一种可能的实现中，下游任务可以包括文本框-网格对齐(bounding-box grid matching，BGM)。
在一种可能的实现中，所述第一文本可以包括第一子文本和第二子文本，所述第二特征表示包括所述第一子文本对应的第一子特征、以及所述第二子文本对应的第二子特征；所述第一子特征不包含所述第一子文本在所述图像中的位置；所述第二子特征包含所述第二子文本在所述图像中的位置；也就是说，在生成第一文本的文本特征时，针对于第一文本中的部分文本(例如第二子文本)，可以将文本以及位置信息生成对应的文本特征(例如第二子特征)，针对于第一文本中的部分文本(例如第一子文本)，可以将文本以及位置信息生成对应的文本特征(例如第一子特征)，相当于所述第一子特征不包含所述第一子文本在所述图像中的位置；所述第二子特征包含所述第二子文本在所述图像中的位置。
在一种可能的实现中，可以通过光学字符识别(optical character recognition，OCR)来识别出图像中所包含的文本以及对应的位置信息，该位置信息可以指示文本所在的区域在图像中的位置，例如，文本所在的区域可以通过矩形框表示，位置信息可以为矩形框的左上点坐标和右下点坐标。
进而，第三特征表示中可以包含第二子文本的位置信息，而不包含第一子文本的位置信息(这里的不包含可以理解为由于输入所导致的，但是可以隐含包含)可以根据第三特征表示预测得到。具体的，可以根据所述第三特征表示，预测所述第一子文本在所述图像中的第一位置；所述第一位置和对应的标注(也就是第一子文本在图像中的真实位置)之间的相似度用于更新所述图像编码器以及所述目标编码器。
在一种可能的实现中，所述图像包括多个图像块；所述第一位置为对所述第一子文本预测所在的图像块；所述标注为所述第一子文本真实所在的图像块。
示例性的，参照图9，本实施例可以分为以下几个步骤：
1)将图片缩放到某个分辨率(如448x448)；
2)将图片区域划分为若干区域(如4x4个grid)；
3)利用OCR得到每个文字的文本框(bounding box)坐标；
4)随机选取一定比例不包含掩码语言建模(mask language modeling，MLM)的文本行(如果下游任务包括MLM的话)，并遮盖住该文本行中每个文字的文本框坐标(例如可以将文本框坐标设置为[0，0，0，0])；
5)判断这些被遮盖住位置信息的文字处于图片中哪个区域
每个单词的特征可以表示为多模态特征的聚合，如：
Ei＝Eword(ti)+Elayout(ti)
其中Eword和Elayout分别表示第i个词的词嵌入和布局信息。
通过遮盖住第i个词的布局信息，然后预测该词在原图中空间分区的类别，则BGM的损失函数可以 定义为：
其中，yij表示第i个词的真实标签，pij表示第i个词是属于第j个图像区域的概率。
本申请在单塔部分提出BGM来提升空间布局感知能力，有助于提升位置敏感的任务中模型的性能。例如在信息提取中，key-value对通常是相邻的成对数据。因为模型在预训练阶段已经学会了对位置以及布局信息的感知能力，所以在信息抽取这类任务中模型会有很好的性能。
此外，下游任务还可以包括MLM，具体的，在模型训练的前馈过程中，可以对文本中的文本单元进行掩码(例如可以为随机掩码)，掩码后的文本单元可以作为预测文本，未被掩码的文本单元可以作为已知文本，模型可以基于未被掩码的文本单元(或者已经预测出的掩码后的文本单元)，依次进行掩码后的文本单元所在文本位置的文本预测，例如，文本可以为“the cat sat on the mat”，对文本进行掩码后，可以得到“the_sat_the mat”，其中，这里的符号“_”的含义是掩码，而不是指下划线在。第一次预测时，可以通过初始标识位以及未被掩码的文本单元，预测其中一个“_”所处的文本位置的文本。
在一种可能的实现中，为了提高图像表征的学习能力，还可以基于图像特征来构建损失，具体的，可以对图像中的部分进行掩码，通过图像编码器得到的结果对掩码区域图像进行图像重建，并基于重建结果和掩码区域的真实图像的像素值之间的差异来更新网络。然而，现有方法图像表征学习能力较弱，具体是由于，为了降低模型的处理算例开销，需要将图像压缩到一个较小的分辨率(例如224*224)，在图像中包含密集型文本的情况下，图像中的像素难以表达出准确的文字信息，训练后的模型的图像表征的学习能力有限。
本申请实施例中，为了提高图像表征的学习能力，可以不对图像而是对图像的图像特征进行掩码，并对掩码区域的图像特征进行特征重建，并基于重建结果和掩码区域的图像特征之间的差异来更新网络，由于图像特征相比像素值本身可以携带更丰富的图像信息以及文本信息，因此可以提高训练后的网络的图像表征的学习能力，上述方式可以称之为掩码图像建模(mask image modeling，MIM)。
在一种可能的实现中，所述第一特征表示包括第三子特征和第四子特征；可以根据所述第四子特征、所述第四子特征在所述第一特征表示中的第二位置、以及所述第三子特征在所述第一特征表示中的第三位置，得到所述第三位置处的特征预测值；所述特征预测值和所述第三子特征之间的相似度用于更新所述图像编码器。
在一种可能的实现中，可以根据所述第四子特征、所述第四子特征在所述第一特征表示中的第二位置、以及所述第三子特征在所述第一特征表示中的第三位置，通过自注意力网络，得到第四特征表示；根据所述第四特征表示，通过预测网络，得到所述第三位置处的特征预测值。
应理解，进行掩码的对象(第一特征表示)可以是图像编码器得到的，也可以是通过对图像编码器得到的图像特征进行一定处理得到的，例如是通过图像区域特征提取模块(例如可以通过感兴趣区域(region of interest，ROI)head实现)对图像中的部分图像区域确定的图像特征。
示例性的，参照图10，本实施例可以分为以下几个步骤：
1)将图片缩放到某个分辨率(如448x448)；
2)计算当前图片的卷积特征图；
3)计算每个文本框区域的视觉特征(ROI features)；
4)随机选取一定比例不包含MLM的文本行(如果下游任务包括MLM的话)，在卷积网络的特征图中遮盖住该文本行对应区域；
5)将所有ROI features(包括被遮盖的ROI特征)送入self-attention；
6)利用contrastive/L1等loss重建被遮盖的ROI特征；
为了增强图像分支的表征学习，先使用ROI对齐的方式提取每个文本行的视觉特征：
其中表示第i个文本行的ROI视觉特征，W和b是网络参数。
可选的，可以利用注意力机制重新计算每个文本行视觉特征的新聚合表示：
H＝Atten(f；φ)＝Atten({fi}；φ)；
其中，H是ROI特征的隐空间表征，Atten是自注意里模块，φ是网络参数。
图像中文本行区域的视觉掩码特征重建损失函数可以表示为：
其中，表示第i个被遮盖的文本框的视觉重建特征，Hi表示所对应的groundtruth labels。
本实施例采用更高的分辨率配合MIM(Masked Image Modeling)来提升文本行区域的视觉表征学习。基于特征图角度重建遮掩视觉特征图，利用相邻unmasked文本行ROI特征帮助重建被遮盖的文本行视觉特征。通过增强视觉表征的学习能力，一方面可以进一步提升多模态特征对齐，另一方面可以提升模型在一些下游任务如图文分类上的性能。
接下来结合实验结果介绍本申请的有益效果：
如表1所示，本申请在公开数据集如信息提取数据集(FUNSD，CORD和SROI)和图文分类数据集(如RVL-CDIP)上均超越SOTA性能。
表1

本申请提供了一种数据处理方法，包括：获取第一特征表示以及第二特征表示，所述第一特征表示为根据图像编码器对图像进行处理得到的图像特征得到，所述第二特征表示为第一文本的文本特征；所述第一文本为所述图像中包括的文本内容；所述第一特征表示和所述第二特征表示之间的相似度用于更新所述图像编码器；根据所述第一特征表示以及所述第二特征表示，通过目标编码器，得到第三特征表示；所述目标编码器用于实现特征表示之间的交互；根据所述第三特征表示，执行下游任务，得到执行结果。
本申请实施例中，通过图文对比学习(text-image contrastive learning，TIC)来增强图文之间的匹配能力，具体的，针对于图像的图像特征和文本的文本特征，通过相似度比对构建的损失，来更新图形编码器(如果文本特征为通过文本编码器得到的，也可以更新文本编码器)。虽然图像的图像特征和图像中文本的文本特征不是同一个模态的特征，但是由于文本是图像中的文本，在图像特征中也会蕴含一定的文本的语义信息，因此，图像特征和文本特征(属于同一个图像)之间在语义维度(或者其他信息维度上)会存在关联。上述相似度可以包含图像中语义信息和文本的语义信息之间的相似度，因此，基于该相似度构建的损失来更新图像编码器以及文本编码器(如果架构中存在文本编码器的话)，能够使得图像编码器和文本编码器提取的特征中蕴含更准确的图像中文本的语义信息，进而增强后续网络的图文匹配能力。
此外，在跨模态的语言模型的架构中，包括图形编码器、文本编码器以及用于提供特征之间交互信息的目标编码器，本申请中，将模型的中间输出(也就是图形编码器和文本编码器的输出)之间的相似度构建的损失来更新图形编码器和文本编码器，使得基于图形编码器和文本编码器输出的特征表示就可以实现下游任务(精度没有基于目标编码器输出的特征表示进行的下游任务高)，而在一些场景中，由于下游任务要处理的数据的数量较大，因此可以使用图形编码器和文本编码器输出的特征表示进行粗排，使用目标编码器输出的特征表示进行精排，以提高召回率。
参照图11，图11为本申请中模型的一个架构示意，其中，采用先双塔对齐图文特征，然后再接单塔构建图文多模态交互模块。在双塔部分，先分别对图像和文字编码提取特征，然后利用粗粒度和细粒度对比学习增强图文整体匹配和区域-文本行匹配能力。在单塔部分提出BGM(Box Grid Matching)来提升空间布局感知能力。为了提升模型对视觉表征的建模能力，在视觉分支采用了更高的分辨率配合MIM(Masked Image Modeling)来提升文本行区域的视觉表征学习。
此外，本申请还提供了一种数据处理方法，所述方法包括：
获取图像；
通过图像编码器，对所述图像进行处理，得到第一特征表示；
通过文本编码器，对所述图像中包含的文本进行处理，得到第二特征表示；
根据所述第一特征表示以及所述第二特征表示，通过目标编码器，得到第三特征表示；所述目标编码器用于实现特征表示之间的交互；
根据所述第三特征表示，执行下游任务，得到执行结果。
参照图12，图12为本申请实施例提供的一种数据处理装置的结构示意，如图12所示，本申请实施例提供的一种数据处理装置1200，包括：
获取模块1201，用于获取第一特征表示以及第二特征表示，所述第一特征表示为根据图像编码器对图像进行处理得到的图像特征得到，所述第二特征表示为第一文本的文本特征；所述第一文本为所述图像中包括的文本内容；
其中，关于获取模块1201的具体描述可以参照上述实施例中步骤701的描述，这里不再赘述。
编码模块1202，用于根据所述第一特征表示以及所述第二特征表示，通过目标编码器，得到第三特征表示；所述目标编码器用于实现特征表示之间的交互；
其中，关于编码模块1202的具体描述可以参照上述实施例中步骤702的描述，这里不再赘述。
任务执行模块1203，用于根据所述第三特征表示，执行下游任务，得到执行结果；所述执行结果和对应的标注之间的相似度以及所述第一特征表示和所述第二特征表示之间的相似度用于更新所述图像编码器。
其中，关于任务执行模块1203的具体描述可以参照上述实施例中步骤703的描述，这里不再赘述。
在一种可能的实现中，所述第二特征表示为通过文本编码器对所述第一文本进行处理得到的文本特征；所述第一特征表示和所述第二特征表示之间的相似度还用于更新所述文本编码器。
在一种可能的实现中，所述第一特征表示和所述第二特征表示之间的相似度与图像中所蕴含的文本语义信息和文本自身的语义信息之间的相似度有关。
在一种可能的实现中，
所述第一文本为所述图像中包含的全部文本；或者，
所述第一文本为所述图像中包含的全部文本中的部分。
在一种可能的实现中，
所述图像为从原始的输入图像中提取的部分图像区域，所述图像包括的文本为所述输入图像包含的文本的部分；或者，
所述图像为原始的输入图像。
在一种可能的实现中，所述第一文本包括第一子文本和第二子文本，所述第二特征表示包括所述第一子文本对应的第一子特征、以及所述第二子文本对应的第二子特征；所述第一子特征不包含所述第一子文本在所述图像中的位置；所述第二子特征包含所述第二子文本在所述图像中的位置；
所述任务执行模块，具体用于：
根据所述第三特征表示，预测所述第一子文本在所述图像中的第一位置；所述第一位置和对应的标注之间的相似度用于更新所述图像编码器以及所述目标编码器。
在一种可能的实现中，所述图像包括多个图像块；所述第一位置为对所述第一子文本预测所在的图像块；所述标注为所述第一子文本真实所在的图像块。
在一种可能的实现中，所述第一特征表示包括第三子特征和第四子特征；所述方法还包括：
预测模块，用于根据所述第四子特征、所述第四子特征在所述第一特征表示中的第二位置、以及所述第三子特征在所述第一特征表示中的第三位置，得到所述第三位置处的特征预测值；所述特征预测值和所述第三子特征之间的相似度用于更新所述图像编码器。
在一种可能的实现中，所述预测模块，具体用于：
根据所述第四子特征、所述第四子特征在所述第一特征表示中的第二位置、以及所述第三子特征在所述第一特征表示中的第三位置，通过自注意力网络，得到第四特征表示；
根据所述第四特征表示，通过预测网络，得到所述第三位置处的特征预测值。
此外，本申请还提供了一种数据处理装置，所述装置包括：
获取模块，用于获取图像；
编码模块，用于通过图像编码器，对所述图像进行处理，得到第一特征表示；
通过文本编码器，对所述图像中包含的文本进行处理，得到第二特征表示；
根据所述第一特征表示以及所述第二特征表示，通过目标编码器，得到第三特征表示；所述目标编码器用于实现特征表示之间的交互；
任务执行模块，用于根据所述第三特征表示，执行下游任务，得到执行结果。
接下来介绍本申请实施例提供的一种执行设备，请参阅图13，图13为本申请实施例提供的执行设备的一种结构示意图，执行设备1300具体可以表现为虚拟现实VR设备、手机、平板、笔记本电脑、智 能穿戴设备、监控数据处理设备或服务器等，此处不做限定。具体的，执行设备1300包括：接收器1301、发射器1302、处理器1303和存储器1304(其中执行设备1300中的处理器1303的数量可以一个或多个，图13中以一个处理器为例)，其中，处理器1303可以包括应用处理器13031和通信处理器13032。在本申请的一些实施例中，接收器1301、发射器1302、处理器1303和存储器1304可通过总线或其它方式连接。
存储器1304可以包括只读存储器和随机存取存储器，并向处理器1303提供指令和数据。存储器1304的一部分还可以包括非易失性随机存取存储器(non-volatile random access memory，NVRAM)。存储器1304存储有处理器和操作指令、可执行模块或者数据结构，或者它们的子集，或者它们的扩展集，其中，操作指令可包括各种操作指令，用于实现各种操作。
处理器1303控制执行设备的操作。具体的应用中，执行设备的各个组件通过总线***耦合在一起，其中总线***除包括数据总线之外，还可以包括电源总线、控制总线和状态信号总线等。但是为了清楚说明起见，在图中将各种总线都称为总线***。
上述本申请实施例揭示的方法可以应用于处理器1303中，或者由处理器1303实现。处理器1303可以是一种集成电路芯片，具有信号的处理能力。在实现过程中，上述方法的各步骤可以通过处理器1303中的硬件的集成逻辑电路或者软件形式的指令完成。上述的处理器1303可以是通用处理器、数字信号处理器(digital signal processing，DSP)、微处理器或微控制器，还可进一步包括专用集成电路(application specific integrated circuit，ASIC)、现场可编程门阵列(field-programmable gate array，FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件。该处理器1303可以实现或者执行本申请实施例中的公开的各方法、步骤及逻辑框图。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。结合本申请实施例所公开的方法的步骤可以直接体现为硬件译码处理器执行完成，或者用译码处理器中的硬件及软件模块组合执行完成。软件模块可以位于随机存储器，闪存、只读存储器，可编程只读存储器或者电可擦写可编程存储器、寄存器等本领域成熟的存储介质中。该存储介质位于存储器1304，处理器1303读取存储器1304中的信息，结合其硬件完成上述方法中涉及模型推理过程的步骤。
接收器1301可用于接收输入的数字或字符信息，以及产生与执行设备的相关设置以及功能控制有关的信号输入。发射器1302可用于通过第一接口输出数字或字符信息；发射器1302还可用于通过第一接口向磁盘组发送指令，以修改磁盘组中的数据；发射器1302还可以包括显示屏等显示设备。
本申请实施例还提供了一种训练设备，请参阅图14，图14是本申请实施例提供的训练设备一种结构示意图，具体的，训练设备1400由一个或多个服务器实现，训练设备1400可因配置或性能不同而产生比较大的差异，可以包括一个或一个以***处理器(central processing units，CPU)1414(例如，一个或一个以上处理器)和存储器1432，一个或一个以上存储应用程序1442或数据1444的存储介质1430(例如一个或一个以上海量存储设备)。其中，存储器1432和存储介质1430可以是短暂存储或持久存储。存储在存储介质1430的程序可以包括一个或一个以上模块(图示没标出)，每个模块可以包括对训练设备中的一系列指令操作。更进一步地，中央处理器1414可以设置为与存储介质1430通信，在训练设备1400上执行存储介质1430中的一系列指令操作。
训练设备1400还可以包括一个或一个以上电源1426，一个或一个以上有线或无线网络接口1450，一个或一个以上输入输出接口1458；或，一个或一个以上操作***1441，例如Windows ServerTM，Mac OS XTM，UnixTM,LinuxTM，FreeBSDTM等等。
本申请实施例中，中央处理器1414，用于执行上述实施例中和模型训练相关的动作。
本申请实施例中还提供一种包括计算机程序产品，当其在计算机上运行时，使得计算机执行如前述执行设备所执行的步骤，或者，使得计算机执行如前述训练设备所执行的步骤。
本申请实施例中还提供一种计算机可读存储介质，该计算机可读存储介质中存储有用于进行信号处理的程序，当其在计算机上运行时，使得计算机执行如前述执行设备所执行的步骤，或者，使得计算机 执行如前述训练设备所执行的步骤。
本申请实施例提供的执行设备、训练设备或终端设备具体可以为芯片，芯片包括：处理单元和通信单元，所述处理单元例如可以是处理器，所述通信单元例如可以是输入/输出接口、管脚或电路等。该处理单元可执行存储单元存储的计算机执行指令，以使执行设备内的芯片执行上述实施例描述的数据处理方法，或者，以使训练设备内的芯片执行上述实施例描述的数据处理方法。可选地，所述存储单元为所述芯片内的存储单元，如寄存器、缓存等，所述存储单元还可以是所述无线接入设备端内的位于所述芯片外部的存储单元，如只读存储器(read-only memory，ROM)或可存储静态信息和指令的其他类型的静态存储设备，随机存取存储器(random access memory，RAM)等。
具体的，请参阅图15，图15为本申请实施例提供的芯片的一种结构示意图，所述芯片可以表现为神经网络处理器NPU 1500，NPU 1500作为协处理器挂载到主CPU(Host CPU)上，由Host CPU分配任务。NPU的核心部分为运算电路1503，通过控制器1504控制运算电路1503提取存储器中的矩阵数据并进行乘法运算。
在一些实现中，运算电路1503内部包括多个处理单元(Process Engine,PE)。在一些实现中，运算电路1503是二维脉动阵列。运算电路1503还可以是一维脉动阵列或者能够执行例如乘法和加法这样的数学运算的其它电子线路。在一些实现中，运算电路1503是通用的矩阵处理器。
举例来说，假设有输入矩阵A，权重矩阵B，输出矩阵C。运算电路从权重存储器1502中取矩阵B相应的数据，并缓存在运算电路中每一个PE上。运算电路从输入存储器1501中取矩阵A数据与矩阵B进行矩阵运算，得到的矩阵的部分结果或最终结果，保存在累加器(accumulator)1508中。
统一存储器1506用于存放输入数据以及输出数据。权重数据直接通过存储单元访问控制器(Direct Memory Access Controller，DMAC)1505，DMAC被搬运到权重存储器1502中。输入数据也通过DMAC被搬运到统一存储器1506中。
BIU为Bus Interface Unit即，总线接口单元1510，用于AXI总线与DMAC和取指存储器(Instruction Fetch Buffer，IFB)1509的交互。
总线接口单元1510(Bus Interface Unit，简称BIU)，用于取指存储器1509从外部存储器获取指令，还用于存储单元访问控制器1505从外部存储器获取输入矩阵A或者权重矩阵B的原数据。
DMAC主要用于将外部存储器DDR中的输入数据搬运到统一存储器1506或将权重数据搬运到权重存储器1502中或将输入数据数据搬运到输入存储器1501中。
向量计算单元1507包括多个运算处理单元，在需要的情况下，对运算电路1503的输出做进一步处理，如向量乘，向量加，指数运算，对数运算，大小比较等等。主要用于神经网络中非卷积/全连接层网络计算，如Batch Normalization(批归一化)，像素级求和，对特征平面进行上采样等。
在一些实现中，向量计算单元1507能将经处理的输出的向量存储到统一存储器1506。例如，向量计算单元1507可以将线性函数；或，非线性函数应用到运算电路1503的输出，例如对卷积层提取的特征平面进行线性插值，再例如累加值的向量，用以生成激活值。在一些实现中，向量计算单元1507生成归一化的值、像素级求和的值，或二者均有。在一些实现中，处理过的输出的向量能够用作到运算电路1503的激活输入，例如用于在神经网络中的后续层中的使用。
控制器1504连接的取指存储器(instruction fetch buffer)1509，用于存储控制器1504使用的指令；
统一存储器1506，输入存储器1501，权重存储器1502以及取指存储器1509均为On-Chip存储器。外部存储器私有于该NPU硬件架构。
其中，上述任一处提到的处理器，可以是一个通用中央处理器，微处理器，ASIC，或一个或多个用于控制上述程序执行的集成电路。
另外需说明的是，以上所描述的装置实施例仅仅是示意性的，其中所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部模块来实 现本实施例方案的目的。另外，本申请提供的装置实施例附图中，模块之间的连接关系表示它们之间具有通信连接，具体可以实现为一条或多条通信总线或信号线。
通过以上的实施方式的描述，所属领域的技术人员可以清楚地了解到本申请可借助软件加必需的通用硬件的方式来实现，当然也可以通过专用硬件包括专用集成电路、专用CPU、专用存储器、专用元器件等来实现。一般情况下，凡由计算机程序完成的功能都可以很容易地用相应的硬件来实现，而且，用来实现同一功能的具体硬件结构也可以是多种多样的，例如模拟电路、数字电路或专用电路等。但是，对本申请而言更多情况下软件程序实现是更佳的实施方式。基于这样的理解，本申请的技术方案本质上或者说对现有技术做出贡献的部分可以以软件产品的形式体现出来，该计算机软件产品存储在可读取的存储介质中，如计算机的软盘、U盘、移动硬盘、ROM、RAM、磁碟或者光盘等，包括若干指令用以使得一台计算机设备(可以是个人计算机，训练设备，或者网络设备等)执行本申请各个实施例所述的方法。
在上述实施例中，可以全部或部分地通过软件、硬件、固件或者其任意组合来实现。当使用软件实现时，可以全部或部分地以计算机程序产品的形式实现。
所述计算机程序产品包括一个或多个计算机指令。在计算机上加载和执行所述计算机程序指令时，全部或部分地产生按照本申请实施例所述的流程或功能。所述计算机可以是通用计算机、专用计算机、计算机网络、或者其他可编程装置。所述计算机指令可以存储在计算机可读存储介质中，或者从一个计算机可读存储介质向另一计算机可读存储介质传输，例如，所述计算机指令可以从一个网站站点、计算机、训练设备或数据中心通过有线(例如同轴电缆、光纤、数字用户线(DSL))或无线(例如红外、无线、微波等)方式向另一个网站站点、计算机、训练设备或数据中心进行传输。所述计算机可读存储介质可以是计算机能够存储的任何可用介质或者是包含一个或多个可用介质集成的训练设备、数据中心等数据存储设备。所述可用介质可以是磁性介质，(例如，软盘、硬盘、磁带)、光介质(例如，DVD)、或者半导体介质(例如固态硬盘(Solid State Disk，SSD))等。

Claims (23)

  1. 一种数据处理方法，其特征在于，包括：
    获取第一特征表示以及第二特征表示，所述第一特征表示为根据图像编码器对图像进行处理得到的图像特征得到，所述第二特征表示为第一文本的文本特征；所述第一文本为所述图像中包括的文本内容；
    根据所述第一特征表示以及所述第二特征表示，通过目标编码器，得到第三特征表示；所述目标编码器用于实现特征表示之间的交互；
    根据所述第三特征表示，执行下游任务，得到执行结果；所述执行结果和对应的标注之间的相似度以及所述第一特征表示和所述第二特征表示之间的相似度用于更新所述图像编码器。
  2. 根据权利要求1所述的方法，其特征在于，所述第二特征表示为通过文本编码器对所述第一文本进行处理得到的文本特征；所述第一特征表示和所述第二特征表示之间的相似度还用于更新所述文本编码器。
  3. 根据权利要求1或2所述的方法，其特征在于，所述第一特征表示和所述第二特征表示之间的相似度与图像中所蕴含的文本语义信息和文本自身的语义信息之间的相似度有关。
  4. 根据权利要求1至3任一所述的方法，其特征在于，
    所述第一文本为所述图像中包含的全部文本；或者，
    所述第一文本为所述图像中包含的全部文本中的部分。
  5. 根据权利要求1至4任一所述的方法，其特征在于，
    所述图像为从原始的输入图像中提取的部分图像区域，所述图像包括的文本为所述输入图像包含的文本的部分；或者，
    所述图像为原始的输入图像。
  6. 根据权利要求1至5任一所述的方法，其特征在于，所述第一文本包括第一子文本和第二子文本，所述第二特征表示包括所述第一子文本对应的第一子特征、以及所述第二子文本对应的第二子特征；所述第一子特征不包含所述第一子文本在所述图像中的位置；所述第二子特征包含所述第二子文本在所述图像中的位置；
    所述根据所述第三特征表示，执行下游任务，包括：
    根据所述第三特征表示，预测所述第一子文本在所述图像中的第一位置；所述第一位置和对应的标注之间的相似度用于更新所述图像编码器以及所述目标编码器。
  7. 根据权利要求6所述的方法，其特征在于，所述图像包括多个图像块；所述第一位置为对所述第一子文本预测所在的图像块；所述标注为所述第一子文本真实所在的图像块。
  8. 根据权利要求1至7任一所述的方法，其特征在于，所述第一特征表示包括第三子特征和第四子特征；所述方法还包括：
    根据所述第四子特征、所述第四子特征在所述第一特征表示中的第二位置、以及所述第三子特征在所述第一特征表示中的第三位置，得到所述第三位置处的特征预测值；所述特征预测值和所述第三子特征之间的相似度用于更新所述图像编码器。
  9. 根据权利要求8所述的方法，其特征在于，所述根据所述第四子特征、所述第四子特征在所述第一特征表示中的第二位置、以及所述第三子特征在所述第一特征表示中的第三位置，得到所述第三位置处的特征预测值，包括：
    根据所述第四子特征、所述第四子特征在所述第一特征表示中的第二位置、以及所述第三子特征在所述第一特征表示中的第三位置，通过自注意力网络，得到第四特征表示；
    根据所述第四特征表示，通过预测网络，得到所述第三位置处的特征预测值。
  10. 一种数据处理方法，其特征在于，所述方法包括：
    获取图像；
    通过图像编码器，对所述图像进行处理，得到第一特征表示；
    通过文本编码器，对所述图像中包含的文本进行处理，得到第二特征表示；
    根据所述第一特征表示以及所述第二特征表示，通过目标编码器，得到第三特征表示；所述目标编码器用于实现特征表示之间的交互；
    根据所述第三特征表示，执行下游任务，得到执行结果。
  11. 一种数据处理装置，其特征在于，包括：
    获取模块，用于获取第一特征表示以及第二特征表示，所述第一特征表示为根据图像编码器对图像进行处理得到的图像特征得到，所述第二特征表示为第一文本的文本特征；所述第一文本为所述图像中包括的文本内容；
    编码模块，用于根据所述第一特征表示以及所述第二特征表示，通过目标编码器，得到第三特征表示；所述目标编码器用于实现特征表示之间的交互；
    任务执行模块，用于根据所述第三特征表示，执行下游任务，得到执行结果；所述执行结果和对应的标注之间的相似度以及所述第一特征表示和所述第二特征表示之间的相似度用于更新所述图像编码器。
  12. 根据权利要求11所述的装置，其特征在于，所述第二特征表示为通过文本编码器对所述第一文本进行处理得到的文本特征；所述第一特征表示和所述第二特征表示之间的相似度还用于更新所述文本编码器。
  13. 根据权利要求11或12所述的装置，其特征在于，所述第一特征表示和所述第二特征表示之间的相似度与图像中所蕴含的文本语义信息和文本自身的语义信息之间的相似度有关。
  14. 根据权利要求11至13任一所述的装置，其特征在于，
    所述第一文本为所述图像中包含的全部文本；或者，
    所述第一文本为所述图像中包含的全部文本中的部分。
  15. 根据权利要求11至14任一所述的装置，其特征在于，
    所述图像为从原始的输入图像中提取的部分图像区域，所述图像包括的文本为所述输入图像包含的文本的部分；或者，
    所述图像为原始的输入图像。
  16. 根据权利要求11至15任一所述的装置，其特征在于，所述第一文本包括第一子文本和第二子文本，所述第二特征表示包括所述第一子文本对应的第一子特征、以及所述第二子文本对应的第二子特征；所述第一子特征不包含所述第一子文本在所述图像中的位置；所述第二子特征包含所述第二子文本在所述图像中的位置；
    所述任务执行模块，具体用于：
    根据所述第三特征表示，预测所述第一子文本在所述图像中的第一位置；所述第一位置和对应的标注之间的相似度用于更新所述图像编码器以及所述目标编码器。
  17. 根据权利要求16所述的装置，其特征在于，所述图像包括多个图像块；所述第一位置为对所述第一子文本预测所在的图像块；所述标注为所述第一子文本真实所在的图像块。
  18. 根据权利要求11至17任一所述的装置，其特征在于，所述第一特征表示包括第三子特征和第四子特征；所述装置还包括：
    预测模块，用于根据所述第四子特征、所述第四子特征在所述第一特征表示中的第二位置、以及所述第三子特征在所述第一特征表示中的第三位置，得到所述第三位置处的特征预测值；所述特征预测值和所述第三子特征之间的相似度用于更新所述图像编码器。
  19. 根据权利要求18所述的装置，其特征在于，所述预测模块，具体用于：
    根据所述第四子特征、所述第四子特征在所述第一特征表示中的第二位置、以及所述第三子特征在所述第一特征表示中的第三位置，通过自注意力网络，得到第四特征表示；
    根据所述第四特征表示，通过预测网络，得到所述第三位置处的特征预测值。
  20. 一种数据处理装置，其特征在于，所述装置包括：
    获取模块，用于获取图像；
    编码模块，用于通过图像编码器，对所述图像进行处理，得到第一特征表示；
    通过文本编码器，对所述图像中包含的文本进行处理，得到第二特征表示；
    根据所述第一特征表示以及所述第二特征表示，通过目标编码器，得到第三特征表示；所述目标编码器用于实现特征表示之间的交互；
    任务执行模块，用于根据所述第三特征表示，执行下游任务，得到执行结果。
  21. 一种计算机存储介质，其特征在于，所述计算机存储介质存储有一个或多个指令，所述指令在由一个或多个计算机执行时使得所述一个或多个计算机执行权利要求1至10中任一项所述方法的操作。
  22. 一种计算机程序产品，其特征在于，包括计算机可读指令，当所述计算机可读指令在计算机设备上运行时，使得所述计算机设备执行如权利要求1至10任一所述的方法。
  23. 一种***，包括至少一个处理器，至少一个存储器；所述处理器、所述存储器通过通信总线连接并完成相互间的通信；
    所述至少一个存储器用于存储代码；
    所述至少一个处理器用于执行所述代码，以执行如权利要求1至10任一所述的方法。
PCT/CN2023/114002 2022-08-22 2023-08-21 一种数据处理方法及其装置 WO2024041479A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202211005409.3A CN115512005A (zh) 2022-08-22 2022-08-22 一种数据处理方法及其装置
CN202211005409.3 2022-08-22

Publications (1)

Publication Number Publication Date
WO2024041479A1 true WO2024041479A1 (zh) 2024-02-29

Family

ID=84501082

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2023/114002 WO2024041479A1 (zh) 2022-08-22 2023-08-21 一种数据处理方法及其装置

Country Status (2)

Country Link
CN (1) CN115512005A (zh)
WO (1) WO2024041479A1 (zh)

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115512005A (zh) * 2022-08-22 2022-12-23 华为技术有限公司 一种数据处理方法及其装置
CN116320459B (zh) * 2023-01-08 2024-01-23 南阳理工学院 一种基于人工智能的计算机网络通信数据处理方法及***
CN115712740B (zh) * 2023-01-10 2023-06-06 苏州大学 多模态蕴含增强图像文本检索的方法和***
CN116258939A (zh) * 2023-02-27 2023-06-13 阿里巴巴达摩院(杭州)科技有限公司 图像处理模型的模型训练方法及装置
CN116226688B (zh) * 2023-05-10 2023-10-31 粤港澳大湾区数字经济研究院(福田) 数据处理、图文检索、图像分类方法及相关设备

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113516143A (zh) * 2020-11-26 2021-10-19 腾讯科技（深圳）有限公司 文本图像匹配方法、装置、计算机设备及存储介质
CN113836333A (zh) * 2021-09-18 2021-12-24 北京百度网讯科技有限公司 图文匹配模型的训练方法、实现图文检索的方法、装置
CN114283430A (zh) * 2021-12-03 2022-04-05 苏州大创科技有限公司 跨模态图文匹配训练方法及装置、存储介质、电子设备
CN114429566A (zh) * 2022-01-20 2022-05-03 北京沃东天骏信息技术有限公司 一种图像语义理解方法、装置、设备及存储介质
US20220172080A1 (en) * 2020-12-02 2022-06-02 International Business Machines Corporation Learning unpaired multimodal feature matching for semi-supervised learning
CN114707007A (zh) * 2022-06-07 2022-07-05 苏州大学 一种图像文本检索方法、装置及计算机存储介质
CN115512005A (zh) * 2022-08-22 2022-12-23 华为技术有限公司 一种数据处理方法及其装置

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113516143A (zh) * 2020-11-26 2021-10-19 腾讯科技（深圳）有限公司 文本图像匹配方法、装置、计算机设备及存储介质
US20220172080A1 (en) * 2020-12-02 2022-06-02 International Business Machines Corporation Learning unpaired multimodal feature matching for semi-supervised learning
CN113836333A (zh) * 2021-09-18 2021-12-24 北京百度网讯科技有限公司 图文匹配模型的训练方法、实现图文检索的方法、装置
CN114283430A (zh) * 2021-12-03 2022-04-05 苏州大创科技有限公司 跨模态图文匹配训练方法及装置、存储介质、电子设备
CN114429566A (zh) * 2022-01-20 2022-05-03 北京沃东天骏信息技术有限公司 一种图像语义理解方法、装置、设备及存储介质
CN114707007A (zh) * 2022-06-07 2022-07-05 苏州大学 一种图像文本检索方法、装置及计算机存储介质
CN115512005A (zh) * 2022-08-22 2022-12-23 华为技术有限公司 一种数据处理方法及其装置

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
SONG SIBO, WAN JIANQIANG, YANG ZHIBO, TANG JUN, CHENG WENQING, BAI XIANG, YAO CONG: "Vision-Language Pre-Training for Boosting Scene Text Detectors", ARXIV (CORNELL UNIVERSITY), CORNELL UNIVERSITY LIBRARY, ARXIV.ORG, ITHACA, 28 April 2022 (2022-04-28), Ithaca, XP093142990, [retrieved on 20240319], DOI: 10.48550/arxiv.2204.13867 *

Also Published As

Publication number Publication date
CN115512005A (zh) 2022-12-23

Similar Documents

Publication Publication Date Title
Zhang et al. A spatial attentive and temporal dilated (SATD) GCN for skeleton‐based action recognition
WO2024041479A1 (zh) 一种数据处理方法及其装置
WO2020228376A1 (zh) 文本处理方法、模型训练方法和装置
WO2021238281A1 (zh) 一种神经网络的训练方法、图像分类***及相关设备
Song et al. An efficient convolutional neural network for small traffic sign detection
Sun et al. Fast object detection based on binary deep convolution neural networks
WO2023236977A1 (zh) 一种数据处理方法及相关设备
CN111950596A (zh) 一种用于神经网络的训练方法以及相关设备
WO2022253074A1 (zh) 一种数据处理方法及相关设备
Wang et al. TRC‐YOLO: A real‐time detection method for lightweight targets based on mobile devices
CN113761153B (zh) 基于图片的问答处理方法、装置、可读介质及电子设备
WO2023284716A1 (zh) 一种神经网络搜索方法及相关设备
WO2023231794A1 (zh) 一种神经网络参数量化方法和装置
WO2024083121A1 (zh) 一种数据处理方法及其装置
WO2024001806A1 (zh) 一种基于联邦学习的数据价值评估方法及其相关设备
WO2022111387A1 (zh) 一种数据处理方法及相关装置
Liu et al. Research of animals image semantic segmentation based on deep learning
WO2021169366A1 (zh) 数据增强方法和装置
WO2024067884A1 (zh) 一种数据处理方法及相关装置
CN116737895A (zh) 一种数据处理方法及相关设备
WO2024046473A1 (zh) 一种数据处理方法及其装置
WO2024114659A1 (zh) 一种摘要生成方法及其相关设备
WO2024017282A1 (zh) 一种数据处理方法及其装置
WO2023185541A1 (zh) 一种模型训练方法及其相关设备
WO2023197857A1 (zh) 一种模型切分方法及其相关设备

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23856569

Country of ref document: EP

Kind code of ref document: A1