WO2022057309A1 - 肺部特征识别方法、装置、计算机设备及存储介质 - Google Patents

肺部特征识别方法、装置、计算机设备及存储介质 Download PDF

Info

Publication number
WO2022057309A1
WO2022057309A1 PCT/CN2021/096366 CN2021096366W WO2022057309A1 WO 2022057309 A1 WO2022057309 A1 WO 2022057309A1 CN 2021096366 W CN2021096366 W CN 2021096366W WO 2022057309 A1 WO2022057309 A1 WO 2022057309A1
Authority
WO
WIPO (PCT)
Prior art keywords
lung
text
image
feature vector
feature
Prior art date
Application number
PCT/CN2021/096366
Other languages
English (en)
French (fr)
Inventor
朱昭苇
孙行智
胡岗
Original Assignee
平安科技（深圳）有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 平安科技（深圳）有限公司 filed Critical 平安科技（深圳）有限公司
Publication of WO2022057309A1 publication Critical patent/WO2022057309A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/56—Extraction of image or video features relating to colour
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/25—Fusion techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/25—Fusion techniques
    • G06F18/253—Fusion techniques of extracted features
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0012—Biomedical image inspection
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
    • G16H50/20—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for computer-aided diagnosis, e.g. based on medical expert systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/25—Fusion techniques
    • G06F18/259—Fusion by voting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30004—Biomedical image processing
    • G06T2207/30061—Lung
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V2201/00—Indexing scheme relating to image or video recognition or understanding
    • G06V2201/03—Recognition of patterns in medical or anatomical images

Definitions

  • the present application relates to the field of image classification of artificial intelligence, and in particular, to a method, device, computer equipment and storage medium for identifying lung features.
  • the identification of lung features mainly relies on medical personnel to manually identify lung image information based on their own experience. Because the movement of lung tissue is uneven and complex, the identification process not only costs medical personnel At the same time, in the process of identification, only the lung image information is often identified, and the main complaint information (text description for the lung image information) of the lung image information is not combined for identification. It is easy to lose the information of lung tissue movement, resulting in low accuracy and low efficiency.
  • the present application provides a lung feature identification method, device, computer equipment and storage medium, which realizes a lung feature identification model including a lung image identification model, a lung text identification model, and a lung fusion identification model, and uses attention It realizes the automatic, rapid and accurate identification of lung features, improves the accuracy and reliability of identification, and improves the efficiency of identification.
  • This application is applicable to fields such as smart medical care, and can further promote the construction of smart cities.
  • a lung feature recognition method comprising:
  • Acquiring data to be identified wherein the data to be identified includes an image of the lung to be identified and a text description of the lung to be identified;
  • the lung feature identification model includes a lung image identification model, a lung text identification model and a lung fusion identification model;
  • Lung image feature extraction is performed on the to-be-recognized lung image by the lung image recognition model to generate a lung image feature vector and image recognition result.
  • the text description performs lung text feature extraction to generate lung text feature vectors and text recognition results;
  • the image recognition result, the text recognition result and the fusion recognition result are voted on by the lung feature recognition model to obtain the lung feature recognition result corresponding to the data to be recognized; the lung feature The recognition result indicates the lung feature category of the data to be recognized.
  • a lung feature identification device comprising:
  • a receiving module for acquiring data to be identified, wherein the data to be identified includes an image of the lung to be identified and a text description of the lung to be identified;
  • an input module for inputting the data to be identified into a lung feature identification model, the lung feature identification model comprising a lung image identification model, a lung text identification model and a lung fusion identification model;
  • the first recognition module is used for performing lung image feature extraction on the to-be-recognized lung image through the lung image recognition model, generating a lung image feature vector and an image recognition result, and simultaneously through the lung text recognition model performing lung text feature extraction on the description of the to-be-recognized lung text to generate a lung text feature vector and a text recognition result;
  • the second recognition module is used to fuse the lung image feature vector and the lung text feature vector using the attention mechanism through the lung fusion recognition model, and extract and recognize the fused features to obtain a fusion recognition result;
  • a voting module configured to vote on the image recognition result, the text recognition result and the fusion recognition result through the lung feature recognition model to obtain the lung feature recognition result corresponding to the data to be recognized;
  • the lung feature identification result indicates the lung feature category of the data to be identified.
  • a computer device comprising a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, the processor implementing the following steps when executing the computer-readable instructions:
  • Acquire data to be identified wherein the data to be identified includes an image of the lung to be identified and a text description of the lung to be identified; input the data to be identified into a lung feature recognition model, where the lung feature identification model includes lung Image recognition model, lung text recognition model and lung fusion recognition model;
  • Lung image feature extraction is performed on the to-be-recognized lung image by the lung image recognition model to generate a lung image feature vector and image recognition result.
  • the text description performs lung text feature extraction to generate lung text feature vectors and text recognition results;
  • the image recognition result, the text recognition result and the fusion recognition result are voted on by the lung feature recognition model to obtain the lung feature recognition result corresponding to the data to be recognized; the lung feature The recognition result indicates the lung feature category of the data to be recognized.
  • One or more readable storage media storing computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the following steps:
  • Acquire data to be identified wherein the data to be identified includes an image of the lung to be identified and a text description of the lung to be identified; input the data to be identified into a lung feature recognition model, where the lung feature identification model includes lung Image recognition model, lung text recognition model and lung fusion recognition model;
  • Lung image feature extraction is performed on the to-be-recognized lung image by the lung image recognition model to generate a lung image feature vector and image recognition result.
  • the text description performs lung text feature extraction to generate lung text feature vectors and text recognition results;
  • the image recognition result, the text recognition result and the fusion recognition result are voted on by the lung feature recognition model to obtain the lung feature recognition result corresponding to the data to be recognized; the lung feature The recognition result indicates the lung feature category of the data to be recognized.
  • the lung feature identification method, device, computer equipment and storage medium obtain the data to be identified; the data to be identified includes the image of the lung to be identified and the text description of the lung to be identified; the data to be identified is input to a lung feature recognition model including a lung image recognition model, a lung text recognition model and a lung fusion recognition model; the lung image feature extraction is performed on the to-be-recognized lung image by the lung image recognition model, Generate lung image feature vectors and image recognition results, and perform lung text feature extraction on the description of the to-be-recognized lung text through the lung text recognition model to generate lung text feature vectors and text recognition results; The lung fusion recognition model uses the attention mechanism to fuse the lung image feature vector and the lung text feature vector, and extracts and recognizes the fused features to obtain a fusion recognition result; through the lung feature recognition model The image recognition result, the text recognition result and the fusion recognition result are voted, and the lung feature recognition result corresponding to the data to be recognized is obtained.
  • the recognition to be recognized by the lung image recognition model is realized.
  • Lung image get the image recognition result, identify the text description of the lung to be recognized through the lung text recognition model, get the text recognition result, and then combine the lung image to be recognized and the text description to be recognized, use the attention mechanism, through the lung fusion
  • the recognition model extracts the image and text fusion features for recognition, and obtains the fusion recognition result.
  • the text recognition result and the fusion recognition result voting is carried out, and the lung feature recognition result is obtained. Recognize lung images and text descriptions of the lungs to be recognized, and automatically, quickly and accurately identify lung features through the multimodal model-based lung feature recognition model, improve the recognition accuracy and reliability, and improve the recognition effectiveness.
  • FIG. 1 is a schematic diagram of an application environment of a lung feature identification method in an embodiment of the present application
  • FIG. 2 is a flowchart of a lung feature identification method in an embodiment of the present application.
  • step S30 of the lung feature identification method in an embodiment of the present application is a flowchart of step S30 of the lung feature identification method in an embodiment of the present application
  • step S30 of the lung feature identification method in another embodiment of the present application is a flowchart of step S30 of the lung feature identification method in another embodiment of the present application.
  • step S40 of the lung feature identification method in an embodiment of the present application
  • step S50 of the lung feature identification method in an embodiment of the present application is a flowchart of step S50 of the lung feature identification method in an embodiment of the present application.
  • FIG. 7 is a schematic block diagram of a lung feature identification device in an embodiment of the present application.
  • FIG. 8 is a schematic diagram of a computer device in an embodiment of the present application.
  • the lung feature identification method provided by the present application can be applied in the application environment as shown in FIG. 1 , wherein the client (computer device) communicates with the server through the network.
  • the client computer equipment
  • the server includes but is not limited to various personal computers, notebook computers, smart phones, tablet computers, cameras and portable wearable devices.
  • the server can be implemented as an independent server or a server cluster composed of multiple servers.
  • a method for identifying lung features is provided, and its technical solution mainly includes the following steps S10-S50:
  • S10 Acquire data to be identified, wherein the data to be identified includes an image of the lung to be identified and a text description of the lung to be identified.
  • the lung image to be identified is an image collected by a lung imaging device, and the lung imaging device can be selected according to requirements, for example, the lung imaging device is a CT device, an X-ray machine, or a three-dimensional projection device, etc. etc.
  • the lung text description is a description of the lung features in the to-be-recognized lung image, that is, the lung text is described as the main complaint information for the to-be-recognized lung image
  • the lung features The features reflected by the movement of lung tissue, such as lung features including pleural concave features, air bronchus features, lung vacuole features, lung spur features, lung ground glass-like features, etc., after collecting the to-be-identified lung image, And after inputting the text description of the lung to be recognized for the image of the lung to be recognized, the image of the lung to be recognized and the text description of the lung to be recognized are determined as the data to be recognized, and a recognition request is triggered.
  • the identification request is a request for
  • the lung feature recognition model includes a lung image recognition model, a lung text recognition model, and a lung fusion recognition model.
  • the lung feature recognition model is a multimodal model that has been trained, and the lung feature recognition model can recognize the lung features of the data to be identified, and the lung feature recognition model includes lungs. part image recognition model, lung text recognition model and lung fusion recognition model, the lung image recognition model is to obtain the image recognition result by extracting the lung image features in the lung image to be recognized, and performing image recognition, And generate a lung image feature vector for the lung fusion recognition model, the lung image feature is the feature of the image space embodied by the movement of the lung tissue, and the network structure of the lung image recognition model can be based on the needs of image recognition.
  • the network structure of the lung image recognition model is VGG16, VGG19, GoogleNet or ResNet, etc.
  • the network structure of the lung image recognition model is the network structure of VGG19; the lung text recognition model is By extracting the lung text features in the text description of the lungs to be identified, and performing text recognition, the text recognition results are obtained, and a lung text feature vector for the lung fusion recognition model is generated, where the lung text features are lungs
  • the characteristics of the text space reflected by the movement of the external tissue, the network structure of the lung text recognition model can be set according to the needs of language recognition, for example, the network structure of the lung text recognition model is TextCNN, LSTM or BERT, etc., as a preference,
  • the network structure of the described lung text recognition model selects the network structure of TextCNN; the described lung fusion recognition model is to use the attention mechanism to fuse the described lung image feature vector and the described lung text feature vector, and extract the fused
  • the image-text fusion feature in the lung image feature vector and the lung text feature vector, and the fusion recognition result is identified, and the image-text fusion feature is the lung image feature vector and the lung
  • the method before the step S20, that is, before the input of the data to be identified into the lung feature identification model, the method includes:
  • Obtain a lung sample set where the lung sample set includes a plurality of lung samples, the lung samples include a lung image and a lung text description associated with the lung image, and the lung samples are associated with the lung image.
  • a lung feature class label association
  • the lung sample set is a collection of the lung samples
  • the historical collection of the lung samples includes a lung image and a lung text description associated with the lung image.
  • the lung sample is associated with a lung feature class label
  • the lung feature class label is a label related to the lung feature class marked on the lung sample
  • the lung image is a historically collected through the lung
  • the image picture of the lung collected by the photographing device is a description of the lung feature in the lung image associated therewith
  • the lung feature category is a classification of the lung feature
  • the lung feature categories include a pleural indentation feature class corresponding to a pleural indentation feature, an air bronchus feature class corresponding to an air bronchus feature, a lung vacuolar feature class corresponding to a pulmonary vacuole feature, and a lung burr feature corresponding to a pleural indentation feature class.
  • Lung spur feature class and lung ground glass features corresponding to lung ground glass features.
  • the multimodal model includes a lung sample image recognition model, a lung sample text recognition model and a lung sample fusion recognition model.
  • the multimodal model is to match the similarity between images and texts, that is, to measure the similarity between an image and a piece of text (the global similarity between the image and the text), and identify the implicit relationship between the image and the text. The characteristics of the relationship are determined, and the classification result of the fusion of an image and a text is determined.
  • the multimodal model includes the initial parameters, and the initial parameters include the lung sample image recognition model and the lung sample text recognition model.
  • the parameters of the model and the lung sample fusion recognition model can be transferred directly from the parameters in the multimodal recognition models in other fields to the initial parameters in the multimodal model by means of transfer learning, simplifying
  • the training process shortens the training time and improves the training efficiency.
  • the multimodal model includes a lung sample image recognition model, a lung sample text recognition model and a lung sample fusion recognition model.
  • the lung image recognition model The lung sample image recognition model that has been trained, the lung text recognition model is the lung sample text recognition model that has been trained, and the lung fusion recognition model is the lung sample fusion that has been trained. Identify the model.
  • the lung image feature is the feature of the image space embodied by the movement of lung tissue
  • the lung sample image feature vector is a vector matrix with the lung image feature
  • the image sample recognition result is all
  • the lung sample image recognition model identifies the results of the lung features in the lung image by the similarity of the image space based on the extracted lung image features
  • the lung sample text feature vector is a feature vector with the lung image.
  • a vector matrix of image features, and the text sample recognition result is that the lung sample text recognition model performs text space similarity according to the extracted lung text features to identify the lung features in the lung text description. result.
  • the image feature vector of the lung sample and the text feature vector of the lung sample are fused through the attention mechanism, and the learning to extract the image-text fusion feature is to capture the implicit similarity between the image and the text. Feature extraction, and local similarity measurement and extraction.
  • sample identification result and the lung feature category label are input into the loss function of the multimodal model, and the loss value is calculated through the loss function.
  • the convergence condition may be a condition that the loss value is small and will not decrease after 6,000 calculations, that is, the loss value is small and will not decrease after 6,000 calculations.
  • the convergence condition can also be the condition that the loss value is less than the set threshold, that is, when the loss value is less than the set threshold.
  • the training is stopped, and the multimodal model after convergence is recorded as a lung feature recognition model. In this way, when the loss value does not reach the preset convergence condition, the multimodal model is continuously adjusted.
  • the initial parameters in the model are triggered, and the lung image feature extraction is performed on the lung image through the lung sample image recognition model to generate a lung sample image feature vector and an image sample recognition result.
  • Part of the sample text recognition model extracts the lung text features from the lung text description, and generates the lung sample text feature vector and the text sample recognition results, which can continuously move closer to the accurate results, so that the accuracy of the recognition is higher. Come higher. In this way, the lung feature recognition of the multimodal model can be optimized, and the accuracy and reliability of the lung feature recognition are improved.
  • the lung image recognition model performs channel splitting and convolution on the to-be-recognized lung image, thereby extracting the lung image features
  • the lung image features are images embodied by the movement of lung tissue.
  • the lung image recognition model includes a plurality of convolutional layers
  • the convolutional layers of the lung image recognition model can be marked as image convolutional layers, and through each of the lung image recognition model
  • the image convolution layer convolves the to-be-identified lung image according to different convolution kernels, and generates the lung image feature vector corresponding to each image convolution layer, and the lung image feature vector has the lung image
  • the dimension of each described lung image feature vector is different according to the difference of each image convolution layer
  • the image recognition result is the lung image feature extracted by the lung image recognition model according to the
  • the lung text recognition model performs word vector conversion on the description of the lung text to be identified, and then performs convolution to extract the lung text features.
  • the lung text feature is the feature of the text space embodied by the movement of lung tissue
  • the lung text recognition model includes a plurality of convolution layers, and the convolution layer of the lung text recognition model can be marked as text convolution Layer, through each text convolution layer in the lung image recognition model, the description of the lung text to be recognized is convolved according to different convolution kernels, and the lung text features corresponding to each text convolution layer are generated.
  • vector the lung text feature vector is a vector matrix with the lung text feature, the dimension of each described lung text feature vector is different according to the difference of each text convolution layer, and the text recognition result is the
  • the lung text recognition model identifies the result of the lung features by performing text space similarity based on the extracted lung text features.
  • the lung image feature extraction is performed on the to-be-recognized lung image by the lung image recognition model, and a lung image feature vector is generated. and image recognition results, including:
  • the lung image recognition model is a network model constructed based on VGG19.
  • the lung image to be identified is an image of three channels: red channel, green channel, and blue channel, that is, the lung image to be identified includes the red channel image corresponding to the red channel, and the green channel image corresponding to the red channel.
  • the green channel image corresponding to the channel and the blue channel image corresponding to the blue channel, through channel splitting, the to-be-identified lung image is split into the red channel image, the green channel image and the
  • the blue channel image, the red channel image is an image that reflects the degree of redness of each pixel through pixel values in the range of 0 to 255
  • the green channel image is an image that reflects the redness of each pixel through pixel values in the range of 0 to 255.
  • An image with a green level, and the blue channel image is an image in which the blue level of each pixel is represented by pixel values ranging from 0 to 255.
  • the lung image recognition model is a network model constructed based on VGG19, and the convolution depth of the lung image can be set to 19, that is, a network model with 19-level convolution layers.
  • the red channel image is convolved by the lung image recognition model to obtain the red feature vector
  • the red feature vector is the vector embodied in the red space extracted from the lung image features
  • the lung image recognition model convolves the green channel image to obtain the green feature vector.
  • the recognition model convolves the blue channel image to obtain the blue feature vector, where the blue feature vector is the vector embodied in the blue space extracted from the lung image features, and the red feature vector,
  • the green feature vector and the blue feature vector are determined as the lung image feature vector.
  • S303 Perform image recognition on the lung image feature vector by using the lung image recognition model to obtain the image recognition result.
  • image recognition is performed on the lung image feature vector by the lung image recognition model, and the image recognition is to perform fully connected classification according to the extracted lung image feature vector to obtain each lung feature category.
  • the probability distribution of so as to output the recognized image recognition result.
  • the present application realizes that the lung image to be recognized is split into a red channel image, a green channel image and a blue channel image through the lung image recognition model; the lung image recognition model is a network model constructed based on VGG19 ; Carry out convolution extraction on the red channel image, the green channel image and the blue channel image respectively by the lung image recognition model to obtain the lung image feature vector; by the lung image recognition model to The lung image feature vector is used for image recognition to obtain the image recognition result.
  • the constructed network model performs convolution on each channel image to extract the lung image features, obtains the lung image feature vector, and outputs the image recognition result according to the lung image feature vector, which can extract the lung image in the lung image to be identified.
  • the lung feature categories are identified through the extracted lung image features, which provides a data basis for subsequent identification and improves the accuracy and reliability of identification.
  • the lung text feature extraction is performed on the description of the lung text to be identified by the lung text recognition model, and the lung text feature is generated.
  • Vector and text recognition results including:
  • the word segmentation is to use a word dictionary to split the description of the lung text to be identified into individual words, and the word dictionary contains word vectors corresponding to all medical terms and words related to the lungs. , and then convert the split words into their corresponding word vectors, which can be converted by the conversion method of word2vec or Glove, and then splicing the converted word vectors to form the text word vector.
  • the pulmonary text recognition model is a network model constructed based on TextCNN, that is, the pulmonary text recognition model has the network structure of TextCNN, and the convolution depth of the pulmonary text recognition model is set to 19, that is, it has 19
  • the network model of the hierarchical convolution layer, the convolution depth of the lung text recognition model is the same as the convolution depth of the lung image recognition model, so as to facilitate the recognition of the subsequent lung fusion recognition model.
  • the channel expansion is a process of expanding the text word vector of a single channel to a vector matrix of a preset dimension and copying the vector matrix until the number of preset channels, that is, expanding the text word vector to the same size as the text word vector.
  • the vector matrix of the same dimension of the lung image feature vector, the expansion method can be set according to the needs, and the vector matrix is copied into a vector matrix with the same number of channels as the lung image feature vector, so as to obtain the same vector matrix as the vector matrix. Describe the first text word vector, the second text word vector and the third text word vector.
  • the first text word vector is convolved by the lung text recognition model to obtain the first text feature vector
  • the second text word vector is processed by the lung text recognition model.
  • the convolution kernel for convolving the vector, the convolution kernel for convolving the second text word vector, and the convolution kernel for convolving the third text word vector may be different, that is, from different text spaces.
  • the dimension of lung text feature vector is extracted, and the first text word vector, the second text word vector and the third text word vector are determined as the lung text feature vector.
  • text recognition is performed on the lung text feature vector by the lung text recognition model, and the text recognition is to perform a fully connected classification according to the extracted lung text feature vector to obtain each lung feature category.
  • the probability distribution of so as to output the recognized text recognition result.
  • the present application implements word segmentation for the description of the lung text to be identified through the lung text recognition model, and constructs a text word vector corresponding to the description of the lung text to be identified; the lung text recognition model is based on A network model constructed by TextCNN; channel expansion of the text word vector to generate a first text word vector, a second text word vector and a third text word vector;
  • the word vector, the second text word vector, and the third text word vector are extracted by convolution to obtain a lung text feature vector, and text recognition is performed on the lung text feature vector by the lung text recognition model, Obtaining the text recognition result, in this way, it is realized that the first text word vector, the second text word vector and the third text word vector are generated by segmenting the description of the lung text to be identified and
  • the lung feature category is identified through the extracted lung text features, which provides a data basis for subsequent identification and improves the accuracy and reliability of identification.
  • the attention mechanism is a mechanism learned by an additional feedforward neural network in neural network learning and recognition through the attention weight, and the lung image feature vector and the lung image feature vector and The implicit relationship between the lung text feature vectors, that is, the lung image feature vector and the lung text feature vector are carried out according to the weight parameters corresponding to each convolution layer learned through the attention mechanism.
  • Weighted fusion so as to obtain the fusion feature vector corresponding to each convolution layer, convolve all the fusion feature vectors, and extract the image-text fusion features, that is, extract the fused features, the image
  • the text fusion feature is an implicit feature associated between the lung image feature vector and the lung text feature vector, that is, the global similarity between the lung image feature vector and the lung text feature vector.
  • the feature is identified according to the extracted image-text fusion features, that is, the probability distribution of each lung feature category is classified by full connection, so as to output the fusion recognition result.
  • the lung image feature vector and the lung text feature vector are fused by using the attention mechanism through the lung fusion recognition model. , and extract image-text fusion features for recognition, and obtain fusion recognition results, including:
  • the attention mechanism technology is to enhance the useful information in the feature vector, that is, the weight parameters corresponding to each convolutional layer according to the useful vector of the lung image feature vector and the lung text feature vector.
  • a weighted average is performed and fused to generate a fused feature vector corresponding to each convolutional layer.
  • the convolution depth in the lung fusion recognition model is the same as the convolution depth of the lung image recognition model or the lung text recognition model, and the convolution depth in the lung fusion recognition model is preferably 19 levels.
  • the lung image feature vector includes a red feature vector, a green feature vector and a blue feature vector;
  • the lung text feature vector includes a first text feature vector, a second text feature vector feature vector and third text feature vector;
  • the lung image recognition model, the lung text recognition model and the lung fusion recognition model all have the same convolution level, and each of the three models is provided with a convolution layer corresponding to each convolution level. ;
  • the described lung image feature vector and the described lung text feature vector are weighted and fused by the weight parameters corresponding to each convolutional layer in the described lung fusion recognition model to obtain a fusion corresponding to each convolutional layer.
  • eigenvectors including:
  • the red feature vector and the first text feature vector corresponding to the same convolution level are weighted according to the first weight parameter of the convolution level, that is, the red feature vector and the first text feature
  • Each vector value in the vector is weighted and averaged according to the first weight parameter to obtain the first fusion feature vector, and the red feature vector, the first text feature vector and the first fusion feature vector have the same dimensions .
  • the green feature vector and the second text feature vector corresponding to the same convolution level are weighted according to the second weight parameter of the convolution level, that is, the green feature vector and the second text feature
  • the green feature vector and the second text feature vector are weighted according to the second weight parameter of the convolution level, that is, the green feature vector and the second text feature
  • Each vector value in the vector is weighted and averaged according to the second weight parameter to obtain the second fusion feature vector, and the green feature vector, the second text feature vector and the second fusion feature vector have the same dimensions .
  • the blue feature vector and the third text feature vector corresponding to the same convolution level are weighted according to the third weight parameter of the convolution level, that is, the blue feature vector and the third
  • the third weight parameter of the convolution level that is, the blue feature vector and the third
  • Each vector value in the text feature vector is weighted and averaged according to the third weight parameter to obtain the third fusion feature vector, the blue feature vector, the third text feature vector and the third fusion feature vector dimensions are the same.
  • the execution order of the steps S4011, S4012 and S4013 is not limited, and may be executed in series or in parallel, and the first weight parameter, the second weight parameter and the third weight parameter may be the same, or can all be different.
  • the weighted average is to average the first fusion feature vector, the second fusion feature vector and the third fusion feature vector after weighting, and the The first fusion feature vector, the second fusion feature vector and the third fusion feature vector are weighted and averaged to obtain the fusion feature vector corresponding to each convolution layer.
  • the extraction process of the image-text fusion feature may be to convolve the fusion feature vector of the convolutional layer of the first layer, and then perform the convolution with the fusion feature vector of the convolutional layer of the next layer of the convolutional layer.
  • the transfer feature vector is obtained by superposition, and then the transfer feature vector is convolved, and the transfer feature vector is continuously superimposed with the fusion feature vector of the convolution layer of the next layer to obtain the transfer feature vector, and the superimposed transfer feature vector is convolved until a one-dimensional The feature vector extraction process of .
  • the identification is performed according to the extracted image-text fusion feature by the lung fusion recognition model, and the identification is to obtain the probability distribution of each lung feature category according to the extracted image-text fusion feature, thereby outputting The identified fusion identification result.
  • the present application realizes the weighted fusion of the lung image feature vector and the lung text feature vector by using the attention mechanism technology and the weight parameters corresponding to each convolution layer in the lung fusion recognition model. , obtain the fusion feature vector corresponding to each convolution layer; perform the image-text fusion feature extraction on the fusion feature vector by the lung fusion recognition model; The image and text fusion features are used for recognition, and the fusion recognition result is obtained.
  • the attention mechanism can be used to enhance the useful information in the image and the text, and the global similarity between the image and the text can be better captured.
  • the lung image feature vector and the lung text feature vector are weighted and fused, and the image and text fusion features are extracted for identification, which can improve the accuracy and reliability of lung feature identification.
  • the voting is to perform a weighted average of the probability values corresponding to the same lung feature category in the image recognition result, the text recognition result and the fusion recognition result, and finally determine that the probability value is the highest. and the lung feature category with the highest probability value is used as the lung feature identification result, and the lung feature identification result includes the identified lung feature category and the probability value corresponding to the category, so
  • the lung feature recognition result indicates the lung feature category of the data to be identified, and the lung feature is the feature embodied by the movement of lung tissue.
  • the lung feature category is a classification of the lung feature, for example, the lung feature category includes the pleural depression feature class corresponding to the pleural depression feature, The air bronchus feature class corresponding to the air bronchus feature, the lung vacuole feature class corresponding to the lung vacuole feature, the lung spur feature class corresponding to the lung spur feature, and the lung ground glass feature corresponding to the lung ground glass feature.
  • the present application realizes by acquiring the data to be recognized in the recognition request; the data to be recognized includes the lung image to be recognized and the text description of the lung to be recognized; the data to be recognized is input into a recognition model containing a lung image , the lung feature recognition model of the lung text recognition model and the lung fusion recognition model; the lung image feature extraction is performed on the to-be-recognized lung image by the lung image recognition model, and the lung image feature vector and the image are generated.
  • Recognition results while performing lung text feature extraction on the description of the to-be-recognized lung text by the lung text recognition model, to generate a lung text feature vector and a text recognition result; using the attention through the lung fusion recognition model
  • the mechanism fuses the lung image feature vector and the lung text feature vector, and extracts image-text fusion features for recognition, and obtains a fusion recognition result; through the lung feature recognition model, the image recognition result, the text
  • the identification results and the fusion identification results are voted for, and the lung feature identification results corresponding to the data to be identified are obtained. In this way, the identification of the lung images to be identified through the lung image identification model is realized, and the image identification results are obtained.
  • the lung text recognition model recognizes the text description of the lungs to be recognized, and obtains the text recognition result, and then combines the lung image to be recognized and the text description to be recognized, and uses the attention mechanism to extract the image and text fusion features through the lung fusion recognition model for recognition. Obtain the fusion recognition result, and finally vote according to the image recognition result, the text recognition result and the fusion recognition result, and obtain the lung feature recognition result, which realizes the combination of the lung image to be recognized and the lung text to be recognized. Described, through the lung feature recognition model based on the multimodal model, the lung features are automatically, quickly and accurately identified, the recognition accuracy and reliability are improved, and the recognition efficiency is improved.
  • step S50 the image recognition result, the text recognition result and the fusion recognition result are voted on by the lung feature recognition model. , to obtain the lung feature identification results corresponding to the data to be identified, including:
  • the weight parameters corresponding to the last layer of the convolutional layer in the lung fusion recognition model include the image weights and all of the image weights provided to the lung image feature vector corresponding to the last layer of the convolutional layer. Describe the text weights of the lung text feature vector.
  • the obtained image weight and the text weight are kept unchanged, the image weight is used as the voting parameter of the image recognition result, and the text weight is used as the vote for the document recognition result.
  • Voting parameter a value of one is used as the voting parameter of the fusion identification result.
  • the final probability distribution of each lung feature category is obtained through weighted average, and the lung feature category with the highest probability value is determined as the lung feature identification result.
  • the weight parameters corresponding to the last layer of the convolutional layer in the lung fusion recognition model are obtained; voting parameters are determined according to the obtained weight parameters; according to the voting parameters, Carry out the voting on the image recognition result, the text recognition result and the fusion recognition result to obtain the lung feature recognition result.
  • the above fusion recognition results are objectively voted, and the lung feature categories are finally identified, which improves the accuracy and reliability of lung feature recognition.
  • a device for identifying lung features is provided, and the device for identifying lung features corresponds to the method for identifying lung features in the above embodiments.
  • the lung feature identification device includes a receiving module 11 , an input module 12 , a first identification module 13 , a second identification module 14 and a voting module 15 .
  • the detailed description of each functional module is as follows:
  • the receiving module 11 is configured to receive the identification request and obtain the data to be identified in the identification request; the data to be identified includes the lung image to be identified and the text description of the lung to be identified; the description of the lung text to be identified is A description of the lung features in the to-be-identified lung image;
  • the input module 12 is used to input the data to be recognized into the lung feature recognition model;
  • the lung feature recognition model includes a lung image recognition model, a lung text recognition model and a lung fusion recognition model;
  • the first recognition module 13 is used for performing lung image feature extraction on the to-be-recognized lung image by the lung image recognition model, generating a lung image feature vector and an image recognition result, and simultaneously identifying the lung text through the lung image.
  • the model performs lung text feature extraction on the description of the to-be-recognized lung text, and generates a lung text feature vector and a text recognition result;
  • the second recognition module 14 is configured to use the attention mechanism to fuse the lung image feature vector and the lung text feature vector through the lung fusion recognition model, and extract the image text fusion feature for recognition, and obtain a fusion recognition result ;
  • the voting module 15 is used for voting on the image recognition result, the text recognition result and the fusion recognition result through the lung feature recognition model, and obtains the lung feature recognition result corresponding to the data to be recognized ;
  • the lung feature identification result indicates the lung feature category of the data to be identified.
  • Each module in the above-mentioned lung feature identification device may be implemented in whole or in part by software, hardware and combinations thereof.
  • the above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.
  • a computer device is provided, and the computer device may be a server, and its internal structure diagram may be as shown in FIG. 8 .
  • the computer device includes a processor, memory, a network interface and a database connected by a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities.
  • the memory of the computer device includes a readable storage medium, an internal memory.
  • the readable storage medium stores an operating system, computer readable instructions and a database.
  • the internal memory provides an environment for the execution of the operating system and computer-readable instructions in the readable storage medium.
  • the network interface of the computer device is used to communicate with an external terminal through a network connection.
  • the computer-readable instructions when executed by a processor, implement a lung feature identification method.
  • the readable storage medium provided in this embodiment includes a non-volatile readable storage medium and a volatile readable storage medium.
  • a computer device including a memory, a processor, and computer-readable instructions stored on the memory and executable on the processor, and the processor implements the lungs in the above embodiments when the computer-readable instructions are executed. feature recognition method.
  • one or more readable storage media storing computer-readable instructions are provided, and the readable storage media provided in this embodiment include non-volatile readable storage media and volatile readable storage media medium; computer-readable instructions are stored on the readable storage medium, and when the computer-readable instructions are executed by one or more processors, cause the one or more processors to implement the lung feature identification method in the foregoing embodiment.
  • Nonvolatile memory may include read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory.
  • Volatile memory may include random access memory (RAM) or external cache memory.
  • RAM is available in various forms such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous chain Road (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • Health & Medical Sciences (AREA)
  • General Physics & Mathematics (AREA)
  • Medical Informatics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Evolutionary Biology (AREA)
  • Evolutionary Computation (AREA)
  • General Engineering & Computer Science (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Public Health (AREA)
  • Artificial Intelligence (AREA)
  • General Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biomedical Technology (AREA)
  • Quality & Reliability (AREA)
  • Radiology & Medical Imaging (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Databases & Information Systems (AREA)
  • Pathology (AREA)
  • Multimedia (AREA)
  • Epidemiology (AREA)
  • Primary Health Care (AREA)
  • Apparatus For Radiation Diagnosis (AREA)
  • Image Analysis (AREA)
  • Measuring And Recording Apparatus For Diagnosis (AREA)

Abstract

一种肺部特征识别方法、装置、计算机设备及存储介质，涉及人工智能领域，所述方法包括：通过获取包括待识别肺部图像和待识别肺部文本描述的待识别数据；通过肺部图像识别模型进行肺部图像特征提取，生成肺部图像特征向量和图像识别结果，同时通过肺部文本识别模型进行肺部文本特征提取，生成肺部文本特征向量和文本识别结果；通过肺部融合识别模型使用注意力机制融合肺部图像特征向量和肺部文本特征向量，并提取图像文本融合特征进行识别，得到融合识别结果；通过投票表决得到肺部特征识别结果。该方法实现了准确地识别出肺部特征，提高了识别准确率和可靠性。该方法适用于智慧医疗等领域，可进一步推动智慧城市的建设。

Description

肺部特征识别方法、装置、计算机设备及存储介质
本申请要求于2020年9月21日提交中国专利局、申请号为202010991495.4，发明名称为“肺部特征识别方法、装置、计算机设备及存储介质”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及人工智能的图像分类领域，尤其涉及一种肺部特征识别方法、装置、计算机设备及存储介质。
背景技术
在目前医疗体系下，肺部特征的识别主要依靠医务人员根据自己的经验对肺部影像信息进行人工判别，因为肺部组织运动是不均匀、复杂的，所以在判别过程中不仅需耗费医务人员的时间、精力，而且存在判断错误的风险，同时在判别过程中往往仅对肺部影像信息进行识别，未结合该肺部影像信息的主诉信息(针对肺部影像信息的文本描述)进行识别，容易丢失肺部组织运动的信息，导致准确率不高，效率低下。
发明内容
本申请提供一种肺部特征识别方法、装置、计算机设备及存储介质，实现了通过包括肺部图像识别模型、肺部文本识别模型、肺部融合识别模型的肺部特征识别模型，并运用注意力机制，以及结合待识别肺部图像和待识别肺部文本描述进行识别，实现了自动地、快速地、准确地识别出肺部特征，提高了识别准确率和可靠性，提升了识别效率。本申请适用于智慧医疗等领域，可进一步推动智慧城市的建设。
一种肺部特征识别方法，包括：
获取待识别数据，其中，所述待识别数据包括待识别肺部图像和待识别肺部文本描述；
将所述待识别数据输入至肺部特征识别模型，所述肺部特征识别模型包括肺部图像识别模型、肺部文本识别模型和肺部融合识别模型；
通过所述肺部图像识别模型对所述待识别肺部图像进行肺部图像特征提取，生成肺部图像特征向量和图像识别结果，同时通过所述肺部文本识别模型对所述待识别肺部文本描述进行肺部文本特征提取，生成肺部文本特征向量和文本识别结果；
通过所述肺部融合识别模型使用注意力机制融合所述肺部图像特征向量和所述肺部文本特征向量，并对融合后的特征进行提取以及识别，得到融合识别结果；
通过所述肺部特征识别模型对所述图像识别结果、所述文本识别结果和所述融合识别结果进行投票表决，获得与所述待识别数据对应的肺部特征识别结果；所述肺部特征识别结果表明了所述待识别数据的肺部特征类别。
一种肺部特征识别装置，包括：
接收模块，用于获取待识别数据，其中，所述待识别数据包括待识别肺部图像和待识别肺部文本描述；
输入模块，用于将所述待识别数据输入至肺部特征识别模型，所述肺部特征识别模型包括肺部图像识别模型、肺部文本识别模型和肺部融合识别模型；
第一识别模块，用于通过所述肺部图像识别模型对所述待识别肺部图像进行肺部图像特征提取，生成肺部图像特征向量和图像识别结果，同时通过所述肺部文本识别模型对所述待识别肺部文本描述进行肺部文本特征提取，生成肺部文本特征向量和文本识别结果；
第二识别模块，用于通过所述肺部融合识别模型使用注意力机制融合所述肺部图像特征向量和所述肺部文本特征向量，并对融合后的特征进行提取以及识别，得到融合识别结果；
表决模块，用于通过所述肺部特征识别模型对所述图像识别结果、所述文本识别结果和所述融合识别结果进行投票表决，获得与所述待识别数据对应的肺部特征识别结果；所述肺部特征识别结果表明了所述待识别数据的肺部特征类别。
一种计算机设备，包括存储器、处理器以及存储在所述存储器中并可在所述处理器上运行的计算机可读指令，所述处理器执行所述计算机可读指令时实现如下步骤：
获取待识别数据，其中，所述待识别数据包括待识别肺部图像和待识别肺部文本描述；将所述待识别数据输入至肺部特征识别模型，所述肺部特征识别模型包括肺部图像识别模型、肺部文本识别模型和肺部融合识别模型；
通过所述肺部图像识别模型对所述待识别肺部图像进行肺部图像特征提取，生成肺部图像特征向量和图像识别结果，同时通过所述肺部文本识别模型对所述待识别肺部文本描述进行肺部文本特征提取，生成肺部文本特征向量和文本识别结果；
通过所述肺部融合识别模型使用注意力机制融合所述肺部图像特征向量和所述肺部文本特征向量，并对融合后的特征进行提取以及识别，得到融合识别结果；
通过所述肺部特征识别模型对所述图像识别结果、所述文本识别结果和所述融合识别结果进行投票表决，获得与所述待识别数据对应的肺部特征识别结果；所述肺部特征识别结果表明了所述待识别数据的肺部特征类别。
一个或多个存储有计算机可读指令的可读存储介质，所述计算机可读指令被一个或多个处理器执行时，使得所述一个或多个处理器执行如下步骤：
获取待识别数据，其中，所述待识别数据包括待识别肺部图像和待识别肺部文本描述；将所述待识别数据输入至肺部特征识别模型，所述肺部特征识别模型包括肺部图像识别模型、肺部文本识别模型和肺部融合识别模型；
通过所述肺部图像识别模型对所述待识别肺部图像进行肺部图像特征提取，生成肺部图像特征向量和图像识别结果，同时通过所述肺部文本识别模型对所述待识别肺部文本描述进行肺部文本特征提取，生成肺部文本特征向量和文本识别结果；
通过所述肺部融合识别模型使用注意力机制融合所述肺部图像特征向量和所述肺部文本特征向量，并对融合后的特征进行提取以及识别，得到融合识别结果；
通过所述肺部特征识别模型对所述图像识别结果、所述文本识别结果和所述融合识别结果进行投票表决，获得与所述待识别数据对应的肺部特征识别结果；所述肺部特征识别结果表明了所述待识别数据的肺部特征类别。
本申请提供的肺部特征识别方法、装置、计算机设备及存储介质，通过获取待识别数据；所述待识别数据包括待识别肺部图像和待识别肺部文本描述；将所述待识别数据输入至包含有肺部图像识别模型、肺部文本识别模型和肺部融合识别模型的肺部特征识别模型；通过所述肺部图像识别模型对所述待识别肺部图像进行肺部图像特征提取，生成肺部图像特征向量和图像识别结果，同时通过所述肺部文本识别模型对所述待识别肺部文本描述进行肺部文本特征提取，生成肺部文本特征向量和文本识别结果；通过所述肺部融合识别模型使用注意力机制融合所述肺部图像特征向量和所述肺部文本特征向量，并对融合后的特征进行提取以及识别，得到融合识别结果；通过所述肺部特征识别模型对所述图像识别结果、所述文本识别结果和所述融合识别结果进行投票表决，获得与所述待识别数据对应的肺部特征识别结果，如此，实现了通过肺部图像识别模型识别待识别肺部图像，得到图像识别结果，通过肺部文本识别模型识别待识别肺部文本描述，得到文本识别结果，再结合待识别肺部图像和待识别文本描述，运用注意力机制，通过肺部融合识别模型提取图像文本融合特征进行识别，得到融合识别结果，最后根据所述图像识别结果、所述文本识别结 果和所述融合识别结果进行投票表决，得出肺部特征识别结果，实现了结合待识别肺部图像和待识别肺部文本描述，通过基于多模态模型的肺部特征识别模型自动地、快速地、准确地识别出肺部特征，提高了识别准确率和可靠性，提升了识别效率。
本申请的一个或多个实施例的细节在下面的附图和描述中提出，本申请的其他特征和优点将从说明书、附图以及权利要求变得明显。
附图说明
为了更清楚地说明本申请实施例的技术方案，下面将对本申请实施例的描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本申请的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动性的前提下，还可以根据这些附图获得其他的附图。
图1是本申请一实施例中肺部特征识别方法的应用环境示意图；
图2是本申请一实施例中肺部特征识别方法的流程图；
图3是本申请一实施例中肺部特征识别方法的步骤S30的流程图；
图4是本申请另一实施例中肺部特征识别方法的步骤S30的流程图；
图5是本申请一实施例中肺部特征识别方法的步骤S40的流程图；
图6是本申请一实施例中肺部特征识别方法的步骤S50的流程图；
图7是本申请一实施例中肺部特征识别装置的原理框图；
图8是本申请一实施例中计算机设备的示意图。
具体实施方式
下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例是本申请一部分实施例，而不是全部的实施例。基于本申请中的实施例，本领域普通技术人员在没有作出创造性劳动前提下所获得的所有其他实施例，都属于本申请保护的范围。
本申请提供的肺部特征识别方法，可应用在如图1的应用环境中，其中，客户端(计算机设备)通过网络与服务器进行通信。其中，客户端(计算机设备)包括但不限于为各种个人计算机、笔记本电脑、智能手机、平板电脑、摄像头和便携式可穿戴设备。服务器可以用独立的服务器或者是多个服务器组成的服务器集群来实现。
在一实施例中，如图2所示，提供一种肺部特征识别方法，其技术方案主要包括以下步骤S10-S50：
S10，获取待识别数据，其中，所述待识别数据包括待识别肺部图像和待识别肺部文本描述。
可理解地，所述待识别肺部图像为通过肺部拍摄设备采集到的图像，所述肺部拍摄设备可以根据需求选择，比如肺部拍摄设备为CT设备、X光机或三维投影设备等等，所述肺部文本描述为针对所述待识别肺部图像中的肺部特征的描述，即所述肺部文本描述为针对所述待识别肺部图像的主诉信息，所述肺部特征为肺部组织运动体现的特征，比如肺部特征包括胸膜凹陷特征、气支气管特征、肺空泡特征、肺毛刺特征、肺部毛玻璃样特征等等，在采集完所述待识别肺部图像，而且针对所述待识别肺部图像输入完所述待识别肺部文本描述之后，将所述待识别肺部图像和所述待识别肺部文本描述确定为待识别数据，触发识别请求，所述识别请求为对所述待识别数据进行肺部特征识别的请求，接收到所述识别请求，获取所述识别请求中的待识别数据。
S20，将所述待识别数据输入至肺部特征识别模型，所述肺部特征识别模型包括肺部图像识别模型、肺部文本识别模型和肺部融合识别模型。
可理解地，所述肺部特征识别模型为训练完成的多模态模型，所述肺部特征识别模型 能够实现识别出所述待识别数据的肺部特征，所述肺部特征识别模型包括肺部图像识别模型、肺部文本识别模型和肺部融合识别模型，所述肺部图像识别模型为通过提取所述待识别肺部图像中的肺部图像特征，并进行图像识别出图像识别结果，并生成用于肺部融合识别模型的肺部图像特征向量，所述肺部图像特征为肺部组织运动体现的图像空间的特征，所述肺部图像识别模型的网络结构可以根据图像识别的需求设定，比如肺部图像识别模型的网络结构为VGG16、VGG19、GoogleNet或ResNet等等，作为优选，所述肺部图像识别模型的网络结构选择VGG19的网络结构；所述肺部文本识别模型为通过提取所述待识别肺部文本描述中的肺部文本特征，并进行文本识别出文本识别结果，并生成用于肺部融合识别模型的肺部文本特征向量，所述肺部文本特征为肺部组织运动体现的文本空间的特征，所述肺部文本识别模型的网络结构可以根据语言识别的需求设定，比如肺部文本识别模型的网络结构为TextCNN、LSTM或BERT等等，作为优选，所述肺部文本识别模型的网络结构选择TextCNN的网络结构；所述肺部融合识别模型为运用注意力机制融合所述肺部图像特征向量和所述肺部文本特征向量，并提取融合后的所述肺部图像特征向量和所述肺部文本特征向量中的图像文本融合特征，并识别出融合识别结果，所述图像文本融合特征为所述肺部图像特征向量和所述肺部文本特征向量之间关联的隐含特征，也即所述肺部图像特征向量和所述肺部文本特征向量之间的全局相似性特征，所述肺部融合识别模型的网络结构可以根据需求设定，比如肺部融合识别模型的网络结构为DenseNet、Deep LearningNet或者LeNet等等，作为优选，所述肺部融合识别模型的网络结构为DenseNet的网络结构。
在一实施例中，所述步骤S20之前，即所述将所述待识别数据输入至肺部特征识别模型之前，包括：
S201，获取肺部样本集，所述肺部样本集包括多个肺部样本，所述肺部样本包括肺部影像和与所述肺部影像关联的肺部文本描述，所述肺部样本与一个肺部特征类别标签关联。
可理解地，所述肺部样本集为所述肺部样本的集合，所述肺部样本历史收集的包含肺部影像和与所述肺部影像关联的肺部文本描述的样本，一个所述肺部样本与一个肺部特征类别标签关联，所述肺部特征类别标签为对所述肺部样本标注的与肺部特征类别相关的标签，所述肺部影像为历史收集到的通过肺部拍摄设备采集的肺部的影像图片，所述肺部文本描述为针对与其关联的所述肺部影像中的肺部特征的描述，所述肺部特征类别为对所述肺部特征的分类，比如，所述肺部特征类别包括与胸膜凹陷特征对应的胸膜凹陷特征类、与气支气管特征对应的气支气管特征类、与肺空泡特征对应的肺空泡特征类、与肺毛刺特征对应的肺毛刺特征类和与肺部毛玻璃样特征对应的肺部毛玻璃样特征。
S202，将所述肺部样本输入含有初始参数的多模态模型；所述多模态模型包括肺部样本图像识别模型、肺部样本文本识别模型和肺部样本融合识别模型。
可理解地，所述多模态模型为通过图像文本匹配相似性，即度量一幅图像和一段文本的相似性(图像和文本之间的全局相似性)，识别出图像和文本之间隐含关系的特征，确定出一幅图像和一段文本融合的分类结果，所述多模态模型包含所述初始参数，所述初始参数包含所述肺部样本图像识别模型、所述肺部样本文本识别模型和所述肺部样本融合识别模型的参数，可以通过迁移学习的方式，将其他领域的多模态识别模型中的参数直接迁移至所述多模态模型中的所述初始参数中，简化训练过程，缩短了训练的时间，提高了训练的效率，所述多模态模型包括肺部样本图像识别模型、肺部样本文本识别模型和肺部样本融合识别模型，所述肺部图像识别模型为训练完成的所述肺部样本图像识别模型，所述肺部文本识别模型为训练完成的所述肺部样本文本识别模型，所述肺部融合识别模型为训练完成的所述肺部样本融合识别模型。
S203，通过所述肺部样本图像识别模型对所述肺部影像进行所述肺部图像特征提取，生成肺部样本图像特征向量和图像样本识别结果，同时通过所述肺部样本文本识别模型对 所述肺部文本描述进行所述肺部文本特征提取，生成肺部样本文本特征向量和文本样本识别结果。
可理解地，所述肺部图像特征为肺部组织运动体现的图像空间的特征，所述肺部样本图像特征向量为具有所述肺部图像特征的向量矩阵，所述图像样本识别结果为所述肺部样本图像识别模型根据提取的所述肺部图像特征进行图像空间的相似性识别出所述肺部影像中肺部特征的结果，所述肺部样本文本特征向量为具有所述肺部图像特征的向量矩阵，所述文本样本识别结果为所述肺部样本文本识别模型根据提取的所述肺部文本特征进行文本空间的相似性识别出所述肺部文本描述中的肺部特征的结果。
S204，通过所述肺部样本融合识别模型使用注意力机制融合所述肺部样本图像特征向量和所述肺部样本文本特征向量，并学习提取所述图像文本融合特征以及识别，得到融合样本识别结果。
可理解地，通过所述注意力机制融合所述肺部样本图像特征向量和所述肺部样本文本特征向量，所述学习提取所述图像文本融合特征为通过捕捉图像和文本之间隐含相似特征的提取、以及局部相似性度量并提取。
S205，对所述图像样本识别结果、所述文本样本识别结果和所述融合样本识别结果进行投票表决，得到样本识别结果。
S206，根据所述样本识别结果和所述肺部特征类别标签，确定出损失值。
可理解地，将所述样本识别结果和所述肺部特征类别标签输入所述多模态模型的损失函数中，通过所述损失函数计算出所述损失值。
S207，在所述损失值未达到预设的收敛条件时，迭代更新所述多模态模型的初始参数，直至所述损失值达到所述预设的收敛条件时，将收敛之后的所述多模态模型记录为肺部特征识别模型。
可理解地，所述收敛条件可以为所述损失值经过了6000次计算后值为很小且不会再下降的条件，即在所述损失值经过6000次计算后值为很小且不会再下降时，停止训练，并将收敛之后的所述多模态模型记录为肺部特征识别模型；所述收敛条件也可以为所述损失值小于设定阈值的条件，即在所述损失值小于设定阈值时，停止训练，并收敛之后的所述多模态模型记录为肺部特征识别模型，如此，在所述损失值未达到预设的收敛条件时，不断调整所述多模态模型中的初始参数，并触发通过所述肺部样本图像识别模型对所述肺部影像进行所述肺部图像特征提取，生成肺部样本图像特征向量和图像样本识别结果，同时通过所述肺部样本文本识别模型对所述肺部文本描述进行所述肺部文本特征提取，生成肺部样本文本特征向量和文本样本识别结果的步骤，可以不断向准确的结果靠拢，让识别的准确率越来越高。如此，能够优化多模态模型的肺部特征识别，提高了肺部特征识别的准确性和可靠性。
S30，通过所述肺部图像识别模型对所述待识别肺部图像进行肺部图像特征提取，生成肺部图像特征向量和图像识别结果，同时通过所述肺部文本识别模型对所述待识别肺部文本描述进行肺部文本特征提取，生成肺部文本特征向量和文本识别结果。
可理解地，所述肺部图像识别模型对所述待识别肺部图像进行通道拆分及卷积，从而提取所述肺部图像特征，所述肺部图像特征为肺部组织运动体现的图像空间的特征，所述肺部图像识别模型包含有多个卷积层，可以将所述肺部图像识别模型的卷积层标记为图像卷积层，通过所述肺部图像识别模型中的各图像卷积层对所述待识别肺部图像按照不同卷积核进行卷积，生成与各个图像卷积层对应的所述肺部图像特征向量，所述肺部图像特征向量为具有所述肺部图像特征的向量矩阵，各所述肺部图像特征向量的维度根据各图像卷积层的不同而不同，所述图像识别结果为所述肺部图像识别模型根据提取的所述肺部图像特征进行图像空间的相似性识别出肺部特征的结果，所述肺部文本识别模型对所述待识别肺部文本描述进行词向量转换，再进行卷积，从而提取所述肺部文本特征，所述肺部文本 特征为肺部组织运动体现的文本空间的特征，所述肺部文本识别模型包含有多个卷积层，可以将所述肺部文本识别模型的卷积层标记为文本卷积层，通过所述肺部图像识别模型中的各文本卷积层对所述待识别肺部文本描述按照不同卷积核进行卷积，生成与各个文本卷积层对应的所述肺部文本特征向量，所述肺部文本特征向量为具有所述肺部文本特征的向量矩阵，各所述肺部文本特征向量的维度根据各文本卷积层的不同而不同，所述文本识别结果为所述肺部文本识别模型根据提取的所述肺部文本特征进行文本空间的相似性识别出肺部特征的结果。
在一实施例中，如图3所示，所述步骤S30中，即所述通过所述肺部图像识别模型对所述待识别肺部图像进行肺部图像特征提取，生成肺部图像特征向量和图像识别结果，包括：
S301，通过所述肺部图像识别模型将所述待识别肺部图像拆分成红色通道图像、绿色通道图像和蓝色通道图像；所述肺部图像识别模型为基于VGG19构建的网络模型。
可理解地，所述待识别肺部图像为红色通道、绿色通道和蓝色通道等三个通道的图像，即所述待识别肺部图像包括与红色通道对应的所述红色通道图像、与绿色通道对应的所述绿色通道图像和与蓝色通道对应的所述蓝色通道图像，通过通道拆分，将所述待识别肺部图像拆分成所述红色通道图像、所述绿色通道图像和所述蓝色通道图像，所述红色通道图像为通过0至255范围的像素值体现各个像素点的红色程度的图像，所述绿色通道图像为通过0至255范围的像素值体现各个像素点的绿色程度的图像，所述蓝色通道图像为通过0至255范围的像素值体现各个像素点的蓝色程度的图像。
其中，所述肺部图像识别模型为基于VGG19构建的网络模型，并可将所述肺部图像的卷积深度设置为19，即具有19层级的卷积层的网络模型。
S302，通过所述肺部图像识别模型分别对所述红色通道图像、所述绿色通道图像和所述蓝色通道图像进行卷积提取，得到与所述红色通道图像对应的红色特征向量、与所述绿色通道图像对应的绿色特征向量和与所述蓝色通道图像对应的蓝色特征向量。
可理解地，通过所述肺部图像识别模型对所述红色通道图像进行卷积，得到所述红色特征向量，所述红色特征向量为提取所述肺部图像特征中红色空间体现的向量，通过所述肺部图像识别模型对所述绿色通道图像进行卷积，得到所述绿色特征向量，所述绿色特征向量为提取所述肺部图像特征中绿色空间体现的向量，通过所述肺部图像识别模型对所述蓝色通道图像进行卷积，得到所述蓝色特征向量，所述蓝色特征向量为提取所述肺部图像特征中蓝色空间体现的向量，将所述红色特征向量、所述绿色特征向量和所述蓝色特征向量确定为所述肺部图像特征向量。
S303，通过所述肺部图像识别模型对所述肺部图像特征向量进行图像识别，得到所述图像识别结果。
可理解地，通过所述肺部图像识别模型对所述肺部图像特征向量进行图像识别，所述图像识别为根据提取的所述肺部图像特征向量进行全连接分类，获得各肺部特征类别的概率分布，从而输出识别到的所述图像识别结果。
本申请实现了通过所述肺部图像识别模型将所述待识别肺部图像拆分成红色通道图像、绿色通道图像和蓝色通道图像；所述肺部图像识别模型为基于VGG19构建的网络模型；通过所述肺部图像识别模型分别对所述红色通道图像、所述绿色通道图像和所述蓝色通道图像进行卷积提取，得到肺部图像特征向量；通过所述肺部图像识别模型对所述肺部图像特征向量进行图像识别，得到所述图像识别结果，如此，实现了通过将所述待识别肺部图像拆分成红色通道图像、绿色通道图像和蓝色通道图像，并基于VGG19构建的网络模型对各通道图像进行卷积提取肺部图像特征，得到肺部图像特征向量，并根据肺部图像特征向量输出图像识别结果，能够提取所述待识别肺部图像中的肺部图像特征，通过提取到的肺部图像特征识别出肺部特征类别，为后续识别提供了数据基础，提高了识别的准确 率和可靠性。
在一实施例中，如图4所示，所述步骤S30中，即所述通过所述肺部文本识别模型对所述待识别肺部文本描述进行肺部文本特征提取，生成肺部文本特征向量和文本识别结果，包括：
S304，通过所述肺部文本识别模型对所述待识别肺部文本描述进行分词，并构建与所述待识别肺部文本描述对应的文本词向量，所述肺部文本识别模型为基于TextCNN构建的网络模型。
可理解地，所述分词为运用词语字典，将所述待识别肺部文本描述拆分成单个的词语，所述词语字典为包含有与所有医学术语及与肺部相关的词语对应的词向量，然后将拆分成的单个的词语转换成与其对应的词向量，可通过word2vec或Glove的转换方式进行转换，再将转换后的词向量进行拼接，拼接成所述文本词向量。
其中，所述肺部文本识别模型为基于TextCNN构建的网络模型，即所述肺部文本识别模型具有TextCNN的网络结构，将所述肺部文本识别模型的卷积深度设置为19，即具有19层级的卷积层的网络模型，所述肺部文本识别模型的卷积深度与所述肺部图像识别模型的卷积深度相同，以便后续肺部融合识别模型的识别。
S305，将所述文本词向量进行通道扩充，生成第一文本词向量、第二文本词向量和第三文本词向量。
可理解地，所述通道扩充为根据单个通道的所述文本词向量扩充至预设维度的向量矩阵并复制向量矩阵直至预设通道个数的过程，即将所述文本词向量扩充至与所述肺部图像特征向量相同维度的向量矩阵，扩充的方式可以根据需求设定，将该向量矩阵复制成与所述肺部图像特征向量相同通道数的向量矩阵，从而得到与该向量矩阵相同的所述第一文本词向量、所述第二文本词向量和所述第三文本词向量。
S306，通过所述肺部文本识别模型分别对所述第一文本词向量、所述第二文本词向量和所述第三文本词向量进行卷积提取，得到与所述第一文本词向量对应的第一文本特征向量、与所述第二文本词向量对应的第二文本特征向量和与所述第三文本词向量对应的第三文本特征向量。
可理解地，通过所述肺部文本识别模型对所述第一文本词向量进行卷积，得到所述第一文本特征向量，通过所述肺部文本识别模型对所述第二文本词向量进行卷积，得到所述第二文本特征向量，通过所述肺部文本识别模型对所述第三文本词向量进行卷积，得到所述第三文本特征向量，其中，对所述第一文本词向量进行卷积的卷积核、对所述第二文本词向量进行卷积的卷积核和对所述第三文本词向量进行卷积的卷积核可以不相同，即从不同的文本空间的维度提取肺部文本特征向量，将所述第一文本词向量、所述第二文本词向量和所述第三文本词向量确定为所述肺部文本特征向量。
S307，通过所述肺部文本识别模型对所述肺部文本特征向量进行文本识别，得到所述文本识别结果。
可理解地，通过所述肺部文本识别模型对所述肺部文本特征向量进行文本识别，所述文本识别为根据提取的所述肺部文本特征向量进行全连接分类，获得各肺部特征类别的概率分布，从而输出识别到的所述文本识别结果。本申请实现了通过所述肺部文本识别模型对所述待识别肺部文本描述进行分词，并构建与所述待识别肺部文本描述对应的文本词向量；所述肺部文本识别模型为基于TextCNN构建的网络模型；将所述文本词向量进行通道扩充，生成第一文本词向量、第二文本词向量和第三文本词向量；通过所述肺部文本识别模型分别对所述第一文本词向量、所述第二文本词向量和所述第三文本词向量进行卷积提取，得到肺部文本特征向量，通过所述肺部文本识别模型对所述肺部文本特征向量进行文本识别，得到所述文本识别结果，如此，实现了通过将所述待识别肺部文本描述进行分词及构建文本词向量，并通道扩充生成第一文本词向量、第二文本词向量和第三文本词向量， 通过基于TextCNN构建的网络模型提取肺部文本特征，得到肺部文本特征向量，并根据肺部文本特征向量输出文本识别结果，能够提取所述待识别肺部文本描述中的肺部文本特征，通过提取到的肺部文本特征识别出肺部特征类别，为后续识别提供了数据基础，提高了识别的准确率和可靠性。
S40，通过所述肺部融合识别模型使用注意力机制融合所述肺部图像特征向量和所述肺部文本特征向量，并对融合后的特征进行提取以及识别，得到融合识别结果。
可理解地，所述注意力机制为通过注意力权重在神经网络学习及识别中一个额外的前馈神经网络来学习的机制，通过所述注意力机制能够挖掘出所述肺部图像特征向量和所述肺部文本特征向量之间的隐含关系，即将所述肺部图像特征向量和所述肺部文本特征向量按照通过所述注意力机制学习到的与各卷积层对应的权重参数进行加权融合，从而得到与各卷积层对应的所述融合特征向量，对所有所述融合特征向量进行卷积，提取出所述图像文本融合特征，即对融合后的特征进行提取，所述图像文本融合特征为所述肺部图像特征向量和所述肺部文本特征向量之间关联的隐含特征，也即所述肺部图像特征向量和所述肺部文本特征向量之间的全局相似性特征，根据提取到的所述图像文本融合特征进行识别，即进行全连接分类出各肺部特征类别的概率分布，从而输出所述融合识别结果。
在一实施例中，如图5所示，所述步骤S40中，即所述通过所述肺部融合识别模型使用注意力机制，融合所述肺部图像特征向量和所述肺部文本特征向量，并提取图像文本融合特征进行识别，得到融合识别结果，包括：
S401，运用注意力机制技术，通过与所述肺部融合识别模型中的各卷积层对应的权重参数，将所述肺部图像特征向量以及所述肺部文本特征向量进行加权融合，得到与各卷积层对应的融合特征向量。
可理解地，所述注意力机制技术为增强特征向量中有用的信息，即对所述肺部图像特征向量和所述肺部文本特征向量中有用的向量按照的各卷积层对应的权重参数进行加权平均，并融合生成与各卷积层对应的融合特征向量。
其中，所述肺部融合识别模型中的卷积深度与所述肺部图像识别模型或所述肺部文本识别模型的卷积深度相同，所述肺部融合识别模型中的卷积深度优选为19层级。
在一实施例中，所述步骤S401中，所述肺部图像特征向量包括红色特征向量、绿色特征向量和蓝色特征向量；所述肺部文本特征向量包括第一文本特征向量、第二文本特征向量和第三文本特征向量；
所述肺部图像识别模型、所述肺部文字识别模型和所述肺部融合识别模型均具有相同的卷积层级，并且三个模型中均设有与每一个卷积层级对应的卷积层；
所述通过所述肺部融合识别模型中的与各卷积层对应的权重参数将所述肺部图像特征向量和所述肺部文本特征向量进行加权融合，得到与各卷积层对应的融合特征向量，包括：
S4011，将与相同卷积层级对应的所述红色特征向量和所述第一文本特征向量按照与该卷积层级对应的第一权重参数进行融合，得到第一融合特征向量。
可理解地，将相同卷积层级对应的所述红色特征向量和所述第一文本特征向量按照与该卷积层级第一权重参数进行加权，即将所述红色特征向量和所述第一文本特征向量中的各向量值按照所述第一权重参数进行加权平均，得到所述第一融合特征向量，所述红色特征向量、所述第一文本特征向量和所述第一融合特征向量的维度相同。
S4012，将与相同卷积层级对应的所述绿色特征向量和所述第二文本特征向量按照与该卷积层级对应的第二权重参数进行融合，得到第二融合特征向量。
可理解地，将相同卷积层级对应的所述绿色特征向量和所述第二文本特征向量按照与该卷积层级第二权重参数进行加权，即将所述绿色特征向量和所述第二文本特征向量中的各向量值按照所述第二权重参数进行加权平均，得到所述第二融合特征向量，所述绿色特 征向量、所述第二文本特征向量和所述第二融合特征向量的维度相同。
S4013，将与相同卷积层级对应的所述蓝色特征向量和所述第三文本特征向量按照与该卷积层级对应的第三权重参数进行融合，得到第三融合特征向量。
可理解地，将相同卷积层级对应的所述蓝色特征向量和所述第三文本特征向量按照与该卷积层级第三权重参数进行加权，即将所述蓝色特征向量和所述第三文本特征向量中的各向量值按照所述第三权重参数进行加权平均，得到所述第三融合特征向量，所述蓝色特征向量、所述第三文本特征向量和所述第三融合特征向量的维度相同。
其中，所述步骤S4011、S4012和S4013的执行顺序不做限制，可以串行执行也可以并行执行，所述第一权重参数、所述第二权重参数和所述第三权重参数可以相同，也可以均不相同。
S4014，将与相同卷积层级对应的所述第一融合特征向量、所述第二融合特征向量和所述第三融合特征向量进行加权平均，得到所述融合特征向量。
可理解地，所述加权平均为将所述第一融合特征向量、所述第二融合特征向量和所述第三融合特征向量进行加权之后取平均值，将与相同卷积层级对应的所述第一融合特征向量、所述第二融合特征向量和所述第三融合特征向量进行加权平均从而得到与各卷积层对应的所述融合特征向量。
S402，通过所述肺部融合识别模型对所述融合特征向量进行所述图像文本融合特征的提取。
可理解地，所述图像文本融合特征的提取过程可以为将第一层的卷积层的融合特征向量进行卷积，再与该卷积层的下一层的卷积层的融合特征向量进行叠加得到中转特征向量，再对该中转特征向量进行卷积，不断与下一层的卷积层的融合特征向量进行叠加得到中转特征向量，对叠加后的中转特征向量进行卷积直至得到一维的特征向量的提取过程。
S403，通过所述肺部融合识别模型根据提取的所述图像文本融合特征进行识别，获得所述融合识别结果。
可理解地，通过所述肺部融合识别模型根据提取的所述图像文本融合特征进行识别，所述识别为根据提取的所述图像文本融合特征，获得各肺部特征类别的概率分布，从而输出识别到的所述融合识别结果。
本申请实现了通过运用注意力机制技术，通过与所述肺部融合识别模型中的各卷积层对应的权重参数，将所述肺部图像特征向量以及所述肺部文本特征向量进行加权融合，得到与各卷积层对应的融合特征向量；通过所述肺部融合识别模型对所述融合特征向量进行所述图像文本融合特征的提取；通过所述肺部融合识别模型根据提取的所述图像文本融合特征进行识别，获得所述融合识别结果，如此，实现了运用注意力机制能够增强图像和文本中有用的信息，并且能够更好捕捉到图像和文本之间的全局相似性，将所述肺部图像特征向量以及所述肺部文本特征向量进行加权融合，并提取图像文本融合特征进行识别，能够提高肺部特征识别的准确率，以及可靠性。
S50，通过所述肺部特征识别模型对所述图像识别结果、所述文本识别结果和所述融合识别结果进行投票表决，获得与所述待识别数据对应的肺部特征识别结果；所述肺部特征识别结果表明了所述待识别数据的肺部特征类别。
可理解地，所述投票表决为对所述图像识别结果、所述文本识别结果和所述融合识别结果中的与相同的肺部特征类别对应的概率值进行加权平均，最终确定出概率值最高的肺部特征类别，将该概率值最高的肺部特征类别作为所述肺部特征识别结果，所述肺部特征识别结果包括识别出的肺部特征类别和与该类别对应的概率值，所述肺部特征识别结果表明了所述待识别数据的肺部特征类别，所述肺部特征为肺部组织运动体现的特征，比如，肺部特征包括胸膜凹陷特征、气支气管特征、肺空泡特征、肺毛刺特征、肺部毛玻璃样特征等等，所述肺部特征类别为对所述肺部特征的分类，比如，所述肺部特征类别包括与胸 膜凹陷特征对应的胸膜凹陷特征类、与气支气管特征对应的气支气管特征类、与肺空泡特征对应的肺空泡特征类、与肺毛刺特征对应的肺毛刺特征类和与肺部毛玻璃样特征对应的肺部毛玻璃样特征。
本申请实现了通过获取所述识别请求中的待识别数据；所述待识别数据包括待识别肺部图像和待识别肺部文本描述；将所述待识别数据输入至包含有肺部图像识别模型、肺部文本识别模型和肺部融合识别模型的肺部特征识别模型；通过所述肺部图像识别模型对所述待识别肺部图像进行肺部图像特征提取，生成肺部图像特征向量和图像识别结果，同时通过所述肺部文本识别模型对所述待识别肺部文本描述进行肺部文本特征提取，生成肺部文本特征向量和文本识别结果；通过所述肺部融合识别模型使用注意力机制融合所述肺部图像特征向量和所述肺部文本特征向量，并提取图像文本融合特征进行识别，得到融合识别结果；通过所述肺部特征识别模型对所述图像识别结果、所述文本识别结果和所述融合识别结果进行投票表决，获得与所述待识别数据对应的肺部特征识别结果，如此，实现了通过肺部图像识别模型识别待识别肺部图像，得到图像识别结果，通过肺部文本识别模型识别待识别肺部文本描述，得到文本识别结果，再结合待识别肺部图像和待识别文本描述，运用注意力机制，通过肺部融合识别模型提取图像文本融合特征进行识别，得到融合识别结果，最后根据所述图像识别结果、所述文本识别结果和所述融合识别结果进行投票表决，得出肺部特征识别结果，实现了结合待识别肺部图像和待识别肺部文本描述，通过基于多模态模型的肺部特征识别模型自动地、快速地、准确地识别出肺部特征，提高了识别准确率和可靠性，提升了识别效率。
在一实施例中，如图6所示，所述步骤S50中，即所述通过所述肺部特征识别模型对所述图像识别结果、所述文本识别结果和所述融合识别结果进行投票表决，获得与所述待识别数据对应的肺部特征识别结果，包括：
S501，获取与所述肺部融合识别模型中的最后一层所述卷积层对应的权重参数。
可理解地，与所述肺部融合识别模型中的最后一层所述卷积层对应的权重参数包括提供给与最后一层卷积层对应的所述肺部图像特征向量的图像权重和所述肺部文本特征向量的文本权重。
S502，根据获取的所述权重参数，确定出投票表决参数。
可理解地，将获取的所述图像权重和所述文本权重保持不变，将所述图像权重作为所述图像识别结果的投票表决参数，将所述文本权重作为所述文办识别结果的投票表决参数，将数值一作为所述融合识别结果的投票表决参数。
S503，按照所述投票表决参数，对所述图像识别结果、所述文本识别结果和所述融合识别结果进行所述投票表决，获得所述肺部特征识别结果。
可理解地，根据所述图像识别结果的投票表决参数、所述图像识别结果、所述文办识别结果的投票表决参数、所述文本识别结果、所述融合识别结果的投票表决参数和所述融合识别，通过加权平均，得到各肺部特征类别的最终概率分布，将概率值最高的肺部特征类别确定为所述肺部特征识别结果。
本申请实现了通过获取与所述肺部融合识别模型中的最后一层所述卷积层对应的权重参数；根据获取的所述权重参数，确定出投票表决参数；按照所述投票表决参数，对所述图像识别结果、所述文本识别结果和所述融合识别结果进行所述投票表决，获得所述肺部特征识别结果，如此，通过对所述图像识别结果、所述文本识别结果和所述融合识别结果进行客观地投票表决，最终识别出肺部特征类别，提高了肺部特征识别的准确率和可靠性。
在一实施例中，提供一种肺部特征识别装置，该肺部特征识别装置与上述实施例中肺部特征识别方法一一对应。如图7所示，该肺部特征识别装置包括接收模块11、输入模块12、第一识别模块13、第二识别模块14和表决模块15。各功能模块详细说明如下：
接收模块11，用于接收到识别请求，获取所述识别请求中的待识别数据；所述待识别数据包括待识别肺部图像和待识别肺部文本描述；所述待识别肺部文本描述为针对所述待识别肺部图像中的肺部特征的描述；
输入模块12，用于将所述待识别数据输入至肺部特征识别模型；所述肺部特征识别模型包括肺部图像识别模型、肺部文本识别模型和肺部融合识别模型；
第一识别模块13，用于通过所述肺部图像识别模型对所述待识别肺部图像进行肺部图像特征提取，生成肺部图像特征向量和图像识别结果，同时通过所述肺部文本识别模型对所述待识别肺部文本描述进行肺部文本特征提取，生成肺部文本特征向量和文本识别结果；
第二识别模块14，用于通过所述肺部融合识别模型使用注意力机制融合所述肺部图像特征向量和所述肺部文本特征向量，并提取图像文本融合特征进行识别，得到融合识别结果；
表决模块15，用于通过所述肺部特征识别模型对所述图像识别结果、所述文本识别结果和所述融合识别结果进行投票表决，获得与所述待识别数据对应的肺部特征识别结果；所述肺部特征识别结果表明了所述待识别数据的肺部特征类别。
关于肺部特征识别装置的具体限定可以参见上文中对于肺部特征识别方法的限定，在此不再赘述。上述肺部特征识别装置中的各个模块可全部或部分通过软件、硬件及其组合来实现。上述各模块可以硬件形式内嵌于或独立于计算机设备中的处理器中，也可以以软件形式存储于计算机设备中的存储器中，以便于处理器调用执行以上各个模块对应的操作。
在一个实施例中，提供了一种计算机设备，该计算机设备可以是服务器，其内部结构图可以如图8所示。该计算机设备包括通过***总线连接的处理器、存储器、网络接口和数据库。其中，该计算机设备的处理器用于提供计算和控制能力。该计算机设备的存储器包括可读存储介质、内存储器。该可读存储介质存储有操作***、计算机可读指令和数据库。该内存储器为可读存储介质中的操作***和计算机可读指令的运行提供环境。该计算机设备的网络接口用于与外部的终端通过网络连接通信。该计算机可读指令被处理器执行时以实现一种肺部特征识别方法。本实施例所提供的可读存储介质包括非易失性可读存储介质和易失性可读存储介质。
在一个实施例中，提供了一种计算机设备，包括存储器、处理器及存储在存储器上并可在处理器上运行的计算机可读指令，处理器执行计算机可读指令时实现上述实施例中肺部特征识别方法。
在一个实施例中，提供了一个或多个存储有计算机可读指令的可读存储介质，本实施例所提供的可读存储介质包括非易失性可读存储介质和易失性可读存储介质；该可读存储介质上存储有计算机可读指令，该计算机可读指令被一个或多个处理器执行时，使得一个或多个处理器实现上述实施例中肺部特征识别方法。
本领域普通技术人员可以理解实现上述实施例方法中的全部或部分流程，是可以通过计算机可读指令来指令相关的硬件来完成，所述的计算机可读指令可存储于一非易失性计算机可读取存储介质或易失性可读存储介质中，该计算机可读指令在执行时，可包括如上述各方法的实施例的流程。其中，本申请所提供的各实施例中所使用的对存储器、存储、数据库或其它介质的任何引用，均可包括非易失性和/或易失性存储器。非易失性存储器可包括只读存储器(ROM)、可编程ROM(PROM)、电可编程ROM(EPROM)、电可擦除可编程ROM(EEPROM)或闪存。易失性存储器可包括随机存取存储器(RAM)或者外部高速缓冲存储器。作为说明而非局限，RAM以多种形式可得，诸如静态RAM(SRAM)、动态RAM(DRAM)、同步DRAM(SDRAM)、双数据率SDRAM(DDRSDRAM)、增强型SDRAM(ESDRAM)、同步链路(Synchlink)DRAM(SLDRAM)、存储器总线(Rambus)直接RAM(RDRAM)、直接存储器总线动态RAM(DRDRAM)、以及存储器总线动态RAM(RDRAM)等。
所属领域的技术人员可以清楚地了解到，为了描述的方便和简洁，仅以上述各功能单元、模块的划分进行举例说明，实际应用中，可以根据需要而将上述功能分配由不同的功能单元、模块完成，即将所述装置的内部结构划分成不同的功能单元或模块，以完成以上描述的全部或者部分功能。
以上所述实施例仅用以说明本申请的技术方案，而非对其限制；尽管参照前述实施例对本申请进行了详细的说明，本领域的普通技术人员应当理解：其依然可以对前述各实施例所记载的技术方案进行修改，或者对其中部分技术特征进行等同替换；而这些修改或者替换，并不使相应技术方案的本质脱离本申请各实施例技术方案的精神和范围，均应包含在本申请的保护范围之内。

Claims (20)

  1. 一种肺部特征识别方法，其中，包括：
    获取待识别数据，其中，所述待识别数据包括待识别肺部图像和待识别肺部文本描述；将所述待识别数据输入至肺部特征识别模型，所述肺部特征识别模型包括肺部图像识别模型、肺部文本识别模型和肺部融合识别模型；
    通过所述肺部图像识别模型对所述待识别肺部图像进行肺部图像特征提取，生成肺部图像特征向量和图像识别结果，同时通过所述肺部文本识别模型对所述待识别肺部文本描述进行肺部文本特征提取，生成肺部文本特征向量和文本识别结果；
    通过所述肺部融合识别模型使用注意力机制融合所述肺部图像特征向量和所述肺部文本特征向量，并对融合后的特征进行提取以及识别，得到融合识别结果；
    通过所述肺部特征识别模型对所述图像识别结果、所述文本识别结果和所述融合识别结果进行投票表决，获得与所述待识别数据对应的肺部特征识别结果；所述肺部特征识别结果表明了所述待识别数据的肺部特征类别。
  2. 如权利要求1所述的肺部特征识别方法，其中，所述通过所述肺部图像识别模型对所述待识别肺部图像进行肺部图像特征提取，生成肺部图像特征向量和图像识别结果，包括：
    通过所述肺部图像识别模型将所述待识别肺部图像拆分成红色通道图像、绿色通道图像和蓝色通道图像，所述肺部图像识别模型为基于VGG19构建的网络模型；
    通过所述肺部图像识别模型分别对所述红色通道图像、所述绿色通道图像和所述蓝色通道图像进行卷积提取，得到与所述红色通道图像对应的红色特征向量、与所述绿色通道图像对应的绿色特征向量和与所述蓝色通道图像对应的蓝色特征向量；
    通过所述肺部图像识别模型对所述红色特征向量、绿色特征向量和蓝色特征向量进行图像识别，得到所述图像识别结果。
  3. 如权利要求1所述的肺部特征识别方法，其中，所述通过所述肺部文本识别模型对所述待识别肺部文本描述进行肺部文本特征提取，生成肺部文本特征向量和文本识别结果，包括：
    通过所述肺部文本识别模型对所述待识别肺部文本描述进行分词，并构建与所述待识别肺部文本描述对应的文本词向量，所述肺部文本识别模型为基于TextCNN构建的网络模型；
    将所述文本词向量进行通道扩充，生成第一文本词向量、第二文本词向量和第三文本词向量；
    通过所述肺部文本识别模型分别对所述第一文本词向量、所述第二文本词向量和所述第三文本词向量进行卷积提取，得到与所述第一文本词向量对应的第一文本特征向量、与所述第二文本词向量对应的第二文本特征向量和与所述第三文本词向量对应的第三文本特征向量；
    通过所述肺部文本识别模型对所述第一文本特征向量、第二文本特征向量和第三文本特征向量进行文本识别，得到所述文本识别结果。
  4. 如权利要求1所述的肺部特征识别方法，其中，所述通过所述肺部融合识别模型使用注意力机制，融合所述肺部图像特征向量和所述肺部文本特征向量，并对融合后的特征进行提取以及识别，得到融合识别结果，包括：
    运用注意力机制技术，通过与所述肺部融合识别模型中的各卷积层对应的权重参数，将所述肺部图像特征向量以及所述肺部文本特征向量进行加权融合，得到与各卷积层对应的融合特征向量；
    通过所述肺部融合识别模型对所述融合特征向量进行所述图像文本融合特征的提取；
    通过所述肺部融合识别模型根据提取的所述图像文本融合特征进行识别，获得所述融合识别结果。
  5. 如权利要求4所述的肺部特征识别方法，其中，所述肺部图像特征向量包括红色特征向量、绿色特征向量和蓝色特征向量；所述肺部文本特征向量包括第一文本特征向量、第二文本特征向量和第三文本特征向量；
    所述肺部图像识别模型、所述肺部文字识别模型和所述肺部融合识别模型均具有相同的卷积层级，并且三个模型中均设有与每一个卷积层级对应的卷积层；
    所述通过所述肺部融合识别模型中的与各卷积层对应的权重参数将所述肺部图像特征向量和所述肺部文本特征向量进行加权融合，得到与各卷积层对应的融合特征向量，包括：
    将与相同卷积层级对应的所述红色特征向量和所述第一文本特征向量按照与该卷积层级对应的第一权重参数进行融合，得到第一融合特征向量；
    将与相同卷积层级对应的所述绿色特征向量和所述第二文本特征向量按照与该卷积层级对应的第二权重参数进行融合，得到第二融合特征向量；
    将与相同卷积层级对应的所述蓝色特征向量和所述第三文本特征向量按照与该卷积层级对应的第三权重参数进行融合，得到第三融合特征向量；
    将与相同卷积层级对应的所述第一融合特征向量、所述第二融合特征向量和所述第三融合特征向量进行加权平均，得到所述融合特征向量。
  6. 如权利要求4所述的肺部特征识别方法，其中，所述通过所述肺部特征识别模型对所述图像识别结果、所述文本识别结果和所述融合识别结果进行投票表决，获得与所述待识别数据对应的肺部特征识别结果，包括：
    获取与所述肺部融合识别模型中的最后一层所述卷积层对应的权重参数；
    根据获取的所述权重参数，确定出投票表决参数；
    按照所述投票表决参数，对所述图像识别结果、所述文本识别结果和所述融合识别结果进行所述投票表决，获得所述肺部特征识别结果。
  7. 如权利要求1所述的肺部特征识别方法，其中，所述将所述待识别数据输入至肺部特征识别模型之前，包括：
    获取肺部样本集，所述肺部样本集包括多个肺部样本，所述肺部样本包括肺部影像和与所述肺部影像关联的肺部文本描述，所述肺部样本与一个肺部特征类别标签关联；
    将所述肺部样本输入含有初始参数的多模态模型；所述多模态模型包括肺部样本图像识别模型、肺部样本文本识别模型和肺部样本融合识别模型；
    通过所述肺部样本图像识别模型对所述肺部影像进行所述肺部图像特征提取，生成肺部样本图像特征向量和图像样本识别结果，同时通过所述肺部样本文本识别模型对所述肺部文本描述进行所述肺部文本特征提取，生成肺部样本文本特征向量和文本样本识别结果；
    通过所述肺部样本融合识别模型使用注意力机制融合所述肺部样本图像特征向量和所述肺部样本文本特征向量，并学习提取所述图像文本融合特征以及识别，得到融合样本识别结果；
    对所述图像样本识别结果、所述文本样本识别结果和所述融合样本识别结果进行投票表决，得到样本识别结果；
    根据所述样本识别结果和所述肺部特征类别标签，确定出损失值；
    在所述损失值未达到预设的收敛条件时，迭代更新所述多模态模型的初始参数，直至所述损失值达到所述预设的收敛条件时，将收敛之后的所述多模态模型记录为肺部特征识别模型。
  8. 一种肺部特征识别装置，其中，包括：
    接收模块，用于获取待识别数据，其中，所述待识别数据包括待识别肺部图像和待识 别肺部文本描述；
    输入模块，用于将所述待识别数据输入至肺部特征识别模型，所述肺部特征识别模型包括肺部图像识别模型、肺部文本识别模型和肺部融合识别模型；
    第一识别模块，用于通过所述肺部图像识别模型对所述待识别肺部图像进行肺部图像特征提取，生成肺部图像特征向量和图像识别结果，同时通过所述肺部文本识别模型对所述待识别肺部文本描述进行肺部文本特征提取，生成肺部文本特征向量和文本识别结果；
    第二识别模块，用于通过所述肺部融合识别模型使用注意力机制融合所述肺部图像特征向量和所述肺部文本特征向量，并对融合后的特征进行提取以及识别，得到融合识别结果；
    表决模块，用于通过所述肺部特征识别模型对所述图像识别结果、所述文本识别结果和所述融合识别结果进行投票表决，获得与所述待识别数据对应的肺部特征识别结果；所述肺部特征识别结果表明了所述待识别数据的肺部特征类别。
  9. 一种计算机设备，包括存储器、处理器以及存储在所述存储器中并可在所述处理器上运行的计算机可读指令，其中，所述处理器执行所述计算机可读指令时实现如下步骤：
    获取待识别数据，其中，所述待识别数据包括待识别肺部图像和待识别肺部文本描述；将所述待识别数据输入至肺部特征识别模型，所述肺部特征识别模型包括肺部图像识别模型、肺部文本识别模型和肺部融合识别模型；
    通过所述肺部图像识别模型对所述待识别肺部图像进行肺部图像特征提取，生成肺部图像特征向量和图像识别结果，同时通过所述肺部文本识别模型对所述待识别肺部文本描述进行肺部文本特征提取，生成肺部文本特征向量和文本识别结果；
    通过所述肺部融合识别模型使用注意力机制融合所述肺部图像特征向量和所述肺部文本特征向量，并对融合后的特征进行提取以及识别，得到融合识别结果；
    通过所述肺部特征识别模型对所述图像识别结果、所述文本识别结果和所述融合识别结果进行投票表决，获得与所述待识别数据对应的肺部特征识别结果；所述肺部特征识别结果表明了所述待识别数据的肺部特征类别。
  10. 如权利要求9所述的计算机设备，其中，所述通过所述肺部图像识别模型对所述待识别肺部图像进行肺部图像特征提取，生成肺部图像特征向量和图像识别结果，包括：
    通过所述肺部图像识别模型将所述待识别肺部图像拆分成红色通道图像、绿色通道图像和蓝色通道图像，所述肺部图像识别模型为基于VGG19构建的网络模型；
    通过所述肺部图像识别模型分别对所述红色通道图像、所述绿色通道图像和所述蓝色通道图像进行卷积提取，得到与所述红色通道图像对应的红色特征向量、与所述绿色通道图像对应的绿色特征向量和与所述蓝色通道图像对应的蓝色特征向量；
    通过所述肺部图像识别模型对所述红色特征向量、绿色特征向量和蓝色特征向量进行图像识别，得到所述图像识别结果。
  11. 如权利要求9所述的计算机设备，其中，所述通过所述肺部文本识别模型对所述待识别肺部文本描述进行肺部文本特征提取，生成肺部文本特征向量和文本识别结果，包括：
    通过所述肺部文本识别模型对所述待识别肺部文本描述进行分词，并构建与所述待识别肺部文本描述对应的文本词向量，所述肺部文本识别模型为基于TextCNN构建的网络模型；
    将所述文本词向量进行通道扩充，生成第一文本词向量、第二文本词向量和第三文本词向量；
    通过所述肺部文本识别模型分别对所述第一文本词向量、所述第二文本词向量和所述第三文本词向量进行卷积提取，得到与所述第一文本词向量对应的第一文本特征向量、与所述第二文本词向量对应的第二文本特征向量和与所述第三文本词向量对应的第三文本特征向量；
    通过所述肺部文本识别模型对所述第一文本特征向量、第二文本特征向量和第三文本特征向量进行文本识别，得到所述文本识别结果。
  12. 如权利要求9所述的计算机设备，其中，所述通过所述肺部融合识别模型使用注意力机制，融合所述肺部图像特征向量和所述肺部文本特征向量，并对融合后的特征进行提取以及识别，得到融合识别结果，包括：
    运用注意力机制技术，通过与所述肺部融合识别模型中的各卷积层对应的权重参数，将所述肺部图像特征向量以及所述肺部文本特征向量进行加权融合，得到与各卷积层对应的融合特征向量；
    通过所述肺部融合识别模型对所述融合特征向量进行所述图像文本融合特征的提取；
    通过所述肺部融合识别模型根据提取的所述图像文本融合特征进行识别，获得所述融合识别结果。
  13. 如权利要求11所述的计算机设备，其中，所述肺部图像特征向量包括红色特征向量、绿色特征向量和蓝色特征向量；所述肺部文本特征向量包括第一文本特征向量、第二文本特征向量和第三文本特征向量；
    所述肺部图像识别模型、所述肺部文字识别模型和所述肺部融合识别模型均具有相同的卷积层级，并且三个模型中均设有与每一个卷积层级对应的卷积层；
    所述通过所述肺部融合识别模型中的与各卷积层对应的权重参数将所述肺部图像特征向量和所述肺部文本特征向量进行加权融合，得到与各卷积层对应的融合特征向量，包括：
    将与相同卷积层级对应的所述红色特征向量和所述第一文本特征向量按照与该卷积层级对应的第一权重参数进行融合，得到第一融合特征向量；
    将与相同卷积层级对应的所述绿色特征向量和所述第二文本特征向量按照与该卷积层级对应的第二权重参数进行融合，得到第二融合特征向量；
    将与相同卷积层级对应的所述蓝色特征向量和所述第三文本特征向量按照与该卷积层级对应的第三权重参数进行融合，得到第三融合特征向量；
    将与相同卷积层级对应的所述第一融合特征向量、所述第二融合特征向量和所述第三融合特征向量进行加权平均，得到所述融合特征向量。
  14. 如权利要求11所述的计算机设备，其中，所述通过所述肺部特征识别模型对所述图像识别结果、所述文本识别结果和所述融合识别结果进行投票表决，获得与所述待识别数据对应的肺部特征识别结果，包括：
    获取与所述肺部融合识别模型中的最后一层所述卷积层对应的权重参数；
    根据获取的所述权重参数，确定出投票表决参数；
    按照所述投票表决参数，对所述图像识别结果、所述文本识别结果和所述融合识别结果进行所述投票表决，获得所述肺部特征识别结果。
  15. 一个或多个存储有计算机可读指令的可读存储介质，其中，所述计算机可读指令被一个或多个处理器执行时，使得所述一个或多个处理器执行如下步骤：
    获取待识别数据，其中，所述待识别数据包括待识别肺部图像和待识别肺部文本描述；将所述待识别数据输入至肺部特征识别模型，所述肺部特征识别模型包括肺部图像识别模型、肺部文本识别模型和肺部融合识别模型；
    通过所述肺部图像识别模型对所述待识别肺部图像进行肺部图像特征提取，生成肺部图像特征向量和图像识别结果，同时通过所述肺部文本识别模型对所述待识别肺部文本描述进行肺部文本特征提取，生成肺部文本特征向量和文本识别结果；
    通过所述肺部融合识别模型使用注意力机制融合所述肺部图像特征向量和所述肺部文本特征向量，并对融合后的特征进行提取以及识别，得到融合识别结果；
    通过所述肺部特征识别模型对所述图像识别结果、所述文本识别结果和所述融合识别 结果进行投票表决，获得与所述待识别数据对应的肺部特征识别结果；所述肺部特征识别结果表明了所述待识别数据的肺部特征类别。
  16. 如权利要求15所述的可读存储介质，其中，所述通过所述肺部图像识别模型对所述待识别肺部图像进行肺部图像特征提取，生成肺部图像特征向量和图像识别结果，包括：
    通过所述肺部图像识别模型将所述待识别肺部图像拆分成红色通道图像、绿色通道图像和蓝色通道图像，所述肺部图像识别模型为基于VGG19构建的网络模型；
    通过所述肺部图像识别模型分别对所述红色通道图像、所述绿色通道图像和所述蓝色通道图像进行卷积提取，得到与所述红色通道图像对应的红色特征向量、与所述绿色通道图像对应的绿色特征向量和与所述蓝色通道图像对应的蓝色特征向量；
    通过所述肺部图像识别模型对所述红色特征向量、绿色特征向量和蓝色特征向量进行图像识别，得到所述图像识别结果。
  17. 如权利要求15所述的可读存储介质，其中，所述通过所述肺部文本识别模型对所述待识别肺部文本描述进行肺部文本特征提取，生成肺部文本特征向量和文本识别结果，包括：
    通过所述肺部文本识别模型对所述待识别肺部文本描述进行分词，并构建与所述待识别肺部文本描述对应的文本词向量，所述肺部文本识别模型为基于TextCNN构建的网络模型；
    将所述文本词向量进行通道扩充，生成第一文本词向量、第二文本词向量和第三文本词向量；
    通过所述肺部文本识别模型分别对所述第一文本词向量、所述第二文本词向量和所述第三文本词向量进行卷积提取，得到与所述第一文本词向量对应的第一文本特征向量、与所述第二文本词向量对应的第二文本特征向量和与所述第三文本词向量对应的第三文本特征向量；
    通过所述肺部文本识别模型对所述第一文本特征向量、第二文本特征向量和第三文本特征向量进行文本识别，得到所述文本识别结果。
  18. 如权利要求15所述的可读存储介质，其中，所述通过所述肺部融合识别模型使用注意力机制，融合所述肺部图像特征向量和所述肺部文本特征向量，并对融合后的特征进行提取以及识别，得到融合识别结果，包括：
    运用注意力机制技术，通过与所述肺部融合识别模型中的各卷积层对应的权重参数，将所述肺部图像特征向量以及所述肺部文本特征向量进行加权融合，得到与各卷积层对应的融合特征向量；
    通过所述肺部融合识别模型对所述融合特征向量进行所述图像文本融合特征的提取；
    通过所述肺部融合识别模型根据提取的所述图像文本融合特征进行识别，获得所述融合识别结果。
  19. 如权利要求17所述的可读存储介质，其中，所述肺部图像特征向量包括红色特征向量、绿色特征向量和蓝色特征向量；所述肺部文本特征向量包括第一文本特征向量、第二文本特征向量和第三文本特征向量；
    所述肺部图像识别模型、所述肺部文字识别模型和所述肺部融合识别模型均具有相同的卷积层级，并且三个模型中均设有与每一个卷积层级对应的卷积层；
    所述通过所述肺部融合识别模型中的与各卷积层对应的权重参数将所述肺部图像特征向量和所述肺部文本特征向量进行加权融合，得到与各卷积层对应的融合特征向量，包括：
    将与相同卷积层级对应的所述红色特征向量和所述第一文本特征向量按照与该卷积层级对应的第一权重参数进行融合，得到第一融合特征向量；
    将与相同卷积层级对应的所述绿色特征向量和所述第二文本特征向量按照与该卷积 层级对应的第二权重参数进行融合，得到第二融合特征向量；
    将与相同卷积层级对应的所述蓝色特征向量和所述第三文本特征向量按照与该卷积层级对应的第三权重参数进行融合，得到第三融合特征向量；
    将与相同卷积层级对应的所述第一融合特征向量、所述第二融合特征向量和所述第三融合特征向量进行加权平均，得到所述融合特征向量。
  20. 如权利要求17所述的可读存储介质，其中，所述通过所述肺部特征识别模型对所述图像识别结果、所述文本识别结果和所述融合识别结果进行投票表决，获得与所述待识别数据对应的肺部特征识别结果，包括：
    获取与所述肺部融合识别模型中的最后一层所述卷积层对应的权重参数；
    根据获取的所述权重参数，确定出投票表决参数；
    按照所述投票表决参数，对所述图像识别结果、所述文本识别结果和所述融合识别结果进行所述投票表决，获得所述肺部特征识别结果。
PCT/CN2021/096366 2020-09-21 2021-05-27 肺部特征识别方法、装置、计算机设备及存储介质 WO2022057309A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202010991495.4A CN111832581B (zh) 2020-09-21 2020-09-21 肺部特征识别方法、装置、计算机设备及存储介质
CN202010991495.4 2020-09-21

Publications (1)

Publication Number Publication Date
WO2022057309A1 true WO2022057309A1 (zh) 2022-03-24

Family

ID=72918738

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/096366 WO2022057309A1 (zh) 2020-09-21 2021-05-27 肺部特征识别方法、装置、计算机设备及存储介质

Country Status (2)

Country Link
CN (1) CN111832581B (zh)
WO (1) WO2022057309A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114863182A (zh) * 2022-05-23 2022-08-05 北京百度网讯科技有限公司 图像分类方法、图像分类模型的训练方法及装置

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111832581B (zh) * 2020-09-21 2021-01-29 平安科技（深圳）有限公司 肺部特征识别方法、装置、计算机设备及存储介质
CN112070069A (zh) * 2020-11-10 2020-12-11 支付宝(杭州)信息技术有限公司 遥感图像的识别方法和装置
CN113469049B (zh) * 2021-06-30 2024-05-10 平安科技（深圳）有限公司 一种疾病信息识别方法、***、装置及存储介质
CN113409306A (zh) * 2021-07-15 2021-09-17 推想医疗科技股份有限公司 一种检测装置、训练方法、训练装置、设备和介质

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107392224A (zh) * 2017-06-12 2017-11-24 天津科技大学 一种基于三通道卷积神经网络的作物病害识别算法
CN108334605A (zh) * 2018-02-01 2018-07-27 腾讯科技（深圳）有限公司 文本分类方法、装置、计算机设备及存储介质
CN108399409A (zh) * 2018-01-19 2018-08-14 北京达佳互联信息技术有限公司 图像分类方法、装置及终端
CN109543714A (zh) * 2018-10-16 2019-03-29 北京达佳互联信息技术有限公司 数据特征的获取方法、装置、电子设备及存储介质
CN111460889A (zh) * 2020-02-27 2020-07-28 平安科技（深圳）有限公司 一种基于语音及图像特征的异常行为识别方法、装置及设备
CN111563551A (zh) * 2020-04-30 2020-08-21 支付宝(杭州)信息技术有限公司 一种多模态信息融合方法、装置及电子设备
CN111832581A (zh) * 2020-09-21 2020-10-27 平安科技（深圳）有限公司 肺部特征识别方法、装置、计算机设备及存储介质

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107665736B (zh) * 2017-09-30 2021-05-25 百度在线网络技术（北京）有限公司 用于生成信息的方法和装置
CN108334843B (zh) * 2018-02-02 2022-03-25 成都国铁电气设备有限公司 一种基于改进AlexNet的燃弧识别方法
CN110188343B (zh) * 2019-04-22 2023-01-31 浙江工业大学 基于融合注意力网络的多模态情感识别方法

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107392224A (zh) * 2017-06-12 2017-11-24 天津科技大学 一种基于三通道卷积神经网络的作物病害识别算法
CN108399409A (zh) * 2018-01-19 2018-08-14 北京达佳互联信息技术有限公司 图像分类方法、装置及终端
CN108334605A (zh) * 2018-02-01 2018-07-27 腾讯科技（深圳）有限公司 文本分类方法、装置、计算机设备及存储介质
CN109543714A (zh) * 2018-10-16 2019-03-29 北京达佳互联信息技术有限公司 数据特征的获取方法、装置、电子设备及存储介质
CN111460889A (zh) * 2020-02-27 2020-07-28 平安科技（深圳）有限公司 一种基于语音及图像特征的异常行为识别方法、装置及设备
CN111563551A (zh) * 2020-04-30 2020-08-21 支付宝(杭州)信息技术有限公司 一种多模态信息融合方法、装置及电子设备
CN111832581A (zh) * 2020-09-21 2020-10-27 平安科技（深圳）有限公司 肺部特征识别方法、装置、计算机设备及存储介质

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114863182A (zh) * 2022-05-23 2022-08-05 北京百度网讯科技有限公司 图像分类方法、图像分类模型的训练方法及装置

Also Published As

Publication number Publication date
CN111832581B (zh) 2021-01-29
CN111832581A (zh) 2020-10-27

Similar Documents

Publication Publication Date Title
WO2022057309A1 (zh) 肺部特征识别方法、装置、计算机设备及存储介质
US11348249B2 (en) Training method for image semantic segmentation model and server
WO2019228317A1 (zh) 人脸识别方法、装置及计算机可读介质
US11842487B2 (en) Detection model training method and apparatus, computer device and storage medium
US11961233B2 (en) Method and apparatus for training image segmentation model, computer device, and storage medium
WO2021135499A1 (zh) 损伤检测模型训练、车损检测方法、装置、设备及介质
WO2020228446A1 (zh) 模型训练方法、装置、终端及存储介质
EP3327582A1 (en) Method and apparatus for completing a knowledge graph
WO2019007041A1 (zh) 基于多视图联合嵌入空间的图像-文本双向检索方法
WO2022142450A1 (zh) 用于图像分割模型训练和图像分割的方法及装置
CN111950329A (zh) 目标检测及模型训练方法、装置、计算机设备和存储介质
CN110930417A (zh) 图像分割模型的训练方法和装置、图像分割方法和装置
CN111062871A (zh) 一种图像处理方法、装置、计算机设备及可读存储介质
CN111368672A (zh) 一种用于遗传病面部识别模型的构建方法及装置
WO2020238353A1 (zh) 数据处理方法和装置、存储介质及电子装置
US20230237771A1 (en) Self-supervised learning method and apparatus for image features, device, and storage medium
CN112395979A (zh) 基于图像的健康状态识别方法、装置、设备及存储介质
CN110705489B (zh) 目标识别网络的训练方法、装置、计算机设备和存储介质
WO2021032062A1 (zh) 图像处理模型生成方法、图像处理方法、装置及电子设备
WO2021031704A1 (zh) 对象追踪方法、装置、计算机设备和存储介质
CN111583184A (zh) 图像分析方法、网络、计算机设备和存储介质
CN115050064A (zh) 人脸活体检测方法、装置、设备及介质
CN113343981A (zh) 一种视觉特征增强的字符识别方法、装置和设备
CN112102235A (zh) 人体部位识别方法、计算机设备和存储介质
WO2022063076A1 (zh) 对抗样本的识别方法及装置

Legal Events

Date Code Title Description
NENP Non-entry into the national phase

Ref country code: DE

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21868146

Country of ref document: EP

Kind code of ref document: A1

122 Ep: pct application non-entry in european phase

Ref document number: 21868146

Country of ref document: EP

Kind code of ref document: A1