CN108549850B - Image identification method and electronic equipment - Google Patents

Image identification method and electronic equipment Download PDF

Info

Publication number
CN108549850B
CN108549850B CN201810260038.0A CN201810260038A CN108549850B CN 108549850 B CN108549850 B CN 108549850B CN 201810260038 A CN201810260038 A CN 201810260038A CN 108549850 B CN108549850 B CN 108549850B
Authority
CN
China
Prior art keywords
image
text information
information
decoding
coding
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201810260038.0A
Other languages
Chinese (zh)
Other versions
CN108549850A (en
Inventor
田疆
李聪
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Lenovo Beijing Ltd
Original Assignee
Lenovo Beijing Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Lenovo Beijing Ltd filed Critical Lenovo Beijing Ltd
Priority to CN201810260038.0A priority Critical patent/CN108549850B/en
Publication of CN108549850A publication Critical patent/CN108549850A/en
Application granted granted Critical
Publication of CN108549850B publication Critical patent/CN108549850B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/10—Terrestrial scenes
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/049—Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Physics & Mathematics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • General Engineering & Computer Science (AREA)
  • Evolutionary Computation (AREA)
  • Molecular Biology (AREA)
  • Computational Linguistics (AREA)
  • Software Systems (AREA)
  • Mathematical Physics (AREA)
  • Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computing Systems (AREA)
  • General Health & Medical Sciences (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Evolutionary Biology (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Multimedia (AREA)
  • Image Analysis (AREA)

Abstract

The invention discloses an image identification method, which comprises the following steps: acquiring image information and first text information; and generating second text information based on the image information and the first text information, wherein the second text information is used for representing the image information and the text information content. The invention also discloses an electronic device.

Description

Image identification method and electronic equipment
Technical Field
The present invention relates to image recognition technologies, and in particular, to an image recognition method and an electronic device.
Background
In the prior art, in the process of identifying the image, only simple judgment can be made on the composition of the image, or a person operating the image can make judgment on the image, so that the identification efficiency is low, and the identification error rate is high.
Disclosure of Invention
The embodiment of the invention provides an image identification method and electronic equipment, which can be used for coding and decoding according to visual features of an image and acquired text information while realizing image identification to obtain second text information describing and fusing the visual features and the text information.
The technical scheme of the embodiment of the invention is realized as follows:
the embodiment of the invention provides an image identification method, which comprises the following steps:
acquiring image information and first text information;
and generating second text information based on the image information and the first text information, wherein the second text information is used for representing the image information and the text information content.
In the foregoing solution, the acquiring the image information and the first text information includes:
extracting visual features from the image;
and coding at least two different types of text information of the image to obtain a coding result representing the semantics of the text information.
In the foregoing solution, the generating second text information based on the image information and the first text information includes:
decoding is carried out based on the visual features and the coding result, and second text information which fuses the visual features and the text information to describe the image is obtained.
In the above scheme, the extracting visual features from the image includes:
processing the image intersection through a convolution layer and a maximum pooling layer of a convolution neural network model to obtain a down-sampling result of the image;
and processing the down-sampling result through an average pooling layer of the convolutional neural network model to obtain the visual characteristics of the image.
In the above scheme, the method further comprises:
and processing the image visual characteristics through an average pooling layer of the convolutional neural network model to obtain the label representing the classification of the image.
In the above scheme, the encoding at least two different types of text information of the image includes:
performing word-level coding on at least two types of text information of the picture through a neural network model corresponding to different types of text information;
and carrying out sentence-level coding on the coding result of the word level.
In the above scheme, the decoding based on the visual features and the encoding result includes:
decoding the coding result at statement level through a first decoder model;
and performing word level decoding through the decoding result of the sentence level of the second decoder model.
In the above scheme, the method further comprises:
distributing corresponding weights to the visual features and the coding results through an attention model;
and inputting the first weight matrix, the second weight matrix, the coding result and the visual features into the first decoder for decoding.
In the above scheme, the method further comprises:
training a convolutional neural network model for extracting visual features from the image based on image samples and classification labels of the image samples;
training a first decoder model based on the statement samples and the corresponding decoding results;
a second decoder model is trained based on the word samples and the corresponding decoding results.
In the above-mentioned scheme, the first step of the method,
when the image is a medical image of a diseased part, the first text information comprises an indication and a clinical report of the diseased part, and the second text information comprises a diagnosis result of the diseased part.
An embodiment of the present invention further provides an electronic device, where the electronic device includes:
the information acquisition module is used for acquiring the image and the first text information;
and the information processing module is used for generating second text information based on the image information and the first text information, and the second text information is used for representing the image information and the text information.
In the above-mentioned scheme, the first step of the method,
the information acquisition module is used for extracting visual features from the image;
the information processing module is used for coding at least two different types of text information of the image to obtain a coding result representing the semantics of the text information.
In the above-mentioned scheme, the first step of the method,
and the information processing module is used for decoding based on the visual characteristics and the coding result to obtain second text information which is fused with the visual characteristics and the text information to describe the image.
In the above-mentioned scheme, the first step of the method,
the information acquisition module is used for processing the image intersection through a convolution layer and a maximum pooling layer of a convolution neural network model to obtain a down-sampling result of the image;
and the information acquisition module is used for processing the down-sampling result through an average pooling layer of the convolutional neural network model to obtain the visual characteristics of the image.
In the above-mentioned scheme, the first step of the method,
the information acquisition module is used for processing the image visual characteristics through an average pooling layer of the convolutional neural network model to obtain the classified labels representing the images.
In the above-mentioned scheme, the first step of the method,
the information processing module is used for carrying out word-level coding on at least two types of text information of the picture through the neural network models corresponding to the text information of different types;
and the information processing module is used for coding the coding result of the word level at the statement level.
In the above-mentioned scheme, the first step of the method,
the information processing module is used for decoding the coding result at a statement level through a first decoder model;
and the information processing module is used for decoding the word level according to the decoding result of the statement level of the second decoder model.
In the above-mentioned scheme, the first step of the method,
the information processing module is further used for distributing corresponding weights to the visual features and the coding results through an attention model;
the information processing module is further configured to input the first weight matrix, the second weight matrix, the encoding result, and the visual characteristic into the first decoder for decoding.
In the above solution, the electronic device further includes:
the training module is used for training a convolutional neural network model for extracting visual features from the image based on an image sample and a classification label of the image sample;
the training module is used for training a first decoder model based on the statement sample and the corresponding decoding result;
the training module is configured to train a second decoder model based on the word samples and the corresponding decoding results.
In the above-mentioned scheme, the first step of the method,
when the image is a medical image of a diseased part, the text information comprises an indication and a clinical report of the diseased part, and the second text information comprises a diagnosis result of the diseased part.
The present invention also provides an electronic device, including:
a memory for storing executable instructions;
a processor, configured to execute the executable instructions stored in the memory, to:
acquiring image information and first text information;
and generating second text information based on the image information and the first text information, wherein the second text information is used for representing the image information and the text information content.
The acquiring of the image information and the first text information includes:
extracting visual features from the image;
and coding at least two different types of text information of the image to obtain a coding result representing the semantics of the text information.
Generating second text information based on the image information and the first text information, including:
decoding is carried out based on the visual features and the coding result, and second text information which fuses the visual features and the text information to describe the image is obtained.
The visual feature extraction from the image comprises:
processing the image intersection through a convolution layer and a maximum pooling layer of a convolution neural network model to obtain a down-sampling result of the image;
and processing the down-sampling result through an average pooling layer of the convolutional neural network model to obtain the visual characteristics of the image.
The method further comprises the following steps:
and processing the image visual characteristics through an average pooling layer of the convolutional neural network model to obtain the label representing the classification of the image.
The encoding of at least two different types of textual information of the image includes:
performing word-level coding on at least two types of text information of the picture through a neural network model corresponding to different types of text information;
and carrying out sentence-level coding on the coding result of the word level.
The decoding based on the visual features and the encoding results comprises:
decoding the coding result at statement level through a first decoder model;
and performing word level decoding through the decoding result of the sentence level of the second decoder model.
The method further comprises the following steps:
distributing corresponding weights to the visual features and the coding results through an attention model;
and inputting the first weight matrix, the second weight matrix, the coding result and the visual features into the first decoder for decoding.
The method further comprises the following steps:
training a convolutional neural network model for extracting visual features from the image based on image samples and classification labels of the image samples;
training a first decoder model based on the statement samples and the corresponding decoding results;
a second decoder model is trained based on the word samples and the corresponding decoding results.
When the image is a medical image of a diseased part, the first text information comprises an indication and a clinical report of the diseased part, and the second text information comprises a diagnosis result of the diseased part.
In the embodiment of the invention, the second text information capable of representing the image information and the text information content is generated through the acquired image information and the acquired first text information, so that the automatic identification of the image is realized, and the output second text information represents the information of the image and the text information content, so that a reader of the second text information can more clearly know the image and the first text information and form visual experience.
Drawings
FIG. 1 is a schematic flow chart of an alternative image recognition method provided by an embodiment of the present invention;
FIG. 2 is an alternative schematic diagram of an electronic device according to an embodiment of the invention;
FIG. 3 is an alternative schematic diagram of an electronic device according to an embodiment of the invention;
FIG. 4 is an alternative schematic diagram of an electronic device provided by an embodiment of the invention;
FIG. 5 is a schematic diagram of processing images at a convolutional layer and a pooling layer by an Activation Function (Activation Function);
FIG. 6 is a schematic flow chart of an alternative image recognition method according to an embodiment of the present invention;
fig. 7 is an alternative structural schematic diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
The present invention will be described in further detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
Fig. 1 is a schematic flow chart of an alternative image recognition method according to an embodiment of the present invention, and as shown in fig. 1, the steps shown in the alternative flow chart of the image recognition method according to the embodiment of the present invention are described.
Step 101: image information and first text information are acquired.
Step 102: generating second text information based on the image information and the first text information;
wherein the second text information is used for representing the image information and the text information content.
In an embodiment of the present invention, the acquiring the image information and the first text information includes: extracting visual features from the image; and coding at least two different types of text information of the image to obtain a coding result representing the semantics of the text information. By the technical scheme shown in the embodiment, accurate extraction of image information and text information can be realized, specifically, the image information can be a photo or a medical image, and the first text information can be text information of at least two different information sources.
In one embodiment of the present invention, the generating of the second text information based on the image information and the first text information includes: decoding is carried out based on the visual features and the coding result, and second text information which fuses the visual features and the text information to describe the image is obtained. By the technical scheme shown in this embodiment, the visual feature and the encoding result are decoded, and second text information is obtained and used for fusing the visual feature and the text information, so that fusion of the extracted information is achieved.
In one embodiment of the present invention, the extracting visual features from the image includes: processing the image intersection through a convolution layer (alternating volumetric layer) and a pooling layer (pooling layer) of a convolution neural network model to obtain a down-sampling result of the image; and processing the down-sampling result through an average pooling layer of the convolutional neural network model to obtain the visual characteristics of the image. Through the technical scheme shown in the embodiment, through the cross processing of the convolutional layer and the maximum pooling layer of the convolutional neural network model, each small block in the neural network is deeply analyzed by the convolutional layer, so that the characteristic with higher abstraction degree is obtained, the size of the matrix can be reduced, the number of nodes in the final full-connection layer is further reduced, and the aim of reducing parameters in the whole neural network is fulfilled.
In an embodiment of the present invention, the image visual features may be further processed through an average pooling layer of the convolutional neural network model, so as to obtain the label representing the classification of the image. Through the technical scheme shown in the embodiment, the label for representing the classification of the image is obtained, so that the classification processing of a plurality of images or the classification of different features of the same image is realized.
In one embodiment of the present invention, the encoding of at least two different types of text information of the image includes: performing word-level coding on at least two types of text information of the picture through a neural network model corresponding to different types of text information; and carrying out sentence-level coding on the coding result of the word level. Through the technical scheme shown in this embodiment, word-level coding and sentence-level coding can be respectively performed on at least two types of text information of the picture through a Bi-directional long and short time memory recurrent neural network (Bi-directional LSTM RNN), wherein the same coder model can be used for the word-level coding or the sentence-level coding of the at least two types of text information of the picture.
In one embodiment of the present invention, the decoding based on the visual feature and the encoding result includes: decoding the coding result at statement level through a first decoder model; and performing word level decoding through the decoding result of the sentence level of the second decoder model. With the technical solution shown in this embodiment, when the first decoder model is a statement decoder, the second decoder model is a Long Short-Term Memory (LSTM) network.
In an embodiment of the present invention, further, the visual features and the encoding results may be assigned with corresponding weights through an attention model; and inputting the first weight matrix, the second weight matrix, the coding result and the visual features into the first decoder for decoding.
In one embodiment of the invention, a convolutional neural network model for extracting visual features from an image is trained based on an image sample and a classification label of the image sample; training a first decoder model based on the statement samples and the corresponding decoding results; a second decoder model is trained based on the word samples and the corresponding decoding results. Through the technical scheme shown in the embodiment, the neural network model and different decoders can be trained in a targeted manner.
In an embodiment of the invention, when the image is a medical image of a patient site, the text information includes an indication of the patient site and a clinical report, and the second text information includes a diagnosis result of the patient site. By the technical scheme shown in the embodiment, the medical image of the diseased part, the indication of the diseased part and the diagnosis result of the diseased part which is obtained by fusing the medical image of the diseased part and the clinical report can be output by natural language.
Fig. 2 is an optional structural schematic diagram of the electronic device according to the embodiment of the present invention, and as shown in fig. 2, an optional structural diagram of the electronic device according to the embodiment of the present invention is provided, and the following describes the modules related to fig. 2 respectively.
An information obtaining module 201, configured to obtain an image and first text information;
an information processing module 202, configured to generate second text information based on the image information and the first text information, where the second text information is used to represent the image information and the text information content.
In an embodiment of the present invention, the information obtaining module 201 is configured to extract visual features from the image; the information processing module 202 is configured to encode at least two different types of text information of the image to obtain an encoding result representing semantics of the text information. By the technical scheme shown in the embodiment, accurate extraction of image information and text information can be realized, specifically, the image information can be a photo or a medical image, and the first text information can be text information of at least two different information sources.
In an embodiment of the present invention, the information processing module 202 is configured to perform decoding based on the visual feature and the encoding result, and obtain second text information that fuses the visual feature and the text information to describe the image. By the technical scheme shown in this embodiment, the visual feature and the encoding result are decoded, and second text information is obtained and used for fusing the visual feature and the text information, so that fusion of the extracted information is achieved.
In an embodiment of the present invention, the information obtaining module 201 is configured to perform cross processing on the image through a convolutional layer and a maximum pooling layer of a convolutional neural network model to obtain a down-sampling result of the image; the information obtaining module 202 is configured to process the downsampling result through an average pooling layer of the convolutional neural network model to obtain a visual feature of the image. Through the technical scheme shown in the embodiment, through the cross processing of the convolutional layer and the maximum pooling layer of the convolutional neural network model, each small block in the neural network is deeply analyzed by the convolutional layer, so that the characteristic with higher abstraction degree is obtained, the size of the matrix can be reduced, the number of nodes in the final full-connection layer is further reduced, and the aim of reducing parameters in the whole neural network is fulfilled.
In an embodiment of the present invention, the information obtaining module 201 is configured to process the image visual features through an average pooling layer of the convolutional neural network model to obtain the labels representing the classifications of the images. Through the technical scheme shown in the embodiment, the label for representing the classification of the image is obtained, so that the classification processing of a plurality of images or the classification of different features of the same image is realized.
In an embodiment of the present invention, the information processing module 202 is configured to perform word-level encoding on at least two types of text information of the picture through a neural network model corresponding to different types of text information; the information processing module 202 is configured to perform statement-level coding on the word-level coding result. Through the technical scheme shown in this embodiment, word-level coding and sentence-level coding can be respectively performed on at least two types of text information of the picture through a Bi-directional long and short time memory recurrent neural network (Bi-directional LSTM RNN), wherein the same coder model can be used for the word-level coding or the sentence-level coding of the at least two types of text information of the picture.
In an embodiment of the present invention, the information processing module 202 is configured to perform statement level decoding on the encoding result through a first decoder model; the information processing module 202 is configured to perform word-level decoding according to a decoding result at a statement level of the second decoder model. With the technical solution shown in this embodiment, when the first decoder model is a statement decoder, the second decoder model is a Long Short-Term Memory network (LSTM Long Short-Term Memory).
In an embodiment of the present invention, the information processing module 202 is further configured to assign corresponding weights to the visual features and the encoding results through an attention model; the information processing module 202 is further configured to input the first weight matrix, the second weight matrix, the encoding result, and the visual characteristic into the first decoder for decoding.
In one embodiment of the present invention, the electronic device further includes: the training module is used for training a convolutional neural network model for extracting visual features from the image based on an image sample and a classification label of the image sample; the training module is used for training a first decoder model based on the statement sample and the corresponding decoding result; the training module is configured to train a second decoder model based on the word samples and the corresponding decoding results. Through the technical scheme shown in the embodiment, the neural network model and different decoders can be trained in a targeted manner.
In an embodiment of the invention, when the image is a medical image of a patient site, the text information includes an indication of the patient site and a clinical report, and the second text information includes a diagnosis result of the patient site. By the technical scheme shown in the embodiment, the medical image of the diseased part, the indication of the diseased part and the diagnosis result of the diseased part which is obtained by fusing the medical image of the diseased part and the clinical report can be output by natural language.
Fig. 3 is an optional structural schematic diagram of the electronic device according to the embodiment of the present invention, and as shown in fig. 3, an optional structural diagram of the electronic device according to the embodiment of the present invention is provided, and the following describes the modules related to fig. 3 respectively.
The image encoder 301 is configured to perform intersection processing on the image through a convolution layer and a maximum pooling layer of a convolutional neural network model to obtain a down-sampling result of the image; and processing the down-sampling result through an average pooling layer of the convolutional neural network model to obtain the visual characteristics of the image.
The text encoder 302 is configured to obtain the first text information and encode the obtained first text information.
A text decoder 303, configured to generate second text information based on the image visual feature and the first text information, where the second text information is used to characterize the image visual feature and the text information content. The information processing procedures of the image encoder 301, the text encoder 302 and the text decoder 303 are as shown in fig. 4.
Fig. 4 is an optional structural schematic diagram of the electronic device according to the embodiment of the present invention, and as shown in fig. 4, an optional structural diagram of the electronic device according to the embodiment of the present invention is described below with reference to the modules in fig. 4.
The first neural network 401 is configured to encode first type text information in the first text information to obtain an encoding result representing semantics of the first type text information.
A first text decoder 402, configured to perform decoding processing on the encoding result of the first neural network 401, so as to output a first type of text information in the first text information through a natural language.
The second neural network 403 is configured to encode first information in second type text information in the first text information to obtain an encoding result representing semantics of the first information in the second type text information, and specifically, the first information in the second type text information in the first text information includes at least two sentences.
A second text decoder 404, configured to perform decoding processing on the encoding result of the second neural network 403, so as to output the first information in the second type text information in the first text information through natural language.
The third neural network 405 is configured to encode second information in second type text information in the first text information to obtain an encoding result representing semantics of the second information in the second type text information, and specifically, the second information in the second type text information in the first text information includes at least two sentences.
A third text decoder 406, configured to perform a decoding process on the encoding result of the third neural network 405 to output, through a natural language, second information in the second type of text information in the first text information.
In an embodiment of the present invention, the first neural network 401, the second neural network 403, and the third neural network 405 may all use a Bi-directional long-term memory recurrent neural network (Bi-directional LSTM RNN), and the encoder models corresponding to different types of text information may be the same, that is, the first text decoder 402, the second text decoder 404, and the third text decoder 406 may be the same type of decoders.
In an embodiment of the present invention, through the word-level encoding and sentence-level encoding of the first neural network 401, the second neural network 403, and the third neural network 405, a sentence-level encoding result that fuses at least two different sources of the first text information may be obtained.
A convolutional neural network 407 for extracting visual features from the image.
In one embodiment of the present invention, the illustrated solution may support images in any format, including but not limited to JPG, PNG, TIF, BMP, etc. Of course, in order to ensure the uniformity and processing rate of image processing during implementation, when the sample image is received, the sample image may be converted into a uniform format supported by the system, and then the sample image is processed accordingly. Of course, in order to adapt to the processing performance of the system, the sample images with different sizes may be cut into fixed-size images supported by the system, and then the images are processed accordingly.
In one embodiment of the present invention, the extracting visual features from the image includes: processing the image intersection through a convolution layer and a maximum pooling layer of a convolution neural network model to obtain a down-sampling result of the image; and processing the down-sampling result through an average pooling layer of the convolutional neural network model to obtain the visual characteristics of the image. In an alternative configuration of the electronic device shown in fig. 4, when the medical image of the patient site is identified by the convolutional neural network, the first text information includes an indication of the patient site and a clinical report, and the clinical report includes at least two statements.
In an embodiment of the present invention, the image visual features may be further processed through an average pooling layer of the convolutional neural network model to obtain the labels representing the classifications of the images, and according to the technical solution shown in this embodiment, the labels representing the classifications of the images may be obtained, so as to implement classification processing on multiple images or classification on different features of the same image.
In one embodiment of the present invention, the visual processing of the image by the convolutional neural network is implemented by processing an Activation Function (Activation Function) of the image with an original size of 256 pixels by 256 pixels as shown in fig. 5.
And the attention model 408 is used for assigning corresponding weights to the visual features and the coding results to obtain a first weight matrix and a second weight matrix, wherein the weight matrices are used for representing the significance of the weight characterization target features.
And a fourth text decoder 409, configured to receive the first weight matrix, the second weight matrix, the encoding result, and the visual characteristics, and perform corresponding decoding.
An information generator 410 for sending the processing result of the fourth text decoder to a second neural network.
The second neural network includes:
a first decoder model 411, configured to perform statement-level decoding on the encoded result;
and the second decoder model 412 is configured to perform word-level decoding on the sentence-level decoding result to obtain second text information that fuses the visual features and the text information to describe the image.
In one embodiment of the present invention, the electronic device further includes: the training module is used for training a convolutional neural network model for extracting visual features from the image based on an image sample and a classification label of the image sample; the training module is used for training a first decoder model based on the statement sample and the corresponding decoding result; the training module is configured to train a second decoder model based on the word samples and the corresponding decoding results. Through the technical scheme shown in the embodiment, the neural network model and different decoders can be trained in a targeted manner.
In an embodiment of the invention, when it is judged that the accuracies of training the neural network model and different decoders for multiple times tend to be stable and do not mutate any more, the trained neural network model and different decoders reach a stable state at the moment, and the training does not need to be continued.
In an embodiment of the present invention, a training iteration number of 2000 times may be preset, and then, when the iteration number of model training reaches 2000 times, it may be determined that the currently trained neural network model and different decoders have reached a stable state, and the training may be stopped.
Fig. 5 is a schematic diagram of processing an image at a convolution layer and a pooling layer by an Activation Function (Activation Function), and as shown in fig. 5, performing convolution processing and pooling processing on an image with an original size of 256 pixels by 256 pixels at the convolution layer and the pooling layer respectively by the Activation Function to obtain visual characteristics of the image. Through the technical scheme shown in the embodiment, through the cross processing of the convolutional layer and the maximum pooling layer of the convolutional neural network model, each small block in the neural network is deeply analyzed by the convolutional layer, so that the characteristic with higher abstraction degree is obtained, the size of the matrix can be reduced, the number of nodes in the final full-connection layer is further reduced, and the aim of reducing parameters in the whole neural network is fulfilled.
Fig. 6 is an alternative flowchart of the image recognition method according to the embodiment of the present invention, and as shown in fig. 6, the steps shown in the alternative flowchart of the image recognition method according to the embodiment of the present invention are described.
Step 601: and processing the image intersection through a convolution layer and a maximum pooling layer of the convolution neural network model to obtain a down-sampling result of the image.
Wherein the image is a facial feature image of a person.
Step 602: and processing the down-sampling result through an average pooling layer of the convolutional neural network model to obtain the visual characteristics of the image.
Thus, the extraction of the corresponding visual feature of the facial feature of the person in the image can be realized. And processing the down-sampling result through an average pooling layer of the convolutional neural network model to obtain the visual characteristics of the image. Through the technical scheme shown in the embodiment, through the cross processing of the convolutional layer and the maximum pooling layer of the convolutional neural network model, each small block in the neural network is analyzed by the convolutional layer more deeply, so that the characteristic with higher abstraction degree is obtained, the size of the matrix can be reduced, the number of nodes in the final full-connection layer is further reduced, and the purpose of reducing parameters in the whole neural network is achieved, so that the convolutional layer is more applicable to the condition that the number of the face images is large.
In an embodiment of the present invention, the image visual features of the facial features may be further processed through an average pooling layer of the convolutional neural network model, so as to obtain the labels representing the classifications of the facial feature images. Through the technical scheme shown in the embodiment, the label for representing the classification of the facial image is obtained, so that the classification processing of a plurality of facial images or the classification of different features of the same facial image is realized.
Step 603: and performing word-level coding on at least two types of text information of the picture through the neural network models corresponding to different types of first text information, and performing statement-level coding on the word-level coding result.
Step 604: and distributing corresponding weights to the visual features and the coding results through an attention model, and inputting the first weight matrix, the second weight matrix, the coding results and the visual features into the first decoder for decoding.
In one embodiment of the present invention, the encoding of at least two different types of text information of the image includes: performing word-level coding on at least two types of text information of the picture through a neural network model corresponding to different types of text information; and carrying out sentence-level coding on the coding result of the word level. Through the technical scheme shown in this embodiment, word-level coding and sentence-level coding can be respectively performed on at least two types of text information of the picture through a Bi-directional long-and-short-term memory recurrent neural network ((Bi-directional LSTM RNN)), wherein the same coder model can be used for the word-level coding or the sentence-level coding of the at least two types of text information of the picture.
Step 605: and performing sentence-level decoding on the encoding result and performing word-level decoding on the decoding result of the second decoder model in the first decoder model to form second text information which fuses the visual features and the text information to describe the image.
Fig. 7 is an alternative structure diagram of an electronic device according to an embodiment of the present invention, and as shown in fig. 7, the electronic device 700 may be a mobile phone, a computer, a digital broadcast terminal, an information transceiver device, a game console, a tablet device, a medical device, an exercise device, a personal digital assistant, etc. with image recognition function. The electronic device 700 shown in fig. 7 includes: at least one processor 701, a memory 702, at least one network interface 704, and a user interface 703. The various components in the electronic device 700 are coupled together by a bus system 705. It is understood that the bus system 705 is used to enable communications among the components. The bus system 705 includes a power bus, a control bus, and a status signal bus in addition to a data bus. But for clarity of illustration the various busses are labeled in figure 7 as the bus system 705.
The user interface 703 may include, among other things, a display, a keyboard, a mouse, a trackball, a click wheel, a key, a button, a touch pad, or a touch screen.
It will be appreciated that the memory 702 can be either volatile memory or nonvolatile memory, and can include both volatile and nonvolatile memory. Among them, the nonvolatile Memory may be a Read Only Memory (ROM), a Programmable Read Only Memory (PROM), an Erasable Programmable Read-Only Memory (EPROM), an Electrically Erasable Programmable Read-Only Memory (EEPROM), a magnetic random access Memory (FRAM), a Flash Memory (Flash Memory), a magnetic surface Memory, an optical disk, or a Compact Disc Read-Only Memory (CD-ROM); the magnetic surface storage may be disk storage or tape storage. Volatile Memory can be Random Access Memory (RAM), which acts as external cache Memory. By way of illustration and not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), Enhanced Synchronous Dynamic Random Access Memory (Enhanced DRAM), Synchronous Dynamic Random Access Memory (SLDRAM), Direct Memory (DRmb Access), and Random Access Memory (DRAM). The memory 702 described in connection with the embodiments of the invention is intended to comprise these and any other suitable types of memory.
Memory 702 in embodiments of the present invention includes, but is not limited to: the ternary content addressable memory, static random access memory, may be capable of storing a variety of data such as image data, text data image recognition programs, etc. to support the operation of the electronic device 700. Examples of such data include: any computer programs for operating on the electronic device 700, such as an operating system 7021 and application programs 7022, image data, text data, image recognition programs, and the like. The operating system 7021 includes various system programs, such as a framework layer, a core library layer, a driver layer, and the like, for implementing various basic services and processing hardware-based tasks. The application 7022 may include various applications, such as a client with an image recognition function, an application, or the like, for implementing various application services including acquiring image information and first text information, and generating second text information based on the image information and the first text information. A program for implementing the power adjustment method according to an embodiment of the present invention may be included in the application program 7022.
The method disclosed in the above embodiments of the present invention may be applied to the processor 701, or implemented by the processor 701. The processor 701 may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be implemented by integrated logic circuits of hardware or operations in the form of software in the processor 701. The Processor 701 may be a general purpose Processor, a Digital Signal Processor (DSP), or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, or the like. The processor 701 may implement or perform the methods, steps, and logic blocks disclosed in embodiments of the present invention. A general purpose processor may be a microprocessor or any conventional processor or the like. The steps of the method disclosed by the embodiment of the invention can be directly implemented by a hardware decoding processor, or can be implemented by combining hardware and software modules in the decoding processor. The software modules may be located in a storage medium located in the memory 702, and the processor 701 may read the information in the memory 702 and perform the steps of the aforementioned methods in conjunction with its hardware.
In an exemplary embodiment, the electronic Device 700 may be implemented by one or more Application Specific Integrated Circuits (ASICs), DSPs, Programmable Logic Devices (PLDs), Complex Programmable Logic Devices (CPLDs), Field Programmable Gate Arrays (FPGAs), general purpose processors, controllers, Micro Controllers (MCUs), microprocessors (microprocessors), or other electronic components for performing the power adjustment method.
In an exemplary embodiment, the present invention further provides a computer readable storage medium, such as the memory 702 comprising a computer program, which is executable by the processor 701 of the electronic device 700 to perform the steps of the foregoing method. The computer readable storage medium can be Memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface Memory, optical disk, or CD-ROM; or may be a variety of devices including one or any combination of the above memories, such as a mobile phone, computer, tablet device, personal digital assistant, etc.
An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, performs:
image information and first text information are acquired.
And generating second text information based on the image information and the first text information, wherein the second text information is used for representing the image information and the text information content.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, embodiments of the present invention may take the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware aspects. Furthermore, embodiments of the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including magnetic disk storage, optical storage, and the like) having computer-usable program code embodied in the medium.
Embodiments of the present invention are described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program operations. These computer programs may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the operations performed by the processor of the computer or other programmable data processing apparatus produce means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program operations may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the operations stored in the computer-readable memory produce an article of manufacture including operating means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program operations may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the operations executed on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The above description is only exemplary of the present invention and should not be taken as limiting the scope of the present invention, and any modifications, equivalents, improvements, etc. that are within the spirit and principle of the present invention should be included in the present invention.

Claims (8)

1. An image recognition method, characterized in that the method comprises:
extracting visual features from the image;
coding at least two different types of text information of the image to obtain a coding result representing the semantics of the text information;
decoding is carried out based on the visual features and the coding result, and second text information which is used for fusing the visual features and the text information to describe the image is obtained, wherein the second text information is used for representing the image information and the text information content.
2. The method of claim 1, wherein the extracting visual features from the image comprises:
processing the image intersection through a convolution layer and a maximum pooling layer of a convolution neural network model to obtain a down-sampling result of the image;
and processing the down-sampling result through an average pooling layer of the convolutional neural network model to obtain the visual characteristics of the image.
3. The method of claim 1, wherein encoding at least two different types of textual information for the image comprises:
performing word-level coding on at least two types of text information of the image through a neural network model corresponding to different types of text information;
and carrying out sentence-level coding on the coding result of the word level.
4. The method of claim 1, wherein the decoding based on the visual characteristic and the encoding result comprises:
decoding the coding result at statement level through a first decoder model;
and performing word-level decoding on the sentence-level decoding result through a second decoder model.
5. The method of claim 4, further comprising:
distributing corresponding weights to the visual features and the coding results through an attention model to obtain a first weight matrix and a second weight matrix;
inputting the first weight matrix, the second weight matrix, the coding result and the visual features into the first decoder model for decoding.
6. The method of claim 1, further comprising:
training a convolutional neural network model for extracting visual features from the image based on image samples and classification labels of the image samples;
training a decoding first decoder model for performing statement level decoding based on the statement samples and the corresponding decoding results;
a second decoder model for word-level decoding is trained based on the word samples and corresponding decoding results.
7. An electronic device, characterized in that the electronic device comprises:
the information acquisition module is used for extracting visual features from the image;
the information acquisition module is also used for coding at least two different types of text information of the image to obtain a coding result representing the semantics of the text information;
and the information processing module is used for decoding based on the visual features and the coding result to obtain second text information which is used for fusing the visual features and the text information to describe the image, and the second text information is used for representing the image information and the text information content.
8. An electronic device, characterized in that the electronic device comprises:
a memory for storing executable instructions;
a processor for executing the executable instructions stored by the memory to perform the image recognition method of claims 1 to 6.
CN201810260038.0A 2018-03-27 2018-03-27 Image identification method and electronic equipment Active CN108549850B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201810260038.0A CN108549850B (en) 2018-03-27 2018-03-27 Image identification method and electronic equipment

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201810260038.0A CN108549850B (en) 2018-03-27 2018-03-27 Image identification method and electronic equipment

Publications (2)

Publication Number Publication Date
CN108549850A CN108549850A (en) 2018-09-18
CN108549850B true CN108549850B (en) 2021-07-16

Family

ID=63517392

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201810260038.0A Active CN108549850B (en) 2018-03-27 2018-03-27 Image identification method and electronic equipment

Country Status (1)

Country Link
CN (1) CN108549850B (en)

Families Citing this family (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109543714B (en) * 2018-10-16 2020-03-27 北京达佳互联信息技术有限公司 Data feature acquisition method and device, electronic equipment and storage medium
CN109614613B (en) * 2018-11-30 2020-07-31 北京市商汤科技开发有限公司 Image description statement positioning method and device, electronic equipment and storage medium
CN109933320B (en) * 2018-12-28 2021-05-18 联想(北京)有限公司 Image generation method and server
CN112231275B (en) * 2019-07-14 2024-02-27 阿里巴巴集团控股有限公司 Method, system and equipment for classifying multimedia files, processing information and training models
CN110717514A (en) * 2019-09-06 2020-01-21 平安国际智慧城市科技股份有限公司 Session intention identification method and device, computer equipment and storage medium
US11380116B2 (en) * 2019-10-22 2022-07-05 International Business Machines Corporation Automatic delineation and extraction of tabular data using machine learning
CN111191715A (en) * 2019-12-27 2020-05-22 深圳市商汤科技有限公司 Image processing method and device, electronic equipment and storage medium
CN111755118B (en) * 2020-03-16 2024-03-08 腾讯科技（深圳）有限公司 Medical information processing method, device, electronic equipment and storage medium
CN111460791B (en) * 2020-03-30 2023-12-01 北京百度网讯科技有限公司 Text classification method, device, equipment and storage medium
CN111861949B (en) * 2020-04-21 2023-07-04 北京联合大学 Multi-exposure image fusion method and system based on generation countermeasure network
CN112686263B (en) * 2020-12-29 2024-04-16 科大讯飞股份有限公司 Character recognition method, character recognition device, electronic equipment and storage medium
CN117710510B (en) * 2024-02-04 2024-06-11 支付宝(杭州)信息技术有限公司 Image generation method and device

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106126596A (en) * 2016-06-20 2016-11-16 中国科学院自动化研究所 A kind of answering method based on stratification memory network
CN106227721A (en) * 2016-08-08 2016-12-14 中国科学院自动化研究所 Chinese Prosodic Hierarchy prognoses system
CN106339591A (en) * 2016-08-25 2017-01-18 汤平 Breast cancer prevention self-service health cloud service system based on deep convolutional neural network
CN106897573A (en) * 2016-08-01 2017-06-27 12西格玛控股有限公司 Use the computer-aided diagnosis system for medical image of depth convolutional neural networks
CN106980683A (en) * 2017-03-30 2017-07-25 中国科学技术大学苏州研究院 Blog text snippet generation method based on deep learning
WO2017163230A1 (en) * 2016-03-24 2017-09-28 Ramot At Tel-Aviv University Ltd. Method and system for converting an image to text
CN107665736A (en) * 2017-09-30 2018-02-06 百度在线网络技术（北京）有限公司 Method and apparatus for generating information
CN107818306A (en) * 2017-10-31 2018-03-20 天津大学 A kind of video answering method based on attention model

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2017163230A1 (en) * 2016-03-24 2017-09-28 Ramot At Tel-Aviv University Ltd. Method and system for converting an image to text
CN106126596A (en) * 2016-06-20 2016-11-16 中国科学院自动化研究所 A kind of answering method based on stratification memory network
CN106897573A (en) * 2016-08-01 2017-06-27 12西格玛控股有限公司 Use the computer-aided diagnosis system for medical image of depth convolutional neural networks
CN106227721A (en) * 2016-08-08 2016-12-14 中国科学院自动化研究所 Chinese Prosodic Hierarchy prognoses system
CN106339591A (en) * 2016-08-25 2017-01-18 汤平 Breast cancer prevention self-service health cloud service system based on deep convolutional neural network
CN106980683A (en) * 2017-03-30 2017-07-25 中国科学技术大学苏州研究院 Blog text snippet generation method based on deep learning
CN107665736A (en) * 2017-09-30 2018-02-06 百度在线网络技术（北京）有限公司 Method and apparatus for generating information
CN107818306A (en) * 2017-10-31 2018-03-20 天津大学 A kind of video answering method based on attention model

Also Published As

Publication number Publication date
CN108549850A (en) 2018-09-18

Similar Documents

Publication Publication Date Title
CN108549850B (en) Image identification method and electronic equipment
CN109874029B (en) Video description generation method, device, equipment and storage medium
JP7112536B2 (en) Method and apparatus for mining entity attention points in text, electronic device, computer-readable storage medium and computer program
CN110705214B (en) Automatic coding method and device
CN110956018B (en) Training method of text processing model, text processing method, text processing device and storage medium
CN110288980A (en) Audio recognition method, the training method of model, device, equipment and storage medium
CN111598979B (en) Method, device and equipment for generating facial animation of virtual character and storage medium
CN111914076B (en) User image construction method, system, terminal and storage medium based on man-machine conversation
CN111274797A (en) Intention recognition method, device and equipment for terminal and storage medium
CN114676234A (en) Model training method and related equipment
CN111985243B (en) Emotion model training method, emotion analysis device and storage medium
CN113516152A (en) Image description method based on composite image semantics
CN116385584A (en) Poster generation method, device and system and computer readable storage medium
CN116050425A (en) Method for establishing pre-training language model, text prediction method and device
CN115757725A (en) Question and answer processing method and device, computer equipment and storage medium
CN113723077B (en) Sentence vector generation method and device based on bidirectional characterization model and computer equipment
CN110390015B (en) Data information processing method, device and system
CN110750637B (en) Text abstract extraction method, device, computer equipment and storage medium
CN115810215A (en) Face image generation method, device, equipment and storage medium
CN111723186A (en) Knowledge graph generation method based on artificial intelligence for dialog system and electronic equipment
CN116956953A (en) Translation model training method, device, equipment, medium and program product
CN114155417B (en) Image target identification method and device, electronic equipment and computer storage medium
CN115050371A (en) Speech recognition method, speech recognition device, computer equipment and storage medium
CN115132182A (en) Data identification method, device and equipment and readable storage medium
CN113392722A (en) Method and device for recognizing emotion of object in video, electronic equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant