CN110689523A - Personalized image information evaluation method based on meta-learning and information data processing terminal - Google Patents
Personalized image information evaluation method based on meta-learning and information data processing terminal Download PDFInfo
- Publication number
- CN110689523A CN110689523A CN201910824591.7A CN201910824591A CN110689523A CN 110689523 A CN110689523 A CN 110689523A CN 201910824591 A CN201910824591 A CN 201910824591A CN 110689523 A CN110689523 A CN 110689523A
- Authority
- CN
- China
- Prior art keywords
- aesthetic
- image
- meta
- training
- evaluation
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
- G06Q10/063—Operations research, analysis or management
- G06Q10/0639—Performance analysis of employees; Performance analysis of enterprise or organisation operations
- G06Q10/06393—Score-carding, benchmarking or key performance indicator [KPI] analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30168—Image quality inspection
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Business, Economics & Management (AREA)
- General Physics & Mathematics (AREA)
- Human Resources & Organizations (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Life Sciences & Earth Sciences (AREA)
- Health & Medical Sciences (AREA)
- Development Economics (AREA)
- Strategic Management (AREA)
- Software Systems (AREA)
- Mathematical Physics (AREA)
- Quality & Reliability (AREA)
- Computing Systems (AREA)
- Entrepreneurship & Innovation (AREA)
- Economics (AREA)
- Educational Administration (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Molecular Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- General Health & Medical Sciences (AREA)
- Game Theory and Decision Science (AREA)
- General Business, Economics & Management (AREA)
- Tourism & Hospitality (AREA)
- Operations Research (AREA)
- Marketing (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Image Analysis (AREA)
Abstract
The invention belongs to the technical field of image processing and computer vision, and discloses a meta-learning-based personalized image information evaluation method and an information data processing terminal, wherein an aesthetic prior meta-model is learned from personalized aesthetic evaluation tasks of a large number of individual users by utilizing the thought of meta-learning, so that the experience knowledge of the individual users on the image aesthetics can be effectively captured; in the meta-learning training process, a two-stage gradient optimization method from a support set to a query set is adopted, so that the network model can effectively learn the adaptive capacity from a training sample to a testing sample, and rapid adaptation and transfer learning can be performed in the face of a new personalized aesthetic evaluation task. According to the method, the individual image aesthetic evaluation is rapidly learned through a small number of training samples of unknown individual users; the generalization performance of good personalized image aesthetic evaluation can be obtained without elaborately designing a network structure; the performance is better than that of the current mainstream personalized image aesthetic evaluation method.
Description
Technical Field
The invention belongs to the technical field of image processing and computer vision, and particularly relates to a meta-learning-based personalized image information evaluation method and an information data processing terminal.
Background
Currently, the closest prior art: with the rapid development of social media and the popularization of mobile devices, the data volume of visual media such as images and videos is increasing day by day in the big data age, and how to process such multimedia data has become a research hotspot and focus in a plurality of interdisciplinary fields such as computer vision and human psychology. Where image aesthetic evaluation is an important research direction therein. The image aesthetic evaluation aims to simulate human to calculate and evaluate the aesthetic perception of the image by using a computer system; therefore, the aesthetic feeling of the image is automatically evaluated, so that the aesthetic feeling evaluation of the image is embodied by human intelligence, and the intelligence relates to a plurality of crossed fields such as image processing, computer vision, psychology and the like, and is a challenging research subject. The image aesthetic evaluation method can automatically evaluate the aesthetic value of human beings to the image, so the method has important application value in the fields of image recommendation systems, image enhancement, image retrieval, personal album management and the like.
Currently, there are three main tasks for image aesthetic evaluation tasks: aesthetic classifications, aesthetic scores, and aesthetic distributions. The aesthetic classification is to divide the image into two types of 'good' and 'bad'; the aesthetic evaluation is to give an aesthetic quality score of the image; the aesthetic distribution is a histogram of the aesthetic distribution giving the image. However, these research methods are all popular image aesthetic evaluation methods, and the popular image aesthetic evaluation refers to an aesthetic evaluation result that most people agree with, and is generally obtained by performing comprehensive aesthetic evaluation on the same image by a plurality of people. While these approaches have made some progress, there are still a number of critical issues that need to be broken through. First, the traditional image aesthetic evaluation methods mainly use traditional manual aesthetic features to model the aesthetic scores of the images, and the image aesthetic evaluation relates to higher-level image content understanding, and these methods are difficult to extract more effective high-level semantic features, thus leading to inaccurate aesthetic evaluation results of the images. Secondly, the strong capability of the deep learning method in the aspect of extracting high-level semantic features of images in recent years promotes the further development of the image aesthetic evaluation method based on large-scale data driving. At present, most of image aesthetic evaluation methods mainly focus on popular image aesthetic evaluation, but due to differences of culture, education, age, gender and the like, the evaluation standards of the individual for beauty often have great differences, so that the personalized image aesthetic evaluation for individual users is more in line with the actual situation. Meanwhile, in real life, it is difficult to obtain the aesthetic evaluation data of a large number of images by the user individual, so that a personalized image aesthetic evaluation model cannot be established by using a large-scale data training mode, and the research progress of the personalized image aesthetic evaluation method is slow at present.
In 2017, Ren et al of the university of rogues in the united states issued a data set (FLICKR-AES) for personalized image aesthetic evaluation, and firstly proposed a personalized image aesthetic evaluation method, which first established a popular aesthetic evaluation model of an image as a prior model by using a deep neural network, established a personalized aesthetic difference evaluation model by using a support vector machine (SVR) by using the relevance of image aesthetic attributes and contents to personalized and popular aesthetic differences, and finally summed up the results of the popular evaluation model and the personalized aesthetic difference evaluation model to finally obtain a personalized aesthetic evaluation result. Then, the method has the following defects: (1) the popular aesthetic evaluation model serving as the prior model eliminates the aesthetic difference among individuals during training, and effective prior knowledge is difficult to obtain; (2) the traditional learning method based on the support vector machine has the problems of difficult convergence and weak migration capability, and is difficult to quickly migrate from a prior model to a new personalized aesthetic evaluation task.
In summary, the problems of the prior art are as follows:
(1) the popular aesthetic evaluation model as the prior model eliminates the aesthetic difference among individuals during training, and effective prior knowledge is difficult to obtain.
(2) The traditional learning method based on the support vector machine has the problems of difficult convergence and weak migration capability, and is difficult to quickly migrate from a prior model to a new personalized aesthetic evaluation task.
The difficulty of solving the technical problems is as follows:
the main difficulty of the above technical problem lies in how to learn the common prior knowledge model from the aesthetics of different individuals, so the image aesthetic evaluation result of an individual is retained in the training process of the prior knowledge model, and therefore an effective aesthetic prior model is learned from the aesthetic evaluation tasks of a large number of individual users by using the learning thought of the society in meta-learning.
The significance of solving the technical problems is as follows:
the technical method is difficult to quickly transfer the established priori knowledge model to the personalized image aesthetic evaluation model through a small number of training samples, and is not beneficial to application in real life. Because a large number of training samples are generally difficult to obtain for an individual user, it is of great practical significance to quickly construct a personalized image aesthetic evaluation model for the individual user under the condition of limited training data.
Disclosure of Invention
Aiming at the problems in the prior art, the invention provides a meta-learning-based personalized image information evaluation method and an information data processing terminal.
The invention is realized in such a way that a personalized image information evaluation method based on meta-learning comprises the following steps:
a first step of collecting an image aesthetic data set containing a plurality of individual user aesthetic scores;
secondly, dividing the training data of the aesthetic evaluation task of each user into a support set and a query set, and preprocessing the image data;
thirdly, constructing a deep convolutional neural network as an aesthetic evaluation model of the image to be trained;
inputting the preprocessed image data and the aesthetic scores into the network model constructed in the third step for meta-learning mode training to obtain an aesthetic prior meta-model;
fifthly, collecting individualized aesthetic scoring data of the image by the user to be tested, and carrying out fine tuning training on the aesthetic prior meta-model obtained in the fourth step by utilizing the preprocessed image and the aesthetic scoring of the user to obtain an individualized image aesthetic evaluation model conforming to the test user;
and sixthly, automatically realizing the individual aesthetic evaluation of the image by using the established individual image aesthetic evaluation model for the aesthetic image to be tested.
Further, the method of collecting an image aesthetic data set containing a plurality of individual user aesthetic scores in the first step comprises:
(1) collecting professional image aesthetic data sets, acquiring images and aesthetic scoring data of the images by users, and normalizing all aesthetic scores to [0,1 ];
(2) the aesthetic evaluation of images by each user is considered as an independent subtask with individual users as research targets.
Further, the second step of dividing the training data of the aesthetic evaluation task of each user into a support set and a query set, and the method for preprocessing the image data comprises the following steps:
(1) dividing the personalized aesthetic evaluation data of each user into a support set and a query set according to the proportion of 8: 2;
(2) scaling the size of the image to a predetermined size, the predetermined size being consistent with the required input size of the constructed deep convolutional neural network;
(3) and carrying out normalization operation on the image data, counting the mean value of the sample images in the training data, and carrying out mean value removing operation on all the sample images to obtain the preprocessed images.
Further, the method for constructing the deep convolutional neural network as the aesthetic evaluation model of the image to be trained in the third step comprises the following steps:
(1) the constructed deep convolutional neural network consists of a basic network model, two full-connection layers and an output layer;
(2) wherein the base network model removes the ResNet18 convolutional network portion of the full link layer;
(3) the two full connection layers respectively consist of 1024 nodes and 512 nodes;
(4) the output layer is a prediction result of the user on the image aesthetic evaluation, and the Sigmoid activation function is used as the activation function of the output layer.
Further, the fourth step inputs the preprocessed image data and the aesthetic scores into the constructed network model for training in a meta-learning manner, and the method for obtaining the aesthetic prior meta-model through training comprises the following steps:
(1) taking the image aesthetic evaluation task of each user as a training target, and respectively inputting the image data of the support set and the image data of the query set into the constructed network model for prediction, wherein the parameters of the network model come from a pre-training network;
(2) the meta-learning training mode is a two-stage gradient optimization method, firstly, updating network model parameters by using image data of a support set in an aesthetic task, and then, performing secondary gradient updating on images of a query set by using the updated network model;
(3) training network model parameters by using image aesthetic evaluation tasks of a large number of users in training data to obtain an aesthetic prior meta-model;
(4) the network model training adopts a random gradient descent method SGD to carry out parameter optimization, and a loss function formula is as follows:
wherein, ynAndrespectively providing a real result and a prediction result of the aesthetic scoring of the user on the image, wherein N is the number of images for performing the aesthetic scoring on all training users; parametrizing network model through gradient optimization methodAnd training the number until the calculated loss function result is smaller than a threshold value, and finally obtaining the aesthetic prior meta-model of the image.
Further, the method for collecting the personalized aesthetic scoring data of the image by the user to be tested in the fifth step, and performing fine tuning training on the obtained aesthetic prior meta-model by using the preprocessed image and the aesthetic scoring of the user to obtain the personalized image aesthetic evaluation model conforming to the tested user comprises the following steps:
(1) collecting aesthetic score data of a test user on the image, normalizing the aesthetic score to [0,1], and preprocessing the image;
(2) performing fine tuning training on the obtained aesthetic prior meta-model by using the image data and the corresponding aesthetic score labels to obtain an individualized image aesthetic evaluation model which accords with a test user;
(3) the fine tuning training process adopts a random gradient descent method SGD to carry out parameter optimization, and a loss function formula is as follows:
wherein, ymAnd
the true and predicted results of the aesthetic scoring of the images for the individual users, respectively, M is the number of images that the test user performs the aesthetic scoring. And carrying out fine tuning training on the network model parameters by a gradient optimization method until the calculated loss function result is smaller than a threshold value, and finally obtaining the personalized image aesthetic evaluation model which accords with the test user.
Another object of the present invention is to provide an information data processing terminal to which the meta learning based personalized image information evaluation method is applied.
In summary, the advantages and positive effects of the invention are: the method has the advantages that the aesthetic prior meta-model is learned from personalized aesthetic evaluation tasks of a large number of individual users by utilizing the thought of meta-learning, so that the experience knowledge of the individual users on the image aesthetics can be effectively captured; in the meta-learning training process, a two-stage gradient optimization method from a support set to a query set is adopted, so that the network model can effectively learn the adaptive capacity from a training sample to a testing sample, and rapid adaptation and transfer learning can be performed in the face of a new personalized aesthetic evaluation task.
Compared with the prior art, the technical scheme of the invention has the following beneficial technical effects:
1. compared with the prior art that the public aesthetic evaluation is utilized to learn the aesthetic prior knowledge, the aesthetic evaluation of the individual user is regarded as an independent subtask, the network model is trained by utilizing a two-stage gradient optimization method based on meta-learning, and the personalized image aesthetic evaluation can be rapidly learned through a small amount of training samples of unknown individual users.
2. The method is independent of the model, the network structure is not required to be designed elaborately, the method can be suitable for all deep convolutional neural networks, and good generalization performance of the personalized image aesthetic evaluation can be obtained only by explicitly training model parameters through two-stage gradient optimization strategies.
3. Aiming at the small sample learning characteristic of the personalized image aesthetic evaluation, the invention solves the problem of few training samples of the personalized aesthetic evaluation by using a few sample element learning strategy, learns a personalized aesthetic priori knowledge model through a wide personalized aesthetic evaluation task, and proves that the performance of the method of the invention is better than that of the conventional mainstream personalized image aesthetic evaluation method through the experimental result on an image aesthetic database.
Drawings
Fig. 1 is a flowchart of a method for evaluating personalized image information based on meta learning according to an embodiment of the present invention.
Fig. 2 is a flowchart of an implementation of the method for evaluating personalized image information based on meta-learning according to the embodiment of the present invention.
Fig. 3 is a structural diagram of a method for evaluating personalized image information based on meta learning according to an embodiment of the present invention.
FIG. 4 is a diagram of the personalized evaluation result of a test user in the FLICKR-AES database according to the embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is further described in detail with reference to the following embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
Aiming at the problems in the prior art, the invention provides a personalized image information evaluation method based on meta-learning and an information data processing terminal, and the invention is described in detail below with reference to the accompanying drawings.
As shown in fig. 1, the method for evaluating personalized image information based on meta learning according to the embodiment of the present invention includes the following steps:
s101: gathering an image aesthetic data set containing a large number of individual user aesthetic scores;
s102: dividing training data of an aesthetic evaluation task of each user into a support set and a query set, and preprocessing image data;
s103: constructing a deep convolutional neural network as an aesthetic evaluation model of the image to be trained;
s104: inputting the preprocessed image data and the aesthetic scores into the network model constructed in S103 to carry out meta-learning mode training, and training to obtain an aesthetic prior meta-model;
s105: collecting individualized aesthetic scoring data of the image by the user to be tested, and performing fine tuning training on the aesthetic prior meta-model obtained in the step S104 by using the preprocessed image and the aesthetic scoring of the user to obtain an individualized image aesthetic evaluation model conforming to the user to be tested;
s106: and for the aesthetic images to be tested, automatically realizing the individual aesthetic evaluation of the images by utilizing the established individual image aesthetic evaluation model.
The technical solution of the present invention is further described below with reference to the accompanying drawings.
Aiming at the small sample characteristic of the personalized image aesthetic evaluation method, the invention designs the personalized image aesthetic evaluation method based on meta-learning through the meta-learning thought. The invention aims at the personalized aesthetic evaluation of images, and aims to solve the problem that an aesthetic prior model constructed by the current personalized aesthetic evaluation method cannot well reflect the mode of an individual user for the aesthetic quality of the images; secondly, performing two-stage gradient optimization on the deep convolutional neural network from support set data to query set data by utilizing a large number of personalized aesthetic evaluation tasks to obtain an aesthetic prior meta-model; and finally, fine-tuning the trained aesthetic prior meta-model by a small amount of training data of unknown individual users to obtain a personalized image aesthetic evaluation model of the trained aesthetic prior meta-model. The specific implementation method is shown in fig. 2, and the embodiment includes a training user data acquisition and preprocessing module, an aesthetic prior model training module, a testing user data acquisition and preprocessing module, and a personalized image aesthetic evaluation module. The training user data acquisition and prediction module comprises parts of collecting training user data, constructing a large number of aesthetic evaluation subtasks, dividing an image data set into a support set and a query set, preprocessing an image and the like; the aesthetic prior model training module comprises a two-stage gradient optimization part for constructing a deep convolutional neural network and a deep network model based on meta-learning; the test user data acquisition and preprocessing comprises the steps of collecting test user data, acquiring a small amount of training data and preprocessing images; the personalized image aesthetic evaluation module is used for carrying out fine tuning training on the aesthetic prior meta-model by utilizing a small amount of training data of a test user to obtain a personalized aesthetic evaluation model.
1. Training user data acquisition and preprocessing module
Collecting training user data: the training user data collected by the method of the present invention is from users in the existing aesthetic image data set FLICKR-AES and the aesthetic scores of the users on the images, wherein the training set comprises 173 users for making an aesthetic score on 35263 images in total, each image has collected the aesthetic scores of five users, and the average aesthetic score made by each user is 1019 images. The aesthetic scores of the images range from 1 to 5, and for the convenience of calculation, the invention normalizes the aesthetic scores of the images of the users to be between 0 and 1.
A number of aesthetic evaluation subtasks were constructed: unlike the traditional method of training and modeling by using images and popular aesthetic scores, the invention takes the aesthetic evaluation task of each user as an independent subtask, namely, takes the aesthetic evaluation of images of 173 users as 173 subtasks to construct a meta-training set (corresponding to D in FIG. 2)meta-train) Such a constructed data set preserves the personalized aesthetic appreciation of images by individual users relative to the popular aesthetic appreciation by training and modeling the scoring of the image aesthetic appreciation by multiple people.
The image dataset is divided into a support set and a query set: because a metadata set containing a large number of personalized image aesthetic evaluation tasks is constructed, the evaluated images of each user need to be divided into a support set and a query set by using the concept of meta-learning, the support set mainly has the function of optimizing a network model in the training process, and the query set mainly has the function of verifying whether the network model optimized by using the support set can be better applied to untrained data to perform secondary correction on model parameters after the support set is trained.
Image preprocessing: the image preprocessing mainly comprises size normalization of the image and image de-averaging operation. Since the input size of the deep convolutional neural network is fixed, scaling operations must be performed on sample images of different sizes, the present invention first scales the sizes of all sample images to 256 × 256 × 3, and then randomly crops the scaled images according to the size of 224 × 224 × 3 to adapt to the network input size, where 3 represents 3 color channels of a color image, i.e., three color channels of RGB; the convolutional neural network training model is utilized, mean value removing operation needs to be carried out on training data, so that the image data of training can be guaranteed to be distributed near the mean value, and the specific process is as follows: firstly, the mean value of sample images in training data is counted, and then the mean value removing operation is carried out on all the sample images to obtain preprocessed images.
2. Aesthetic prior model training module
Constructing a deep convolutional neural network: deep convolutional neural network f in FIG. 3θThe network structure model used by the invention is characterized in that the convolution network part is a ResNet18 network with a full connection layer removed, the full connection layer containing 1024 nodes is generated by utilizing Global Average Pooling (GAP) operation, and then two full connection layers and an output layer are constructed after the full connection layer; the two fully-connected layers are respectively composed of 1024 nodes and 512 nodes, and the output layer is the predicted image aesthetic score. In order to achieve a more rapid and stable training effect, a BN layer and a Dropout layer are added after each full-connection layer, so that the prediction score is [0,1]]Finally using the Sigmoid activation function as the activation function of the output layer.
The deep network model is based on two-stage gradient optimization of meta-learning: due to the constructed meta-training set Dmeta-trainThe invention comprises a large number of personalized image aesthetic evaluation tasks which basically belong to the small sample learning problem, so the invention effectively processes the small sample learning idea by utilizing a large number of learning tasks by means of meta-learning and obtains an aesthetic prior meta-model by carrying out network model training based on two-stage gradient optimization on the personalized image aesthetic evaluation tasks of 173 users in a meta-training set. The model training process comprises the following steps:
(1) randomly selecting image training data of one user, inputting the preprocessed support set image data into a deep convolution neural network to obtain a predicted image aesthetic score, and performing back propagation calculation on gradient updating network parameters by using a Euclidean distance between the two as a loss function in order to keep the predicted aesthetic score of a network model consistent with the aesthetic score of the user;
(2) in order to verify whether the network model trained by using the image data of the user support set can effectively perform effective aesthetic evaluation on unknown image data, the invention uses the network model updated in the step (1) of inputting the image data of the user query set to perform secondary gradient calculation updating to correct network parameters, and the two-stage gradient optimization method can effectively use a small amount of existing training samples (support sets) to learn the model and can quickly adapt to the prediction of the unknown samples (query sets).
(3) And then randomly selecting another user to train the network model by repeating the steps (1) and (2) until the image data of each user in the meta-training set trains the network model 50 times.
The network model training adopts a random gradient descent method SGD to carry out parameter optimization, and a loss function formula is as follows:
wherein, ynAnd
the actual result and the predicted result of the aesthetic scoring of the image by the user are respectively, and N is the number of images for all training users to perform the aesthetic scoring. And training the network model parameters by a gradient optimization method until the calculated loss function result is less than 0.0001, and finally obtaining the aesthetic prior meta-model of the image. The obtained aesthetic prior model optimizes the tasks of the aesthetic evaluation of the personalized images of a large number of users, obtains the prior knowledge of the general users in the aesthetic process of the images, and can be quickly adapted to the aesthetic evaluation of the personalized images of unknown users.
3. Test user data acquisition and preprocessing
Collecting test user data: the test user data collected by the method is from an existing aesthetic image data set FLICKR-AES and the aesthetic scores of users on images, wherein the test set comprises 37 users for performing aesthetic scores on 4737 images in total, each image collects the aesthetic scores of five users, the number of images subjected to the aesthetic scores by each test user is 105-171, and the average value of the number of the images is 137. The aesthetic scores of the images range from 1 to 5, and for the convenience of calculation, the invention normalizes the aesthetic scores of the images of the users to be between 0 and 1.
A small amount of training data is acquired: in order to perform personalized image aesthetic evaluation on an unknown test user by using an aesthetic prior meta-model, a small amount of training data of the user is required to perform fine tuning training on a previous model, a certain number of images for aesthetic scoring of the user are used as training samples, and then the rest of images are used as test samples to verify the performance of the method.
Image preprocessing: the image preprocessing mainly comprises size normalization of the image and image de-averaging operation. Since the input size of the deep convolutional neural network is fixed, scaling operations must be performed on sample images of different sizes, the present invention first scales the sizes of all sample images to 256 × 256 × 3, and then randomly crops the scaled images according to the size of 224 × 224 × 3 to adapt to the network input size, where 3 represents 3 color channels of a color image, i.e., three color channels of RGB; the convolutional neural network training model is utilized, mean value removing operation needs to be carried out on training data, so that the image data of training can be guaranteed to be distributed near the mean value, and the specific process is as follows: firstly, the mean value of sample images in training data is counted, and then the mean value removing operation is carried out on all the sample images to obtain preprocessed images.
Individualized image aesthetic evaluation module
According to the aesthetic prior meta-model obtained by training in the step 2, the invention utilizes a small amount of training samples of the test user to carry out fine tuning training on the model parameters, and then the personalized image aesthetic evaluation model which accords with the user can be obtained. The fine tuning training process adopts a random gradient descent method SGD to carry out parameter optimization, and a loss function formula is as follows:
wherein, ymAnd
the real result and the predicted result of the aesthetic scoring of the image for the individual user are respectively, and M is the number of training samples for the test user to perform the aesthetic scoring. By means of gradient optimisationAnd carrying out fine tuning training on the network model parameters until the calculated loss function result is less than 0.0001, and finally obtaining the personalized image aesthetic evaluation model which accords with the test user.
Finally, for the test sample image of the test user, the image aesthetic evaluation result which is in line with the aesthetic sense of the test user can be predicted by calling the personalized image aesthetic evaluation model of the test user.
The technical effects of the present invention will be described in detail below in conjunction with performance tests and experimental analysis.
The personalized aesthetic image datasets of the present invention are all from the literature: ren J, Shen X, Lin Z, et al. The aesthetic scores for individuals in the FLICKR-AES image dataset ranged from 1 point to 5 points, and for ease of calculation, the present invention normalized all aesthetic scores to between 0 points to 1 points.
In order to prove the effect of the invention, the user to be tested is subjected to personalized image aesthetic evaluation and compared with other personalized image aesthetic evaluation methods.
In order to verify the correctness of the invention, a test user is selected from a FLICKR-AES image database for verification. Fig. 4 shows the test image, the aesthetic score of the user, the Ren method, and the aesthetic evaluation result of the method of the present invention, and it can be seen from the test result that the result obtained by the method of the present invention in evaluating the personalized aesthetic evaluation of the image is more consistent with the aesthetic evaluation result of the user than the Ren method, and the personalized aesthetic score of the image according to the individual user can be evaluated more accurately.
The method is compared with a Ren method in a FLICKR-AES data set document to carry out personalized aesthetic prediction performance of the image, because the most important index in image aesthetics is the ranking Correlation of the subjective and objective prediction results, the method uses a Spearman Rank Order Correlation Coefficient (SROCC) to measure the performance of the two methods, the SROCC is used for quantitatively measuring the ranking Correlation of the aesthetic score prediction results and the real results, and the larger the SROCC value is, the better the prediction performance of the method is. The invention trains and tests the individualized aesthetic evaluation result of each individual user to the image in the FLICKR-AES image test set in two ways, specifically, 10 or 100 images which are subjected to aesthetic scoring by each user are respectively selected randomly as training samples, the rest images are used as test samples for testing, in order to eliminate random selection errors, the experiment is repeated for 50 times, the SROCC average value is taken as the evaluation performance of each test user, and finally, the average result of the individualized image aesthetic evaluation performance of 37 individual users in the test set is taken as the overall prediction performance.
Table 1 shows a comparison of the personalized aesthetic evaluation performance of the two methods. As can be seen from the table, the method for the invention to have the overall prediction performance higher than Ren on 37 individuals in the FLICKR-AES image test set shows that the method has good prediction performance on the personalized image aesthetic evaluation aiming at individual users.
Table 1: personalized image aesthetic evaluation performance comparison
The method provided by the invention has the advantages that the personalized image aesthetic evaluation model based on the meta-learning can well simulate the personalized aesthetic of an individual user to an image, the aesthetic prior meta-model based on the two-stage gradient optimization can effectively capture the prior knowledge of the general user to the image aesthetic, the training samples of a small number of unknown users can be quickly adapted to the personalized image aesthetic evaluation conforming to the user, the experiment shows that the method provided by the invention has better aesthetic evaluation performance compared with the previous method, and the method can be widely applied to the personalized visual aesthetic analysis.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents and improvements made within the spirit and principle of the present invention are intended to be included within the scope of the present invention.
Claims (7)
1. A personalized image information evaluation method based on meta-learning is characterized by comprising the following steps:
a first step of collecting an image aesthetic data set containing a plurality of individual user aesthetic scores;
secondly, dividing the training data of the aesthetic evaluation task of each user into a support set and a query set, and preprocessing the image data;
thirdly, constructing a deep convolutional neural network as an aesthetic evaluation model of the image to be trained;
inputting the preprocessed image data and the aesthetic scores into the network model constructed in the third step for meta-learning mode training to obtain an aesthetic prior meta-model;
fifthly, collecting individualized aesthetic scoring data of the image by the user to be tested, and carrying out fine tuning training on the aesthetic prior meta-model obtained in the fourth step by utilizing the preprocessed image and the aesthetic scoring of the user to obtain an individualized image aesthetic evaluation model conforming to the test user;
and sixthly, automatically realizing the individual aesthetic evaluation of the image by using the established individual image aesthetic evaluation model for the aesthetic image to be tested.
2. The meta-learning based personalized image information rating method of claim 1, wherein the method of collecting image aesthetic data sets containing a large number of individual user aesthetic scores in the first step comprises:
(1) collecting professional image aesthetic data sets, acquiring images and aesthetic scoring data of the images by users, and normalizing all aesthetic scores to [0,1 ];
(2) the aesthetic evaluation of images by each user is considered as an independent subtask with individual users as research targets.
3. The meta-learning based personalized image information evaluation method of claim 1, wherein the second step divides training data of an aesthetic evaluation task of each user into a support set and a query set, and the preprocessing method of the image data comprises:
(1) dividing the personalized aesthetic evaluation data of each user into a support set and a query set according to the proportion of 8: 2;
(2) scaling the size of the image to a predetermined size, the predetermined size being consistent with the required input size of the constructed deep convolutional neural network;
(3) and carrying out normalization operation on the image data, counting the mean value of the sample images in the training data, and carrying out mean value removing operation on all the sample images to obtain the preprocessed images.
4. The method for evaluating personalized image information based on meta-learning according to claim 1, wherein the method for constructing the deep convolutional neural network as the aesthetic evaluation model of the image to be trained in the third step comprises:
(1) the constructed deep convolutional neural network consists of a basic network model, two full-connection layers and an output layer;
(2) wherein the base network model removes the ResNet18 convolutional network portion of the full link layer;
(3) the two full connection layers respectively consist of 1024 nodes and 512 nodes;
(4) the output layer is a prediction result of the user on the image aesthetic evaluation, and the Sigmoid activation function is used as the activation function of the output layer.
5. The meta-learning based personalized image information evaluation method of claim 1, wherein in the fourth step, the preprocessed image data and the aesthetic score are input into the constructed network model for training in a meta-learning manner, and the method for obtaining the aesthetic prior meta-model by training comprises the following steps:
(1) taking the image aesthetic evaluation task of each user as a training target, and respectively inputting the image data of the support set and the image data of the query set into the constructed network model for prediction, wherein the parameters of the network model come from a pre-training network;
(2) the meta-learning training mode is a two-stage gradient optimization method, firstly, updating network model parameters by using image data of a support set in an aesthetic task, and then, performing secondary gradient updating on images of a query set by using the updated network model;
(3) training network model parameters by using image aesthetic evaluation tasks of a large number of users in training data to obtain an aesthetic prior meta-model;
(4) the network model training adopts a random gradient descent method SGD to carry out parameter optimization, and a loss function formula is as follows:
wherein, ynAnd
respectively providing a real result and a prediction result of the aesthetic scoring of the user on the image, wherein N is the number of images for performing the aesthetic scoring on all training users; and training the network model parameters by a gradient optimization method until the calculated loss function result is less than a threshold value, and finally obtaining an aesthetic prior meta-model of the image.
6. The method for evaluating personalized image information based on meta-learning according to claim 1, wherein the fifth step of collecting personalized aesthetic scoring data of the image by the user to be tested, and performing fine tuning training on the obtained aesthetic prior meta-model by using the preprocessed image and the aesthetic scoring of the user to obtain the personalized image aesthetic evaluation model conforming to the user to be tested comprises:
(1) collecting aesthetic score data of a test user on the image, normalizing the aesthetic score to [0,1], and preprocessing the image;
(2) performing fine tuning training on the obtained aesthetic prior meta-model by using the image data and the corresponding aesthetic score labels to obtain an individualized image aesthetic evaluation model which accords with a test user;
(3) the fine tuning training process adopts a random gradient descent method SGD to carry out parameter optimization, and a loss function formula is as follows:
wherein, ymAnd
respectively carrying out real result and prediction result on the aesthetic scoring of the image for the individual user, wherein M is the number of images for carrying out the aesthetic scoring on the test user; and carrying out fine tuning training on the network model parameters by a gradient optimization method until the calculated loss function result is smaller than a threshold value, and finally obtaining the personalized image aesthetic evaluation model which accords with the test user.
7. An information data processing terminal applying the meta-learning based personalized image information evaluation method according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910824591.7A CN110689523A (en) | 2019-09-02 | 2019-09-02 | Personalized image information evaluation method based on meta-learning and information data processing terminal |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910824591.7A CN110689523A (en) | 2019-09-02 | 2019-09-02 | Personalized image information evaluation method based on meta-learning and information data processing terminal |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110689523A true CN110689523A (en) | 2020-01-14 |
Family
ID=69107645
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910824591.7A Pending CN110689523A (en) | 2019-09-02 | 2019-09-02 | Personalized image information evaluation method based on meta-learning and information data processing terminal |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110689523A (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111695421A (en) * | 2020-04-30 | 2020-09-22 | 北京迈格威科技有限公司 | Image recognition method and device and electronic equipment |
| CN112115265A (en) * | 2020-09-25 | 2020-12-22 | 中国科学院计算技术研究所苏州智能计算产业技术研究院 | Small sample learning method in text classification |
| CN112364990A (en) * | 2020-10-29 | 2021-02-12 | 北京语言大学 | Method and system for realizing grammar error correction and less sample field adaptation through meta-learning |
| CN112992174A (en) * | 2021-02-03 | 2021-06-18 | 深圳壹秘科技有限公司 | Voice analysis method and voice recording device thereof |
| CN113255743A (en) * | 2021-05-12 | 2021-08-13 | 展讯通信（上海）有限公司 | Image processing method, image processing device, electronic equipment and storage medium |
| CN113313683A (en) * | 2021-05-28 | 2021-08-27 | 西安电子科技大学 | Non-reference video quality evaluation method based on meta-migration learning |
| CN116090553A (en) * | 2023-04-10 | 2023-05-09 | 环球数科集团有限公司 | Artificial intelligence automatic processing system based on meta learning |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107610123A (en) * | 2017-10-11 | 2018-01-19 | 中共中央办公厅电子科技学院 | A kind of image aesthetic quality evaluation method based on depth convolutional neural networks |
| CN109902912A (en) * | 2019-01-04 | 2019-06-18 | 中国矿业大学 | A kind of personalized image aesthetic evaluation method based on character trait |
| CN109978836A (en) * | 2019-03-06 | 2019-07-05 | 华南理工大学 | User individual image esthetic evaluation method, system, medium and equipment based on meta learning |
-
2019
- 2019-09-02 CN CN201910824591.7A patent/CN110689523A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107610123A (en) * | 2017-10-11 | 2018-01-19 | 中共中央办公厅电子科技学院 | A kind of image aesthetic quality evaluation method based on depth convolutional neural networks |
| CN109902912A (en) * | 2019-01-04 | 2019-06-18 | 中国矿业大学 | A kind of personalized image aesthetic evaluation method based on character trait |
| CN109978836A (en) * | 2019-03-06 | 2019-07-05 | 华南理工大学 | User individual image esthetic evaluation method, system, medium and equipment based on meta learning |
Non-Patent Citations (2)
| Title |
|---|
| AI研习社-译站: "图像样本不够用？元学习帮你解决", 《雷锋网：HTTPS://WWW.LEIPHONE.COM/CATEGORY/AI/CCXZRHOPMEJN4CUD.HTML》 * |
| CHELSEA FINN，ET AL.: "Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks", 《ARXIV》 * |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111695421A (en) * | 2020-04-30 | 2020-09-22 | 北京迈格威科技有限公司 | Image recognition method and device and electronic equipment |
| CN111695421B (en) * | 2020-04-30 | 2023-09-22 | 北京迈格威科技有限公司 | Image recognition method and device and electronic equipment |
| CN112115265A (en) * | 2020-09-25 | 2020-12-22 | 中国科学院计算技术研究所苏州智能计算产业技术研究院 | Small sample learning method in text classification |
| CN112364990A (en) * | 2020-10-29 | 2021-02-12 | 北京语言大学 | Method and system for realizing grammar error correction and less sample field adaptation through meta-learning |
| CN112992174A (en) * | 2021-02-03 | 2021-06-18 | 深圳壹秘科技有限公司 | Voice analysis method and voice recording device thereof |
| CN113255743A (en) * | 2021-05-12 | 2021-08-13 | 展讯通信（上海）有限公司 | Image processing method, image processing device, electronic equipment and storage medium |
| CN113313683A (en) * | 2021-05-28 | 2021-08-27 | 西安电子科技大学 | Non-reference video quality evaluation method based on meta-migration learning |
| CN116090553A (en) * | 2023-04-10 | 2023-05-09 | 环球数科集团有限公司 | Artificial intelligence automatic processing system based on meta learning |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110689523A (en) | Personalized image information evaluation method based on meta-learning and information data processing terminal | |
| CN111738301B (en) | Long-tail distribution image data identification method based on double-channel learning | |
| CN106095893B (en) | A kind of cross-media retrieval method | |
| CN109902912B (en) | Personalized image aesthetic evaluation method based on character features | |
| CN109165692B (en) | User character prediction device and method based on weak supervised learning | |
| CN110728656A (en) | Meta-learning-based no-reference image quality data processing method and intelligent terminal | |
| CN111291604A (en) | Face attribute identification method, device, storage medium and processor | |
| CN110516095A (en) | Weakly supervised depth Hash social activity image search method and system based on semanteme migration | |
| CN112687374B (en) | Psychological crisis early warning method based on text and image information joint calculation | |
| CN111414461A (en) | Intelligent question-answering method and system fusing knowledge base and user modeling | |
| CN110378736B (en) | Method for evaluating experience satisfaction degree of tourists on natural resources through facial expression recognition | |
| CN110309850A (en) | Vision question and answer prediction technique and system based on language priori problem identification and alleviation | |
| CN112668486A (en) | Method, device and carrier for identifying facial expressions of pre-activated residual depth separable convolutional network | |
| CN109543637A (en) | A kind of face identification method, device, equipment and readable storage medium storing program for executing | |
| CN116304035A (en) | Multi-notice multi-crime name relation extraction method and device in complex case | |
| CN115309860A (en) | False news detection method based on pseudo twin network | |
| CN114511759A (en) | Method and system for identifying categories and determining characteristics of skin state images | |
| CN112396091B (en) | Social media image popularity prediction method, system, storage medium and application | |
| CN108509588B (en) | Lawyer evaluation method and recommendation method based on big data | |
| CN110347825A (en) | The short English film review classification method of one kind and device | |
| CN112116669B (en) | Image aesthetic prediction method based on color and harmonic plane composition | |
| CN111242131B (en) | Method, storage medium and device for identifying images in intelligent paper reading | |
| CN113743079A (en) | Text similarity calculation method and device based on co-occurrence entity interaction graph | |
| CN117195027A (en) | Cluster weighted clustering integration method based on member selection | |
| Chen et al. | Intelligent teaching evaluation system integrating facial expression and behavior recognition in teaching video |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200114 |