CN113239635A

CN113239635A - Model evaluation method and device

Info

Publication number: CN113239635A
Application number: CN202110666839.9A
Authority: CN
Inventors: 李增兴; 李昀; 梁绍文; 邓卫兵
Original assignee: Bank of China Ltd
Current assignee: Bank of China Ltd
Priority date: 2021-06-16
Filing date: 2021-06-16
Publication date: 2021-08-10

Abstract

The invention discloses a model evaluation method and a model evaluation device, wherein a first verification node can obtain a first model to be evaluated sent by a first training node in a block chain network, verify the first model to be evaluated to obtain a first evaluation score, respectively obtain a second evaluation score sent by at least one second verification node, count an evaluation score group, determine the evaluation scores of which the number is not less than a preset threshold value in the evaluation score group as target evaluation scores, and determine the target evaluation scores as consensus evaluation scores of all verification nodes on the first model to be evaluated. According to the method, the first training node and the plurality of verification nodes are arranged in the same block chain network, the plurality of verification nodes are used for verifying the model, the consensus mechanism of the block chain network is referred, the consensus evaluation result of each verification node is determined according to the evaluation result of the model trained by each verification node, and the accuracy, reliability and stability of model evaluation can be effectively improved.

Description

Model evaluation method and device

Technical Field

The invention relates to the technical field of computers, in particular to a model evaluation method and a model evaluation device.

Background

With the development of computer technology, artificial intelligence training technology is continuously improved.

In distributed artificial intelligence training, a training node can train a model and can send the trained model to a verification node for verification to evaluate the performance of the model. If the verification is passed, the model performance can be determined to be qualified, and the method can be popularized and applied.

However, in the prior art, errors may occur in the process of verifying the model, so that the evaluation accuracy is low.

Disclosure of Invention

In view of the above problems, the present invention provides a method and an apparatus for evaluating a model, which overcome or at least partially solve the above problems, and the technical solution is as follows:

a model evaluation method is applied to a first verification node, and comprises the following steps:

in a block chain network, a first verification node obtains a first model to be evaluated sent by a first training node, wherein the block chain network comprises a plurality of verification nodes;

the first verification node verifies the first model to be evaluated to obtain a first evaluation score;

in the block chain network, the first verification node respectively obtains second evaluation scores sent by at least one second verification node, wherein the second verification node is different from the first verification node, and the second evaluation scores are obtained by verifying the first model to be evaluated by the second verification node;

the first verification node counts an evaluation score group, and determines evaluation scores of which the number is not less than a preset threshold value in the evaluation score group as target evaluation scores, wherein the evaluation score group comprises the first evaluation scores and the second evaluation scores;

and the first verification node determines the target evaluation score as a consensus evaluation score of the first to-be-evaluated model by the first verification node and each second verification node.

A model evaluation apparatus applied to a first verification node, the apparatus comprising: a first obtaining unit, a first verifying unit, a second obtaining unit, a third obtaining unit, a first statistic unit, a first determining unit and a second determining unit, wherein:

the first obtaining unit is configured to perform: in a block chain network, obtaining a first model to be evaluated sent by a first training node, wherein the block chain network comprises a plurality of verification nodes;

the first authentication unit configured to perform: verifying the first model to be evaluated;

the second obtaining unit is configured to perform: obtaining a first evaluation score;

the third obtaining unit is configured to perform: in the block chain network, respectively obtaining second evaluation scores sent by at least one second verification node, wherein the second verification node is different from the first verification node, and the second evaluation scores are obtained by verifying the first model to be evaluated by the second verification node;

the first statistical unit configured to perform: counting an evaluation score group, wherein the evaluation score group comprises the first evaluation score and each second evaluation score;

the first determination unit is configured to perform: determining the evaluation scores of which the number is not less than a preset threshold value in the evaluation score group as target evaluation scores;

the second determination unit configured to perform: and determining the target evaluation score as a consensus evaluation score of the first verification node and each second verification node on the first model to be evaluated.

In the model evaluation method and apparatus provided in this embodiment, a first verification node may obtain a first model to be evaluated sent by a first training node in a blockchain network, the blockchain network includes a plurality of verification nodes, the first verification node verifies the first model to be evaluated to obtain a first evaluation score, the first verification node obtains at least one second evaluation score sent by a second verification node in the blockchain network, respectively, where the second verification node is different from the first verification node, the second evaluation score is obtained by the second verification node verifying the first model to be evaluated, the first verification node performs statistics on an evaluation score group, and determines the evaluation scores whose number is not less than a preset threshold in the evaluation score group as target evaluation scores, where the evaluation score group includes the first evaluation score and each second evaluation score, and the first verification node determines the target evaluation score as a consensus evaluation score of the first verification node and each second verification node on the first model to be evaluated. According to the method, the first training node and the plurality of verification nodes are arranged in the same block chain network, the models are verified by the plurality of verification nodes, the consensus mechanism of the block chain network is referred, the consensus evaluation result of each verification node is determined according to the evaluation result of each verification node on the trained model of the training node, the situations that when the models are verified by only using a single verification node, the single verification node has problems (such as network failure, power failure or attack by hackers) to cause verification errors and low evaluation accuracy can be avoided, and the accuracy, reliability and stability of model evaluation can be effectively improved.

The foregoing description is only an overview of the technical solutions of the present invention, and the following detailed description of the present invention is provided to enable the technical means of the present invention to be more clearly understood, and to enable the above and other objects, features, and advantages of the present invention to be more clearly understood.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.

FIG. 1 is a flow chart illustrating a first model evaluation method provided by an embodiment of the invention;

FIG. 2 is a flow chart illustrating a second model evaluation method provided by an embodiment of the invention;

FIG. 3 is a flow chart illustrating a fifth method for evaluating a model according to an embodiment of the present invention;

FIG. 4 is a flow chart illustrating a sixth model evaluation method according to an embodiment of the present invention;

FIG. 5 is a schematic structural diagram of a first model evaluation apparatus according to an embodiment of the present invention;

fig. 6 is a schematic structural diagram illustrating a sixth model evaluation apparatus according to an embodiment of the present invention.

Detailed Description

Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention can be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

As shown in fig. 1, the present embodiment proposes a first model evaluation method. The method may be applied to a first authentication node, and may comprise the steps of:

s101, in a block chain network, a first verification node obtains a first model to be evaluated, which is sent by a first training node, and the block chain network comprises a plurality of verification nodes;

the first verification node may be a node used for verifying the trained model to be evaluated.

The first training node may be a node used for training and well training a model to be evaluated. It should be noted that, after the model to be evaluated is trained, the first training node may send the trained model to be evaluated to the verification node for verification.

Specifically, the nodes (e.g., the first training node and the first verification node) in this embodiment may be electronic devices, such as a mobile phone, a desktop computer, a server, and the like.

The first model to be evaluated may be a model to be evaluated that is trained by the first training node. The model type of the first model to be evaluated is not limited, and the model may be a face recognition model, a commodity detection model, or a data calculation model, for example.

It should be noted that, the invention may set the first training node and the plurality of verification nodes (including the first verification node) in the same blockchain network, verify the model to be evaluated by using the plurality of verification nodes, refer to the consensus mechanism of the blockchain network, determine the consensus evaluation result of each verification node on the model to be evaluated according to the evaluation result of each verification node on the model to be evaluated trained by the training node, and avoid the situations of verification error and low evaluation accuracy caused by a problem (such as network disconnection, power outage or hacking attack) of a single verification node when the model to be evaluated is verified by using only the single verification node.

It is understood that the other verification nodes except the first verification node may also be electronic devices (such as a mobile phone, a desktop computer, and a server), and may also be nodes used for verifying the trained model to be evaluated.

In particular, the first authentication node may be an authentication node in a blockchain network. It should be noted that, in this embodiment, the verification nodes other than the first verification node in the blockchain network may be collectively referred to as the second verification node, so as to distinguish the first verification node from other verification nodes in the blockchain network. The first verification node and the second verification node may be the same type of electronic device or different types of electronic devices, and each second verification node may also be the same or different types of electronic devices.

Specifically, the first verification node may receive the first to-be-evaluated model sent by the first training node through the blockchain network.

Optionally, each verification node may send the network address in the blockchain to the first training node first, and then the first training node may send the first model to be evaluated to each network address in the blockchain by broadcasting, that is, to each verification node. It should be noted that each verification node may verify the first model to be evaluated after receiving the first model to be evaluated, and the final verification result of the first model to be evaluated may be determined according to the verification result of each verification node on the first model to be evaluated.

Optionally, as shown in fig. 2, in the second model evaluation method proposed in this embodiment, the step S101 may include steps S201, S202, and S203. Wherein:

s201, a first verification node obtains a model file and a digital signature which are sent by a first training node, wherein the model file comprises a first model to be evaluated;

the model file may be a file in which a first model to be evaluated is stored.

Wherein the digital signature may be a digital signature used by the first training node.

S202, the first verification node verifies whether the digital signature is correct, and if so, the step S203 is executed; otherwise, execution of step S203 is prohibited.

The first verification node may verify the digital signature after receiving the model file and the digital signature sent by the first training node, so as to determine whether the model file is tampered by a third party during transmission, that is, determine whether the model data or structure of the first model to be evaluated is tampered. The first verification node may determine whether to extract and verify the first model to be evaluated from the model file according to a verification result of the digital signature.

S203, the first verification node extracts a first model to be evaluated from the model file.

If the verification result of the digital signature by the first verification node is correct, the model file can be determined not to be tampered in the transmission process, namely the model data or the structure of the first model to be evaluated is determined not to be tampered, and the first model to be evaluated can be extracted from the model file for verification; if the verification result of the digital signature is failure, the model file can be determined to be tampered in the transmission process, that is, the model data or the structure of the first model to be evaluated is determined to be possibly tampered, so that the extraction of the first model to be evaluated from the model file and the verification of the first model to be evaluated can be prohibited, an erroneous verification result generated when the tampered first model to be evaluated is verified can be avoided, and the consumption of verification resources can be reduced.

It should be noted that, if the verification result of the digital signature by the first verification node is failure, a message may be sent to the first training node, so that the first training node may send the model file and the digital signature to the first verification node again.

S102, verifying the first model to be evaluated by the first verification node;

the first verification node may verify the first model to be evaluated after receiving the first model to be evaluated.

It should be noted that the first verification node may verify the first model to be evaluated by testing performance indexes such as accuracy of the first model to be evaluated and Area Under the ROC Curve (AUC).

The accuracy may be an accuracy of a processing result output by the model when the model processes the related service. For example, when the first model to be evaluated is a face recognition model, the accuracy may be the accuracy of the face class output by the first model to be evaluated when face recognition is performed.

S103, the first verification node obtains a first evaluation score;

the first evaluation score may be an evaluation score generated when the first verification node verifies the first model to be evaluated. The first evaluation score may be a performance indicator score, such as accuracy and AUC, of the first model to be evaluated.

Specifically, the first verification node may generate a first evaluation score when verifying the first model to be evaluated.

Optionally, in the third model evaluation method provided in this embodiment, the step S102 may include:

the first verification node imports the first model to be evaluated into a verification program so that the verification program uses the test data set to test the first model to be evaluated;

at this time, the step S103 may include:

the first verification node obtains model test accuracy output by the verification program, and the model test accuracy is determined as a first evaluation score.

The verification program may be a program for verifying the model to be evaluated. The verification program may be set in the first verification node, or may be set in the third-party electronic device, which is not limited in this embodiment of the present invention.

Specifically, after the first verification node obtains the first model to be evaluated, the first model to be evaluated may be imported into a verification program for verification.

When the verification program verifies the first model to be evaluated, the verification program may use the prepared test data set to test the first model to be evaluated. In particular, the verification program may use the test data set to test the performance indicator of the first model to be evaluated.

The model test accuracy may be the accuracy, that is, the accuracy of the processing result output when the related service is processed.

Specifically, the verification program may input the test data in the test data set to the first model to be evaluated, obtain a processing result output by the first model to be evaluated according to the test data, and determine whether the processing result of the first model to be evaluated on the test data is correct by comparing whether the processing result is consistent with the real result, and if not, the test may be recorded as a test failure, and if so, the test may be recorded as a test success. The verification program may calculate the model test accuracy by counting the test results of the first model to be evaluated on each test data in the test data set (for example, the ratio of the number of successful tests to the total number of tests may be determined as the model test accuracy).

Specifically, the first verification node may obtain the model test accuracy rate sent by the verification program after the verification program completes verification of the first model to be evaluated. The first verification node may directly determine the model test accuracy as a first evaluation score for the first model to be evaluated.

S104, in the block chain network, the first verification node respectively obtains second evaluation scores sent by at least one second verification node, wherein the second verification node is different from the first verification node, and the second evaluation scores are obtained by verifying the first model to be evaluated by the second verification node;

it should be noted that each second verification node may also receive the first model to be evaluated, which is broadcast by the first training node through the blockchain network.

After receiving the first model to be evaluated, each second verification node can verify the first model to be evaluated to obtain a corresponding second evaluation score.

In this embodiment, the evaluation scores obtained by the verification nodes except the first verification node for the verification of the first model to be evaluated may be collectively referred to as the second evaluation score. It is understood that each second evaluation score may be the same or different. For example, the second evaluation score 01 obtained by the second verification node 01 through verification of the first model to be evaluated is different from the second evaluation score 02 obtained by the second verification node 02 through verification of the first model to be evaluated.

Specifically, after a certain second verification node completes verification of the first model to be evaluated, the first verification node may receive the evaluation score of the first model to be evaluated, which is sent by the second verification node.

It should be noted that, in the blockchain network, each verification node may receive the evaluation score of the first model to be evaluated, which is sent by other verification nodes. For example, when the blockchain network includes a first verification node, a second verification node 01 and a second verification node 02, the first verification node may receive evaluation scores of the first model to be evaluated sent by the two second verification nodes, the second verification node 01 may receive evaluation scores of the first model to be evaluated sent by the first verification node and the second verification node 02, and the second verification node 02 may receive evaluation scores of the first model to be evaluated sent by the first verification node and the second verification node 01.

It is understood that the transmission process between the first training node and the second verification node with respect to the first model to be evaluated may also be performed by means of a model file and a digital signature.

Optionally, in the fourth model evaluation method provided in this embodiment, step S104 may include:

in a block chain network, a first verification node respectively obtains a second evaluation score and a digital signature sent by at least one second verification node;

the first verification node respectively verifies whether the digital signature sent by each second verification node is correct, and if so, the corresponding second evaluation score can be obtained.

The first verification node may obtain the second evaluation scores and the digital signatures sent by each second verification node in the blockchain network, and may verify the digital signatures sent by each second verification node, respectively, and if the verification result is correct, the corresponding second evaluation scores may be saved, otherwise, the corresponding second evaluation scores may be prohibited from being saved, and a message may be sent to the corresponding second verification nodes so that the corresponding second verification nodes may re-verify the first model to be evaluated. For example, when a second verification node 01 is arranged in the blockchain network, the first verification node may obtain a second evaluation score 01 and a digital signature 01 sent by the second verification node 01, the first verification node may verify the digital signature 01, if the verification result of the digital signature 01 is correct, the first verification node may correspondingly store the second evaluation score 01, if the verification result of the digital signature 01 is incorrect, the first verification node may prohibit storing the second evaluation score 01, and send a message to the second verification node 01 so that the second verification node 01 re-verifies the first model to be evaluated.

It should be noted that, after a certain second verification node verifies the first model to be evaluated again, the first verification node may obtain a new second evaluation score and a new digital signature sent by the second verification node, and verify the new digital signature again until the first verification node verifies that the verification result of the digital signature of the second verification node is correct, and the first verification node may store the corresponding evaluation score sent by the second verification node.

If the verification result of the digital signature of a certain second verification node continuously has errors for multiple times, the second verification node can be prohibited from being reused to continuously verify the first model to be evaluated, the second verification node is prohibited from being used to participate in consensus evaluation of the first model to be evaluated, and technicians can be reminded that the second verification node has problems.

S105, counting an evaluation score group by the first verification node, wherein the evaluation score group comprises a first evaluation score and each second evaluation score;

the evaluation score group may be a set of the first evaluation score and each second evaluation score.

Specifically, the first verification node may count the number of each evaluation score in the evaluation score group after all the evaluation scores sent by other verification nodes are received.

S106, determining the evaluation scores of which the number is not less than a preset threshold value in the evaluation score group as target evaluation scores by the first verification node;

specifically, the first verification node may rank the number of each statistical evaluation score, and record the highest number value and the corresponding evaluation score. Then, the first verification node may compare the recorded highest quantity value with a preset threshold, and if the highest quantity value is not less than the preset threshold, the first verification node may determine an evaluation score corresponding to the highest quantity value as a target evaluation score; otherwise, the first verification point may consider that an error (e.g., a verification error caused by a network outage, a power outage, a hacking attack, or the like) occurs in the verification of the first model to be evaluated by one or some verification nodes, may prohibit determining the final evaluation score of the first model to be evaluated, avoid that each verification node performs an error consensus on the evaluation scores, and may return to execute step S101, re-receive the first model to be evaluated and perform verification again, where other verification nodes in the time zone block chain network may also re-receive and verify the first model to be evaluated, and re-perform consensus on the evaluation scores.

It should be noted that the specific value of the preset threshold is not limited in the present invention. For example, the preset threshold may be a certain proportion (e.g., 50%) of the number of verification nodes participating in the scoring.

And S107, the first verification node determines the target evaluation score as a consensus evaluation score of the first verification node and each second verification node on the first model to be evaluated.

Specifically, after the target evaluation score is obtained, the target evaluation score can be determined as the consensus evaluation score of each verification node in the block chain network.

It is understood that each verification node in the blockchain network can be used as a first verification node to perform the method shown in fig. 1, and determine the consensus evaluation score for the first model to be evaluated.

It should be further noted that, in the present invention, the first training node and the plurality of verification nodes are arranged in the same blockchain network, the plurality of verification nodes are used to verify the model, and a consensus mechanism of the blockchain network is referenced, and a consensus evaluation result of each verification node is determined according to an evaluation result of each verification node on the model trained by the training node, so that a situation that when the model is verified only by using a single verification node, a single verification node has a problem (e.g., network failure, power failure or hacking attack) to cause a verification error and a low evaluation accuracy can be avoided, accuracy, reliability and stability of model evaluation can be effectively improved, and fairness of model evaluation can also be improved.

In the model evaluation method provided in this embodiment, a first verification node may obtain a first model to be evaluated sent by a first training node in a blockchain network, the blockchain network includes a plurality of verification nodes, the first verification node verifies the first model to be evaluated to obtain a first evaluation score, the first verification node obtains second evaluation scores sent by at least one second verification node in the blockchain network, respectively, where the second verification nodes are different from the first verification node, the second evaluation scores are obtained by the second verification node verifying the first model to be evaluated, the first verification node performs statistics on an evaluation score group, and determines evaluation scores whose number is not less than a preset threshold value in the evaluation score group as target evaluation scores, where the evaluation score group includes the first evaluation scores and the second evaluation scores, the first verification node evaluates the target evaluation scores, and determining consensus evaluation scores of the first verification node and each second verification node on the first model to be evaluated. According to the method, the first training node and the plurality of verification nodes are arranged in the same block chain network, the models are verified by the plurality of verification nodes, the consensus mechanism of the block chain network is referred, the consensus evaluation result of each verification node is determined according to the evaluation result of each verification node on the trained model of the training node, the situations that when the models are verified by only using a single verification node, the single verification node has problems (such as network failure, power failure or attack by hackers) to cause verification errors and low evaluation accuracy can be avoided, and the accuracy, reliability and stability of model evaluation can be effectively improved.

Based on the steps shown in fig. 2, as shown in fig. 3, the present embodiment proposes a fifth model evaluation method. In the method, a first private key and a first public key which are matched with each other are stored in a first training node, a digital signature is obtained by encrypting a first hash value by the first training node through the first private key, and the first hash value is obtained by calculating the hash value of a model file through the first training node. At this time, the step S202 may include:

s301, the first verification node decrypts the digital signature by using a first public key to obtain a first hash value, wherein the first public key is issued to the first verification node by the first training node;

the first training node may perform hash value calculation on the model file by using a hash function to obtain a first hash value. Then, the first training node may encrypt the first hash value using the first private key to obtain an encrypted first hash value, determine the encrypted first hash value as a digital signature, and then send the model file and the determined digital signature to the first verification node.

Specifically, after obtaining the model file and the digital signature sent by the first training node, the first verification node may decrypt the digital signature using the first public key issued by the first training node, so as to obtain the first hash value.

S302, the first verification node calculates the hash value of the model file to obtain a second hash value;

the first verification node may perform hash value calculation on the model file by using the hash function after obtaining the model file and the digital signature sent by the first training node, so as to obtain a second hash value calculation.

S303, the first verification node compares whether the second hash value is consistent with the first hash value, if so, the step S304 is executed; otherwise, execution of step S304 is prohibited to reduce consumption of the authentication resource.

It should be noted that the first verification node may determine whether the model file is tampered with by a third party in the transmission process by comparing whether the second hash value is consistent with the first hash value, that is, determine whether the model data or the result of the first model to be evaluated in the model file is tampered with by the third party, and determine the verification result of the digital signature.

S304, the first verification node determines that the verification result is correct.

If the comparison result of the second hash value and the first hash value is consistent, the first verification node can determine that the model file is not tampered by a third party in the transmission process, namely, the model data or the result of the first model to be evaluated in the model file is not tampered by the third party, and can determine that the verification result of the digital signature sent by the first training node is correct.

If the comparison result of the second hash value and the first hash value is inconsistent, the first verification node may determine that the model file is tampered by a third party in the transmission process, that is, it is determined that the model data or the structure of the first model to be evaluated is possibly tampered, and it may be determined that the verification result of the digital signature sent by the first training node is an error.

It should be noted that, if the first verification node determines that the verification result of the digital signature sent by the first training node is correct, the first model to be evaluated may be extracted from the model file sent by the first training node and verified; if the first verification node determines that the verification result of the digital signature sent by the first training node is wrong, the first verification node can prohibit the first model to be evaluated from being extracted from the model file sent by the first training node, prohibit the first model to be evaluated from being verified, reduce the consumption of verification resources, and avoid the wrong verification result generated when the tampered first model to be evaluated is verified.

The model evaluation method provided in this embodiment may determine whether the model file is tampered during the transmission process by verifying the digital signature of the first training node, extract the model from the model file for verification when the verification result is correct, prohibit extracting the first model to be evaluated from the model file transmitted by the first training node, prohibit verifying the first model to be evaluated, reduce consumption of verification resources, and avoid an erroneous verification result generated when the tampered first model to be evaluated is verified when the verification result is erroneous.

Based on the steps shown in fig. 1, as shown in fig. 4, the present embodiment proposes a sixth model evaluation method. In the method, a plurality of training nodes are included in the blockchain network, and the method may further include the following steps:

s401, a first verification node obtains a consensus evaluation score of at least one second model to be evaluated, wherein each second model to be evaluated is well trained by a training node in a block chain network;

it can be understood that the first verification node may use the method shown in fig. 1 to obtain consensus evaluation scores of models to be evaluated trained by different training nodes, and determine an optimal model from the models to be evaluated according to the consensus evaluation scores of the models to be evaluated.

A plurality of training nodes (including a first training node) may be disposed in the blockchain network. In this embodiment, all training nodes except the first training node may be collectively referred to as a second training node, so as to distinguish the first training node from other training nodes. The training nodes may be the same type of electronic device or different types of electronic devices.

Specifically, each training node can train different types of models to obtain corresponding trained models to be evaluated. For example, when a first training node and a second training node 01 are arranged in the block chain network, for a model for face recognition, the first training node may be used for training a neural network model to obtain a trained neural network model to be evaluated; the second training node 01 may be configured to train the deep learning model to obtain a trained deep learning model to be evaluated. Of course, each training node may also train the same type of model.

Specifically, the model trained by each training node can be used as the model to be evaluated.

It should be noted that, in this embodiment, all models to be evaluated except the first model to be evaluated may be collectively referred to as the second evaluation model, so as to distinguish the first model to be evaluated from other models to be evaluated.

The first model to be evaluated and the second model to be evaluated may be the same type or different types of models, for example, both the first model to be evaluated and each of the second model to be evaluated may be models for performing face recognition; the second evaluation models may be the same type or different types of models, for example, the second evaluation model 01 and the second evaluation model 02 are different types of models.

Specifically, after training the model to be evaluated, each training node may broadcast the trained model to be evaluated to each verification node, and each verification node may verify the model to be evaluated sent by each training node to obtain a corresponding evaluation score. Then, the first verification node may receive the evaluation scores of the models to be evaluated sent by each verification node, respectively count the evaluation scores of the same model to be evaluated, and respectively determine consensus evaluation scores of the models to be evaluated. For example, the first verification node may receive the evaluation scores of the first model to be evaluated and the second model 01 to be evaluated, which are sent by each verification node, count the evaluation scores of the first model to be evaluated by each verification node, determine the consensus evaluation score of the first model to be evaluated, count the evaluation scores of the second model 01 to be evaluated by each verification node, and determine the consensus evaluation score of the second model 01 to be evaluated.

S402, sequencing the consensus evaluation scores of the first model to be evaluated and the second models to be evaluated by the first verification node, and determining the highest consensus evaluation score in the sequenced consensus evaluation scores as a target consensus evaluation score;

it should be noted that, when each model to be evaluated is a model for processing the same service, the present invention may determine the optimal model for processing the service from each model to be evaluated according to the consensus evaluation score of each model to be evaluated.

Specifically, the first verification node may rank the consensus evaluation scores of the models to be evaluated, determine the consensus evaluation score with the highest score, and determine the consensus evaluation score with the highest score as the target consensus evaluation score.

And S403, determining the model to be evaluated corresponding to the target consensus evaluation score as the optimal model by the first verification node.

Specifically, the consensus evaluation score with the highest score, that is, the model to be evaluated corresponding to the target consensus evaluation score, may be determined as the optimal model.

Optionally, after determining the optimal model, the first verification node may publish the optimal model to each node through broadcasting in the blockchain network, so as to promote and apply the optimal model, and improve processing efficiency of corresponding services.

Optionally, when the first verification node issues the optimal model, the first verification node may carry the consensus evaluation score of the optimal model and a digital signature of the evaluation score of each verification node on the optimal model.

Optionally, after determining the optimal model, the first verification node may award a reward to a training node that trained the optimal model.

In order to better illustrate the implementation process of the method shown in fig. 4, the invention proposes example 1 for illustration.

Example 1: for face recognition service, in order to obtain a face recognition model with good recognition performance, a technician can input three different types of face recognition models into a target block chain network for training and evaluation.

Wherein, be provided with three training nodes in the target block chain network: first training node, second training node 01 and second training node 02 are provided with ten verification nodes: first authentication node, second authentication node 01, second authentication node 02 … … second authentication node 08 and second authentication node 09.

Specifically, the technician may input three different types of face recognition models to the first training node, the second training node 01, and the second training node 02, respectively, for training.

After the first training node trains the face recognition model, a trained first model to be evaluated can be obtained, and the first model to be evaluated can be broadcasted to ten verification nodes for verification; after the face recognition model is trained by the second training node 01, a trained second model 01 to be evaluated can be obtained, and the second model 01 to be evaluated can be broadcasted to ten verification nodes for verification; after the face recognition model is trained by the second training node 02, the trained second model 02 to be evaluated can be obtained, and the second model 02 to be evaluated can be broadcasted to ten verification nodes for verification.

Then, the first verification node can respectively verify the first model to be evaluated, the second model 01 to be evaluated and the third model to be evaluated to obtain corresponding evaluation scores; the second verification node 01 can respectively verify the first model to be evaluated, the second model 01 to be evaluated and the second model 02 to be evaluated to obtain corresponding evaluation scores; the second verification node 02 may respectively verify the first model to be evaluated, the second model 01 to be evaluated, and the second model 02 to be evaluated to obtain corresponding evaluation scores; by analogy, the second verification nodes 03 to 09 can respectively verify the first model to be evaluated, the second model 01 to be evaluated and the second model 02 to be evaluated to obtain corresponding evaluation scores.

Then, the first verification node may obtain the evaluation scores of the three models to be evaluated, which are sent by the nine second verification nodes, respectively. The first verification node may count the evaluation scores of the nine verification nodes on the first model to be evaluated, determine the evaluation scores of which the number is not less than 5 (5 is the preset threshold) as target evaluation scores, and determine the target evaluation scores as consensus evaluation scores of each verification node on the first model to be evaluated; the first verification node may count the evaluation scores of the nine verification nodes on the second model 01 to be evaluated, determine the evaluation scores of which the number is not less than 5 (5 is the preset threshold) as target evaluation scores, and determine the target evaluation scores as consensus evaluation scores of each verification node on the second model 01 to be evaluated; by analogy, the first verification node can determine the consensus evaluation scores of the verification nodes on the first model to be evaluated, the second model to be evaluated 01 and the second model to be evaluated 02.

Then, the first verification node may sequence the consensus evaluation scores of the models to be evaluated, that is, sequence the consensus evaluation scores of the first model to be evaluated, the second model to be evaluated 01, and the second model to be evaluated 02, determine the consensus evaluation score with the highest score among the sequenced consensus evaluation scores as a target consensus evaluation score, and determine the model to be evaluated corresponding to the target consensus evaluation score as the optimal face recognition model.

And then, the first verification node can issue the optimal face recognition model to each node to popularize and apply the optimal face recognition model. Meanwhile, the first verification node can reward the training node for training the optimal face recognition model so as to encourage the training node to continue to invest in the model training.

The model evaluation method provided by the embodiment can perform consensus evaluation on models to be evaluated trained by different training nodes, determine the optimal model from the models to be evaluated according to the consensus evaluation score of each model to be evaluated, and distribute the optimal model to each node, so that the optimal model is popularized and applied, and the processing efficiency of related services is improved.

Based on the steps shown in fig. 1, the present embodiment proposes a first model evaluation device as shown in fig. 5. The apparatus may be applied to a first authentication node, and may include: a first obtaining unit 101, a first verifying unit 102, a second obtaining unit 103, a third obtaining unit 104, a first counting unit 105, a first determining unit 106, and a second determining unit 107, wherein:

a first obtaining unit 101 configured to perform: in a block chain network, obtaining a first model to be evaluated sent by a first training node, wherein the block chain network comprises a plurality of verification nodes;

Optionally, in the second model evaluation device provided in this embodiment, the first obtaining unit 101 may include: a fourth obtaining unit, a second verifying unit and a first extracting unit, wherein:

a fourth obtaining unit configured to perform: obtaining a model file and a digital signature sent by a first training node, wherein the model file comprises a first model to be evaluated;

a second verification unit configured to perform: verifying whether the digital signature is correct, and if so, triggering a first extraction unit;

a first extraction unit configured to perform: and extracting a first model to be evaluated from the model file.

The model file may be a file in which a first model to be evaluated is stored.

If the verification result of the digital signature by the first verification node is correct, the model file can be determined not to be tampered in the transmission process, and a first model to be evaluated can be extracted from the model file for verification; if the verification result of the digital signature is failure, the model file can be determined to be tampered in the transmission process, so that the first model to be evaluated can be prohibited from being extracted from the model file and the first model to be evaluated can be prohibited from being verified, the wrong verification result generated when the tampered first model to be evaluated is verified can be avoided, and the consumption of verification resources can be reduced.

A first verification unit 102 configured to perform: verifying the first model to be evaluated;

It should be noted that the first verification node may verify the first model to be evaluated by testing performance indexes such as accuracy and AUC of the first model to be evaluated.

The accuracy may be an accuracy of a processing result output by the model when the model processes the related service.

A second obtaining unit 103 configured to perform: obtaining a first evaluation score;

Optionally, in a third model evaluation device proposed in this embodiment, the first verification unit 102 is configured to perform: importing the first model to be evaluated into a verification program, so that the verification program uses a test data set to test the first model to be evaluated;

a second obtaining unit 103 configured to perform: and obtaining the model test accuracy output by the verification program, and determining the model test accuracy as a first evaluation score.

Specifically, the verification program may input the test data in the test data set to the first model to be evaluated, obtain a processing result output by the first model to be evaluated according to the test data, and determine whether the processing result of the first model to be evaluated on the test data is correct by comparing whether the processing result is consistent with the real result, and if not, the test may be recorded as a test failure, and if so, the test may be recorded as a test success. The verification program can calculate the model test accuracy by counting the test results of the first model to be evaluated on each test data in the test data set.

A third obtaining unit 104 configured to perform: in a block chain network, respectively obtaining second evaluation scores sent by at least one second verification node, wherein the second verification nodes are different from the first verification nodes, and the second evaluation scores are obtained by verifying the first model to be evaluated by the second verification nodes;

In this embodiment, the evaluation scores obtained by the verification nodes except the first verification node for the verification of the first model to be evaluated may be collectively referred to as the second evaluation score. It is understood that each second evaluation score may be the same or different.

It should be noted that, in the blockchain network, each verification node may receive the evaluation score of the first model to be evaluated, which is sent by other verification nodes.

Optionally, in the fourth model evaluation apparatus provided in this embodiment, the third obtaining unit 104 includes: a sixth obtaining unit, a third verifying unit, and a seventh obtaining unit, wherein:

a sixth obtaining unit configured to perform: in a block chain network, respectively obtaining a second evaluation score and a digital signature sent by at least one second verification node;

a third verification unit configured to perform: respectively verifying whether the digital signatures sent by the second verification nodes are correct, and if so, triggering a seventh obtaining unit;

the first verification node may obtain the second evaluation scores and the digital signatures sent by each second verification node in the blockchain network, and may verify the digital signatures sent by each second verification node, respectively, and if the verification result is correct, the corresponding second evaluation scores may be saved, otherwise, the corresponding second evaluation scores may be prohibited from being saved, and a message may be sent to the corresponding second verification nodes so that the corresponding second verification nodes may re-verify the first model to be evaluated.

A first statistical unit 105 configured to perform: counting an evaluation score group, wherein the evaluation score group comprises a first evaluation score and each second evaluation score;

A first determining unit 106 configured to perform: determining the evaluation scores of which the number is not less than a preset threshold value in the evaluation score group as target evaluation scores;

specifically, the first verification node may rank the number of each statistical evaluation score, and record the highest number value and the corresponding evaluation score. Then, the first verification node may compare the recorded highest quantity value with a preset threshold, and if the highest quantity value is not less than the preset threshold, the first verification node may determine an evaluation score corresponding to the highest quantity value as a target evaluation score; otherwise, the first verification point may consider that the verification of the first model to be evaluated by one or some verification nodes is wrong, and may prohibit determining the final evaluation score of the first model to be evaluated, thereby avoiding that each verification node performs wrong consensus on the evaluation scores.

A second determining unit 107 configured to perform: and determining the target evaluation score as a consensus evaluation score of the first verification node and each second verification node on the first model to be evaluated.

The model evaluation device provided by this embodiment can verify the model by using a plurality of verification nodes by setting the first training node and the plurality of verification nodes in the same block chain network, and by referring to the consensus mechanism of the block chain network, determine the consensus evaluation result of each verification node according to the evaluation result of the model trained by each verification node, thereby avoiding the situations of verification error and low evaluation accuracy caused by the occurrence of a problem in a single verification node when the model is verified by only using a single verification node, and effectively improving the accuracy, reliability and stability of model evaluation.

Based on the second model evaluation device, the present embodiment proposes a fifth model evaluation device. In a fifth model evaluation device, a first training node stores a first private key and a first public key which are matched with each other, a digital signature is obtained by encrypting a first hash value by using the first private key by the first training node, and the first hash value is obtained by calculating a hash value of a model file by the first training node;

a second verification unit comprising: decryption unit, fifth acquisition unit, computational element, sixth unit, comparison unit and third determining unit, wherein:

a decryption unit configured to perform: decrypting the digital signature by using a first public key, wherein the first public key is issued to a first verification node by a first training node;

a fifth obtaining unit configured to perform: obtaining a first hash value;

a computing unit configured to perform: carrying out hash value calculation on the model file;

a sixth unit configured to perform: obtaining a second hash value;

an alignment unit configured to perform: comparing whether the second hash value is consistent with the first hash value or not, and if so, triggering a third determining unit;

a third determination unit configured to perform: and determining that the verification result is correct.

The model evaluation device provided in this embodiment may verify the digital signature of the first training node to determine whether the model file is tampered in the transmission process, extract the model from the model file for verification when the verification result is correct, prohibit extracting the first model to be evaluated from the model file transmitted by the first training node when the verification result is incorrect, prohibit verifying the first model to be evaluated, reduce consumption of verification resources, and avoid an incorrect verification result generated when the tampered first model to be evaluated is verified.

Based on the schematic structural diagram shown in fig. 1, as shown in fig. 6, the present embodiment proposes a sixth model evaluation device. In the apparatus, the blockchain network includes a plurality of training nodes, and the apparatus may further include: an eighth obtaining unit 201, a sorting unit 202, a fourth determining unit 203, and a fifth determining unit 204, wherein:

an eighth obtaining unit 201 configured to perform: obtaining consensus evaluation scores of at least one second model to be evaluated, wherein each second model to be evaluated is trained by a training node in the block chain network;

it can be understood that the first verification node may first obtain consensus evaluation scores of the to-be-evaluated models trained by different training nodes, and then determine the optimal model from the to-be-evaluated models according to the consensus evaluation scores of the to-be-evaluated models.

Specifically, each training node can train different types of models to obtain corresponding trained models to be evaluated.

The first model to be evaluated and the second model to be evaluated can be the same type or different types of models.

Specifically, after training the model to be evaluated, each training node may broadcast the trained model to be evaluated to each verification node, and each verification node may verify the model to be evaluated sent by each training node to obtain a corresponding evaluation score. Then, the first verification node may receive the evaluation scores of the models to be evaluated sent by each verification node, respectively count the evaluation scores of the same model to be evaluated, and respectively determine consensus evaluation scores of the models to be evaluated.

A sorting unit 202 configured to perform: sequencing consensus evaluation scores of the first model to be evaluated and each second model to be evaluated;

A fourth determination unit 203 configured to perform: determining the highest consensus evaluation score in the ranked consensus evaluation scores as a target consensus evaluation score;

A fifth determining unit 204 configured to perform: and determining the model to be evaluated corresponding to the target consensus evaluation score as an optimal model.

The model evaluation device provided by this embodiment can perform consensus evaluation on models to be evaluated trained by different training nodes, determine an optimal model from the models to be evaluated according to consensus evaluation scores of the models to be evaluated, and distribute the optimal model to each node, so as to popularize and apply the optimal model and improve the processing efficiency of related services.

It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in the process, method, article, or apparatus that comprises the element.

The above are merely examples of the present invention, and are not intended to limit the present invention. Various modifications and alterations to this invention will become apparent to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A model evaluation method applied to a first verification node, the method comprising:

in a block chain network, the first verification node obtains a first model to be evaluated sent by a first training node, and the block chain network comprises a plurality of verification nodes;

2. The method of claim 1, wherein the first verification node obtains a first to-be-evaluated model sent by a first training node, comprising:

the first verification node obtains a model file and a digital signature which are sent by the first training node, wherein the model file comprises the first model to be evaluated;

and the first verification node verifies whether the digital signature is correct, and if so, the first model to be evaluated is extracted from the model file.

3. The method according to claim 2, wherein the first training node stores a first private key and a first public key which are matched with each other, the digital signature is obtained by encrypting a first hash value by the first training node by using the first private key, and the first hash value is obtained by performing hash value calculation on the model file by the first training node;

the first verification node verifying whether the digital signature is correct comprises:

the first verification node decrypts the digital signature by using the first public key to obtain the first hash value, wherein the first public key is issued to the first verification node by the first training node;

the first verification node calculates a hash value of the model file to obtain a second hash value;

and the first verification node compares whether the second hash value is consistent with the first hash value, and if so, the verification result is determined to be correct.

4. The method of claim 1, wherein the first authentication node obtains the second evaluation scores sent by at least one second authentication node respectively, and comprises:

the first verification node respectively obtains a second evaluation score and a digital signature sent by at least one second verification node;

and the first verification node respectively verifies whether the digital signature sent by each second verification node is correct, and if so, corresponding second evaluation scores are obtained.

5. The method of claim 1, wherein the first verification node verifying the first to-be-evaluated model comprises:

the first verification node leads the first model to be evaluated into a verification program, so that the verification program uses a test data set to test the first model to be evaluated;

the obtaining a first rating score comprises:

and the first verification node obtains the model test accuracy output by the verification program and determines the model test accuracy as the first evaluation score.

6. The method of any of claims 1 to 5, wherein a plurality of training nodes are included in the blockchain network, the method further comprising:

the first verification node obtains a consensus evaluation score of at least one second model to be evaluated, wherein each second model to be evaluated is trained by the training node in the block chain network;

the first verification node ranks the consensus evaluation scores of the first model to be evaluated and each second model to be evaluated, and determines the highest consensus evaluation score in the ranked consensus evaluation scores as a target consensus evaluation score;

and the first verification node determines the model to be evaluated corresponding to the target consensus evaluation score as an optimal model.

7. A model evaluation apparatus applied to a first verification node, the apparatus comprising: a first obtaining unit, a first verifying unit, a second obtaining unit, a third obtaining unit, a first statistic unit, a first determining unit and a second determining unit, wherein:

8. The apparatus of claim 7, wherein the first obtaining unit comprises: a fourth obtaining unit, a second verifying unit and a first extracting unit, wherein:

the fourth obtaining unit is configured to perform: obtaining a model file and a digital signature sent by the first training node, wherein the model file comprises the first model to be evaluated;

the second verification unit configured to perform: verifying whether the digital signature is correct, and if so, triggering the first extraction unit;

the first extraction unit configured to perform: and extracting the first model to be evaluated from the model file.

9. The apparatus according to claim 8, wherein the first training node stores a first private key and a first public key that match, the digital signature is obtained by the first training node encrypting a first hash value by using the first private key, and the first hash value is obtained by the first training node performing hash value calculation on the model file;

the second authentication unit includes: decryption unit, fifth acquisition unit, computational element, sixth unit, comparison unit and third determining unit, wherein:

the decryption unit configured to perform: decrypting the digital signature using the first public key, the first public key being issued by the first training node to the first verification node;

the fifth obtaining unit configured to perform: obtaining the first hash value;

the computing unit configured to perform: carrying out hash value calculation on the model file;

the sixth unit configured to perform: obtaining a second hash value;

the alignment unit is configured to perform: comparing whether the second hash value is consistent with the first hash value or not, and if so, triggering the third determining unit;

the third determination unit is configured to perform: and determining that the verification result is correct.

10. The apparatus of claim 7, wherein the third obtaining unit comprises: a sixth obtaining unit, a third verifying unit, and a seventh obtaining unit, wherein:

the sixth obtaining unit is configured to perform: respectively obtaining a second evaluation score and a digital signature sent by at least one second verification node in the block chain network;

the third verification unit configured to perform: respectively verifying whether the digital signatures sent by the second verification nodes are correct or not, and if so, triggering the seventh obtaining unit;

the seventh obtaining unit configured to perform: a corresponding second evaluation score is obtained.