CN115035133A

CN115035133A - Model training method, image segmentation method and related device

Info

Publication number: CN115035133A
Application number: CN202210613278.0A
Authority: CN
Inventors: 黄烨翀; 陈翼男; 张少霆
Original assignee: Shanghai Sensetime Intelligent Technology Co Ltd
Current assignee: Shanghai Shangtang Shancui Medical Technology Co ltd
Priority date: 2022-05-31
Filing date: 2022-05-31
Publication date: 2022-09-09

Abstract

The application discloses a model training method, an image segmentation method and a related device, comprising the following steps: processing the sample image by using a first processing model to obtain a first segmentation result, and processing the sample image by using a second processing model to obtain a second segmentation result; generating first labeling information by using the first segmentation result, and obtaining second labeling information by using the second segmentation result; parameters of the first process model are adjusted based on the first segmentation result and the first reference information, and parameters of the second process model are adjusted based on the second segmentation result and the second reference information. According to the scheme, the two processing models generate labeling information for each other, so that the two models are supervised interactively during training, label-free data can be fully utilized to train the models, the possibility of overfitting can be reduced compared with the training of a single model, and the generalization capability of the models is improved.

Description

Model training method, image segmentation method and related device

Technical Field

The present application relates to the field of image processing technologies, and in particular, to a model training method, an image segmentation method, and a related apparatus.

Background

The medical image segmentation is a complex and key step in the medical image processing and analyzing process, and aims to segment parts with certain special meanings in the medical image, provide reliable basis for clinical diagnosis and pathology research and assist doctors in giving more accurate diagnosis.

Medical images have high complexity and lack simple linear features, and in addition, the accuracy of the segmentation result is influenced by factors such as partial volume effect, gray level nonuniformity, artifacts, proximity of gray levels among different soft tissues, individual difference of different patients, and different manifestations of the same disease. In recent years, medical image segmentation algorithms based on deep learning are gradually developed, but labels of medical images are difficult to obtain, so that when a model is trained, only part of data is often labeled, and the model training is greatly influenced.

Disclosure of Invention

The application at least provides a model training method, an image segmentation method and a related device.

The application provides a training method of a segmentation model, which comprises the following steps: processing the sample image by using a first processing model to obtain a first segmentation result, and processing the sample image by using a second processing model to obtain a second segmentation result; generating first labeling information based on the first segmentation result, and generating second labeling information based on the second segmentation result; and adjusting parameters of the first processing model based on the first division result and the first reference information, and adjusting parameters of the second processing model based on the second division result and the second reference information, wherein the first reference information comprises second labeling information, and the second reference information comprises first labeling information.

Therefore, the two processing models generate labeling information for each other, so that the two models are supervised interactively during training, the models can be trained by fully utilizing label-free data, the possibility of overfitting can be reduced compared with the training of a single model, and the generalization capability of the models is improved.

Wherein, processing the sample image by using the first processing model to obtain a first segmentation result comprises: performing target segmentation on the sample image by using a first processing model to obtain a first segmentation result about a target object; and processing the sample image by using a second processing model to obtain a second segmentation result, wherein the second segmentation result comprises: performing target segmentation on the sample image by using a second processing model to obtain a second segmentation result about the target object; or performing distance regression on the sample image by using a second processing model to obtain a predicted distance result, wherein the predicted distance result represents the distance between each pixel point in the sample image and a predicted boundary, and the predicted boundary is the boundary of a predicted region corresponding to the target object in the sample image; based on the predicted distance result, a second segmentation result is obtained with respect to the target object.

Therefore, the first processing model and the second processing model adopt the same processing mode for the image, or different processing modes can be adopted between the two models, and if different processing modes are adopted, the information of different angles of the image can be learned between the two models and are complemented, so that the generalization capability of the models is improved.

The prediction distance result is a prediction distance graph, and the value of each pixel point in the prediction distance graph is used for representing the distance between the corresponding pixel point in the sample image and the prediction boundary; based on the predicted distance result, obtaining a second segmentation result about the target object, comprising: and performing binarization processing on the predicted distance map by using a preset binarization mode to obtain a second segmentation result related to the target object.

Therefore, by performing binarization processing on the predicted distance map obtained by distance regression, the region corresponding to the target object and the other region can be roughly distinguished from the predicted distance map, so that a second division result can be obtained for use in subsequent adjustment of the model parameters.

Wherein the first segmentation result and the second segmentation result both have gradient attributes; generating first annotation information based on the first segmentation result, including: acquiring a copy of the first segmentation result, and performing gradient isolation on the copy of the first segmentation result to obtain first labeling information without gradient attribute; generating second labeling information based on the second segmentation result, including: and acquiring a copy of the second segmentation result, and performing gradient isolation on the copy of the second segmentation result to obtain second labeling information without gradient attributes.

Therefore, the first labeling information is generated through the first segmentation result, and the second labeling information is generated through the second segmentation result, so that interactive supervision between the two models is realized, the two models can be complemented, the two models can mutually absorb the advantages of the other model, the generalization capability of the models is improved, and the labeling information can be generated through the gradient isolation operation by utilizing the segmentation information so as to be used for realizing the subsequent interactive supervision.

The first reference information and the second reference information also comprise real labeling information of the sample image; adjusting parameters of the first process model based on the first segmentation result and the first reference information includes: adjusting parameters of the first processing model based on a difference between the first segmentation result and the second labeling information and a difference between the first segmentation result and the real labeling information; adjusting parameters of the second process model based on the second segmentation result and the second reference information comprises: and adjusting the parameters of the second processing model based on the difference between the second segmentation result and the first labeling information and the difference between the second segmentation result and the real labeling information.

Therefore, by adjusting the parameters of the processing model by the difference between the segmentation result and the annotation information and the true annotation information, the model can learn the true annotation information and the information in the annotation information generated based on the segmentation result, thereby performing segmentation more accurately.

Wherein, before adjusting the parameters of the second processing model based on the difference between the second segmentation result and the first annotation information and the difference between the second segmentation result and the real annotation information, the method further comprises: under the condition that the second processing model processes the sample image to output a predicted distance result, performing distance transformation based on the real annotation information to obtain real distance information; the prediction distance result is used for obtaining a second segmentation result and representing the distance between each pixel point in the sample image and the prediction boundary, the prediction boundary is the boundary of a prediction region corresponding to the target object in the sample image, the real distance information represents the distance between each pixel point in the sample image and a real boundary, and the real boundary is the boundary of the real region corresponding to the target object in the sample image; adjusting parameters of the second processing model based on a difference between the second segmentation result and the first annotation information and a difference between the second segmentation result and the real annotation information, including: and adjusting parameters of the second processing model based on the difference between the second segmentation result and the first labeled information, the difference between the second segmentation result and the real labeled information, and the difference between the predicted distance result and the real distance information.

Therefore, the real labeling information is converted into the real distance information to be compared with the predicted distance result, and the parameters of the second processing model are adjusted through the difference between the real labeling information and the predicted distance result, so that the second processing model can learn from the real distance information, and the predicted distance result is closer to the real distance information.

The real distance information is a real distance map, and the value of each pixel point in the real distance map is used for representing the distance between the corresponding pixel point in the sample image and the real boundary; carrying out distance transformation based on the real labeling information to obtain real distance information, wherein the distance transformation comprises the following steps: determining a real area in the sample image as a foreground and determining the rest areas as a background based on the real labeling information; respectively carrying out distance conversion and normalization processing on the foreground and the background in the sample image to obtain a foreground distance map and a background distance map; and performing subtraction processing by using the foreground distance map and the background distance map to obtain a real distance map.

Therefore, the real distance graph can be obtained by performing distance transformation and normalization processing on the foreground and the background and subtracting the foreground and the background, and the real distance graph is used as real distance information for parameter adjustment of the second processing model in the follow-up process, so that the generalization capability of the second processing model is improved.

The application also provides an image segmentation method, which comprises the following steps: acquiring a target image; processing the target image by using the processing model to obtain a segmentation result about the target object; the processing model is a first processing model or a second processing model obtained by training by using any one of the model training methods.

Therefore, the first processing model or the second processing model is obtained by training using the model training method, and the target image is processed, so that the target object can be accurately segmented.

The processing model is a second processing model, and the second processing model is used for carrying out distance regression on the target image; processing the target image with the processing model to obtain a segmentation result about the target object, comprising: performing distance regression on the target image by using the processing model to obtain a predicted distance result, wherein the predicted distance result represents the distance between each pixel point in the target image and a predicted boundary, and the predicted boundary is the boundary of a predicted region corresponding to a target object in the target image; based on the predicted distance result, a segmentation result is obtained with respect to the target object.

Therefore, the second processing model can accurately segment the target object by obtaining an accurate segmentation result of the target object on the basis of the distance regression.

The present application further provides a model training device, comprising: the processing module is used for processing the sample image by utilizing a first processing model to obtain a first segmentation result and processing the sample image by utilizing a second processing model to obtain a second segmentation result; the labeling module is used for generating first labeling information based on the first segmentation result and generating second labeling information based on the second segmentation result; the adjusting module is used for adjusting parameters of the first processing model based on the first segmentation result and the first reference information, and adjusting parameters of the second processing model based on the second segmentation result and the second reference information, wherein the first reference information comprises second labeling information, and the second reference information comprises the first labeling information.

The present application also provides an image segmentation apparatus, including: the device comprises an acquisition module and a processing module, wherein the acquisition module is used for acquiring a target image; the processing module is used for processing the target image by using the processing model to obtain a segmentation result about the target object; the processing model is a first processing model or a second processing model obtained by training by using any one of the model training methods.

The present application further provides an electronic device comprising a memory and a processor coupled to each other, wherein the processor is configured to execute program instructions stored in the memory to implement any of the model training methods or any of the image segmentation methods described above.

The present application also provides a computer readable storage medium having stored thereon program instructions that, when executed by a processor, implement any of the model training methods or any of the image segmentation methods described above.

According to the scheme, the two processing models generate labeling information for each other, so that the two models are supervised interactively during training, label-free data can be fully utilized to train the models, the possibility of overfitting can be reduced compared with the training of a single model, and the generalization capability of the models is improved.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the application.

FIG. 1 is a schematic flow chart diagram of an embodiment of a model training method of the present application;

FIG. 2 is a schematic flow chart diagram illustrating another embodiment of a model training method of the present application;

FIG. 3 is a schematic flow chart diagram illustrating a further embodiment of the model training method of the present application;

FIG. 4 is a schematic flowchart of another embodiment of step S330;

FIG. 5 is a diagram illustrating a conversion process between real annotation information and a real distance map;

FIG. 6 is a diagram illustrating an embodiment of a segmentation model training method according to the present application;

FIG. 7 is a flowchart illustrating an embodiment of an image segmentation method of the present application;

FIG. 8 is a block diagram of an embodiment of the present invention model training device;

FIG. 9 is a block diagram of an embodiment of an image segmentation apparatus according to the present application;

FIG. 10 is a block diagram of an embodiment of an electronic device of the present application;

FIG. 11 is a block diagram of an embodiment of a computer-readable storage medium of the present application.

Detailed Description

The embodiments of the present application will be described in detail below with reference to the drawings.

In the following description, for purposes of explanation and not limitation, specific details are set forth such as particular system structures, interfaces, techniques, etc. in order to provide a thorough understanding of the present application.

The term "and/or" herein is merely an association describing an associated object, meaning that three relationships may exist, e.g., a and/or B, may mean: a exists alone, A and B exist simultaneously, and B exists alone. In addition, the character "/" herein generally indicates that the former and latter associated objects are in an "or" relationship. Further, the term "plurality" herein means two or more than two. In addition, the term "at least one" herein means any one of a plurality or any combination of at least two of a plurality, for example, including at least one of A, B, C, and may mean including any one or more elements selected from the group consisting of A, B and C. A

The model training method and the image segmentation method in the present application may be executed by one electronic device, or may be executed by two electronic devices, respectively, and the electronic devices may be any devices with processing capability, such as a tablet computer, a mobile phone, a computer, and the like.

Referring to fig. 1, fig. 1 is a schematic flow chart of an embodiment of a model training method according to the present application.

Specifically, the model training method may include the steps of:

step S110: and processing the sample image by using the first processing model to obtain a first segmentation result.

In this embodiment, the first processing model and the second processing model are trained, the trained first processing model or the trained second processing model may be used for image segmentation, and both may be used for segmenting the same object in the image, and the first processing model and the second processing model may be independent from each other, or there may be some components that are the same.

It is understood that step S120 needs to be executed after step S110, and step S140 needs to be executed after step S130, besides, the order of execution of steps S110 to step S140 is not limited, for example, step S110 to step S120 and step S130 to step S140 may be executed synchronously, or step S110, step S130, step S120 and step S140 may also be executed in sequence, or step S130, step S140, step S110 and step S120 may also be executed in sequence, etc.

The first processing model and the second processing model are both used for obtaining the segmentation result of the target object in the image, so the first segmentation result is the segmentation result of the target object in the sample image obtained by using the first processing model, the second segmentation result is the segmentation result of the target object in the sample image obtained by using the second processing model, and the segmentation result is used for representing the corresponding segmentation prediction region of the target object in the sample image.

Step S120: and generating first label information based on the first segmentation result.

Step S130: and processing the sample image by using a second processing model to obtain a second segmentation result.

Step S140: and generating second labeling information based on the second segmentation result.

Based on the first segmentation result and the second segmentation result about the target object in the sample image, first annotation information and second annotation information about the target object in the sample image can be generated, and both the first annotation information and the second annotation information can represent a reference area corresponding to the target object in the sample image. The first annotation information and the second annotation information are generated by using the segmentation result, and are not artificially annotated, and generally speaking, the first annotation information and the second annotation information may be pseudo-annotated.

It can be understood that the reference information is a segmentation reference result of the sample image with respect to the target object, and in the model training process, the reference information may be used for comparing with a result output by the model, and parameters of the model may be adjusted based on a difference between the reference information and the result output by the model, so that the model output is more satisfactory.

For the second processing model, the first labeled information may be used as second reference information for comparing with the second segmentation result output by the second processing model to adjust the parameters of the second processing model. Similarly, for the first processing model, the second labeled information may be used as the first reference information for comparing with the first segmentation result output by the first processing model to adjust the parameters of the first processing model.

The first segmentation result is obtained by using the first processing model, and is a prediction result of segmentation of the sample image with respect to the target object by the first processing model, and the second segmentation result is the same as the first segmentation result. The first annotation information and the second annotation information generated by using the first segmentation result and the second segmentation result are actually different from the real annotation information of the sample image about the target object, the real annotation information represents the real area of the target object in the sample image, and the first annotation information and the second annotation information can be pseudo-annotations representing the reference areas corresponding to the target object, and the pseudo-annotations can be simply regarded as the real annotation information containing noise.

Step S150: parameters of the first process model are adjusted based on the first segmentation result and the first reference information, and parameters of the second process model are adjusted based on the second segmentation result and the second reference information.

The first reference information comprises second annotation information, and the second reference information comprises the first annotation information. Specifically, step S150 may include adjusting parameters of the first processing model using a difference between the first segmentation result and the second labeling information, and adjusting parameters of the second processing model using a difference between the second segmentation result and the first labeling information.

It is understood that the above steps S110 to S150 are related to one training, and the first processing model and the second processing model can be trained multiple times by selecting different sample images to perform the above steps multiple times, so that the first processing model and the second processing model meet the preset requirements, thereby completing the training of the models.

In the scheme, the two processing models generate labeling information for each other, so that the two models are supervised interactively during training, the models can be trained by fully utilizing label-free data, the possibility of overfitting can be reduced compared with the training of a single model, and the generalization capability of the models is improved. By the aid of the method, under the condition that only part of labeled data exists, the model which is segmented accurately can be trained, and effectiveness of data training is improved.

Furthermore, the pseudo label can be regarded as real label information with noise, and training by using the pseudo label can be performed when the label has noise, so that the model can have stronger tolerance to the noise and can be accurately segmented when the model has the noise.

In some embodiments, the first reference information and the second reference information further comprise real annotation information of the sample image, respectively, and the real annotation information may represent a real area of the target object in the sample image. Therefore, when the parameters of the first processing model are adjusted, the first processing model can be adjusted by using the difference between the first segmentation result and the second labeling information and the difference between the first segmentation result and the real labeling information. In adjusting the parameters of the second processing model, the second processing model may be adjusted by using the difference between the second segmentation result and the first labeling information and the difference between the second segmentation result and the real labeling information.

Referring to fig. 2, fig. 2 is a schematic flow chart of another embodiment of the model training method of the present application. Specifically, the model training method may include the steps of:

step S210: and performing target segmentation on the sample image by using the first processing model to obtain a first segmentation result about the target object.

Step S110 may be implemented by step S210, and specifically, the first processing model may directly perform target segmentation on the sample image to segment a target object from the sample image, so as to obtain a first segmentation result, for performing an image segmentation task.

Step S220: and generating first label information based on the first segmentation result.

Specifically, the first label information is generated by using the first division result, and the first division result can be used as the first label information for the device. It is understood that the first segmentation result has a gradient property, and the first labeling information does not have a gradient property. In some embodiments, the first segmentation result may not be directly used as the first annotation information, for example, the first segmentation result may be combined with the real annotation information, or the first annotation information may be generated after the first segmentation result is subjected to the preset processing.

Step S230: and performing target segmentation on the sample image by using a second processing model to obtain a second segmentation result about the target object.

Step S130 may be implemented by step S230, and specifically, the second processing model may directly perform target segmentation on the sample image to segment a target object from the sample image, so as to obtain a first segmentation result, for performing an image segmentation task.

Step S240: and generating second labeling information based on the second segmentation result.

Specifically, the second label information is generated by using the second division result, which may be the second label information. It is understood that the first segmentation result has a gradient property, and the first labeling information does not have a gradient property. In some embodiments, the second segmentation result may not be directly used as the second annotation information, for example, the second segmentation result may be combined with the real annotation information, or the second annotation information may be generated after the second segmentation result is subjected to the preset processing.

Step S250: parameters of the first process model are adjusted based on the first segmentation result and the first reference information, and parameters of the second process model are adjusted based on the second segmentation result and the second reference information.

The related content of step S250 can refer to the related description related to step S150, which is not described herein again.

In the embodiment, the two processing models generate labeling information for each other, so that the two models are supervised interactively during training, the models can be trained by fully utilizing label-free data, the possibility of overfitting generated by training of a single model can be reduced, the two models can mutually learn the advantages of each other, and the generalization capability of the models is improved.

Referring to fig. 3, fig. 3 is a schematic flow chart of a model training method according to another embodiment of the present application. Specifically, the model training method may include the steps of:

step S310: and performing target segmentation on the sample image by using the first processing model to obtain a first segmentation result about the target object.

Step S320: and generating first label information based on the first segmentation result.

Step S310 and step S320 may refer to the related description of step S210 and step S220.

The first segmentation result has a gradient attribute, and can be used for calculating loss, performing back propagation and updating parameters of the first processing model. Step S320 may specifically include obtaining a copy of the first segmentation result, and performing gradient isolation on the copy of the first segmentation result to obtain first labeling information without gradient attribute.

It is understood that the specific value of the first segmentation result is not changed by performing the gradient isolation, so the first label information and the first segmentation result have the same value, and the difference between the first label information and the first segmentation result mainly depends on whether the gradient attribute is present. Thus, information that is consistent with the value of the first segmentation result but is not used for calculating the loss and performing back propagation to update the parameters of the first processing model, that is, the first annotation information, is generated through step S320, and this information can be used as the second reference information.

Step S330: and processing the sample image by using a second processing model to obtain a second segmentation result.

Referring to fig. 4, fig. 4 is a schematic flowchart illustrating another embodiment of the step S330 in the present application, specifically, the step S330 may include:

step S431: and performing distance regression on the sample image by using the second processing model to obtain a predicted distance result.

In this embodiment, the first processing model is used to execute a segmentation task and directly perform target segmentation on the sample image to obtain a first segmentation result, and the second processing model is used to execute a distance regression task and obtain a second segmentation result based on a predicted distance result obtained by the distance regression.

The distance regression task is used for predicting the distance between each pixel point in the image and a prediction boundary, wherein the prediction boundary can be the boundary of a prediction region corresponding to a target object in a sample image. Therefore, the predicted distance result obtained by distance regression of the sample image by the second processing model can represent the distance between each pixel point in the sample image and the predicted boundary.

It should be noted that the boundary of the prediction region may be a closed graph, and the distance may also be used to reflect a position relationship between the pixel point and the prediction boundary (prediction region), specifically, the distance may reflect whether the pixel point is on the prediction boundary and whether the pixel point is in a region enclosed by the prediction boundary. For example, if the distance between the pixel point and the prediction boundary is zero, the pixel point is located on the prediction boundary; in addition, whether the pixel point is in the area enclosed by the prediction boundary can be reflected through the direction of the distance, for example, if the prediction distance corresponding to the pixel point is a positive value, the pixel point can be considered to be located in the area enclosed by the prediction boundary, and if the prediction distance corresponding to the pixel point is a negative value, the pixel point can be considered to be located in the area enclosed by the prediction boundary.

Step S432: based on the predicted distance result, a second segmentation result is obtained with respect to the target object.

After the distance relationship between each pixel point in the sample image and the boundary of the prediction region corresponding to the target object is obtained, the prediction region corresponding to the target object can be determined according to the distance relationship, so that the prediction region corresponding to the target object is distinguished from other regions in the sample image, and a second segmentation result is obtained.

It is understood that the second segmentation result may not be obtained by direct segmentation, but may be obtained on the basis of distance regression, and the second segmentation result substantially distinguishes the region corresponding to the target object from other regions, i.e. may be considered as the segmentation of the target object.

In some embodiments, the predicted distance result may be in the form of a predicted distance map, where the value of each pixel in the predicted distance map is used to represent the distance between the corresponding pixel in the sample image and the predicted boundary. Then step S432 may be to perform binarization processing on the predicted distance map by using a preset binarization mode to obtain a second segmentation result about the target object, where the preset binarization mode may be a soft binarization operation, and the result obtained through the soft binarization operation does not only include two values, but also includes continuous numerical values distributed in a preset range.

It can be understood that, in the training process, when performing reverse propagation, derivation is required to be performed on the second segmentation result, the predicted distance result is not directly binarized to obtain the second segmentation result containing two values, but soft binarization operation is performed to make the second segmentation result contain continuous values within a preset range, so as to facilitate gradient propagation. Similarly, when training is performed, the first segmentation result does not only include two values to distinguish the foreground from the background, but also includes continuous numerical values to facilitate gradient backtracking. Certainly, when the model is actually applied, if the second processing model is adopted to perform image segmentation, the result of the predicted distance can be directly binarized, and the obtained segmentation result can only contain two values to distinguish the foreground and the background; if the first processing model is used for image segmentation, the directly obtained first segmentation result may only include two values to distinguish the foreground from the background.

In a specific application scenario, the prediction distance result is a prediction distance map, and the value of each pixel point on the prediction distance map can be obtained by normalizing the prediction distance between the corresponding pixel point and the prediction boundary in the sample image, so that the value of each pixel point on the prediction distance map is distributed between [ -1,1], the value of the pixel point on the prediction boundary is zero, and the numerical value can represent the distance between the corresponding pixel point and the prediction boundary in the sample image, and the larger the absolute value of the numerical value is, the farther the pixel point is from the prediction boundary; and the numerical value can also reflect the position relation between the pixel point and the prediction area, the values of the pixel points in the prediction boundary enclosing area are all positive values, and the values of the pixel points outside the prediction boundary enclosing area are all negative values. On this basis, the preset binarization mode may specifically be that the following binarization function is used to calculate the predicted distance map to obtain a second segmentation result: y-Sigmoid (1000 x), the result of the second segmentation is a continuous number distributed between 0,1, approaching 0 or 1 at most locations, exhibiting a continuous change of 0 to 1 near the prediction boundary.

Step S340: and generating second labeling information based on the second segmentation result.

And the second segmentation result has a gradient attribute, can be used for calculating loss and performing back propagation, and updates the parameters of the second processing model. Step S320 may specifically include obtaining a copy of the second segmentation result, and performing gradient isolation on the copy of the second segmentation result to obtain second labeling information without gradient attribute.

It is understood that the specific value of the second segmentation result is not changed by performing the gradient isolation, so the second label information and the second segmentation result have the same value, and the difference between the two results is mainly whether the gradient attribute is provided. Thus, the information that is consistent with the value of the second segmentation result but is not used for calculating the loss and performing the back propagation to update the parameters of the second processing model, i.e., the first annotation information, is generated in step S340, and the information can be used as the second reference information.

Step S350: parameters of the first process model are adjusted based on the first segmentation result and the first reference information, and parameters of the second process model are adjusted based on the second segmentation result and the second reference information.

Specifically, the first reference information may include second label information, the second reference information may include the first label information, and the step S350 may include adjusting a parameter of the first processing model using a difference between the first division result and the second label information, and adjusting a parameter of the second processing model using a difference between the second division result and the first label information.

In some embodiments, the first reference information and the second reference information may further include real annotation information, and step S350 may include adjusting the first processing model using a difference between the first segmentation result and the second annotation information, and a difference between the first segmentation result and the real annotation information; and adjusting the second processing model by using the difference between the second segmentation result and the first labeling information and the difference between the second segmentation result and the real labeling information.

Further, in the case that the second processing model outputs the result of predicting the distance for the sample image processing, before step S350, the method may further include: and performing distance transformation based on the real labeling information to obtain real distance information. The second reference information may include real distance information in addition to the first label information and the real label information, wherein the first label information and the real label information are segmentation reference information for comparison with the second segmentation result, and the real distance information is distance reference information for comparison with the predicted distance result. Step S350 may include adjusting a parameter of the second processing module based on a difference between the second segmentation result and the first annotation information, a difference between the second segmentation result and the real annotation information, and a difference between the predicted distance result and the real distance information.

More specifically, the real labeling information may represent a real region corresponding to the target object in the sample image, where the real region may be determined as a foreground and the remaining regions are determined as a background. Performing distance transformation based on the real labeling information, and obtaining the real distance information may include: and performing distance transformation and normalization processing on the foreground in the sample image to obtain a foreground distance map, wherein the foreground distance map can be used for representing the distance from the foreground point to the nearest background point. And performing distance transformation and normalization processing on the background in the sample image to obtain a background distance map, wherein the background distance map can be used for representing the distance from the background point to the nearest foreground point. The execution sequence of the foreground and background processing steps is not limited, and then the foreground distance map and the background distance map are used for subtraction processing, so that a real distance map can be obtained and used as real distance information, the value of each pixel point in the real distance map is used for representing the distance between the corresponding pixel point in the sample image and the real boundary of the target object, and the real boundary is the boundary of the real area corresponding to the target object in the sample image.

In the embodiment, the two processing models generate the labeling information for each other, so that the two models are supervised interactively during training, the model can be trained by fully utilizing the label-free data, the overfitting possibility can be reduced compared with the training of a single model, and the generalization capability of the model is improved. Furthermore, different processing modes can be adopted between the two models, and the two models can learn information of images at different angles to complement each other, so that the interactive supervision effect is improved, and the generalization capability of the models is improved.

Furthermore, compared with direct segmentation, the method for obtaining the segmentation result based on the distance regression method can better resist the anatomical structure abnormality caused by noise, burrs and the like may exist on the edge obtained by direct segmentation, and the distance regression method can more accurately judge the edge, reduce false positives or false negatives of the abnormality and improve the generalization capability of the model.

Referring to fig. 5, fig. 5 is a schematic diagram of a conversion process between real labeling information and a real distance map, in a specific application scenario, the real labeling information (segmentation labeling) represents a real area corresponding to a target object in a sample image as a foreground, other areas as a background, and euclidean distance transformation and normalization processing are performed on the foreground to obtain a foreground distance map (normalized foreground distance); the foreground is negated to obtain a background, and Euclidean distance transformation and normalization processing are carried out on the basis to obtain a background distance graph (normalized background distance); and subtracting the background distance map from the foreground distance map to obtain a real distance map (normalized distance map). The numerical value of the pixel point in the real distance map is used for representing the distance between the corresponding pixel point and the real boundary in the sample image, wherein the numerical value is distributed between [ -1,1], the isoline with the numerical value of 0 is superposed with the segmentation contour line of the real area in the real labeling information, the values in the real distance map corresponding to the foreground area are all larger than zero, the values in the real distance map corresponding to the background area are all smaller than zero, and the larger the absolute value of the numerical value is, the farther the pixel point is from the segmentation contour line. The actual distance map obtained through the above operations may be compared with the predicted distance map, and the difference between the two may be used to adjust the parameters of the second processing model.

Referring to fig. 6, fig. 6 is a schematic diagram illustrating an embodiment of a segmentation model training method according to the present application.

The model 1 (a first processing model) is used for executing an image segmentation task to obtain a segmentation prediction 1 (a first segmentation result), and performing gradient isolation on a copy of the segmentation prediction 1 to obtain a pseudo label 1 (first label information); the model 2 (second processing model) is used for executing the distance regression task to obtain a predicted distance result, performing soft binarization operation on the predicted distance result to obtain a segmentation prediction 2 (second segmentation result), and performing gradient isolation on a copy of the segmentation prediction to obtain a pseudo label 2 (second label information). For the model 1, the parameters of the model 1 are adjusted by the difference between the segmentation prediction 1 and the pseudo label 2 and the difference between the segmentation prediction 1 and the segmentation label. For the model 2, the parameters of the model 2 are adjusted by the difference between the segmentation prediction 2 and the pseudo label 1, the difference between the segmentation prediction 2 and the segmentation label, and the difference between the predicted distance result and the normalized distance map (true distance information) obtained based on the segmentation label.

If there is no segmentation label, then for model 1, the parameters of model 1 are adjusted by the difference between segmentation prediction 1 and pseudo label 2, and for model 2, the parameters of model 2 are adjusted by the difference between segmentation prediction 2 and pseudo label 1.

Specifically, adjusting the parameters of model 2 by the difference between the segmentation prediction 2 and the segmentation labeling can be achieved by the following loss function:

loss is Dice (segmentation prediction, segmentation annotation).

Adjusting the parameters of the model 2 by the difference between the predicted distance result and the normalized distance map can be achieved by the following loss function:

lose ═ MSE (predicted distance results, normalized distance map).

Referring to fig. 7, fig. 7 is a flowchart illustrating an embodiment of an image segmentation method according to the present application.

Specifically, the image segmentation method may include the steps of:

step S710: and acquiring a target image.

It is understood that the target image may be a medical image containing an image of the target object, and the types of medical images include, but are not limited to: three-dimensional CT (Computed Tomography), MRI (Magnetic Resonance Imaging), and the like.

Step S720: the target image is processed using the processing model to obtain a segmentation result for the target object.

The processing model is a first processing model or a second processing model obtained by training based on any model training method. After training of the first processing model and the second processing model is completed, the segmentation accuracy of the two models may have a certain difference, and the better one of the two models is selected to be applied in the image segmentation method, wherein the first processing model is used for segmenting the target image, and the second processing model is used for segmenting the target image or performing distance regression on the target image. The target object can be a preset tissue, an organ, a focus and the like, the target object can be accurately segmented from the medical image by using the segmentation model, and the segmentation result can be used for auxiliary medical treatment subsequently.

If the second processing model is used for distance regression on the target image, then the processing of the target image by the processing model to obtain the segmentation result about the target object comprises: and performing distance regression on the target image by using the processing model to obtain a predicted distance result, and obtaining a segmentation result about the target object based on the predicted distance result. The prediction distance result represents the distance between each pixel point in the target image and a prediction boundary, and the prediction boundary is the boundary of a prediction area corresponding to a target object in the target image. Specifically, obtaining the segmentation result about the target object based on the predicted distance result may obtain the segmentation result by performing binarization processing on the predicted distance result, and specifically, for example, the binarization function may be to make y equal to 1 when x satisfies a preset condition, and otherwise make y equal to 0. For the processing description of the related steps, reference may be made to the related contents in the aforementioned model training method, which is not described herein again.

In the above embodiment, the first processing model or the second processing model is obtained by training using the model training method, and the target image is processed, so that the target object can be accurately segmented.

Referring to fig. 8, fig. 8 is a schematic diagram of a framework of an embodiment of the present application model training apparatus.

In this embodiment, the model training apparatus 80 includes a processing module 81, an annotating module 82 and an adjusting module 83, where the processing module 81 may be configured to process the sample image by using a first processing model to obtain a first segmentation result, and process the sample image by using a second processing model to obtain a second segmentation result. The annotation module 82 can be configured to generate first annotation information based on the first segmentation result and generate second annotation information based on the second segmentation result. The adjusting module 83 may be configured to adjust parameters of the first processing model based on the first segmentation result and the first reference information, and adjust parameters of the second processing model based on the second segmentation result and the second reference information, wherein the first reference information includes the second annotation information, and the second reference information includes the first annotation information.

The processing module 81 may be configured to process the sample image by using a first processing model to obtain a first segmentation result, and specifically includes: and performing target segmentation on the sample image by using the first processing model to obtain a first segmentation result about the target object. The processing module 81 may be configured to process the sample image by using a second processing model to obtain a second segmentation result, which specifically includes: performing target segmentation on the sample image by using a second processing model to obtain a second segmentation result about the target object; or performing distance regression on the sample image by using a second processing model to obtain a predicted distance result, wherein the predicted distance result represents the distance between each pixel point in the sample image and a predicted boundary, and the predicted boundary is the boundary of a predicted region corresponding to the target object in the sample image; based on the predicted distance result, a second segmentation result is obtained with respect to the target object.

According to the scheme, the first processing model and the second processing model adopt the same processing mode for the image, or different processing modes can be adopted between the two models, and if different processing modes are adopted, the information of different angles of the image can be learned between the two models to complement each other, so that the generalization capability of the models is improved.

The prediction distance result is a prediction distance graph, and the value of each pixel point in the prediction distance graph is used for representing the distance between the corresponding pixel point in the sample image and the prediction boundary; the processing module 81 may be configured to obtain a second segmentation result about the target object based on the predicted distance result, and specifically includes: and performing binarization processing on the predicted distance map by using a preset binarization mode to obtain a second segmentation result related to the target object.

In the above-described aspect, the region corresponding to the target object and the other regions can be roughly distinguished from the predicted distance map by performing binarization processing on the predicted distance map obtained by distance regression, so that a second division result can be obtained for use in subsequent adjustment of the model parameters.

Wherein the first segmentation result and the second segmentation result both have gradient attributes; the labeling module 82 may be configured to generate the first labeling information based on the first segmentation result, which specifically includes: and acquiring a copy of the first segmentation result, and performing gradient isolation on the copy of the first segmentation result to obtain first labeling information without gradient attributes. The labeling module 82 may be configured to generate second labeling information based on the second segmentation result, which specifically includes: and acquiring a copy of the second segmentation result, and performing gradient isolation on the copy of the second segmentation result to obtain second labeling information without gradient attributes.

According to the scheme, the first labeling information is generated through the first segmentation result, and the second labeling information is generated through the second segmentation result, so that interactive supervision between the two models is realized, the two models can be complemented, the two models can mutually absorb the advantages of the other model, the generalization capability of the models is improved, and the labeling information can be generated through the gradient isolation operation by utilizing the segmentation information so as to be used for realizing the subsequent interactive supervision.

The first reference information and the second reference information also comprise real labeling information of the sample image; the adjusting module 83 may be configured to adjust parameters of the first processing model based on the first segmentation result and the first reference information, and specifically includes: and adjusting parameters of the first processing model based on the difference between the first segmentation result and the second labeling information and the difference between the first segmentation result and the real labeling information. The adjusting module 83 may be configured to adjust parameters of the second processing model based on the second segmentation result and the second reference information, and specifically includes: and adjusting the parameters of the second processing model based on the difference between the second segmentation result and the first labeling information and the difference between the second segmentation result and the real labeling information.

According to the scheme, the parameters of the processing model are adjusted through the difference between the segmentation result and the labeling information and the real labeling information, so that the model can learn the real labeling information and the information in the labeling information generated based on the segmentation result, and the segmentation is carried out more accurately.

The model training device can further comprise a transformation module, wherein the transformation module is used for performing distance transformation based on the real annotation information to obtain the real distance information under the condition that the second processing model outputs a predicted distance result for processing the sample image before adjusting the parameters of the second processing model based on the difference between the second segmentation result and the first annotation information and the difference between the second segmentation result and the real annotation information; the prediction distance result is used for obtaining a second segmentation result and represents the distance between each pixel point in the sample image and the prediction boundary, the prediction boundary is the boundary of a prediction area corresponding to the target object in the sample image, the real distance information represents the distance between each pixel point in the sample image and a real boundary, and the real boundary is the boundary of the real area corresponding to the target object in the sample image. The adjusting module 83 may be configured to adjust parameters of the second processing model based on a difference between the second segmentation result and the first annotation information and a difference between the second segmentation result and the real annotation information, and specifically includes: and adjusting parameters of the second processing model based on the difference between the second segmentation result and the first labeled information, the difference between the second segmentation result and the real labeled information, and the difference between the predicted distance result and the real distance information.

According to the scheme, the real labeling information is converted into the real distance information to be compared with the predicted distance result, and the parameters of the second processing model are adjusted through the difference between the real labeling information and the predicted distance result, so that the second processing model can learn from the real distance information, and the predicted distance result is closer to the real distance information.

The real distance information is a real distance map, and the value of each pixel point in the real distance map is used for representing the distance between a corresponding pixel point and a real boundary in the sample image; the transformation module may be configured to perform distance transformation based on the real annotation information to obtain real distance information, and specifically includes: determining a real area in the sample image as a foreground and determining the rest areas as a background based on the real labeling information; respectively carrying out distance conversion and normalization processing on the foreground and the background in the sample image to obtain a foreground distance map and a background distance map; and performing subtraction processing by using the foreground distance map and the background distance map to obtain a real distance map.

According to the scheme, the real distance graph can be obtained by performing distance conversion and normalization processing on the foreground and the background and subtracting the foreground and the background, and the real distance graph is used as real distance information for parameter adjustment of the second processing model in the follow-up process, so that the generalization capability of the second processing model is improved.

Referring to fig. 9, fig. 9 is a schematic diagram of a frame of an embodiment of an image segmentation apparatus according to the present application.

In this embodiment, the image segmentation apparatus 90 includes an obtaining module 91 and a processing module 92, where the obtaining module 91 is configured to obtain a target image; the processing module 92 is configured to process the target image by using the processing model to obtain a segmentation result about the target object; the processing model is a first processing model or a second processing model obtained by training by using any one of the model training methods.

According to the scheme, the first processing model or the second processing model is obtained by training through the model training method, the target image is processed, and the target object can be accurately segmented.

The processing model is a second processing model, and the second processing model is used for carrying out distance regression on the target image; the processing module 92 may be configured to process the target image by using the processing model to obtain a segmentation result about the target object, and specifically includes: performing distance regression on the target image by using the processing model to obtain a predicted distance result, wherein the predicted distance result represents the distance between each pixel point in the target image and a predicted boundary, and the predicted boundary is the boundary of a predicted region corresponding to a target object in the target image; based on the predicted distance result, a segmentation result is obtained with respect to the target object.

According to the scheme, the second processing model can accurately segment the target object by obtaining the accurate segmentation result of the target object on the basis of distance regression.

Referring to fig. 10, fig. 10 is a schematic diagram of a frame of an embodiment of an electronic device according to the present application.

In this embodiment, the electronic device 100 includes a memory 101 and a processor 102 coupled to each other, and the processor 102 is configured to execute program instructions stored in the memory 101 to implement the steps of any of the above-described model training method embodiments or the steps of any of the above-described image segmentation method embodiments. In one particular implementation scenario, electronic device 100 may include, but is not limited to: a microcomputer, a server, and the electronic device 100 may further include a mobile device such as a notebook computer, a tablet computer, and the like, which is not limited herein.

In particular, the processor 102 is configured to control itself and the memory 101 to implement the steps of any of the above-described embodiments of the model training method or the steps of any of the above-described embodiments of the image segmentation method. Processor 102 may also be referred to as a CPU (Central Processing Unit). The processor 102 may be an integrated circuit chip having signal processing capabilities. The Processor 102 may also be a general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic, discrete hardware components. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. Additionally, the processor 102 may be commonly implemented by integrated circuit chips.

Referring to fig. 11, fig. 11 is a block diagram illustrating an embodiment of a computer-readable storage medium according to the present application.

In this embodiment, the computer readable storage medium 110 stores program instructions 111 capable of being executed by a processor, and the program instructions 111 are used for implementing the steps of any of the above-described embodiments of the model training method or the steps of any of the above-described embodiments of the image segmentation method.

The computer-readable storage medium 110 may be a medium that can store program data, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk, or may be a server that stores the program data, and the server can transmit the stored program data to other devices for operation, or can self-operate the stored program data.

In some embodiments, computer-readable storage medium 110 may also be a memory as shown in FIG. 10.

In some embodiments, functions of or modules included in the apparatus provided in the embodiments of the present disclosure may be used to execute the method described in the above method embodiments, and specific implementation thereof may refer to the description of the above method embodiments, and for brevity, will not be described again here.

The foregoing description of the various embodiments is intended to highlight various differences between the embodiments, and the same or similar parts may be referred to each other, and for brevity, will not be described again herein.

In the several embodiments provided in the present application, it should be understood that the disclosed method and apparatus may be implemented in other manners. For example, the above-described apparatus embodiments are merely illustrative, and for example, a division of a module or a unit is merely one type of logical division, and an actual implementation may have another division, for example, a unit or a component may be combined or integrated with another system, or some features may be omitted, or not implemented. In addition, the shown or discussed coupling or direct coupling or communication connection between each other may be through some interfaces, indirect coupling or communication connection between devices or units, and may be in an electrical, mechanical or other form.

In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.

The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be substantially implemented or contributed to by the prior art, or all or part of the technical solution may be embodied in a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, a network device, or the like) or a processor (processor) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.

If the technical scheme of the application relates to personal information, a product applying the technical scheme of the application clearly informs personal information processing rules before processing the personal information, and obtains personal independent consent. If the technical scheme of the application relates to sensitive personal information, a product applying the technical scheme of the application obtains individual consent before processing the sensitive personal information, and simultaneously meets the requirement of 'express consent'. For example, at a personal information collection device such as a camera, a clear and significant identifier is set to inform that the personal information collection range is entered, the personal information is collected, and if the person voluntarily enters the collection range, the person is regarded as agreeing to collect the personal information; or on the device for processing the personal information, under the condition of informing the personal information processing rule by using obvious identification/information, obtaining personal authorization by modes of popping window information or asking a person to upload personal information of the person by himself, and the like; the personal information processing rule may include information such as a personal information processor, a personal information processing purpose, a processing method, and a type of personal information to be processed.

Claims

1. A method of model training, comprising:

processing the sample image by using a first processing model to obtain a first segmentation result, and processing the sample image by using a second processing model to obtain a second segmentation result;

generating first labeling information based on the first segmentation result, and generating second labeling information based on the second segmentation result;

adjusting parameters of the first processing model based on the first segmentation result and first reference information, and adjusting parameters of the second processing model based on the second segmentation result and second reference information, wherein the first reference information comprises the second labeling information, and the second reference information comprises the first labeling information.

2. The method of claim 1, wherein processing the sample image using the first processing model to obtain the first segmentation result comprises:

performing target segmentation on the sample image by using the first processing model to obtain a first segmentation result about a target object; and the number of the first and second groups,

the processing the sample image by using the second processing model to obtain a second segmentation result includes:

performing target segmentation on the sample image by using the second processing model to obtain a second segmentation result about the target object; alternatively, the first and second electrodes may be,

performing distance regression on the sample image by using the second processing model to obtain a predicted distance result, wherein the predicted distance result represents the distance between each pixel point in the sample image and a predicted boundary, and the predicted boundary is the boundary of a predicted region corresponding to a target object in the sample image; and obtaining a second segmentation result about the target object based on the prediction distance result.

3. The method according to claim 2, wherein the predicted distance result is a predicted distance map, and a value of each pixel point in the predicted distance map is used for representing a distance between a corresponding pixel point in the sample image and the predicted boundary;

the obtaining a second segmentation result about the target object based on the predicted distance result comprises:

and carrying out binarization processing on the predicted distance map by using a preset binarization mode to obtain a second segmentation result related to the target object.

4. The method according to any one of claims 1 to 3, wherein the first and second segmentation results each have gradient properties;

the generating of the first labeling information based on the first segmentation result comprises:

acquiring a copy of the first segmentation result, and performing gradient isolation on the copy of the first segmentation result to obtain the first labeling information without gradient attribute;

generating second labeling information based on the second segmentation result comprises:

and acquiring a copy of the second segmentation result, and performing gradient isolation on the copy of the second segmentation result to obtain the second labeling information without gradient attribute.

5. The method according to any one of claims 1 to 4, wherein the first reference information and the second reference information further include real annotation information of the sample image, respectively; the adjusting parameters of the first processing model based on the first segmentation result and first reference information comprises:

adjusting parameters of the first processing model based on a difference between the first segmentation result and the second annotation information and a difference between the first segmentation result and the real annotation information;

the adjusting parameters of the second processing model based on the second segmentation result and second reference information comprises:

adjusting parameters of the second processing model based on a difference between the second segmentation result and the first annotation information and a difference between the second segmentation result and the real annotation information.

6. The method of claim 5, wherein before the adjusting the parameters of the second processing model based on the difference between the second segmentation result and the first annotation information and the difference between the second segmentation result and the real annotation information, the method further comprises:

under the condition that the second processing model outputs a prediction distance result for the sample image processing, performing distance transformation based on the real annotation information to obtain real distance information; the prediction distance result is used for obtaining the second segmentation result and represents a distance between each pixel point in the sample image and a prediction boundary, the prediction boundary is a boundary of a prediction region corresponding to a target object in the sample image, the real distance information represents a distance between each pixel point in the sample image and a real boundary, and the real boundary is a boundary of a real region corresponding to the target object in the sample image;

the adjusting parameters of the second processing model based on the difference between the second segmentation result and the first annotation information and the difference between the second segmentation result and the real annotation information comprises:

the adjusting parameters of the second processing model based on a difference between the second segmentation result and the first annotation information, a difference between the second segmentation result and the true annotation information, and a difference between the predicted distance result and the true distance information.

7. The method according to claim 6, wherein the real distance information is a real distance map, and the value of each pixel point in the real distance map is used for representing the distance between the corresponding pixel point in the sample image and the real boundary;

the distance transformation based on the real labeling information to obtain real distance information comprises the following steps:

determining the real area in the sample image as a foreground and the rest areas as a background based on the real labeling information;

respectively carrying out distance conversion and normalization processing on the foreground and the background in the sample image to obtain a foreground distance map and a background distance map;

and performing subtraction processing by using the foreground distance map and the background distance map to obtain the real distance map.

8. A method of image segmentation, the method comprising:

acquiring a target image;

processing the target image by using a processing model to obtain a segmentation result about a target object; wherein the treatment model is a first treatment model or a second treatment model trained using the method of any one of claims 1-7.

9. The method of claim 8, wherein the process model is the second process model and the second process model is used to perform distance regression on the target image;

the processing the target image by using the processing model to obtain a segmentation result about the target object comprises:

performing distance regression on the target image by using the processing model to obtain a predicted distance result, wherein the predicted distance result represents the distance between each pixel point in the target image and a predicted boundary, and the predicted boundary is the boundary of a predicted region corresponding to a target object in the target image;

and obtaining a segmentation result about the target object based on the prediction distance result.

10. A model training apparatus, comprising:

the processing module is used for processing the sample image by utilizing a first processing model to obtain a first segmentation result and processing the sample image by utilizing a second processing model to obtain a second segmentation result;

the labeling module is used for generating first labeling information based on the first segmentation result and generating second labeling information based on the second segmentation result;

and an adjusting module, configured to adjust parameters of the first processing model based on the first segmentation result and first reference information, and adjust parameters of the second processing model based on the second segmentation result and second reference information, where the first reference information includes the second labeled information, and the second reference information includes the first labeled information.

11. An image segmentation apparatus, comprising:

the acquisition module is used for acquiring a target image;

the processing module is used for processing the target image by using a processing model to obtain a segmentation result about a target object; wherein the treatment model is a first treatment model or a second treatment model trained using the method of any one of claims 1-7.

12. An electronic device comprising a memory and a processor coupled to each other, the processor being configured to execute program instructions stored in the memory to implement the method of any of claims 1 to 7 or any of claims 8-9.

13. A computer readable storage medium having stored thereon program instructions, which when executed by a processor implement the method of any of claims 1 to 7 or any of claims 8 to 9.