CN114882996B

CN114882996B - Hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning

Info

Publication number: CN114882996B
Application number: CN202210265257.4A
Authority: CN
Inventors: 黄炳升; 林楚旋; 陈嘉; 陈佳兆; 龙廷玉; 陈玉莹; 冯仕庭; 王猛; 周小琦; 董帜; 王霁朏; 彭振鹏
Original assignee: Shenzhen University
Current assignee: Shenzhen University
Priority date: 2022-03-17
Filing date: 2022-03-17
Publication date: 2023-04-07
Anticipated expiration: 2042-03-17
Also published as: CN114882996A

Abstract

The application discloses a hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning, which comprises the following steps: controlling a first feature extraction module, a second feature extraction module and a third feature extraction module to determine a first feature map, a second feature and a third feature map based on the MR image to be predicted; and the control prediction module determines a CK19 expression type and an MVI type of the MR image based on the first feature map, the second feature map and the third feature map. According to the method and the device, the abundant image characteristics carried by the MR image to be predicted are directly extracted through the prediction network model, and the problem that the prediction performance of the model is influenced due to subjectivity in image characteristic analysis can be avoided. Meanwhile, the CK19 expression features are extracted through the first feature extraction module, the shared features of CK19 expression and MVI are extracted through the second feature extraction module, the MVI features are extracted through the third feature extraction module, task parameters of a shared task and task parameters of an independent task are separated, and the problem that prediction results are unstable due to complex parameter sharing can be solved.

Description

Hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning

Technical Field

The application relates to the technical field of biomedical engineering, in particular to a hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning.

Background

Primary liver cancer is the sixth most common cancer worldwide, and the incidence rate thereof is on the rise worldwide. Hepatocellular carcinoma is the most common one of primary liver cancers, accounting for 90% of all primary liver malignancies. Hepatocellular Carcinoma (HCC) has high mortality and strong heterogeneity, and has significant global differences in morbidity and mortality. Surgical resection is currently the primary treatment of choice for HCC, however, most HCC patients treated by surgical resection have a recurrence of neogenetic tumors within 2 years. Among the factors associated with the early recurrence of HCC after surgery, positive expression of Cytokeratin19 (CK 19) and Microvascular Invasion (MVI) are important predictors of early recurrence and poor prognosis of HCC, and these two factors have a correlation that HCC positive expression of CK19 is more likely to develop MVI.

HCC positively expressed by CK19 is more likely to have postoperative recurrent metastasis than HCC negatively expressed by CK19, and CK19 positive expression is usually predictive of strong HCC invasiveness and is one of independent factors of HCC poor prognosis. When HCC patients are found to be at a high risk of poor prognosis, physicians need to reduce the postoperative recurrence rate by expanding the resection range of the tumor. Therefore, preoperative prediction of CK19 expression in liver cancer is crucial to the formulation of treatment strategies and prognosis judgment for patients. According to literature reports, the risk of postoperative recurrence of HCC patients with MVI is high, and the MVI is an important prediction factor of HCC poor prognosis. Therefore, the preoperative diagnosis of the liver cancer MVI has important guiding function in the aspects of selection of treatment modes, prognosis and the like of liver cancer patients.

At present, CK19 expression is clinically diagnosed mainly through pathological examination of preoperative invasive puncture biopsy or operative excision of focus, MVI is diagnosed mainly depending on postoperative pathological examination, and the results of the two examinations serve as diagnostic gold standards of CK19 expression and MVI. The Imaging assessment method before the operation of liver cancer patients, which is commonly used in clinical practice, mainly includes ultrasound, electron computed tomography and Magnetic Resonance Imaging (MRI). Wherein, gadoxeroside disodium (Gadolinium ethybenzyl diethylethylenediamine Pentaacetic Acid, gd-EOB-DTPA) contrast agent enhanced MRI is an important examination method for HCC preoperative diagnosis. The method not only can accurately reflect the pathological condition of liver cancer tissue and provide rich information for evaluation and diagnosis of HCC, but also can reflect the biological behavior and gene expression of HCC through the imaging characteristics, and is favorable for predicting CK19 expression and preoperative MVI. However, currently, doctors generally analyze the imaging characteristics of MRI manually to predict the expression and MVI of the liver cancer CK19, so that the prediction results of the expression and MVI of the liver cancer CK19 are often affected by the clinical experience of the doctors, and the prediction results may be different among different doctors, thereby causing the inaccurate analysis and prediction results.

Thus, the prior art has yet to be improved and enhanced.

Disclosure of Invention

The present application is directed to provide a hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning, which overcome the disadvantages of the prior art.

In order to solve the above technical problem, a first aspect of the embodiments of the present application provides a method for predicting hepatocellular carcinoma CK19 and MVI based on multitask learning, where the method is applied to a trained prediction network model, and the prediction network model includes a first feature extraction module for extracting CK19 expression features, a second feature extraction module for extracting shared features of CK19 expression and MVI, a third feature extraction module for extracting MVI features, and a prediction module; the method comprises the following steps:

controlling the first feature extraction module, the second feature extraction module and the third feature extraction module to determine a first feature map, a second feature map and a third feature map based on the MR image to be predicted;

and controlling the prediction module to determine a CK19 expression type and an MVI type corresponding to the MR image to be predicted based on the first feature map, the second feature map and the third feature map.

The hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning is characterized in that the prediction module comprises a first prediction branch and a second prediction branch, and the first prediction branch and the second prediction branch respectively comprise a convolution unit, a spatial transformation unit and a prediction unit which are sequentially cascaded; the controlling the prediction module to determine the CK19 expression category and the MVI category corresponding to the MR image to be predicted based on the first feature map, the second feature map, and the third feature map specifically includes:

controlling the first prediction branch to determine a CK19 expression category corresponding to the MR image to be predicted based on the first feature map and the second feature map;

and controlling the second prediction branch to determine the MVI category corresponding to the MR image to be predicted based on the second characteristic diagram and the third characteristic diagram.

The hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning is characterized in that the prediction module comprises a first prediction branch and a second prediction branch, the first prediction branch and the second prediction branch respectively comprise a feature transformation unit, a relation inference unit and a prediction unit which are sequentially cascaded, and the feature transformation unit comprises a convolution unit and a space transformation unit which are sequentially cascaded; the controlling the prediction module to determine the CK19 expression type and the MVI type corresponding to the MR image to be predicted based on the first feature map, the second feature map and the third feature map specifically includes:

controlling a feature transformation unit in the first prediction branch to determine a first affine feature map based on the first feature map and the second feature map;

controlling a feature transformation unit in the second prediction branch to determine a second affine feature map based on the second feature map and the third feature map;

controlling a relation inference unit in the first prediction branch to determine a first relation feature map based on the first affine feature map and the second affine feature map, and controlling a prediction unit in the first prediction branch to determine a CK19 expression category corresponding to the MR image to be predicted based on the first relation feature map;

and controlling a relation inference unit in the second prediction branch to determine a second relation feature map based on the first affine feature map and the second affine feature map, and controlling a prediction unit in the second prediction branch to determine the corresponding MVI type of the MR image to be predicted based on the second relation feature map.

The hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning comprises a positioning network block, a network generator and a sampler, wherein the positioning network block is connected with the network generator, the network generator is connected with the sampler, the positioning network block is used for generating an affine transformation coefficient matrix, the network generator is used for generating a sampling grid based on the affine transformation coefficient matrix, and the sampler is used for carrying out position mapping on a feature map input into the spatial transformation unit based on a sampling network so as to obtain an affine feature map.

The hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning is characterized in that the prediction module comprises a first prediction branch and a second prediction branch, and the first prediction branch and the second prediction branch respectively comprise a convolution unit, a relation inference unit and a prediction unit which are sequentially cascaded; the controlling the prediction module to determine the CK19 expression category and the MVI category corresponding to the MR image to be predicted based on the first feature map, the second feature map, and the third feature map specifically includes:

controlling a convolution unit in the first prediction branch to determine a third affine feature map based on the first feature map and the second feature map;

controlling a convolution unit in the second prediction branch to determine a fourth affine feature map based on the second feature map and the third feature map;

controlling a relation inference unit in the first prediction branch to determine a third relation feature map based on the third affine feature map and the fourth affine feature map, and controlling a prediction unit in the first prediction branch to determine a CK19 expression category corresponding to the MR image to be predicted based on the third relation feature map;

and controlling a relation inference unit in the second prediction branch to determine a fourth relation feature map based on the third affine feature map and the fourth affine feature map, and controlling a prediction unit in the second prediction branch to determine the corresponding MVI type of the MR image to be predicted based on the fourth relation feature map.

The hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning is characterized in that the relational inference unit comprises a volume block, a global pooling layer and a multilayer perceptron; the convolution block is used for acquiring a feature relation graph of the two affine feature graphs input into the associated reasoning unit; the global pooling layer is used for converting the characteristic relation graph into a characteristic relation vector; the multilayer perceptron is configured to generate a relational feature map based on the feature relationship vector.

The hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning is characterized in that the CK19 expression category comprises CK19 expression negativity or CK19 expression positivity; the MVI category includes MVI or no MVI.

A second aspect of the embodiments of the present application provides a device for predicting hepatocellular carcinoma CK19 and MVI based on multitask learning, wherein the device is configured with a trained prediction network model, the prediction network model includes a first feature extraction module for extracting CK19 expression features, a second feature extraction module for extracting CK19 expression and MVI shared features, a third feature extraction module for extracting MVI features, and a prediction module, the device includes:

the first control module is used for controlling the first feature extraction module, the second feature extraction module and the third feature extraction module to determine a first feature map, a second feature map and a third feature map based on the MR image to be predicted;

and the second control module is used for controlling the prediction module to determine a CK19 expression type and an MVI type corresponding to the MR image to be predicted based on the first feature map, the second feature map and the third feature map.

A third aspect of embodiments of the present application provides a computer-readable storage medium storing one or more programs, which are executable by one or more processors to implement the steps of the method for predicting hepatocellular carcinoma CK19 and MVI based on multitask learning as described in any one of the above.

A fourth aspect of embodiments of the present application provides a terminal device, including: a processor, a memory, and a communication bus; the memory has stored thereon a computer readable program executable by the processor;

the communication bus realizes connection communication between the processor and the memory;

the processor, when executing the computer readable program, implements the steps in the method for hepatocellular carcinoma CK19 and MVI prediction based on multitask learning as described in any of the above.

Has the advantages that: compared with the prior art, the application provides a hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning, and the method comprises the following steps: controlling the first feature extraction module, the second feature extraction module and the third feature extraction module to determine a first feature map, a second feature map and a third feature map based on the MR image to be predicted; and controlling the prediction module to determine a CK19 expression type and an MVI type corresponding to the MR image to be predicted based on the first feature map, the second feature map and the third feature map. According to the method and the device, the prediction network model based on deep learning is adopted to directly extract rich image features carried by the MR image to be predicted, so that the problem that the prediction performance of the model is influenced due to subjectivity in image feature analysis can be avoided. Meanwhile, the CK19 expression features are extracted through the first feature extraction module, the shared features of CK19 expression and MVI are extracted through the second feature extraction module, the MVI features are extracted through the third feature extraction module, task parameters of a shared task and task parameters of an independent task are separated, and the problem that prediction results of a prediction network model are unstable due to complex parameter sharing can be solved.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without any inventive work.

Fig. 1 is a flowchart of a hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning provided in the present application.

Fig. 2 is a schematic diagram illustrating a training process and a testing process of a prediction network model in the hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning according to the present application.

Fig. 3 is a schematic structural diagram of a first feature extraction module in the hepatocellular carcinoma CK19 and MVI prediction method based on multi-task learning according to the present disclosure.

Fig. 4 is a schematic structural diagram of a convolution block a in the hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning according to the present application.

Fig. 5 is a schematic structural diagram of a first residual unit in the hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning according to the present application.

Fig. 6 is a schematic structural diagram of one implementation of a prediction network model in the hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning according to the present application.

Fig. 7 is a schematic structural diagram of a convolution unit in the hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning according to the present application.

Fig. 8 is a schematic structural diagram of a convolution block c in the hepatocellular carcinoma CK19 and MVI prediction method based on multi-task learning according to the present application.

Fig. 9 is a schematic structural diagram of a spatial transform unit in the hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning according to the present application.

Fig. 10 is a schematic structural diagram of an implementation of a prediction network model in the hepatocellular carcinoma CK19 and MVI prediction method based on multi-task learning according to the present application.

Fig. 11 is a schematic structural diagram of a relationship inference unit in the hepatocellular carcinoma CK19 and MVI prediction method based on multi-task learning according to the present application.

Fig. 12 is a schematic structural diagram of one implementation of a prediction network model in the hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning according to the present application.

Fig. 13 is a schematic structural diagram of a hepatocellular carcinoma CK19 and MVI prediction device based on multitask learning according to the present application.

Fig. 14 is a schematic structural diagram of a terminal device provided in the present application.

Detailed Description

The present application provides a hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning, and in order to make the objects, technical solutions and effects of the present application clearer and clearer, the present application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.

As used herein, the singular forms "a", "an", "the" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. It will be understood that when an element is referred to as being "connected" or "coupled" to another element, it can be directly connected or coupled to the other element or intervening elements may also be present. Further, "connected" or "coupled" as used herein may include wirelessly connected or wirelessly coupled. As used herein, the term "and/or" includes all or any element and all combinations of one or more of the associated listed items.

It will be understood by those within the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

It should be understood that, the sequence numbers and sizes of the steps in this embodiment do not mean the execution sequence, and the execution sequence of each process is determined by its function and inherent logic, and should not constitute any limitation on the implementation process of this embodiment.

The inventor finds that the primary liver cancer is the sixth most common cancer worldwide, and the incidence rate of the primary liver cancer is on the rise worldwide. Hepatocellular carcinoma is the most common one of primary liver cancers, accounting for 90% of all primary liver malignancies. Hepatocellular Carcinoma (HCC) has a high mortality rate and a strong heterogeneity, and there are significant global differences in morbidity and mortality. Surgical resection is currently the primary treatment of choice for HCC, however, most HCC patients treated by surgical resection have a recurrence of neogenetic tumors within 2 years. Among the factors associated with the early recurrence of HCC after surgery, positive expression of Cytokeratin19 (CK 19) and Microvascular Invasion (MVI) are important predictors of early recurrence and poor prognosis of HCC, and these two factors have a correlation that HCC positive expression of CK19 is more likely to develop MVI.

HCC positively expressed by CK19 is more likely to have postoperative recurrent metastasis than HCC negatively expressed by CK19, and CK19 positive expression is usually predictive of strong HCC invasiveness and is one of independent factors of HCC poor prognosis. When HCC patients are found to be at a high risk of poor prognosis, physicians need to reduce the postoperative recurrence rate by expanding the resection range of the tumor. Therefore, preoperatively predicting the CK19 expression of the liver cancer is important for making a treatment strategy and judging prognosis of a patient. According to literature reports, the risk of postoperative recurrence of HCC patients with MVI is high, and the MVI is an important prediction factor of HCC poor prognosis. Therefore, the preoperative diagnosis of the liver cancer MVI has important guiding function in the aspects of selection of treatment modes, prognosis and the like of liver cancer patients.

At present, CK19 expression is clinically diagnosed mainly through pathological examination of preoperative invasive puncture biopsy or operative excision of focus, MVI is diagnosed mainly depending on postoperative pathological examination, and the results of the two examinations serve as diagnostic gold standards of CK19 expression and MVI. The Imaging assessment method before the operation of liver cancer patients, which is commonly used in clinical practice, mainly includes ultrasound, electron computed tomography and Magnetic Resonance Imaging (MRI). Wherein, contrast agent enhanced MRI of gadolinate disodium (Gadolinium ethybenzyl diethylstillnamide Pentaacetic Acid, gd-EOB-DTPA) is an important examination method for HCC preoperative diagnosis. The kit can accurately reflect the pathological condition of liver cancer tissue, provides rich information for evaluation and diagnosis of HCC, and can reflect the biological behavior and gene expression of HCC due to the imaging characteristics, thereby being beneficial to predicting CK19 expression and preoperative MVI. However, currently, doctors generally analyze the imaging characteristics of MRI to predict the expression and MVI of liver cancer CK19, so that the prediction results of liver cancer CK19 expression and MVI are often influenced by clinical experience of doctors, and the prediction results may be different among doctors, thereby resulting in inaccurate analysis and prediction results.

In order to solve the problems, a statistical analysis method and a traditional machine learning method are generally adopted to predict the CK19 expression and MVI of the liver cancer at present, wherein the statistical analysis method is to use the characteristic information of a medical image to perform statistical analysis to predict, and the statistical analysis method has measurement errors in part of characteristic critical value indexes when the MR image is required to be analyzed, so that the result of the statistical analysis may be inaccurate; meanwhile, the analysis of most image characteristics has subjectivity, and the accuracy of the analysis prediction result cannot be ensured. In addition, the statistical learning method is easy to have an overfitting problem, so that the model prediction performance is general, and under the condition that the feature data volume is large, the statistical learning method may not find the correlation among the features, so that the model prediction performance is poor. The machine learning method weakens the convergence problem, and the model prediction capability is strong under the condition of large characteristic data quantity. The accuracy of a prediction model established based on a traditional machine learning method is high, however, the traditional machine learning needs artificial design of features, the artificial design of the features often has the problems of strong data dependence and poor feature generalization, and overfitting is easy to occur on a small sample.

Based on this, in the embodiment of the application, the first feature extraction module, the second feature extraction module and the third feature extraction module are controlled to determine a first feature map, a second feature map and a third feature map based on an MR image to be predicted; and controlling the prediction module to determine a CK19 expression type and an MVI type corresponding to the MR image to be predicted based on the first feature map, the second feature map and the third feature map. According to the method and the device, the prediction network model based on deep learning is adopted to directly extract rich image features carried by the MR image to be predicted, so that the problem that the prediction performance of the model is influenced due to subjectivity in image feature analysis can be avoided. Meanwhile, the CK19 expression features are extracted through the first feature extraction module, the shared features of CK19 expression and MVI are extracted through the second feature extraction module, the MVI features are extracted through the third feature extraction module, task parameters of a shared task and task parameters of an independent task are separated, and the problem that prediction results of a prediction network model are unstable due to complex parameter sharing can be solved.

The following description of the embodiments is provided to further explain the present disclosure by way of example in connection with the appended drawings.

The embodiment provides a hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning, the prediction method is applied to a trained prediction network model, the prediction network model is a neural network model based on deep learning, and feature extraction can be directly carried out on an MR (Magnetic Resonance) image to be predicted so as to extract rich image features carried by the MR image. The prediction network model comprises a first feature extraction module for extracting CK19 (Cytokeratin 19) expression features, a second feature extraction module for extracting shared features of CK19 expression and MVI (microvasculature Invasion), a third feature extraction module for extracting MVI features and a prediction module; the first feature extraction module, the second feature extraction module and the third feature extraction module are parallel and are connected with the prediction module, input items of the first feature extraction module, the second feature extraction module and the third feature extraction module are all MR images to be predicted, and input items of the prediction module are CK19 expression categories and MVI categories corresponding to the MR images to be predicted.

The embodiment provides a hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning, as shown in fig. 1 and 2, the method includes:

and S10, controlling the first feature extraction module, the second feature extraction module and the third feature extraction module to determine a first feature map, a second feature map and a third feature map based on the MR image to be predicted.

Specifically, the first feature extraction module, the second feature extraction module and the third feature extraction module are parallel, the first feature extraction module is used for extracting a CK19 expression feature in the MR image to be predicted, the second feature extraction module is used for extracting a CK19 expression feature and an MVI shared feature in the MR image to be predicted to play a role in shared learning, and the third feature extraction module is used for extracting an MVI feature in the MR image to be predicted. The input items of the first feature extraction module, the second feature extraction module and the third feature extraction module are MR images to be predicted, the output item of the first feature extraction module is a first feature map carrying image features used for reflecting CK19 expression, the output item of the second feature extraction module is a second feature map carrying shared image features used for reflecting CK19 expression and MVI, and the output item of the third feature extraction module is a third feature map carrying image features used for reflecting MVI. In the embodiment, a CK19 expression characteristic, an MVI shared characteristic and an MVI characteristic are independently obtained through a first characteristic extraction module, a second characteristic extraction module and a third characteristic extraction module, and each characteristic extraction module is configured with independent parameters, so that a CK19 expression characteristic extraction task is only influenced by the CK19 expression characteristic extraction task, the MVI characteristic extraction task is only influenced by the MVI characteristic extraction task, and the CK19 expression and MVI shared characteristic extraction task is influenced by the CK19 expression characteristic extraction task and the MVI characteristic extraction task, therefore, the characteristic extraction module consisting of the first characteristic extraction module, the second characteristic extraction module and the third characteristic extraction module can learn not only the characteristic knowledge of the CK19 expression characteristic extraction task and the MVI characteristic extraction task, but also the shared knowledge among the CK19 expression characteristic extraction task and the MVI characteristic extraction task, which is favorable for avoiding that a network is biased to fit to the CK19 expression characteristic extraction task or the MVI characteristic extraction task due to complex parameter sharing, so as to predict that the network model has a negative increase and/or a performance shifts to another performance extraction task, and otherwise, when the CK19 expression task is the MVI characteristic extraction task is decreased, the CK19 expression task, and the MVI shared knowledge is an increase task, and the performance extraction task is considered as a decrease.

Therefore, in the embodiment, the CK19 expression feature extraction task, the MVI feature extraction task, and the CK19 expression and MVI shared feature extraction task are independent of each other, and the task parameter parameters of each task are independent, so that the problems that the negative migration phenomenon exists in the existing multi-task learning and the performance of each task cannot be improved at the same time can be solved. The method is that the existing multi-task learning shares one feature extraction module, and when related association or conflict exists among the multi-tasks and the feature sharing is directly carried out through the shared feature extraction module, the shared task parameters in the shared feature extraction module are complex and low in effectiveness, so that the model performance of the prediction network model is reduced. Meanwhile, in the learning process, the shared feature extraction module is led or influenced by a certain task, so that the prediction network model is more prone to fitting the certain task, the effect of other tasks is possibly negatively influenced, and the performance effect of other tasks is poor. In the embodiment, the CK19 expression feature extraction task, the MVI feature extraction task and the CK19 expression and MVI shared feature extraction task are respectively provided with the feature extraction modules, and the model parameters of the feature extraction modules are independently configured, so that the influence caused by the sharing of the task parameters of each task can be avoided, and meanwhile, each task is not influenced by other tasks, and the model performance of the prediction network model can be improved.

In addition, in this embodiment, in addition to independently setting the feature extraction module for the CK19 expression feature extraction task and the MVI feature extraction task, a shared feature extraction module (i.e., a second feature extraction module) for extracting the CK19 expression and MVI shared features is also provided, and the shared feature extraction module realizes a flexible balance between the CK19 expression feature extraction task and the MVI feature extraction task. In the feature extraction modules of the CK19 expression feature extraction task and the MVI feature extraction task, the feature extraction module corresponding to each task and the shared feature extraction module in charge of shared learning provide feature knowledge and shared feature knowledge of each task, so that the predictive network model can improve the predictive performance and the generalization performance of the CK19 expression feature extraction task and the MVI feature extraction task by utilizing the correlation of the CK19 expression feature extraction task and the MVI feature extraction task.

In an implementation manner of this embodiment, the network structures of the first feature extraction module, the second feature extraction module, and the third feature extraction module are the same. The first feature extraction module is described as an example. As shown in fig. 3, the first feature extraction module includes a convolution block a, a maximum pooling layer, a first residual unit, and a second residual unit, which are sequentially cascaded, where as shown in fig. 4, the convolution block a includes a convolution layer, a Group Normalization layer GN), and an active linear unit layer. In one implementation, the convolution kernel of a convolution layer in convolution block a may be a 3 x 3 convolution kernel.

The network structure of the first residual error unit and the second residual error unit is the same. As shown in fig. 5, the first residual unit includes a convolution block b, a convolution block a, a residual block, a convolution block b, a global average pooling layer, and a convolution layer, which are cascaded in sequence, wherein the network layer included in the convolution block b is the same as the network layer included in the convolution block a, and the difference between the two is that the convolution kernel of the convolution layer in the convolution block b is 1 × 1 convolution kernel. The residual block comprises a first sub-input layer, a second sub-input layer, a first adder, an adaptive pooling layer, a convolution block b, a first convolution layer, a second convolution layer, a softmax layer, a first multiplier, a second multiplier and a second adder, wherein the first sub-input layer is respectively connected with the first adder and the first multiplier, the second sub-input layer is respectively connected with the first adder and the second multiplier, the first adder is sequentially connected with the adaptive pooling layer and the convolution block b, the convolution block b is respectively connected with the first convolution layer and the second convolution layer, the first convolution layer is connected with the first multiplier, the second convolution layer is connected with the second multiplier, and the first multiplier and the second multiplier are both connected with the second adder, wherein input items of the first sub-input layer and the second sub-input layer are output items of the convolution block a, and the first sub-input layer and the second sub-input layer have input and output functions, namely output items of the first sub-input layer and the second sub-input layer are output items of the convolution block a.

S20, controlling the prediction module to determine a CK19 expression type and an MVI type corresponding to the MR image to be predicted based on the first feature map, the second feature map and the third feature map.

Specifically, the prediction module is configured to predict a CK19 expression category and an MVI category corresponding to the MR image to be predicted, where the CK19 expression category includes CK19 expression negativity or CK19 expression positivity, and the MVI category includes MVI or no MVI; the MR image to be predicted is a magnetic resonance image formed by Gd-EOB-DTPA enhanced MRI examination of a liver cancer patient. The prediction module is used for absorbing image features extracted by the first feature extraction module based on the CK19 expression task, extracting image features extracted by the third feature extraction module based on the MVI task, extracting shared image features extracted by the second feature extraction module based on the CK19 expression task and the MVI task, and determining a CK19 expression category and an MVI category corresponding to the MR image to be predicted based on all the extracted image features.

In an implementation manner of this embodiment, as shown in fig. 6, the prediction module includes a first prediction branch and a second prediction branch, where the first prediction branch and the second prediction branch both include a convolution unit, a spatial transform unit, and a prediction unit that are sequentially cascaded. Correspondingly, the controlling the prediction module to determine the CK19 expression category and the MVI category corresponding to the MR image to be predicted based on the first feature map, the second feature map, and the third feature map specifically includes:

Specifically, the first prediction branch is used for predicting a CK19 expression category corresponding to the MR image to be predicted, and the second prediction branch is used for predicting an MVI category corresponding to the MR image to be predicted, where an input item of a convolution unit in the first prediction branch is a fusion graph of the first feature map and the second feature map, and an input item of a convolution unit in the second prediction branch is a fusion graph of the second feature map and the third feature map, where the fusion graph is formed based on channel splicing. That is to say, the fused graph of the first feature map and the second feature map is obtained by splicing the first feature map and the second feature map according to the channel direction, and the fused graph of the second feature map and the third feature map is obtained by splicing the third feature map and the second feature map according to the channel direction.

As shown in fig. 7, the convolution unit includes a convolution block c, a third residual unit, and a fourth residual unit, which are cascaded in sequence; as shown in fig. 8, the convolution block c includes a convolution layer, a Batch Normalization layer (BN), and a Leaky Linear Unit (Leaky ReLU) layer, where the convolution layer is connected to the Batch Normalization layer, the Batch Normalization layer is connected to the Leaky Linear Unit layer, and a convolution kernel of the convolution layer is 3 × 3 convolution kernels. Note that the network configuration of the third residual unit and the fourth residual unit is the same as that of the first residual unit, and reference may be made to the network configuration of the first residual unit, which will not be described in detail here. The prediction unit may include a max pooling layer and a fully-connected layer through which a CK19 expression category or MVI category is output.

The spatial transformation unit is used for carrying out affine transformation on the feature map output by the convolution unit so as to remove noise in the feature map, so that the model performance of the prediction network model can be improved, meanwhile, the spatial transformation unit can enhance the generalization capability of the prediction network module and improve the learning capability of the prediction network model. In one implementation, the affine transformation performed on the feature map by the spatial transformation unit may be represented as:

wherein,

pixel point representing an image, is->

Representing pixels of the image after affine transformation, A _θ Representing a matrix of affine transform coefficients. The affine transform coefficient matrix may include one or more of a translation coefficient, a scaling coefficient, a flipping coefficient, a rotation coefficient, and a shearing coefficient. In the space transformation unit, an affine transformation coefficient matrix is learned through the training process of the prediction network model, the image is transformed into a next expected form, meanwhile, the interested area of the prediction network model for the characteristic diagram can be automatically selected in the training process, and therefore the prediction accuracy of the prediction network model is improved.

As shown in fig. 9, the spatial transform unit may include a positioning network block connected to the network generator, a network generator connected to the sampler, the positioning network block being configured to generate an affine transformation coefficient matrix, the network generator being configured to generate a sampling grid based on the affine transformation coefficient matrix, and a sampler configured to perform position mapping on the feature map input to the spatial transform unit based on the sampling network to obtain an affine feature map.

In one implementation, as shown in fig. 9, the positioning network block includes a first convolutional layer, a second convolutional layer, a first leaky linear activation unit layer, a third convolutional layer, a second leaky linear activation unit layer, a first fully-connected layer, and a second fully-connected layer, which are sequentially cascaded, where a convolution kernel of the first convolutional layer is 1 × 1 convolution kernel, a convolution kernel of the second convolutional layer is 3 × 3 convolution kernel, and a convolution kernel of the third convolutional layer is 5 × 5 convolution kernel. The network generator generates a grid according to the image size of the affine characteristic image determined by the space transformation unit, and then determines the adopted network T corresponding to the output item of the space transformation unit based on the affine transformation coefficient matrix and the generated network _θ (G) In that respect The sampler may include an adder that performs position mapping on the feature map of the input spatial transform unit by using the layer pair to obtain an affine feature map after affine transformation, and may also perform completion of interpolation on a value-free region in the affine feature map by using an interpolation method.

In the embodiment, a spatial transformation unit is added into the prediction network model, the spatial transformation unit absorbs the image features extracted by the first feature extraction module based on the CK19 expression task, the third feature extraction module based on the MVI task, and the shared image features extracted by the second feature extraction module based on the CK19 expression task and the MVI task, and performs affine transformation on the effective image features in the fusion graph of the first feature graph and the second feature graph input into the spatial transformation unit or the fusion of the second feature graph and the third feature graph through the spatial transformation unit, so that the prediction unit can perform CK19 expression prediction and MVI prediction based on the affine feature graph obtained by affine transformation, thereby reducing the noise information of the feature graph and the CK19 expression prediction task or the MVI prediction task, transforming the input features into a form expected by a next layer of the network, contributing to improving the generalization of the network model, and improving the prediction accuracy of the CK19 expression prediction task and the MVI prediction task. Of course, in practical application, the first prediction branch and the second prediction branch may not include a spatial variation unit, that is, the first prediction branch and the second prediction branch may only include a convolution unit and a prediction unit that are cascaded in sequence.

In an implementation manner of this embodiment, as shown in fig. 10, the prediction module includes a first prediction branch and a second prediction branch, where the first prediction branch and the second prediction branch both include a feature transformation unit, a relationship inference unit, and a prediction unit that are sequentially cascaded, and the feature transformation unit includes a convolution unit and a spatial transformation unit that are sequentially cascaded; the controlling the prediction module to determine the CK19 expression type and the MVI type corresponding to the MR image to be predicted based on the first feature map, the second feature map and the third feature map specifically includes:

Specifically, the convolution unit and the spatial transform unit in the prediction unit and the feature transform unit are all the same as those in the foregoing implementation, and specific reference may be made to the description of the foregoing implementation, which is not repeated herein. The description of the relational inference unit is focused here. The relation reasoning unit in the first prediction branch is respectively connected with the relation reasoning unit in the first prediction branch and the feature transformation unit in the second prediction branch, and the relation reasoning unit in the second prediction branch is respectively connected with the feature transformation unit in the first prediction branch and the feature transformation unit in the second prediction branch. That is to say, the input items of the relationship inference unit in the first prediction branch are the first affine feature map determined by the relationship inference unit in the first prediction branch and the second affine feature map determined by the relationship inference unit in the second prediction branch, and the input items of the relationship inference unit in the second prediction branch are the first affine feature map determined by the relationship inference unit in the first prediction branch and the second affine feature map determined by the relationship inference unit in the second prediction branch.

The relationship inference unit is used for determining the correlation between the first affine feature map and the second affine feature map, namely, the relationship inference unit is used for extracting the correlation between the CK19 expression prediction task and the MVI prediction task, and the prediction unit is made to know the correlation, so that the prediction accuracy of the prediction network model is improved. In one implementation, the relational inference unit may be represented as a relational function, where the expression of the relational function may be:

wherein, LR (A) _l ，A _m ) Representing a relational feature graph, A _l Representing a first affine feature map, A _m A second affine feature map is represented which,

and f _□ Is a function for reflecting the pairwise relationship between tasks.

In a specific implementation manner, the relational inference unit comprises a volume block, a global pooling layer and a multi-layer perceptron; the convolution block is used for acquiring a feature relation graph of the two affine feature graphs input into the associated reasoning unit; the global pooling layer is used for mapping the feature relationship graphConverting into a feature relation vector; the multilayer perceptron is configured to generate a relational feature map based on the feature relationship vector. That is, f _□ By means of volume blocks, a global average pooling layer and a multi-layer perceptron,

is realized by a multilayer perceptron. As shown in fig. 11, the convolution block includes a convolution layer, a group normalization layer, an activated linear unit layer with leakage, four convolution blocks c, an adaptive pooling layer, a full-link layer, an activated linear unit layer, a full-link layer, and an activated linear unit layer, which are sequentially cascaded. The network structure of the convolution block c is the same as that of the convolution block c in the convolution unit, and will not be described in detail here. In addition, the convolution kernel of the convolution layer is a 1 × 1 convolution kernel.

In this embodiment, the global average pooling layer and the multilayer perceptron in the relational inference unit both have model parameters that can be trained, and the model parameters are obtained by training in the training process of the prediction network model, that is, end-to-end learning enables the relational inference unit to learn the relationship between the CK19 expression prediction task and the MVI prediction task. Meanwhile, the relational reasoning unit automatically learns the mutual relation between the CK19 expression prediction task and the MVI prediction task in a data-driven mode without inputting any prior knowledge about task relation to the prediction network model.

In the embodiment, the relationship inference unit is arranged in the prediction network model, on a branch of a CK19 expression prediction task, a feature map corresponding to the CK19 expression prediction task and a feature map corresponding to an MVI prediction task are connected in series, the feature maps after being connected in series are input into the relationship inference unit, the relationship inference unit analyzes the feature maps obtained after being connected in series, then the prediction unit performs CK19 expression prediction based on the relationship feature map obtained by the relationship inference unit, the prediction is performed on the branch of the MVI prediction task in the same way, so that the relationship inference unit can influence the feature learning of a network layer at the upstream of a module, an implicit feature capable of performing relationship inference is generated, and the performance of the prediction network model in the CK19 expression prediction task and the MVI prediction task is improved.

In an implementation manner of this embodiment, as shown in fig. 12, in the hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning, the prediction module includes a first prediction branch and a second prediction branch, where the first prediction branch and the second prediction branch each include a convolution unit, a relationship inference unit, and a prediction unit that are sequentially cascaded; the controlling the prediction module to determine the CK19 expression category and the MVI category corresponding to the MR image to be predicted based on the first feature map, the second feature map, and the third feature map specifically includes:

controlling convolution units in the first prediction branch to determine a third affine feature map based on the first feature map and the second feature map;

Specifically, the prediction network model does not include a spatial variation unit, the relationship inference unit is directly connected to the convolution unit, and an output item of the convolution unit is used as an input item of the convolution unit and performs relationship inference to obtain a relationship characteristic diagram, wherein network structures and functions of the convolution unit, the relationship inference unit and the prediction unit are the same as those of the convolution unit, the relationship inference unit and the prediction network, and are not repeated here.

In an implementation manner of this embodiment, before predicting a CK19 expression category and an MVI category corresponding to an MR image to be predicted based on the prediction network model, the prediction network model is trained in advance, wherein in a training process of the prediction network model, a training sample set used by the prediction network model includes a plurality of training MR images, and each of the plurality of training MR images is an MR image obtained by Gd-EOB-DTPA enhanced MRI examination of a liver cancer patient. In a specific implementation manner, each training MR image is obtained by performing Gd-EOB-DTPA enhanced MRI examination within one month before a liver cancer operation, a liver cancer tumor carried by a liver cancer patient corresponding to each training MR image is a single-shot lesion, each training MR image carries a CK19 expression category label and an MVI category label, and before a liver cancer surgical resection treatment, other treatments for HCC are not performed, wherein the CK19 expression category label can be determined by immunohistochemical analysis after the liver cancer operation, and the MVI category label is determined by pathological confirmation after the liver cancer operation.

After the training sample set is obtained, the training sample set may be preprocessed, where the preprocessing may include one or more of resampling, image cropping, image size unification, and data amplification. The resampling is specifically to count the resolution of the training MR images in the received training sample set, and determine the resolution (e.g., 1.19 × 1.19 × 2 mm) corresponding to the training MR image with the most resolution ³ ) As the target resolution, the resolutions of all the training MR images in the training sample set are resampled to the target resolution, so that the resampling operation on excessive data can be avoided, and meanwhile, the generalization of the prediction network model can be improved.

The image cutting is to reduce interference of background information of a training MR image on a network model, wherein the image cutting is to cut each layer of image in the training MR image based on a focus area, wherein the preset focus area can be formed by drawing three layers of gold standards on the cross section of the MR image of a patient based on ITK software by a radiologist, the three layers of gold standards are respectively the top layer, the maximum layer and the bottom layer of a tumor, and finally, a three-dimensional frame containing the whole focus is formed based on the three layers of gold standards.

The image size is uniformly used for adjusting the image size of the cropped training MR images so that the image sizes of all the adjusted training MR images are the same, wherein the adjusted image size may be the maximum image size of all the training MR images. For example, the image size of each slice image in the adjusted training MR image is 224 × 224, which can be achieved by performing a zero padding operation around the image of each slice image of the training MR image.

The data amplification aims to improve the diversity of the training sample set and improve the generalization capability of the prediction network model. Wherein the data augmentation may include (1) image flipping: turning over the original image in the horizontal direction or the vertical direction; (2) image random cropping: image clipping is carried out on the original image, and the clipping amplitude is 0-10%; (3) image scaling: randomly zooming to 70% -110% of the original image; (4) image translation: translating 0-10% in the horizontal direction and the vertical direction; (5) image rotation: rotating the image by a rotation angle within the range of-20 degrees; (6) shear transformation: and (4) cutting and transforming the original image with the amplitude of-16 degrees.

In the training process of the prediction network model, an internal cross validation result is obtained by using a 10-fold cross validation method: the Area (AUC) of a Receiver Operating Characteristic Curve (ROC) of a CK19 expression prediction task Under the ROC can reach 0.87, and the accuracy rate reaches 0.83; the AUC of the MVI prediction task of the liver cancer can reach 0.88, and the accuracy rate reaches 0.85. Then, in order to evaluate the generalization ability of the algorithm, an external independent test set is adopted to carry out external independent test on the prediction network model obtained by training, so as to obtain an external independent verification result: AUC of the CK19 expression prediction task can reach 0.80, and the accuracy rate reaches 0.84; the AUC of the MVI prediction task of the liver cancer can reach 1.00, and the accuracy rate reaches 0.89.

In summary, the present embodiment provides a hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning, the method includes: controlling a first feature extraction module, a second feature extraction module and a third feature extraction module to determine a first feature map, a second feature and a third feature map based on the MR image to be predicted; and the control prediction module determines a CK19 expression type and an MVI type of the MR image based on the first feature map, the second feature map and the third feature map. According to the method and the device, the abundant image characteristics carried by the MR image to be predicted are directly extracted through the prediction network model, and the problem that the prediction performance of the model is influenced due to subjectivity in image characteristic analysis can be avoided. Meanwhile, the CK19 expression features are extracted through the first feature extraction module, the shared features of CK19 expression and MVI are extracted through the second feature extraction module, the MVI features are extracted through the third feature extraction module, task parameters of a shared task and task parameters of an independent task are separated, and the problem that prediction results are unstable due to complex parameter sharing can be solved.

Based on the hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning, the embodiment provides a hepatocellular carcinoma CK19 and MVI prediction device based on multitask learning, the prediction device is configured with a trained prediction network model, the prediction network model comprises a first feature extraction module for extracting CK19 expression features, a second feature extraction module for extracting shared features of CK19 expression and MVI, a third feature extraction module for extracting MVI features, and a prediction module, as shown in fig. 13, the prediction device comprises:

a first control module 100, configured to control the first feature extraction module, the second feature extraction module, and the third feature extraction module to determine a first feature map, a second feature map, and a third feature map based on an MR image to be predicted;

a second control module 200, configured to control the prediction module to determine, based on the first feature map, the second feature map, and the third feature map, a CK19 expression category and an MVI category corresponding to the MR image to be predicted.

Based on the above-mentioned hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning, the present embodiment provides a computer-readable storage medium, where one or more programs are stored, and the one or more programs can be executed by one or more processors to implement the steps in the hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning according to the above-mentioned embodiments.

Based on the hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning, the present application further provides a terminal device, as shown in fig. 14, including at least one processor (processor) 20; a display screen 21; and a memory (memory) 22, and may further include a communication Interface (Communications Interface) 23 and a bus 24. The processor 20, the display 21, the memory 22 and the communication interface 23 can communicate with each other through the bus 24. The display screen 21 is configured to display a user guidance interface preset in the initial setting mode. The communication interface 23 may transmit information. The processor 20 may call logic instructions in the memory 22 to perform the methods in the embodiments described above.

Furthermore, the logic instructions in the memory 22 may be implemented in software functional units and stored in a computer readable storage medium when sold or used as a stand-alone product.

The memory 22, which is a computer-readable storage medium, may be configured to store a software program, a computer-executable program, such as program instructions or modules corresponding to the methods in the embodiments of the present disclosure. The processor 20 executes the functional application and data processing, i.e. implements the method in the above-described embodiments, by executing the software program, instructions or modules stored in the memory 22.

The memory 22 may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to the use of the terminal device, and the like. Further, the memory 22 may include a high speed random access memory and may also include a non-volatile memory. For example, a variety of media that can store program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk, may also be transient storage media.

In addition, the specific processes loaded and executed by the storage medium and the instruction processors in the terminal device are described in detail in the method, and are not stated herein.

Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions in the embodiments of the present application.

Claims

1. A hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning is characterized in that the method is applied to a trained prediction network model, and the prediction network model comprises a first feature extraction module, a second feature extraction module, a third feature extraction module and a prediction module, wherein the first feature extraction module is used for extracting CK19 expression features, the second feature extraction module is used for extracting shared features of CK19 expression and MVI, and the third feature extraction module is used for extracting MVI features; the method comprises the following steps:

controlling the prediction module to determine a CK19 expression category and an MVI category corresponding to the MR image to be predicted based on the first feature map, the second feature map and the third feature map;

the prediction module comprises a first prediction branch and a second prediction branch, the first prediction branch and the second prediction branch respectively comprise a feature transformation unit, a relation inference unit and a prediction unit which are sequentially cascaded, and the feature transformation unit comprises a convolution unit and a space transformation unit which are sequentially cascaded; the controlling the prediction module to determine the CK19 expression category and the MVI category corresponding to the MR image to be predicted based on the first feature map, the second feature map, and the third feature map specifically includes:

2. The method of claim 1, wherein the spatial transform unit comprises a positioning network block, a network generator and a sampler, the positioning network block is connected to the network generator, the network generator is connected to the sampler, the positioning network block is used for generating an affine transform coefficient matrix, the network generator is used for generating a sampling grid based on the affine transform coefficient matrix, and the sampler is used for mapping the position of the feature map input to the spatial transform unit based on the sampling network to obtain an affine feature map.

3. The hepatocellular carcinoma CK19 and MVI prediction method based on multitask learning as claimed in claim 1, wherein the prediction module comprises a first prediction branch and a second prediction branch, and the first prediction branch and the second prediction branch each comprise a convolution unit, a relation inference unit and a prediction unit which are cascaded in sequence; the controlling the prediction module to determine the CK19 expression category and the MVI category corresponding to the MR image to be predicted based on the first feature map, the second feature map, and the third feature map specifically includes:

4. The method for predicting hepatocellular carcinoma CK19 and MVI based on multitask learning according to claim 1 or 3, wherein the relational inference unit comprises a volume block, a global pooling layer and a multi-layer perceptron; the convolution block is used for acquiring a feature relationship diagram of two affine feature diagrams input into the relationship inference unit; the global pooling layer is used for converting the characteristic relation graph into a characteristic relation vector; the multilayer perceptron is configured to generate a relational feature map based on the feature relationship vector.

5. The method of claim 1, wherein the CK19 expression categories comprise CK19 expression negativity or CK19 expression positivity; the MVI category includes MVI or no MVI.

6. A hepatocellular carcinoma CK19 and MVI prediction device based on multitask learning is characterized in that the prediction device is provided with a trained prediction network model, the prediction network model comprises a first feature extraction module used for extracting CK19 expression features, a second feature extraction module used for extracting shared features of CK19 expression and MVI, a third feature extraction module used for extracting MVI features and a prediction module, and the prediction device comprises:

the second control module is used for controlling the prediction module to determine a CK19 expression type and an MVI type corresponding to the MR image to be predicted based on the first feature map, the second feature map and the third feature map;

7. A computer readable storage medium, storing one or more programs, the one or more programs being executable by one or more processors for performing the steps of the method for hepatocellular carcinoma CK19 and MVI prediction based on multitask learning according to any one of claims 1-5.

8. A terminal device, comprising: a processor, a memory, and a communication bus; the memory has stored thereon a computer readable program executable by the processor;

the processor, when executing the computer readable program, implements the steps in the method for predicting hepatocellular carcinoma CK19 and MVI based on multitask learning according to any one of claims 1-5.