CN113192085A

CN113192085A - Three-dimensional organ image segmentation method and device and computer equipment

Info

Publication number: CN113192085A
Application number: CN202110512121.4A
Authority: CN
Inventors: 张耀; 田疆; 张杨
Original assignee: Lenovo Beijing Ltd
Current assignee: Lenovo Beijing Ltd
Priority date: 2021-05-11
Filing date: 2021-05-11
Publication date: 2021-07-30

Abstract

The application provides a three-dimensional organ image segmentation method, a device and a computer device, which are used for extracting the characteristics of a three-dimensional organ image to obtain a plurality of three-dimensional organ characteristic graphs with different scales, calibrating the characteristic graphs on corresponding scales by utilizing the spatial correlation information and the semantic correlation information among the characteristics of different levels to enhance the representation of a target object on the context with different scales, specifically inputting the three-dimensional organ characteristic graphs with different scales, accurately and completely identifying the edge and the category of the target object in a target three-dimensional characteristic graph by utilizing more complete and accurate spatial detail information and target semantic information based on an organ characteristic calibration model obtained by a spatial attention mechanism and a semantic attention mechanism training, segmenting the three-dimensional organ image accordingly, greatly improving the segmentation effect of an organ region, namely, target objects with different sizes, such as tumors with different sizes and the like, contained in the three-dimensional organ image are accurately identified.

Description

Three-dimensional organ image segmentation method and device and computer equipment

Technical Field

The present application relates to the field of image processing technologies, and in particular, to a method and an apparatus for segmenting a three-dimensional organ image, and a computer device.

Background

With the development of various image processing techniques, there is an increasing research on the processing and analysis of medical images. Taking a liver tumor identification scene as an example, at present, an image segmentation technology is generally used to extract a liver tumor region in an acquired liver image, so as to assist in achieving accuracy and efficiency of subsequent image processing.

In practical application, because the sizes of liver tumors in different classes and different stages are different, in the image segmentation processing process, small-sized tumors are easily ignored in the liver feature map of the same size, and the edges of the large-sized tumors obtained by segmentation may be incomplete, so that the recognition integrity and accuracy of the liver tumor region are low.

Disclosure of Invention

In view of the above, the present application provides a three-dimensional organ image segmentation method, including:

acquiring a three-dimensional organ image to be segmented;

extracting the features of the three-dimensional organ image to obtain a plurality of three-dimensional organ feature maps with different scales;

inputting the three-dimensional organ characteristic diagrams with different scales into an organ characteristic calibration model, and outputting a target three-dimensional organ characteristic diagram; the organ feature calibration model is obtained by performing calibration training on sample three-dimensional organ features of different scales based on a space attention mechanism and a semantic attention mechanism;

and utilizing the target three-dimensional organ characteristic diagram to perform segmentation processing on the three-dimensional organ image and outputting an organ region segmentation result of the three-dimensional organ image.

Optionally, the inputting the three-dimensional organ feature maps of different scales into the organ feature calibration model and outputting the target three-dimensional organ feature map includes:

calibrating the three-dimensional organ feature map with a smaller scale by using the spatial correlation information between the two three-dimensional organ feature maps with adjacent scales to obtain an undetermined organ feature map with a corresponding scale;

calibrating the undetermined feature map with a larger scale by utilizing semantic correlation information between the two undetermined organ feature maps with adjacent scales to obtain a candidate organ feature map with a corresponding scale;

and performing fusion processing on the obtained candidate organ characteristic graphs with different scales to obtain a target three-dimensional organ characteristic graph.

Optionally, the calibrating the organ feature map with a smaller scale by using spatial correlation information between two three-dimensional organ feature maps with adjacent scales to obtain the undetermined organ feature map with a corresponding scale includes:

processing the two three-dimensional organ feature maps of adjacent scales to obtain a space attention map aiming at the three-dimensional organ feature map of a smaller scale;

and utilizing the space attention map to calibrate the corresponding three-dimensional organ feature map with the smaller scale to obtain the undetermined organ feature map with the corresponding scale.

Optionally, the processing two three-dimensional organ feature maps of adjacent scales to obtain a spatial attention map for a three-dimensional organ feature map of a smaller scale includes:

merging the two three-dimensional organ feature maps of adjacent scales to obtain a merged feature map;

and inputting the merged feature map into a spatial attention network, and outputting a spatial attention map of a three-dimensional organ feature map with a smaller scale in the two three-dimensional organ feature maps with the adjacent scales.

Optionally, the calibrating the corresponding smaller-scale three-dimensional organ feature map by using the spatial attention map to obtain a corresponding-scale undetermined organ feature map, including:

carrying out down-sampling and normalization processing on the spatial attention diagram to obtain a calibration organ feature diagram;

performing format conversion processing on the three-dimensional organ characteristic diagram with a smaller scale in the two three-dimensional organ characteristic diagrams with adjacent scales to obtain an organ characteristic diagram to be calibrated, wherein the organ characteristic diagram to be calibrated is matched with the format of the corresponding spatial attention diagram;

and performing characteristic product operation on the organ characteristic diagram to be calibrated and the corresponding organ characteristic diagram to be calibrated to obtain the organ characteristic diagram to be determined with the corresponding scale.

Optionally, the calibrating the characteristic map of the undetermined organ with a larger scale by using semantic correlation information between the two characteristic maps of the undetermined organ with adjacent scales to obtain the candidate characteristic map of the organ with the corresponding scale includes:

processing the two obtained characteristic images of the organs to be determined of the adjacent scales to obtain a semantic attention vector aiming at the characteristic image of the organs to be determined of the larger scale;

and utilizing the semantic attention vector to calibrate the corresponding larger-scale to-be-determined organ feature map to obtain a candidate organ feature map of the corresponding scale.

Optionally, the processing the two obtained feature maps of the undetermined organ of the adjacent scales to obtain a semantic attention vector for the feature map of the undetermined organ of the larger scale includes:

performing maximum pooling and average pooling on the two obtained to-be-determined organ feature maps of adjacent scales on channel dimensions respectively, and merging organ feature vectors obtained by processing to obtain semantic organ feature vectors;

and performing regression processing on the semantic organ feature vector to obtain a semantic attention vector aiming at the undetermined organ feature map with a larger scale.

Optionally, the calibrating the corresponding larger-scale to-be-determined organ feature map by using the semantic attention vector to obtain a candidate organ feature map of a corresponding scale includes:

carrying out normalization processing on the semantic attention vector to obtain a calibrated semantic organ feature vector;

performing format conversion processing on the characteristic diagram of the undetermined organ with the larger scale in the two acquired characteristic diagrams of the undetermined organ with the adjacent scales to acquire the characteristic diagram of the undetermined organ to be calibrated, which is matched with the format of the corresponding semantic attention vector;

and performing product operation on the to-be-calibrated undetermined organ feature map and the calibration semantic organ feature vector to obtain a candidate organ feature map with a corresponding scale.

The present application further proposes a three-dimensional organ image segmentation apparatus, the apparatus comprising:

the image acquisition module is used for acquiring a three-dimensional organ image to be segmented;

the characteristic extraction module is used for extracting the characteristics of the three-dimensional organ image to obtain a plurality of three-dimensional organ characteristic graphs with different scales;

the characteristic calibration module is used for inputting the three-dimensional organ characteristic diagrams with different scales into the organ characteristic calibration model and outputting a target three-dimensional organ characteristic diagram; the organ feature calibration model is obtained by performing calibration training on sample three-dimensional organ features of different scales based on a space attention mechanism and a semantic attention mechanism;

and the image segmentation module is used for segmenting the three-dimensional organ image by using the target three-dimensional organ characteristic diagram and outputting an organ region segmentation result of the three-dimensional organ image.

The present application further proposes a computer device, the computer device comprising:

a communication module;

a memory for storing a program for implementing the three-dimensional organ image segmentation method as described above;

and the processor is used for loading and executing the program stored in the memory so as to realize the steps of the three-dimensional organ image segmentation method.

Therefore, the application provides a three-dimensional organ image segmentation method, a three-dimensional organ image segmentation device and a computer device, in order to accurately identify target objects with different sizes contained in a three-dimensional organ image, such as liver tumors with different sizes, and the like, the three-dimensional organ image can be subjected to feature extraction to obtain a plurality of three-dimensional organ feature maps with different scales, the application provides a method for calibrating the feature maps on corresponding scales by utilizing spatial correlation information and semantic correlation information among different levels of features to enhance the representation of the target objects (such as liver tumors and the like) on contexts with different scales, specifically can input the three-dimensional organ feature maps with different scales, and accurately and completely identify the edges and categories of the target objects in the target three-dimensional feature map by utilizing more complete and accurate spatial detail information and target semantic information based on an organ feature calibration model obtained by training of a spatial attention machine and a semantic attention machine, therefore, the three-dimensional organ image is segmented, the organ region segmentation effect is greatly improved, and target objects with different sizes, such as tumors with different sizes and the like, contained in the three-dimensional organ image are accurately identified.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, it is obvious that the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.

Fig. 1 is a schematic flow chart of an alternative example of the three-dimensional organ image segmentation method proposed in the present application;

fig. 2 is a schematic structural diagram of an optional feature extraction model for obtaining three-dimensional organ feature maps of different scales in the three-dimensional organ image segmentation method provided by the present application;

FIG. 3 is a schematic flow chart of yet another alternative example of the three-dimensional organ image segmentation method proposed in the present application;

FIG. 4 is a schematic flow chart of yet another alternative example of the three-dimensional organ image segmentation method proposed in the present application;

FIG. 5 is a schematic flow chart of yet another alternative example of the three-dimensional organ image segmentation method proposed in the present application;

fig. 6 is a schematic flowchart of an alternative example of obtaining a feature map of an organ to be located based on a spatial attention mechanism in the three-dimensional organ image segmentation method provided by the present application;

fig. 7 is a schematic flowchart of another alternative example of obtaining a feature map of an organ to be located based on a spatial attention mechanism in the three-dimensional organ image segmentation method provided by the present application;

FIG. 8 is a schematic flow chart illustrating an alternative example of obtaining a candidate organ feature map based on a semantic attention mechanism in the three-dimensional organ image segmentation method proposed in the present application;

FIG. 9 is a schematic flow chart illustrating another alternative example of obtaining a candidate organ feature map based on a semantic attention mechanism in the three-dimensional organ image segmentation method proposed in the present application;

fig. 10 is a schematic structural diagram of an alternative example of the three-dimensional organ image segmentation apparatus proposed in the present application;

fig. 11 is a schematic structural diagram of still another alternative example of the three-dimensional organ image segmentation apparatus proposed in the present application;

fig. 12 is a schematic hardware configuration diagram of an alternative example of a computer device suitable for the three-dimensional organ image segmentation method and apparatus proposed in the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments, and for convenience of description, only the parts related to the related inventions are shown in the drawings. In case of conflict, the embodiments and features of the embodiments in the present application can be combined with each other, that is, all other embodiments obtained by a person of ordinary skill in the art without any creative effort based on the embodiments in the present application belong to the protection scope of the present application.

It should be understood that "system", "apparatus", "unit" and/or "module" as used herein is a method for distinguishing different components, elements, parts or assemblies at different levels. However, other words may be substituted by other expressions if they accomplish the same purpose.

As used in this application and the appended claims, the terms "a," "an," "the," and/or "the" are not intended to be inclusive in the singular, but rather are intended to be inclusive in the plural unless the context clearly dictates otherwise. In general, the terms "comprises" and "comprising" merely indicate that steps and elements are included which are explicitly identified, that the steps and elements do not form an exclusive list, and that a method or apparatus may include other steps or elements. An element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in the process, method, article, or apparatus that comprises the element.

In the description of the embodiments herein, "/" means "or" unless otherwise specified, for example, a/B may mean a or B; "and/or" herein is merely an association describing an associated object, and means that there may be three relationships, e.g., a and/or B, which may mean: a exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of the present application, "a plurality" means two or more than two. The terms "first", "second" and the like are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature.

Additionally, flow charts are used herein to illustrate operations performed by systems according to embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed in the exact order in which they are performed. Rather, the various steps may be processed in reverse order or simultaneously. Meanwhile, other operations may be added to the processes, or a certain step or several steps of operations may be removed from the processes.

For the description of the background technology, because image segmentation models such as convolutional neural networks extract image features of different levels by using multiple layers of convolutional kernels of different scales, the identification of corresponding large and small organ (such as liver tumor) regions is realized on different convolutional layers, but the image segmentation method directly fuses multiple layers of information without considering the characteristics of different levels and the correlation among the different levels, which will affect the final organ identification effect, namely, reduce the accuracy of three-dimensional organ image segmentation.

In order to improve the above problems, the present application proposes to realize three-dimensional organ image segmentation based on Attention Mechanism (AM) with the application of technologies such as deep learning, machine learning, and computer vision, which are included in Artificial Intelligence (AI), in the field of image processing. The attention mechanism can be intuitively explained by using a human vision mechanism, and can be considered as a resource allocation mechanism, namely, resources are allocated according to the importance degree of an attention object (such as a certain feature extracted from an image), more resources are allocated to the important object, and less resources are allocated to an unimportant or bad object.

Based on this, in the embodiment of the present application, spatial correlation and semantic correlation between organ features proposed by different convolutional layers may be obtained, so as to adjust weights of different organ features according to these correlations, enhance representation of context information of an object to be identified (e.g., a liver tumor) in different scales, that is, increase weights of a network on a region of interest (i.e., a region where the object to be identified is located, such as a liver tumor region) from a channel and a spatial dimension, and decrease weights of a non-region of interest (e.g., a non-tumor region in a liver image, etc.), so as to extract features of the region of interest in different scales more effectively and accurately, that is, improve a three-dimensional organ image segmentation effect, and further improve liver tumor identification accuracy. It should be noted that the three-dimensional organ image segmentation method provided by the present application is not limited to a liver tumor identification scene, and implementation processes of identification scenes of other organs are similar, and detailed descriptions thereof are not given in the present application.

In conjunction with the above description of the technical concept of the three-dimensional organ image segmentation method proposed by the present application, the following describes in detail the three-dimensional organ image segmentation method proposed by the present application in conjunction with an application example of a liver tumor recognition scenario, which includes, but is not limited to, the method steps described in the following embodiments.

Referring to fig. 1, a flow chart of an alternative example of the three-dimensional organ image segmentation method proposed in the present application is schematically illustrated, and the method may be applied to a computer device, which may be a server or a terminal device with certain data processing capability. The server can be an independent physical server, a server cluster integrated by a plurality of physical servers, a cloud server with cloud computing capability and the like; the terminal devices may include, but are not limited to: the mobile terminal comprises a smart phone, a tablet personal computer, an ultra-mobile personal computer (UMPC), a netbook, a Personal Digital Assistant (PDA), an Augmented Reality (AR) device, a Virtual Reality (VR) device, a robot, a desktop computer and the like.

As shown in fig. 1, the three-dimensional organ image segmentation method proposed by the present embodiment may include, but is not limited to, the following steps:

step S11, obtaining a three-dimensional organ image to be segmented;

in the application, the three-dimensional organ image to be segmented may be obtained by scanning a detected object (e.g., a living body with an organ to be identified, such as a sick patient) to obtain a three-dimensional image of a corresponding body part, and since the detected object is different in different application scenes and the corresponding scanned body part is different, the content of the obtained three-dimensional organ image to be segmented may also be changed accordingly.

It can be understood that the three-dimensional organ image is acquired by an independent image acquisition device according to, but not limited to, the above manner, and then is sent to the computer device; the three-dimensional image scanning of the detected object can also be realized by an image collector of the computer equipment to obtain a corresponding three-dimensional organ image to be segmented, the specific implementation process can be determined according to the situation, and the detailed description is omitted in the application.

Step S12, extracting the characteristics of the three-dimensional organ image to obtain a plurality of three-dimensional organ characteristic diagrams with different scales;

in a liver tumor identification scene, for example, due to the fact that the sizes of liver tumors are different, in order to accurately identify the liver tumors of all sizes contained in a three-dimensional organ image, the method and the device provide that feature extraction networks with various different scales are used for extracting features (namely downsampling with different resolutions) of the obtained three-dimensional organ image, and three-dimensional organ feature maps with corresponding scales containing the liver tumor features of different sizes are obtained. Specifically, a convolutional neural network can be used for coding an input three-dimensional organ image and extracting features of different depths of the network, so that the receptive field and the width of shallow feature are small, and local low-level texture features are concerned more; the deep characteristic receptive field and the width are large, and high-level semantic information is concerned, so that the method and the device can extract the characteristics of the three-dimensional organ image by using the convolution neural network with convolution kernels of different scales, and can obtain characteristic images of different scales containing multi-dimensional information.

In order to enable the obtained Feature map to contain more comprehensive and accurate Feature information, the method and the device can select a Pyramid Feature extraction network formed by a full convolution neural network, namely a Feature Pyramid Network (FPN) to realize Feature extraction of the three-dimensional organ image, so as to obtain the Feature map with higher quality, and the method and the device do not limit the specific network structure of the Feature Pyramid network.

Referring to the schematic diagram of the structure of the feature pyramid network shown in fig. 2, the feature pyramid network may include an encoder and a decoder, and the encoder and the decoder may be connected by a bottleneck layer. As with the network architecture shown in fig. 2, the encoder may include a series of convolutional layers and pooling layers to continually reduce the feature size; the decoder is corresponding to the encoder and is also composed of a series of convolution layers and deconvolution layers with corresponding scales so as to continuously enlarge the characteristic scales, combine the outputs of all scales together and output a multi-scale three-dimensional organ characteristic diagram.

It can be seen that, compared with other feature extraction networks such as an RCNN (Region-Convolutional Neural Network) Network, the feature pyramid Network implements horizontal connection on a Convolutional layer (down-sampling) and a deconvolution layer (up-sampling) of the same level to ensure that the output three-dimensional organ feature map of the level contains more detailed and accurate semantic information, position information, and the like, so that targets (such as organs such as liver) of corresponding scales can be detected on feature maps of different resolutions (i.e., three-dimensional organ feature maps of different scales), and meanwhile, since the feature map output by each Convolutional layer is derived from feature fusion of the current layer and a higher level, the obtained feature maps of each scale have sufficient feature expression capability, such as detection of spatial information, high-level semantic information, and the like in an image, and contribute to improvement of image segmentation effect. The horizontal connection may be implemented by using convolution of 1 × 1 × 1, but is not limited to this implementation and may be determined as the case may be.

For example, in the embodiment of the present application, a feature extraction network with 5 convolutional layers of different scales is taken as an example for explanation, as shown in fig. 2, the sizes of convolution kernels of the 5 convolutional layers and corresponding deconvolution layers from top to bottom may be 128 × 128 × 128 × 32, 64 × 64 × 64, 32 × 32 × 128, 16 × 16 × 16 × 256, and 8 × 8 × 8 × 320 in sequence, and according to the working principle of the feature pyramid network, the feature extraction is performed on the input three-dimensional organ image with the levels of convolution kernels of different sizes, so as to obtain a three-dimensional organ feature map of a corresponding scale, and a specific implementation process is not described in detail. The size of the bottleneck layer for realizing the connection between the encoder and the decoder can be 4 multiplied by 320, and the detection and extraction of high-level semantic information are realized according to the bottleneck layer and the deconvolution layer.

It should be noted that the size of each convolution kernel included in the feature pyramid network is not limited to the size shown in fig. 2, and the number of convolution and deconvolution layers included in the feature pyramid network may be determined according to actual requirements, including but not limited to the number of levels shown in fig. 2.

In practical application of the present application, the initial feature extraction model described above may be used to train the sample three-dimensional organ image until a training constraint condition is satisfied (for example, the loss value of each extracted feature map is smaller than a loss threshold or converges, and the present application does not limit the content of the condition, which may be determined by circumstances), so as to obtain a feature extraction model with a corresponding network structure. Therefore, after the three-dimensional organ image to be segmented is actually obtained, the feature extraction model obtained by pre-training can be directly called, the three-dimensional organ image is input into the feature extraction model, a plurality of three-dimensional organ feature maps with different scales are output, and the feature extraction processing efficiency is improved compared with a processing mode of online adjustment of feature extraction network parameters.

In some embodiments, the loss value of the three-dimensional organ feature map obtained at this time can be calculated, so that the network parameters of the feature extraction model are adjusted, the output accuracy of the feature extraction model is further improved, and the image segmentation accuracy is further improved. It should be noted that, the present application does not describe in detail the specific implementation process of the feature extraction model training based on the feature pyramid network, and includes, but is not limited to, the above-described training implementation and application process.

Step S13, inputting a plurality of three-dimensional organ characteristic diagrams with different scales into the organ characteristic calibration model, and outputting a target three-dimensional organ characteristic diagram;

as described above in relation to the determination process of the technical concept of the present application, the present application is intended to achieve accurate segmentation of different sized objects (e.g. liver tumor) in a three-dimensional organ image by taking into account the characteristics and correlations of different levels of image features while fusing multi-level feature information. Therefore, the method for realizing the organ feature calibration is provided by carrying out calibration training on the three-dimensional organ features of the samples with different scales on the basis of a space attention mechanism and a semantic attention mechanism in advance, and a specific training implementation method is not limited by the application.

In the embodiment of the application, as the feature pyramid network can be adopted to extract the features of the three-dimensional organ image to obtain the multi-scale three-dimensional organ feature map, and the input image processing characteristics of the feature pyramid network can be known, the high-level feature map focuses on global high-level semantic information, and the low-level feature map focuses on local low-level texture, the semantic information in the high-level feature map can be introduced into the low-level feature map by calculating the spatial correlation between the high-level features and the bottom-level features in the training implementation process of the organ feature calibration model, as shown in fig. 3; and the spatial relationship of the bottom layer features can be introduced into the high layer features by calculating the semantic correlation between the low layer features and the high layer features, so that the finally obtained candidate three-dimensional organ feature maps with different scales can be ensured to more specifically and accurately comprise the spatial information and the semantic information of the object to be identified.

In the model training process, a supervision signal (such as an acquired three-dimensional organ image) can be added to the feature map obtained in each processing step, and a corresponding loss value is calculated so as to adjust network parameters, such as the network parameters of the feature extraction network, and improve the output reliability and accuracy of the feature extraction network.

Specifically, following the above description, after the extracted three-dimensional organ feature maps of different scales are calibrated based on the spatial attention mechanism to obtain a plurality of undetermined organ feature maps of corresponding scales, the loss value of each undetermined organ feature map can be calculated by using a preset loss function, so as to adjust the network parameters accordingly; similarly, after the multiple undetermined organ feature maps with different scales are obtained through further calibration processing based on the semantic gravity mechanism to obtain multiple candidate three-dimensional organ feature maps with corresponding scales, the loss value of each candidate three-dimensional organ feature map can be calculated by using a preset loss function so as to adjust network parameters.

Then, the finally obtained multiple candidate three-dimensional organ feature maps with different scales can be fused together, for example, fusion processing of the multiple candidate three-dimensional organ feature maps with different scales is realized through one convolution layer, so that one obtained target three-dimensional organ feature map can contain basic target features such as color features and texture features, and also can contain spatial detail features and target semantic features, and the target object can be segmented and identified more accurately.

And step S14, segmenting the three-dimensional organ image by using the target three-dimensional organ feature map, and outputting the organ region segmentation result of the three-dimensional organ image.

As described above, for the multi-layer three-dimensional organ feature maps with different scales, the present application introduces spatial correlation and semantic correlation between features of different levels, so as to adjust the feature weights of the three-dimensional organ features directly extracted on the corresponding scales according to the correlations, and enhance the representation of objects to be identified (such as liver tumors, etc.) on contexts of different scales, so that a plurality of finally obtained candidate three-dimensional organ feature maps with different scales are fused together, and the target object edges can be identified more accurately and completely by using the characteristics of the organ features, the detected spatial detail information and the detected target semantic information, so as to improve the organ region segmentation effect of the three-dimensional organ image, that is, the target objects with different sizes, such as tumors with different sizes, contained in the three-dimensional organ image are identified accurately.

Referring to fig. 4, which is a schematic flow chart of another optional example of the three-dimensional organ image segmentation method proposed in the present application, the present embodiment may be an optional refinement implementation method of the three-dimensional organ image segmentation method described in the foregoing embodiment, but is not limited to the refinement implementation method described in the present embodiment. Referring to the processing flow diagram of the three-dimensional organ image segmentation method shown in fig. 3, the refinement implementation method proposed in this embodiment may include:

step S41, obtaining a three-dimensional organ image to be segmented;

step S42, extracting the characteristics of the three-dimensional organ image to obtain a plurality of three-dimensional organ characteristic diagrams with different scales;

regarding the implementation process of step S41 and step S42, reference may be made to the description of corresponding parts in the foregoing embodiments, which are not described herein again.

Step S43, using the space correlation information between two three-dimensional organ characteristic diagrams of adjacent scales to calibrate the three-dimensional organ characteristic diagram of smaller scale, and obtaining the undetermined organ characteristic diagram of corresponding scale;

referring to the network structure of the organ feature calibration model shown in fig. 3, in the three-dimensional organ feature maps of different scales output by the feature extraction network, for any two three-dimensional organ feature maps of adjacent scales, the spatial detail information and the target semantic information detected in the respective acquisition processes are different, specifically, the target semantic information is focused more on the three-dimensional organ feature map of a higher level, and the spatial detail information is focused more on the three-dimensional organ feature map of a lower level.

Therefore, the embodiment proposes to perform spatial information calibration processing on the three-dimensional organ feature map of the higher level by using spatial correlation information between the two three-dimensional organ feature maps of adjacent levels, so as to enrich spatial detail information of the three-dimensional organ feature map of the corresponding scale, which is beneficial to more accurately determining the target position. For convenience of description, in this embodiment, the calibrated three-dimensional organ feature map may be referred to as a to-be-determined organ feature map of a corresponding scale.

It should be noted that the implementation method of how to obtain the spatial correlation information between two three-dimensional organ feature maps of adjacent scales by using the spatial attention mechanism is not limited in the present application, and may be determined according to the working principle of the spatial attention mechanism, and the embodiment of the present application is not described in detail herein.

Step S44, calibrating the undetermined feature map with a larger scale by using semantic correlation information between two undetermined organ feature maps with adjacent scales to obtain a candidate organ feature map with a corresponding scale;

unlike the processing procedure of spatial information calibration described in step S43, since the target semantic information of the lower-level three-dimensional organ feature map (i.e., the feature map with a larger scale) is relatively less, in order to be able to combine the target semantic information to more accurately identify the target region (e.g., the liver tumor region) in the three-dimensional organ feature map, the embodiment will continue to acquire semantic correlation information between two undetermined organ feature maps with adjacent scales, so as to perform semantic information calibration on the lower-level three-dimensional organ feature map by using the semantic correlation information, so as to obtain a candidate organ feature map containing more detailed and accurate target semantic information.

It should be understood that, for the three-dimensional organ feature maps of different scales extracted directly from the acquired three-dimensional organ image, the present application sequentially utilizes the spatial correlation and semantic correlation of the adjacent hierarchical features to perform calibration processing, so that the finally obtained candidate organ feature maps of each scale all contain reliable and accurate spatial detail information and target semantic information,

moreover, it should be noted that, in the process of calibrating the extracted three-dimensional organ feature maps of different scales, according to needs, the semantic correlation information between two three-dimensional organ feature maps of adjacent scales may be used first to calibrate the three-dimensional organ feature map of a larger scale, and then the calibrated feature maps of adjacent scales are continuously spatially calibrated in the manner described in step S43, so that the implementation process is similar, and the detailed description is omitted in this application. Therefore, in the organ feature calibration model, the execution sequence of the calibration based on the spatial correlation and the execution sequence of the calibration based on the semantic correlation are not limited, and can be determined according to the situation.

Step S45, performing fusion processing on the obtained candidate organ feature maps with different scales to obtain a target three-dimensional organ feature map;

and step S46, segmenting the three-dimensional organ image by using the target three-dimensional organ feature map, and outputting the organ region segmentation result of the three-dimensional organ image.

As described above, the present application can use the obtained target three-dimensional organ feature map to realize complete and accurate identification of the edge of the target object with each size, perform image segmentation according to the identification result, and obtain the organ region segmentation result of the three-dimensional organ image, that is, accurately identify target objects with different sizes in the three-dimensional organ image, such as liver tumors with different sizes, and how to use the feature map to perform the image segmentation process, which is not described in detail in the present application.

Therefore, after the acquired three-dimensional organ feature maps with different scales are obtained by feature extraction, in consideration of the fact that in the process of acquiring the three-dimensional organ feature maps with different scales, the spatial detail information and the target semantic information detected by inputting the corresponding hierarchy network images are different, such as part of the spatial detail information or the target semantic information is lost, therefore, the extracted three-dimensional organ feature maps with the same scales are directly used for image segmentation processing, the segmentation effect of the three-dimensional organ images is reduced, and target objects with different sizes cannot be reliably and completely identified.

In order to solve the above problem, in the embodiment of the present application, it is proposed to acquire spatial correlation information between feature maps of adjacent scales, and calibrate, for a feature map with many details that is output by a high-level network and lost due to resolution reduction, a feature map relatively lacking spatial detail information by using the spatial correlation information, so as to perfect spatial detail information included in the feature map of the scale; in the same way, the feature maps lacking the target semantic information are calibrated by utilizing the semantic correlation information between the feature maps of adjacent scales obtained after the calibration, so that the finally obtained feature maps of different scales can contain relatively complete and accurate space detail information and target semantic information, and thus, the multi-scale feature maps are fused into a target three-dimensional organ feature map for image segmentation, and target objects such as liver tumors of different scales in the three-dimensional organ image can be reliably and accurately identified.

Referring to fig. 5, which is a schematic flow chart of yet another optional example of the three-dimensional organ image segmentation method proposed in the present application, this embodiment may be a further refinement implementation method of the three-dimensional organ image segmentation method described in the foregoing embodiment, and mainly describes a refinement implementation method of the calibration processing procedure of the feature map, but is not limited to the refinement implementation method described in this embodiment. As shown in fig. 5, the refinement implementation method may include:

step S51, obtaining a three-dimensional organ image to be segmented;

step S52, inputting the three-dimensional organ image into a pyramid feature extraction model, and outputting a plurality of three-dimensional organ feature maps with different scales;

in combination with the description of the corresponding part of the above embodiment, the pyramid feature extraction model is obtained by training a sample three-dimensional organ image based on a feature pyramid network, and the specific training implementation process of the embodiment of the present application is not described in detail herein.

It should be noted that, for a feature extraction model that extracts a plurality of three-dimensional organ feature maps with different scales from a three-dimensional organ image, including but not limited to a pyramid feature extraction model, other types of feature extraction networks may also be used for modeling according to actual needs, and details of the application are not described herein.

Step S53, processing two three-dimensional organ feature maps of adjacent scales to obtain a space attention map aiming at the three-dimensional organ feature map of a smaller scale;

step S54, using the space attention map to calibrate the three-dimensional organ characteristic map with a corresponding smaller scale, and obtaining the undetermined organ characteristic map with a corresponding scale;

the attention mechanism (attention) in computer vision (computer vision) is mainly to make the system focus on the interested place, such as the target object to be identified in the three-dimensional organ image collected in the present application, specifically, tumors of various sizes in the three-dimensional liver image.

Specifically, in the embodiment of the present application, soft attention based on a spatial attention mechanism may be adopted to process two three-dimensional organ feature maps of adjacent scales, that is, perform corresponding spatial transformation on spatial domain information in the two three-dimensional organ feature maps of adjacent scales, and extract key information (i.e., information of a region of interest, such as information of a region where a liver tumor is located). It should be noted that the present application is not limited to the specific implementation method for obtaining the spatial attention map.

As described above, the spatial attention map obtained by the present application not only realizes more precise location and identification of the region where the target object is located, but also introduces the target semantic information of the three-dimensional organ feature map with a larger adjacent scale, so that the embodiment calibrates the three-dimensional organ feature map with a smaller scale directly extracted by using the spatial attention map, and can improve noise of an underlying network introduced in the process of extracting the three-dimensional organ feature map with a smaller scale and a lot of spatial detail information lost by a higher-level network due to resolution reduction, so that the undetermined organ feature map with the scale obtained after calibration can more clearly embody more complete spatial detail information and target semantic information of the target object, and the embodiment does not detail the specific implementation process of the calibration process.

Step S55, processing the two obtained characteristic graphs of the undetermined organ with the adjacent scales to obtain a semantic attention vector aiming at the characteristic graph of the undetermined organ with the larger scale;

step S56, using the semantic attention vector to calibrate the corresponding larger-scale to-be-determined organ feature map to obtain a candidate organ feature map of the corresponding scale;

in the calibration of the spatial domain key information of the three-dimensional organ feature map with a smaller scale output by each opposite underlying network, considering that the three-dimensional organ feature map with a larger scale output by the opposite underlying network pays more attention to the global semantic information and ignores information in other aspects, in this embodiment, the semantic attention vector of the undetermined organ feature map with a larger scale is obtained by using two undetermined organ feature maps with adjacent scales, so that the semantic attention vector is introduced into the spatial detail information detected by the underlying network.

Therefore, the semantic attention vector is utilized to calibrate the corresponding large-scale to-be-determined organ feature map, so that not only can semantic information in the feature map output by the underlying network be improved according to target semantic information detected by a high-level network, but also the candidate organ feature map obtained after calibration can contain more accurate spatial detail information.

Step S57, performing fusion processing on the obtained candidate organ feature maps with different scales to obtain a target three-dimensional organ feature map;

and step S58, segmenting the three-dimensional organ image by using the target three-dimensional organ feature map, and outputting the organ region segmentation result of the three-dimensional organ image.

In summary, in the embodiment of the present application, after feature extraction is performed on a three-dimensional organ image by using a pyramid feature extraction model to obtain a plurality of three-dimensional organ feature maps with different scales, in order to solve the problem that networks with different scales may not be able to detect complete and detailed spatial detail information and target semantic information in the image at the same time, and more accurately and reliably identify target objects (such as liver and liver tumor) with different sizes in the three-dimensional organ image, feature advantages, such as spatial detail information or target semantic information, included in feature maps output by adjacent networks with different scales are provided, and processing is performed based on a corresponding attention mechanism to obtain calibration information for calibrating the feature map with the corresponding scale output by the network, so as to enrich the spatial detail information and the target semantic information included in the feature maps, thereby enabling spatial features, target semantic information, included in calibrated candidate organ feature maps to be more accurate and reliable, The semantic features are more accurate and detailed, so that the three-dimensional organ image is segmented according to the semantic features, the organ region segmentation effect of the three-dimensional organ image is greatly improved, the target object in the three-dimensional organ image is accurately and completely identified, and the positioning and identifying requirements of the current application scene on the target object are met.

Referring to fig. 6, which is a schematic flow chart of another optional example of the three-dimensional organ image segmentation method provided in the present application, this embodiment mainly performs detailed description on an implementation process of obtaining a to-be-determined organ feature map of a corresponding scale by calibrating a smaller-scale three-dimensional organ feature map using spatial correlation information between two three-dimensional organ feature maps of adjacent scales in the above embodiment, and for other implementation steps of the three-dimensional organ image segmentation method, reference may be made to descriptions of corresponding parts in the above embodiment, which is not described in detail in this embodiment. As shown in fig. 6, the refinement implementation method proposed in this embodiment may include:

step S61, merging the two three-dimensional organ feature maps with adjacent scales to obtain a merged feature map;

step S62, inputting the merged feature map into a space attention network, and outputting a space attention map of a three-dimensional organ feature map with a smaller scale in two three-dimensional organ feature maps with adjacent scales;

referring to the schematic flow chart of the implementation method for calibrating the three-dimensional organ feature map shown in fig. 7, it can be known from the technical concept of implementing the calibration of the three-dimensional organ feature map by combining the spatial attention mechanism described in the above embodiment, in the process of obtaining the spatial attention map, the embodiment of the present application is expected to determine not only spatial detail information in the three-dimensional organ feature map of the corresponding scale output by the current-level network, but also target semantic information in the three-dimensional organ feature map output by the adjacent low-level network.

Based on this, for a three-dimensional organ feature map of a corresponding scale output by any level network, when obtaining a spatial attention map for calibrating the three-dimensional organ feature map, the embodiment of the present application may perform upsampling on the three-dimensional organ feature map output by the level network to obtain a three-dimensional organ feature map output by an adjacent lower level network, merge the three-dimensional organ feature map with the three-dimensional organ feature map output by the level network (which may be implemented by, but is not limited to, a Concat function), and obtain a merged feature map with more channel dimensions, so that the merged feature map contains more accurate and complete spatial detail information and target semantic information relative to the three-dimensional organ feature map output by the level network.

Then, the merged feature map including the three-dimensional organ feature maps output by two adjacent hierarchical networks is input into the spatial attention network, and compared with the method for directly inputting the three-dimensional organ feature maps output by the hierarchical network into the spatial attention network, the method for acquiring the spatial attention image of the present embodiment can introduce the target semantic information included in the three-dimensional feature maps output by the higher hierarchical network into the three-dimensional organ feature maps output by the underlying network, and simultaneously make full use of the more accurate and complete spatial features included in the three-dimensional organ feature maps output by the lower hierarchical network to accurately identify and position the region where the target object is located in the three-dimensional organ feature maps output by the higher hierarchical network, that is, the spatial detail information of the target object represented by the spatial attention map obtained in the present embodiment is more complete and accurate. It should be noted that the specific network structure of the spatial attention network is not limited in the present application, and may be determined based on the working principle of the spatial attention mechanism.

For example, for a pyramid extraction model, different levels of networks encode input images at resolutions of corresponding sizes, the output three-dimensional organ feature map of corresponding scale can be recorded as Fi, i can represent the level of the network level in the pyramid extraction model, i belongs to [0, L-1 ]]L may represent the number of network levels, such as the feature pyramid extraction network structures shown in fig. 2 and fig. 3, where each of the encoder and decoder includes five layers of networks, i.e., L is 5, these 5 layers of networks are respectively denoted as Level0, Level1, Level2, Level3 and Level4, and the three-dimensional organ feature maps Fi ∈ R of corresponding scales output by the networks of different levels^Ci ^×Di×Hi×WiThat is, the feature map of the ith scale may be represented by a spatial transform matrix, elements in the spatial transform matrix are respectively a channel C, a depth D, a height H, and a width W, and specific values thereof may be determined according to the size of a convolution kernel of the hierarchical convolution network.

Based on this, in the process of obtaining the spatial attention map, in this embodiment, for the merged feature map obtained by merging and including the three-dimensional organ feature maps output by two adjacent hierarchical networks, the merged feature map is input into the pooling layer, and is respectively subjected to maximum pooling (Max processing process shown in fig. 6) and average pooling (Avg processing process shown in fig. 6) to realize feature extraction on the spatial dimension of the three-dimensional organ feature maps of the two adjacent dimensions connected in parallel, and then the two three-dimensional organ feature maps obtained by processing are spliced into one feature map T_i ^spaIs concretely implementedThe embodiment will not be described in detail.

After that, the feature maps obtained by stitching are input to the fully connected layer formed by the multi-layer perceptron MLP for processing, and the obtained feature maps are represented as a spatial attention map, as shown in fig. 7, the multi-layer perceptron MLP of the present example may be formed by a three-layer convolution network, that is, a three-layer perceptron, but is not limited to this network structure, and may be determined as appropriate.

Step S63, down-sampling and normalizing the spatial attention map to obtain a calibration organ feature map;

in succession to the above description, the present application proposes to process the merged feature map after merging the three-dimensional organ features of two adjacent scales based on the spatial attention mechanism, and to regress the merged feature map into an attention map a_i ^mAnd the calibration organ characteristic diagram is recorded as a calibration organ characteristic diagram and used for realizing calibration processing on the three-dimensional organ characteristic diagram with smaller scale output by the high-level network. Therefore, in the calibration process, in order to realize the calculation between the two feature maps, the regression process may be performed on the obtained spatial attention map, specifically, the downsampling (down sample) may be performed on the spatial attention map to reduce the number of pixels included in the feature map, and the normalization process may be performed on the downsampled feature map by using a Sigmoid activation function to obtain a calibration organ feature map a_i ^mThe specific implementation process is not limited in the application.

Wherein the regression-derived attention map, i.e., the calibration organ feature map A_i ^mCan be defined as:

in the formula (1), f_spa() The three-dimensional organ feature graph can represent a three-layer convolution network, namely a three-layer perceptron, theta can mark network parameters of the perceptron, and a specific acquisition process related to the spatial attention graph can be determined according to the working principle of the spatial attention graphAnd then the combined feature map is subjected to space attention processing, target semantic information detected by a larger-scale three-dimensional organ feature map output by a high-level network is introduced, the accuracy of each feature weight contained in the obtained calibration organ feature map is improved, and the combined feature map can be used as a calibration reference feature map of a smaller-scale three-dimensional organ feature map to improve the space detail information and the target semantic information in the smaller-scale three-dimensional organ feature map, thereby being beneficial to completely and accurately identifying the edge of a target object and improving the image segmentation effect.

Step S64, format conversion processing is carried out on the three-dimensional organ characteristic diagram with smaller scale in the two three-dimensional organ characteristic diagrams with adjacent scales, and the organ characteristic diagram to be calibrated matched with the format of the corresponding space attention diagram is obtained;

and step S65, performing feature product operation on the organ feature map to be calibrated and the corresponding organ feature map to be calibrated to obtain the organ feature map to be determined with the corresponding scale.

In practical application, because the normalized organ feature map and the three-dimensional organ feature map of the current scale directly output by the feature extraction model have a certain format difference, and the two feature maps cannot be directly operated, the embodiment proposes that the three-dimensional organ feature map of the current scale, namely the three-dimensional organ feature map of the adjacent scale, is subjected to format conversion processing, for example, a 1 × 1 × 1 convolution layer can be used for processing the three-dimensional organ feature map of the smaller scale, and then the obtained organ feature map to be calibrated and the calibrated organ feature map are subjected to feature product operation, for example, feature product operation is performed on the obtained organ feature map to be calibrated and the calibrated organ feature map

Obtaining a calibrated characteristic diagram F of the undetermined organ with corresponding scale_i ^spaThe detailed calculation process is not described in detail in this application.

Therefore, the implementation process of performing spatial correlation calculation on the three-dimensional organ feature maps of any two adjacent scales, introducing a high-level semantic concept into the bottom-level feature, performing feature calibration on the three-dimensional organ feature map of a smaller scale to obtain the feature map of the undetermined organ can be realized according to a calculation method expressed by the following formula:

in the above formula (1), F_i ^spaCan represent the attention feature after the calibration of the three-dimensional organ feature map of the ith scale, namely the feature contained in the characteristic map of the to-be-determined organ, which represents the feature with the most information quantity in the three-dimensional organ feature map of the ith scale, f_spa(F_i,F_i-1) Can be shown in Table F_iAnd F_i-1As shown in fig. 7, the relationship (a) can be obtained by performing regression processing on a perceptron composed of three layers of convolution networks, but is not limited to this obtaining manner; g_spa(F_i) Can be used to realize F_iThe feature mapping, namely, the format conversion processing in step S64, to obtain a feature map of the organ to be calibrated; c is a regularization factor, 1/C (f)_spa(F_i,F_i-1) May represent the processing procedure described above in step S63, and therefore,

i.e. the above-mentioned calibration organ signature.

In summary, compared with a processing method of directly performing spatial attention processing on a three-dimensional organ feature map of the current scale to calibrate the three-dimensional organ feature map, the embodiment of the present invention provides a method of calculating spatial correlation of features included in three-dimensional organ feature maps of adjacent scales, and introducing target semantic information in a three-dimensional organ feature map of a smaller scale output by a higher-level network into a three-dimensional organ feature map of a larger scale output by an underlying network, so as to improve integrity and accuracy of spatial detail information and target semantic information included in an obtained calibrated organ feature map.

Referring to fig. 8, which is a schematic flow chart of another optional example of the three-dimensional organ image segmentation method provided in the present application, in this embodiment, the semantic correlation information between two undetermined organ feature maps of adjacent scales is mainly used to perform calibration processing on the undetermined organ feature map of a larger scale, so as to obtain a detailed description of the implementation process of the candidate organ feature map of a corresponding scale, and for other implementation steps of the three-dimensional organ image segmentation method, reference may be made to the description of corresponding parts in the above embodiment, which is not described in detail in this embodiment. As shown in fig. 8, the refinement implementation method proposed in this embodiment may include:

step S81, respectively performing maximum pooling and average pooling on the two obtained undetermined organ feature maps of adjacent scales on a channel dimension, and merging the organ feature vectors obtained by processing to obtain semantic organ feature vectors;

referring to the description of the spatial correlation calculation process described above, the present application is applied to the calculation process of semantic correlation information of the two three-dimensional organ feature maps, i.e. to two undetermined organ feature maps (e.g. F) of adjacent scales_i ^spaAnd

) In the implementation process of processing to obtain the semantic attention vector for the undetermined organ feature map of a larger scale, as shown in fig. 9, in the processing stage of the pooling layer, the maximum pooling processing (Max processing shown in fig. 9) and the average pooling processing (Avg processing shown in fig. 9) may be performed on each input undetermined organ feature map, and the maximum pooling processing results of the undetermined organ feature maps of two adjacent scales are merged to obtain a maximum pooled feature vector; merging the average pooling processing results of the two to-be-determined organ feature maps to obtain an average pooling feature vector, splicing the two feature vectors into a feature vector, and recording the feature vector as a semantic organ feature vector T_i ^sem。

It should be noted that how to perform the maximum pooling and average pooling processing on the three-dimensional organ feature map of each scale based on the semantic attention mechanism can be determined according to the working principle of the semantic attention mechanism, and the detailed description of the processing is not provided herein.

Step S82, carrying out regression processing on the semantic organ feature vector to obtain a semantic attention vector aiming at a large-scale undetermined organ feature map;

step S83, carrying out normalization processing on the semantic attention vector to obtain a calibrated semantic organ feature vector;

similar to the above spatial correlation analysis process, the embodiment of the invention can also return to an attention vector a after extracting and processing the features of the feature maps of the to-be-determined organs of two adjacent scales in parallel on the channel dimension based on the semantic attention mechanism_i ^vWhich can be defined as

The present application may refer to this attention vector A_i ^vReferred to as alignment semantic organ feature vectors.

In the embodiment of the present application, as shown in fig. 9, in the process of obtaining the above semantic attention vector, a perceptron f composed of three layers of convolutional networks may still be utilized_semThe regression processing is performed on the semantic organ feature vector, but the method is not limited to the perceptron of the structure, and the method can be used as the case may be.

Then, the semantic attention vector output by the perceptron can be continuously normalized by using the sigmoid activation function to obtain a calibrated semantic organ feature vector A_i ^v. It should be noted that the present application does not detail the type of the sigmoid activation function and the normalization processing principle thereof.

Step S84, format conversion processing is carried out on the undetermined organ characteristic diagram with the larger scale in the two undetermined organ characteristic diagrams with the adjacent scales to obtain the undetermined organ characteristic diagram to be calibrated, which is matched with the format of the corresponding semantic attention vector;

and step S85, performing product operation on the to-be-calibrated undetermined organ feature map and the calibration semantic organ feature vector to obtain a candidate organ feature map with a corresponding scale.

On the larger scale to be determined organ feature map

Before the calibration processing, a 1 × 1 × 1 convolution network can be used for carrying out format conversion processing on the organ feature map to be calibrated, so that the processed to-be-calibrated organ feature map to be determined is matched with the format of the semantic attention vector, and the to-be-calibrated organ feature map to be determined and the semantic attention vector can be subjected to further product operation to realize the to-be-determined organ feature map

The feature calibration process of (1).

Therefore, the embodiment of the application is based on the semantic attention mechanism, and is used for the undetermined organ feature map of the ith scale

Calibrating to obtain candidate organ characteristic diagram with corresponding scale

Can be defined as:

specifically, in combination with the semantic relevance analysis process described above, the embodiment of the present application may calculate a candidate organ feature map of any scale according to the following formula:

the meanings represented by the three parts on the right side of the equation in equation (3) are similar to the meanings represented by the corresponding parts in equation (2), and are not described in detail in this embodiment.

Therefore, the method can be seen in the application, after calibrating the three-dimensional organ features with smaller dimensions based on a space attention mechanism to obtain the undetermined organ feature map with corresponding dimensions, further processing the undetermined organ feature maps with two adjacent dimensions based on a semantic attention mechanism, namely introducing bottom-layer space detail information into high-layer features to calibrate the undetermined organ feature map with larger dimensions, and ensuring that the edges and the types of target objects with corresponding dimensions can be determined according to complete and detailed space detail information and target semantic information in the finally obtained candidate organ feature maps with each dimension, namely improving the accuracy of target object region identification, so that the target three-dimensional organ feature maps with different dimensions are fused according to the candidate organ feature maps with different dimensions to perform image segmentation on the three-dimensional organ images, and the target objects with different dimensions contained in the three-dimensional organ images can be reliably and accurately identified, such as liver tumors of different sizes.

In combination with the calculation process of the spatial correlation and the semantic correlation between the feature maps of adjacent scales described in the above embodiment, in the training process of implementing the organ feature calibration model based on the calculation process, after the spatial correlation calculation is completed for the input sample three-dimensional device feature maps of different scales according to the above manner, deep supervised training may be performed on the obtained undetermined organ feature map of each scale, for example, a loss value of the undetermined organ feature map of the scale is calculated by using a preset loss function, and then, adjustment of network parameters is implemented according to the loss value, so as to improve accuracy of feature extraction and spatial correlation calculation between feature maps of adjacent scales, which is not described in detail in this embodiment of the specific training implementation process.

Similarly, according to the semantic correlation calculation method described above, based on the semantic attention mechanism, the semantic correlation between undetermined feature maps of adjacent scales is calculated, so as to calibrate undetermined feature maps of larger scales, and after candidate organ feature maps are obtained, the network parameter adjustment can still be realized by adopting a deep supervised training mode, so as to improve the accuracy of feature extraction, spatial correlation information between adjacent scale feature maps, and semantic correlation information between adjacent scale feature maps, further improve the three-dimensional organ image segmentation effect, and realize accurate and reliable identification of target objects of different sizes. The training process for realizing the organ characteristic calibration model according to the correlation calculation method is not described in detail in the application.

In still other embodiments provided by the present application, the feature extraction network, the spatial attention network, and the semantic attention network may also constitute an initial image segmentation network, and the initial image segmentation network is trained by using different sample three-dimensional organ images according to the network processing process and the training method described in the above network parts, so as to obtain a three-dimensional organ image segmentation model. It can be understood that the three-dimensional organ image segmentation model may include the above-described feature extraction model (such as the above-described pyramid feature extraction model), an organ feature calibration model, and other sub-models, and the training implementation process of the three-dimensional organ image segmentation model is not described in detail in the present application, and reference may be made to the above-described training process of each sub-model according to the technical concept proposed in the present application, but not limited thereto.

It should be understood that, for any model obtained through the above training, in an actual application, an actually acquired three-dimensional organ image or an obtained feature map is input into the model for processing, and further optimization processing (such as performing one or more model iteration processes) may be performed on network parameters of the model according to a model output result, so as to further improve the output accuracy of the model in a current application scenario, and a specific optimization processing process is not described in detail in this application.

Referring to fig. 10, a schematic structural diagram of an alternative example of the three-dimensional organ image segmentation apparatus proposed in the present application, which may be applied to the computer device as described above, and the present embodiment is not limited as appropriate with respect to the product type of the computer device. As shown in fig. 10, the apparatus may include:

an image acquisition module 101, configured to acquire a three-dimensional organ image to be segmented;

a feature extraction module 102, configured to perform feature extraction on the three-dimensional organ image to obtain a plurality of three-dimensional organ feature maps of different scales;

the feature calibration module 103 is configured to input the multiple three-dimensional organ feature maps with different scales into an organ feature calibration model, and output a target three-dimensional organ feature map;

the organ feature calibration model is obtained by performing calibration training on sample three-dimensional organ features of different scales based on a space attention mechanism and a semantic attention mechanism.

And the image segmentation module 104 is configured to perform segmentation processing on the three-dimensional organ image by using the target three-dimensional organ feature map, and output an organ region segmentation result of the three-dimensional organ image.

In some embodiments, the feature calibration module 103 may include:

the first calibration processing unit is used for calibrating the three-dimensional organ feature map with a smaller scale by utilizing the spatial correlation information between the two three-dimensional organ feature maps with adjacent scales to obtain the undetermined organ feature map with a corresponding scale;

the second calibration processing unit is used for calibrating the undetermined feature map with a larger scale by utilizing semantic correlation information between the two undetermined organ feature maps with adjacent scales to obtain a candidate organ feature map with a corresponding scale;

in a possible implementation manner, as shown in fig. 11, the first calibration processing unit may include:

a spatial attention map obtaining unit 1031, configured to process two three-dimensional organ feature maps at adjacent scales to obtain a spatial attention map for a three-dimensional organ feature map at a smaller scale;

the first calibration unit 1032 is configured to perform calibration processing on the corresponding smaller-scale three-dimensional organ feature map by using the spatial attention map to obtain a corresponding-scale undetermined organ feature map.

Optionally, the spatial attention map obtaining unit 1031 may include:

the first merging processing unit is used for merging the two three-dimensional organ feature maps of adjacent scales to obtain a merged feature map;

and the spatial attention processing unit is used for inputting the merged feature map into a spatial attention network and outputting a spatial attention map of a three-dimensional organ feature map with a smaller scale in the two three-dimensional organ feature maps with adjacent scales.

Accordingly, the first calibration unit 1032 may include:

the first normalization processing unit is used for carrying out down-sampling and normalization processing on the spatial attention diagram to obtain a calibration organ feature diagram;

the first format conversion processing unit is used for carrying out format conversion processing on the three-dimensional organ feature map with the smaller scale in the two three-dimensional organ feature maps with the adjacent scales to obtain the organ feature map to be calibrated, wherein the organ feature map to be calibrated is matched with the format of the corresponding spatial attention map;

and the space calibration unit is used for performing characteristic product operation on the organ characteristic diagram to be calibrated and the corresponding organ characteristic diagram to be calibrated to obtain the organ characteristic diagram to be calibrated with the corresponding scale.

In still other embodiments, as shown in fig. 11, the second calibration processing unit may include:

a semantic attention vector obtaining unit 1033, configured to process the obtained two to-be-determined organ feature maps of adjacent scales, and obtain a semantic attention vector for the to-be-determined organ feature map of a larger scale;

and a second calibration unit 1034, configured to perform calibration processing on the corresponding larger-scale to-be-determined organ feature map by using the semantic attention vector, so as to obtain a candidate organ feature map of the corresponding scale.

Optionally, the semantic attention vector obtaining unit 1033 may include:

a semantic organ feature vector obtaining unit, configured to perform maximum pooling and average pooling on the two obtained to-be-determined organ feature maps of adjacent scales in a channel dimension, and merge the processed organ feature vectors to obtain a semantic organ feature vector;

and the regression processing unit is used for carrying out regression processing on the semantic organ feature vector to obtain a semantic attention vector aiming at the undetermined organ feature map with a larger scale.

Accordingly, the second calibration unit 1034 may include:

the second normalization processing unit is used for performing normalization processing on the semantic attention vector to obtain a calibrated semantic organ feature vector;

the second format conversion processing unit is used for carrying out format conversion processing on the undetermined organ characteristic diagram with the larger scale in the two acquired undetermined organ characteristic diagrams with the adjacent scales to acquire the undetermined organ characteristic diagram to be calibrated, which is matched with the format of the corresponding semantic attention vector;

and the semantic calibration unit is used for performing product operation on the to-be-calibrated undetermined organ feature map and the calibrated semantic organ feature vector to obtain a candidate organ feature map with a corresponding scale.

Based on the analysis, the feature calibration module 103 may further include:

and a fusion processing unit 1035, configured to perform fusion processing on the obtained multiple candidate organ feature maps with different scales to obtain a target three-dimensional organ feature map.

It should be noted that, various modules, units, and the like in the embodiments of the foregoing apparatuses may be stored in the memory as program modules, and the processor executes the program modules stored in the memory to implement corresponding functions, and for the functions implemented by the program modules and their combinations and the achieved technical effects, reference may be made to the description of corresponding parts in the embodiments of the foregoing methods, which is not described in detail in this embodiment.

The present application further provides a computer-readable storage medium, on which a computer program may be stored, where the computer program may be called and loaded by a processor to implement each step of the three-dimensional organ image segmentation method described in the foregoing embodiments, and a specific implementation process may refer to descriptions of corresponding parts in the foregoing embodiments, which is not described in detail in this embodiment.

Referring to fig. 12, which is a schematic diagram illustrating a hardware structure of an alternative example of a computer device suitable for the three-dimensional organ image segmentation method and apparatus provided in the present application, as shown in fig. 12, the computer device may include: a communication module 121, a memory 122, and a processor 123, wherein:

the number of the communication module 121, the memory 122, and the processor 123 may be at least one, and the communication module 121, the memory 122, and the processor 123 may all be connected to a communication bus, so as to implement data interaction therebetween through the communication bus, and a specific implementation process may be determined according to requirements of a specific application scenario, which is not described in detail herein.

The communication module 121 may include a communication module capable of implementing data interaction by using a wireless communication network, such as a WIFI module, a 5G/6G (fifth generation mobile communication network/sixth generation mobile communication network) module, a GPRS module, and the like, and the communication module 121 may further include a communication interface for implementing data interaction between internal components of the computer device, such as a USB interface, a serial/parallel port, and the like, and the specific content included in the communication module 121 is not limited in this application.

In the present embodiment, the memory 122 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device or other volatile solid-state storage device. The processor 123 may be a Central Processing Unit (CPU), an application-specific integrated circuit (ASIC), a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices.

In practical applications of the present embodiment, the memory 122 may be used to store a program for implementing the three-dimensional organ image segmentation method described in any of the above method embodiments; the processor 23 may load and execute a program stored in the memory 122 to implement each step of the three-dimensional organ image segmentation method provided in any one of the above method embodiments of the present application, and the specific implementation process may refer to the description of the corresponding part in the corresponding embodiment above, which is not described again.

It should be understood that the structure of the computer device shown in fig. 12 does not constitute a limitation to the computer device in the embodiment of the present application, and in practical applications, the computer device may include more or less components than those shown in fig. 12, or some components may be combined, and may be determined according to the product type of the computer device, for example, the computer device is a terminal device listed above, and the computer device may further include at least one device such as a touch sensing unit for sensing a touch event on a touch display panel, a keyboard, a mouse, an image collector (such as a camera), a sound collector, and the like; such as at least one output device, e.g., a display, a speaker, a vibration mechanism, a light, etc., which are not listed herein.

Under the condition that the computer device is the terminal device, the terminal device can collect and scan an object to be detected to obtain a three-dimensional organ image to be segmented, then the segmentation processing of the three-dimensional organ image is realized according to the three-dimensional organ image segmentation method described above, target objects with different sizes, such as liver tumors with different sizes and the like, are identified, and the category of the target objects can be determined according to specific application scenes, including but not limited to a liver tumor identification application scene; in still other embodiments, the terminal device may also receive a three-dimensional organ image acquired and transmitted by another device, and perform segmentation processing on the three-dimensional organ image according to the manner described in the above embodiments, which is not limited in this application and may be determined as appropriate.

However, in the case that the computer device is a server, the terminal device having a three-dimensional image acquisition function may generally acquire a three-dimensional organ image to be segmented and send the three-dimensional organ image to the server, and the server segments the three-dimensional organ image according to the three-dimensional organ image segmentation method described in the above embodiment to obtain an organ region segmentation result satisfying application requirements, such as a segmentation result of liver tumors of different sizes, which is fed back to a preset terminal for display, so as to assist in implementing disease diagnosis, determination of a treatment scheme, and the like of an object to be detected, and a specific implementation process of the present application is not described in detail herein.

Finally, it should be noted that, in the present specification, the embodiments are described in a progressive or parallel manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments may be referred to each other. The device and the computer equipment disclosed by the embodiment correspond to the method disclosed by the embodiment, so that the description is relatively simple, and the relevant points can be referred to the method part for description.

The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method of three-dimensional organ image segmentation, the method comprising:

acquiring a three-dimensional organ image to be segmented;

2. The method of claim 1, wherein inputting the plurality of three-dimensional organ feature maps of different scales into an organ feature calibration model and outputting a target three-dimensional organ feature map comprises:

3. The method according to claim 2, wherein the calibrating the organ feature map with a smaller scale by using the spatial correlation information between the two three-dimensional organ feature maps with adjacent scales to obtain the undetermined organ feature map with a corresponding scale comprises:

4. The method of claim 3, wherein said processing two of said three-dimensional organ feature maps at adjacent scales to obtain a spatial attention map for a three-dimensional organ feature map at a smaller scale comprises:

5. The method of claim 3, wherein the using the spatial attention map to perform calibration processing on the corresponding smaller-scale three-dimensional organ feature map to obtain a corresponding-scale undetermined organ feature map comprises:

6. The method according to any one of claims 2 to 5, wherein the step of performing calibration processing on the undetermined organ feature map with a larger scale by using semantic correlation information between two undetermined organ feature maps with adjacent scales to obtain a candidate organ feature map with a corresponding scale includes:

7. The method of claim 6, wherein said processing two of said pending organ feature maps of adjacent scales obtained to obtain a semantic attention vector for a pending organ feature map of a larger scale comprises:

8. The method according to claim 6, wherein said performing calibration processing on the corresponding larger-scale undetermined organ feature map by using the semantic attention vector to obtain a corresponding-scale candidate organ feature map comprises:

9. A three-dimensional organ image segmentation apparatus, the apparatus comprising:

10. A computer device, the computer device comprising:

a communication module;

a memory for storing a program for implementing the three-dimensional organ image segmentation method according to any one of claims 1 to 8;

a processor for loading and executing the program stored in the memory to realize the steps of the three-dimensional organ image segmentation method according to any one of claims 1 to 8.