CN113628346A

CN113628346A - Method for freely browsing VB (visual basic) and method and system for marking

Info

Publication number: CN113628346A
Application number: CN202110743855.3A
Authority: CN
Inventors: 孙加源; 谢芳芳; 宋亮; 唐祺
Original assignee: Shanghai Chest Hospital
Current assignee: Shanghai Chest Hospital
Priority date: 2021-06-30
Filing date: 2021-06-30
Publication date: 2021-11-09
Anticipated expiration: 2041-06-30
Also published as: CN113628346B

Abstract

The invention discloses a method for freely browsing VB (visual basic), which comprises the following steps of 1, performing lung segmentation on a chest CT (computed tomography) image, performing main bronchus segmentation and peripheral bronchus segmentation, and finally connecting the main bronchus segmentation and the peripheral bronchus segmentation to form integral bronchus segmentation so as to obtain a bronchus tree; setting the pixel value of the bronchial tree to 1 and the pixel value of the area of the non-bronchial tree to 0; step 2, extracting a central line to obtain a bronchial central line tree model; step 3, generating triangular patch three-dimensional model data; step 4, obtaining a unique path from the entrance of the bronchus to the end position of each bronchus; and 5, switching paths. The invention also discloses a method for marking VB in free browsing and a related system. The invention enables doctors to freely browse in VB without being limited to a fixed path or a designated position, thereby being capable of freely jumping and switching in all bronchial trees.

Description

Method for freely browsing VB (visual basic) and method and system for marking

Technical Field

The invention belongs to the field of medical instruments, and particularly relates to a method for freely browsing a Virtual Bronchoscope (VB) and a method and a system for marking.

Background

Virtual Bronchoscopy (VB) is a Computed Tomography (CT) based imaging technique. Which performs three-dimensional reconstruction and visualization processing (such as Volume Rendering) based on CT data or extracts the bronchial tubes of the tracheal tree using an algorithm to generate corresponding bronchial images. The bronchus image can accurately display the lumen and the diameter of the bronchus, the left bronchus, the right bronchus and the bronchial tree. By means of these images, non-invasive intraluminal assessments and corresponding bronchial surgical planning can be performed.

In the prior art, a bronchus navigation system or software generally provides a virtual bronchoscope technology for preoperative planning, and can generate a corresponding fixed path along a bronchus for a region of interest (ROI) and provide a virtual bronchus playing image along the path.

Currently known methods can only perform complex operations along a fixed path or on a corresponding 3D model to perform virtual bronchial rendering along a fixed path. In an actual clinical scenario, a doctor needs to repeatedly familiarize and browse and practice various levels of bronchial paths before skillfully using a bronchoscope instrument, and meanwhile, the doctor often needs to not only clearly understand a virtual bronchoscope image of a planned path, but also needs to know the bronchial branches near a focus and needs to perform simulation operation and browsing, and at present, a method for performing random VB browsing on all bronchial branches still does not exist.

In the aspect of anatomy, corresponding naming specifications are provided for all levels of the bronchus, and at present, doctors mainly rely on distinguishing the bronchus branches on the level of CT images, and naming and classifying the three levels of the bronchus branches according to the angles of the bronchus relative to the sagittal plane and the coronal plane of a human body. However, for bronchi below three levels, the angle is more difficult to judge due to too many branches, and the naming of the bronchi by the traditional method is not feasible in practice.

Disclosure of Invention

In order to solve at least one of the above-mentioned technical problems, the present invention discloses a method of freely browsing a Virtual Bronchoscope (VB), comprising:

step 1, performing lung segmentation on a chest CT image, performing main bronchus segmentation and peripheral bronchus segmentation, and finally connecting the main bronchus segmentation and the peripheral bronchus segmentation to form integral bronchus segmentation to obtain a bronchus tree; setting pixel values of the bronchial tree to 1 and pixel values of regions of the non-bronchial tree to 0;

step 2, extracting a central line based on the bronchial tree obtained in the step 1 to obtain a bronchial central line tree model; the centerline coordinate data of each bronchus is composed of a series of points expressed based on three-dimensional space coordinates;

step 3, performing isosurface extraction by using a Marching cube algorithm based on the pixel value marked as 1 in the step 1, thereby generating triangular-surfaced three-dimensional model data;

step 4, according to the bronchial centerline tree model obtained in the step 2, upwards searching and combining from each terminal bronchial tube to obtain a unique path from the entrance of the bronchial tube to the tip position of each bronchial tube;

step 5, virtual bronchoscope rendering and path switching: performing virtual bronchoscope rendering according to the three-dimensional model data generated in the step 3; when virtual bronchoscope browsing is carried out, three-dimensional playing browsing is carried out from the position of a main bronchus, any path connected with the main bronchus is used as an initial path, and a mouse, a keyboard, a touch screen or a remote rod is used, so that a VB image can be played from the initial position of the path to the tail end along the path; when playing, besides displaying VB images, displaying the indication direction of the current path; at the bifurcation part close to the next path, using a mouse, a keyboard, a touch screen or a remote lever to carry out corresponding three-dimensional visual angle transformation, and when the direction of the visual angle transformation and the corresponding bifurcation direction are less than a threshold value gamma and the current visual angle transformation is greater than an angle delta, searching a new path which accords with the current visual angle in the generated paths; after a new path is determined, the current browsing path can be automatically switched to the corresponding bifurcation direction, and meanwhile, the current path display is updated to be a path along the corresponding bifurcation direction, so that browsing is carried out along the bronchial path expected by the user; this step is repeated when the next bronchial bifurcation is encountered.

Further, step 6, using a mouse, a keyboard, a touch screen or a joystick to perform path rollback, so that the VB is displayed in a rollback mode along the current path.

Further, γ is 15 degrees; delta is 20 degrees.

Further, a specific method for searching for a new path that matches the current view angle from among the generated paths in step 5 is as follows: and searching a path which is in accordance with the current position of the bronchus and has the minimum angle difference with the current view angle in the paths which are used as the superior nodes, and using the path as a new path of the current view angle.

Further, the lung segmentation in step 1 comprises: s11, creating a two-dimensional lung template image: extracting the maximum lung area layer images of the N chest CT images on the sagittal position to obtain N maximum lung area layer images on the sagittal position; interpolating the N sagittal lung area maximum layer images to a fixed image size, and creating to obtain a template image; counting the probability that each pixel on the template image belongs to the lung, wherein the probability is obtained by calculating according to formula 1:

p is the number of images belonging to the lung in N chest CT images/N formula 1;

p represents the probability that each pixel on the template image belongs to the lung; setting the pixels with P more than 50% as 1, setting the pixels with P less than or equal to 50% as 0, and taking N as an integer more than 1;

s12, lung segmentation is realized: selecting pixels with pixel density less than-800 HU from 12 edges of the input cuboid three-dimensional image as air segmentation seed points outside the skin, and performing 3D adaptive region growth; the self-adaptation in the 3D self-adaptation region growing means that the upper limit variable T of a region growing threshold is gradually increased, and whether region growing pixels appear in the central region of each layer of the axis position of the cuboid three-dimensional image is detected; if the current region growing threshold value is larger than the human skin threshold value, returning to the previous threshold value, re-growing the region, taking the region growing result as the air outside the skin for segmentation, and setting the corresponding pixel on the original cuboid three-dimensional image as 0 HU;

searching pixels with pixel density of less than-800 HU in the central area of each layer of the axis position of the cuboid three-dimensional image as lung segmentation seed points to carry out 3D self-adaptive area growth, wherein the upper limit of the threshold is the threshold of the last 3D area growth in air segmentation outside the skin; performing connectivity analysis on a region growing result of 3D self-adaptive region growing on the lung segmentation seed points, and extracting a connected region with the largest volume as lung segmentation to obtain a lung segmentation image;

s13, breast CT image judgment: extracting a layer with the largest lung area at the sagittal position from the lung segmentation image obtained in the step S12, and interpolating the layer image to a fixed image size to obtain an I1 image; recording the I1 image and the two-dimensional lung template image created in the step S11 as an I2 image, comparing the images, and determining that the cuboid three-dimensional image input in the step S12 is a chest CT image when the overlapping degree is more than 50%, wherein the cuboid three-dimensional image can be used in the subsequent steps; otherwise, the cuboid three-dimensional image input in the step S12 is not a chest CT image and cannot be used in the subsequent steps; the degree of overlap is calculated by equation 2:

degree of overlap ═ D × 2/(the number of pixels belonging to the lung in the I1 image + the number of pixels belonging to the lung in the I2 image) formula 2;

d represents the number of overlapping pixels, which refers to the number of pixels belonging to the lung on both the I1 image and the I2 image;

step S11 is not in sequence with step S12.

Further, the main bronchus segmentation performed in step 1 comprises:

s21, selecting main bronchus seed points: traversing pixels in the central area of each layer of the axial position of the lung segmentation image, and selecting a pixel point with the maximum coordinate z as an initial point; if a plurality of pixel points with the maximum coordinate z exist at the same time, taking one of the pixel points as an initial point; calculating the distance between each pixel point and the initial point by using a fast marching algorithm (fast marching); starting from the layer where the initial point is located, and performing 2D connected domain analysis layer by layer along the z reducing direction; until a layer is found that meets the following condition:

I. a plurality of connected regions;

II, solving the distance S from each connected region to the initial point; s, accumulating the distances from the pixels in the communication area to an initial point by traversing the pixels in the communication area, and then dividing the distances by the number of the pixels in the communication area to obtain the distance; wherein the maximum distance S_maxAt a minimum distance S_minA phase difference of more than 50 mm;

take the minimum distance S in Condition II_minAll pixel points on the corresponding connected region are used as main bronchus seed points:

s22, roughly dividing the main bronchus: obtaining the rough segmentation of the main bronchus by adopting 3D self-adaptive supervised region growth; the 3D adaptive supervised region growing means starting to perform 3D adaptive region growing from the main bronchus seed point obtained in step S21, and monitoring the region growing volume, and if the region growing volume is larger than the upper limit of the normal adult bronchus volume, stopping the growing; extracting the central line of the region growing pixel, and establishing a bronchial tree structure; judging whether the number of the sub-bronchus is more than 3, or the radius of the sub-bronchus is 20% larger than that of the parent bronchus, or the included angle between the parent bronchus and the sub-bronchus is less than 90 degrees, considering that the sub-bronchus leaks, and setting the pixel density of the original chest CT image of all leaked parts as 0 HU; continuing to perform 3D self-adaptive region growing, and after performing the above steps circularly for 2-5 times (preferably, performing the above steps circularly for 3 times), roughly dividing the final region growing pixel as a main bronchus;

s23, optimizing the main bronchus rough segmentation result: s23-1, extracting the central line of the main bronchus roughly divided by the main bronchus obtained in the step S22;

s23-2, constructing a section of cylinder along the central line of the main bronchus to roughly divide and enclose the main bronchus, wherein the radius of the cylinder is 1.5 times that of the current main bronchus section;

s23-3, roughly dividing the main trachea into seed points in each section of cylinder, taking a variable K as an upper threshold, and taking an initial value as a minimum pixel value of non-main trachea division in a neighborhood to perform 3D adaptive region growth;

s23-4, monitoring whether the growth result exceeds the range of the current cylinder; if the area growing pixel point does not exceed the current cylinder range, recording the average pixel density of the area growing pixel point and the minimum pixel density in the neighborhood, taking the difference value of the area growing pixel point and the minimum pixel density as the score of the secondary growing result, simultaneously making K equal to K +1, and returning to the step S23-3; if the region growing pixel point is outside the current cylinder, executing the next step;

s23-5, obtaining the region growing result with the highest score as the optimization of the main bronchus rough segmentation result of the section; and (4) optimizing each section of the main bronchus rough segmentation to complete the result optimization of the main bronchus rough segmentation.

Further, performing peripheral bronchial segmentation comprises:

s31, extracting the candidate bronchus section of the peripheral bronchus;

s32, extracting candidate bronchial segments of peripheral bronchi;

s33, connecting candidate bronchial segments of the peripheral bronchus;

the step S31 of extracting candidate bronchial sections of the peripheral bronchus includes:

s31-1, taking a variable F as an upper threshold limit of pixel density, wherein the initial value is-1000 HU, F traverses each integer value in the range of [ -1000HU, -500HU ], and the value is used as the upper threshold limit to perform threshold segmentation on the chest CT image;

s31-2, performing 2D connected domain analysis on the segmentation result on the axial position, the coronal position or the sagittal position of the chest CT image, recording the average pixel density and the neighborhood minimum pixel density of the connected regions, and taking the interpolation of the connected regions as the score of the corresponding connected region; if F < -500HU, let F be F +1, jump back to step S31-1, otherwise execute step S31-3;

s31-3, for each connected region, searching an adjacent connected region set of the connected region on the front layer and the rear layer, and obtaining an adjacent front-rear connected region set through permutation and combination; calculating a fitting straight line for each adjacent connected region combination, so that the sum of the distances from all pixel points in the connected region combination to the fitting straight line is minimum;

s31-4, calculating the overlapping degree of each adjacent connected region combination in the direction of the fitted straight line and the median of the connected region score; taking the middle connected region as a candidate bronchus section of the peripheral bronchus, taking the product of the combination overlapping degree and the median of the score of the connected region as the score of the candidate bronchus section of the peripheral bronchus, and taking the direction of a fitting straight line as the direction of the peripheral bronchus on the candidate section, namely the section direction of the candidate bronchus section of the peripheral bronchus;

s31-5, only keeping the peripheral bronchus candidate bronchus section with the highest score for the peripheral bronchus candidate bronchus section set with the overlapping relation;

step S32 peripheral bronchus candidate bronchus section extraction, namely connecting peripheral bronchus candidate bronchus sections meeting the connection condition in the peripheral bronchus candidate bronchus sections obtained in the step S31 to form a peripheral bronchus candidate bronchus section of a section; the connection conditions are as follows:

i. peripheral bronchial candidate bronchial sections are adjacent;

the included angle of the section directions of the candidate bronchus sections of the two peripheral bronchus is less than 60 degrees;

iii, the included angle of a connecting line between the section direction of each peripheral bronchus candidate bronchus section and the central point of the two peripheral bronchus candidate bronchus sections is less than 60 degrees;

when a plurality of peripheral bronchus candidate bronchus sections are connected, the connecting line of the central points of the peripheral bronchus candidate bronchus sections cannot be withdrawn;

v. connected peripheral bronchial candidate bronchial sections having at least one pixel density, both of which are comprised;

the overlapping ratio between the front and rear peripheral bronchial candidate bronchial sections of the current peripheral bronchial candidate bronchial section is smaller than the overlapping ratio between the current peripheral bronchial candidate bronchial section and the front and rear peripheral bronchial candidate bronchial sections, respectively;

the step S33 of inter-bronchial segment connection of the peripheral bronchial candidate includes:

s33-1, if the peripheral bronchus candidate bronchus section spans multiple levels of bronchus, firstly, disconnecting the peripheral bronchus candidate bronchus section into multiple sub-peripheral bronchus candidate bronchus sections at a bifurcation between the two levels, and respectively connecting the sub-peripheral bronchus candidate bronchus section with another peripheral bronchus candidate bronchus section;

s33-2, calculating the distance between the port section of the peripheral bronchial candidate and the port section of another peripheral bronchial candidate, and taking the minimum distance as the distance between the two peripheral bronchial candidate and the connection position;

s33-3, judging whether the two peripheral bronchus candidate bronchus sections are connected in a certain stage of bronchus or connected between two adjacent stages of bronchus;

1) if the connection is carried out in a certain grade of bronchus, the following conditions are required to be met:

A. the included angle between the candidate bronchial segments of the peripheral bronchus is less than 60 degrees;

B. the coincidence rate of the cross sections of the candidate bronchial sections of the peripheral bronchus to be connected in the direction of the connecting line of the gravity centers of the cross sections is more than 60 percent; the calculation method of the section gravity center is as follows: dividing the coordinate sum of all pixels on the section by the number of pixels to serve as the center of gravity of the section;

C. 2D self-adaptive area growth is carried out from one section to the other section layer by layer, and the coincidence rate of each grown section layer and the sections of the front layer and the rear layer in the direction of the connecting line of the gravity centers is more than 60 percent;

2) if the connection between two adjacent bronchi is needed, the following conditions are satisfied:

a. the included angle between the candidate bronchial segments of the peripheral bronchus is less than 90 degrees;

b. the coincidence rate of the cross sections to be connected in the direction of the connecting line of the centers of gravity of the cross sections is more than 50 percent;

c. 2D self-adaptive area growth is carried out from one section to the other section layer by layer, and the coincidence rate of each grown section layer and the sections of the front layer and the rear layer in the direction of the connecting line of the gravity centers is more than 50 percent;

and connecting the peripheral bronchial candidate bronchial segments according to the conditions recorded in the A-C and the a-C to complete the connection between the peripheral bronchial candidate bronchial segments.

Illustrating the meaning expressed by step S33-2: assuming that two peripheral bronchi are A and B, if the two peripheral bronchi have no sub-bronchial section, four distances A1B1, A2B1, A1B2 and A2B2 of two ports (A1 and A2) of A and two ports (B1 and B2) of B are respectively obtained, the minimum distance is taken as the distance between the two bronchial sections, and the corresponding port is taken as the position to be connected. If A has two sub-bronchial segments (C, D) and B has two sub-bronchial segments (E, F), the distances of the four sub-bronchial segments CE, CF, DE and DF are respectively calculated according to the method, and the minimum distance and the corresponding port are taken as the distance and the connection position of A and B.

Further, the bronchial segmentation that connects the main bronchial segmentation and the peripheral bronchial segmentation into a whole includes:

s41, judging whether there is an overlapping pixel: if the peripheral bronchial candidate bronchial segment of the peripheral bronchial segmentation and the main bronchus segmentation have overlapped pixels, the peripheral bronchial candidate bronchial segment can be directly connected with the main bronchus; if the peripheral bronchial candidate bronchial segment and the main bronchus are segmented without overlapping pixels, performing 2D region growth layer by layer on the main bronchus from the cross section of the port of the peripheral bronchial candidate bronchial segment, which is closer to the main bronchus;

s42, processing for the case where no overlapping pixels exist in step S41: if the overlapping ratio of the cross section of the growing area and the cross section of the previous layer is more than or equal to 50%, the candidate bronchial segment of the peripheral bronchus is directly connected with the main bronchus; if the overlapping ratio of the cross section of the growing area and the cross section of the previous layer is less than 50%, the candidate bronchial section of the peripheral bronchus cannot be directly connected with the main bronchus;

s43, processing the case that the overlapping ratio is less than 50% in the step S43: if the length of the peripheral bronchial candidate is less than 10mm, discarding; otherwise, a cylinder is constructed to connect the peripheral bronchial candidate segment and the main bronchus according to the peripheral bronchial candidate bronchial heading and its end section.

The invention also discloses a method for marking the VB in free browsing, which comprises the method for marking the VB in free browsing; further comprising:

when the browser is in the current VB browsing mode, a branch mark can be directly selected from a preset bronchus naming branch template to name and mark the bronchus section of the current browsing position; after selection, the naming mark is directly assigned to the currently browsed bronchial segment and recorded, and all the browsed bronchial segments can be named and marked by combining the step 5; the next time a VB view is taken, the named bronchial segment that has been named can automatically display its named marker when it is viewed close to the bronchus.

In one or more embodiments, a method for marking a free browsing VB includes the above-described method for free browsing VB; further comprising:

naming and marking the centerlines of the bronchial segments when obtaining the unique path from the entrance of the bronchi to the position of each bronchus ending in the step 4; after the naming mark, the naming mark is directly endowed to the currently browsed bronchial segment and recorded; the next time a VB view is taken, the named bronchial segment that has been named can automatically display its named marker when it is viewed close to the bronchus.

The invention also discloses a system for freely browsing VB, which comprises:

the bronchial tree extraction module is used for performing lung segmentation on the chest CT image, performing main bronchus segmentation and peripheral bronchus segmentation, and finally connecting the main bronchus segmentation and the peripheral bronchus segmentation to form integral bronchial segmentation so as to obtain a bronchial tree; setting pixel values of the bronchial tree to 1 and pixel values of regions of the non-bronchial tree to 0;

the center line extraction module is used for extracting a center line based on the bronchial tree to obtain a bronchial center line tree model; the centerline coordinate data of each bronchus is composed of a series of points expressed based on three-dimensional space coordinates;

the three-dimensional data generation module is used for performing isosurface extraction on the basis of the pixel value marked as 1 in the step 1 by using a Marching cube algorithm so as to generate triangular-surfaced three-dimensional model data;

the unique path generating module is used for upwards searching and combining from each tail end bronchus according to the bronchus central line tree model to obtain a unique path from the entrance of the bronchus to the end position of each bronchus;

a virtual bronchoscope rendering and path switching module for virtual bronchoscope rendering and path switching: rendering the virtual bronchoscope according to the three-dimensional model data generated by the three-dimensional data generation module; when virtual bronchoscope browsing is carried out, three-dimensional playing browsing is carried out from the position of a main bronchus, any path connected with the main bronchus is used as an initial path, and a mouse, a keyboard, a touch screen or a remote rod is used, so that a VB image can be played from the initial position of the path to the tail end along the path; when playing, besides displaying VB images, displaying the indication direction of the current path; at the bifurcation part close to the next path, using a mouse, a keyboard, a touch screen or a remote lever to carry out corresponding three-dimensional visual angle transformation, and when the direction of the visual angle transformation and the corresponding bifurcation direction are less than a threshold value gamma and the current visual angle transformation is greater than an angle delta, searching a new path which accords with the current visual angle in the generated paths; after a new path is determined, the current browsing path can be automatically switched to the corresponding bifurcation direction, and meanwhile, the current path display is updated to be a path along the corresponding bifurcation direction, so that browsing is carried out along the bronchial path expected by the user; when the next bronchial bifurcation is encountered, the path switching is repeated.

Further, still include: and the backspacing display module is used for backspacing the path by using a mouse, a keyboard, a touch screen or a remote rod so that the VB is backspacing displayed along the current path.

Further, γ is 15 degrees; delta is 20 degrees.

Further, the specific method for searching a new path conforming to the current viewing angle in the generated plurality of paths by the virtual bronchoscope rendering and path switching module is as follows: and searching a path which is in accordance with the current position of the bronchus and has the minimum angle difference with the current view angle in the paths which are used as the superior nodes, and using the path as a new path of the current view angle.

Further, the bronchial tree extraction module comprises:

the single-layer sagittal lung template image generation module is used for extracting the lung area maximum layer images of the N chest CT images on the sagittal position to obtain N sagittal lung area maximum layer images; interpolating the N sagittal lung area maximum layer images to a fixed image size, and creating to obtain a template image; counting the probability that each pixel on the template image belongs to the lung, wherein the probability is obtained by calculating according to formula 1:

the chest CT image judgment module comprises a lung segmentation module and a segmentation result and sagittal lung template comparison module;

the lung segmentation module is used for inputting a cuboid three-dimensional image; selecting pixels with pixel density less than-800 HU from 12 edges of the input cuboid three-dimensional image as air segmentation seed points outside the skin, and performing 3D adaptive region growth; the self-adaptation in the 3D self-adaptation region growing means that the upper limit variable T of a region growing threshold is gradually increased, and whether region growing pixels appear in the central region of each layer of the axis position of the cuboid three-dimensional image is detected; if the current region growing threshold value is larger than the human skin threshold value, returning to the previous threshold value, re-growing the region, taking the region growing result as the air outside the skin for segmentation, and setting the corresponding pixel on the original cuboid three-dimensional image as 0 HU; searching pixels with pixel density of less than-800 HU in the central area of each layer of the axis position of the cuboid three-dimensional image as lung segmentation seed points to carry out 3D self-adaptive area growth, wherein the upper limit of the threshold is the threshold of the last 3D area growth in air segmentation outside the skin; performing connectivity analysis on a region growing result of 3D self-adaptive region growing on the lung segmentation seed points, and extracting a connected region with the largest volume as lung segmentation to obtain a lung segmentation image;

the segmentation result and sagittal lung template comparison module is used for extracting a layer with the maximum lung area at the sagittal position from the lung segmentation image obtained by the lung segmentation module, and interpolating the layer of image to a fixed image size, and marking the layer of image as an I1 image; recording the template image created by the I1 image and the single-layer sagittal lung template image generation module as an I2 image, comparing, and determining that the input image is a chest CT image when the overlapping degree is more than 50%, wherein the input image can be used for extracting the bronchial tree; otherwise, the input image is not the chest CT image and cannot be used for extracting the bronchial tree; the degree of overlap is calculated by equation 2:

d denotes the number of overlapping pixels, and refers to the number of pixels belonging to the lung on both the I1 image and the I2 image.

Further, the value range of N is 20-60; the central area is a square area which takes the central point of the shaft-shaped position image as the center and has the side length of 80-120 pixels. Preferably, N ranges from 30 to 50. The central area is a square area which takes the central point of the shaft-shaped position image as the center and has the side length of 90-110 pixels. More preferably, N is 50. The central area is a square area with the center of the shaft-like position image as the center and the side length of 100 pixels. The fixed image size was 300 × 400 and the interlayer spacing was 1.0mm × 1.0 mm.

Further, the bronchial tree extraction module further comprises:

the main bronchus segmentation module is used for extracting and obtaining main bronchus segmentation from lung segmentation;

wherein, main bronchus is cut apart the module and is included:

the main bronchus seed point selecting module is used for traversing pixels in the central area of each layer of the axial position of the lung segmentation image and selecting a pixel point with the maximum coordinate z as an initial point; if a plurality of pixel points with the maximum coordinate z exist at the same time, taking one of the pixel points as an initial point; calculating the distance between each pixel point and the initial point by using a fast traveling algorithm; starting from the layer where the initial point is located, and performing 2D connected domain analysis layer by layer along the z reducing direction; until a layer is found that meets the following condition:

I. a plurality of connected regions;

take the minimum distance S in Condition II_minAll pixel points on the corresponding communication area are used as main bronchus seed points;

the main bronchus rough segmentation module is used for obtaining main bronchus rough segmentation by using 3D self-adaptive supervised region growing; the 3D adaptive supervised region growing means starting to perform 3D adaptive region growing from the main bronchus seed point obtained in step S21, and monitoring the region growing volume, and if the region growing volume is larger than the upper limit of the normal adult bronchus volume, stopping the growing; extracting the central line of the region growing pixel, and establishing a bronchial tree structure; judging whether the number of the sub-bronchus is more than 3, or the radius of the sub-bronchus is 20% larger than that of the parent bronchus, or the included angle between the parent bronchus and the sub-bronchus is less than 90 degrees, considering that the sub-bronchus leaks, and setting the pixel density of the original chest CT image of all leaked parts as 0 HU; continuing to execute 3D self-adaptive region growing, and after the 3D self-adaptive region growing is executed for 2-5 times in a circulating mode, roughly dividing the growing pixels of the last region as a main bronchus;

the main bronchus rough segmentation result optimization module realizes the main bronchus rough segmentation result optimization through the following modes:

s23-1, extracting the central line of the obtained main bronchus roughly divided;

Further, the device also comprises a peripheral bronchus segmentation module used for peripheral bronchus segmentation.

This peripheral bronchus is cut apart the module and is included:

the peripheral bronchus candidate section extraction module is used for taking a variable F as an upper threshold limit of pixel density, taking an initial value of the variable F as-1000 HU, traversing each integer value in the range of [ -1000HU, -500HU ], taking the value as the upper threshold limit, and performing threshold segmentation on the chest CT image; 2D connected domain analysis is carried out on the segmentation result on the axial position, the coronal position or the sagittal position of the chest CT image, the average pixel density and the neighborhood minimum pixel density of the connected regions are recorded, and the interpolation values of the connected regions are taken as the scores of the corresponding connected regions; if F is less than-500 HU, making F equal to F +1, repeatedly executing threshold segmentation on the chest CT image by using the value as the upper threshold limit; 2D connected domain analysis is carried out on the segmentation result on the axial position, the coronal position or the sagittal position of the chest CT image, the average pixel density and the neighborhood minimum pixel density of the connected regions are recorded, the interpolation of the connected regions is taken as the score operation of the corresponding connected region, otherwise, the adjacent connected region set of the connected region is searched on the front layer and the back layer of each connected region, and the set of the adjacent front and back connected region combination is obtained through arrangement and combination; calculating a fitting straight line for each adjacent connected region combination, so that the sum of the distances from all pixel points in the connected region combination to the fitting straight line is minimum; calculating the overlapping degree of each adjacent connected region combination in the direction of the fitted straight line and the median of the scores of the connected regions; taking the middle connected region as a candidate bronchus section of the peripheral bronchus, taking the product of the combination overlapping degree and the median of the score of the connected region as the score of the candidate bronchus section of the peripheral bronchus, and taking the direction of a fitting straight line as the direction of the peripheral bronchus on the candidate section, namely the section direction of the candidate bronchus section of the peripheral bronchus; for the peripheral bronchus candidate bronchus section sets with overlapping relation, only the peripheral bronchus candidate bronchus section with the highest score is reserved;

the peripheral bronchus candidate bronchus section extracting module is used for connecting peripheral bronchus candidate bronchus sections meeting the connecting condition in the obtained peripheral bronchus candidate bronchus sections to form a peripheral bronchus candidate bronchus section of a section; the connection conditions are as follows:

i. peripheral bronchial candidate bronchial sections are adjacent;

the peripheral bronchus candidate bronchus section connecting module is used for connecting any two sections of peripheral bronchus candidate bronchus sections, if the peripheral bronchus candidate bronchus sections cross multi-stage bronchus, firstly, the peripheral bronchus candidate bronchus sections are disconnected at a bifurcation between two stages to form a plurality of sections of sub-peripheral bronchus candidate bronchus sections, and the sub-peripheral bronchus candidate bronchus sections are respectively connected with another peripheral bronchus candidate bronchus section; calculating the distance between the port section of the peripheral bronchial candidate bronchial segment and the port section of another peripheral bronchial candidate bronchial segment, and taking the minimum distance as the distance between the two peripheral bronchial candidate bronchial segments and the connection position; cutting off whether the two peripheral bronchus candidate bronchus sections are connected in a certain stage of bronchus or connected between two adjacent stages of bronchus;

b. the coincidence rate of the section shapes to be connected in the direction of the connecting line of the gravity centers is more than 50 percent;

c. 2D area growth is carried out from one section to the other section layer by layer, and the coincidence rate of each layer of section and the sections of the front layer and the rear layer in the gravity center connecting line direction is larger than 50%;

Further, the system also comprises a peripheral bronchus and main bronchus connecting module, which is used for connecting the extracted main bronchus segmentation with the peripheral bronchus segmentation to form a bronchus tree;

the peripheral bronchus and main bronchus connecting module completes connection to form a bronchus tree by the following means:

The invention also discloses a system for marking the VB in free browsing, which comprises the system for marking the VB in free browsing; the system also comprises a bronchus branch manual marking module which is used for directly selecting a branch mark from a preset bronchus naming branch template to name and mark the bronchus section at the current browsing position when the system is in the current VB browsing mode; after selection, the naming mark is directly endowed to the currently browsed bronchial segment and recorded, and the naming mark can be carried out on all browsed bronchial segments by combining a virtual bronchoscope rendering and path switching module; the next time a VB view is taken, the named bronchial segment that has been named can automatically display its named marker when it is viewed close to the bronchus.

In one or more embodiments, a system for marking a free-browsing VB includes the system for marking a free-browsing VB as described above; the system also comprises a bronchial branch manual marking module which is used for naming and marking the central line of each bronchial segment when the unique path generating module obtains the unique path from the entrance of the bronchus to the terminal position of each bronchus; after the naming mark, the naming mark is directly endowed to the currently browsed bronchial segment and recorded; the next time a VB view is taken, the named bronchial segment that has been named can automatically display its named marker when it is viewed close to the bronchus.

The invention also discloses a computer readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method as described above.

The invention provides a method which can enable a doctor to freely browse in a Virtual Bronchoscope (VB), is not limited to a fixed path or a designated position, can freely jump and switch in all bronchial trees, and can mark and name the positions of currently browsed bronchial branches at any time while freely browsing, thereby providing a method for intuitively marking and naming any bronchial branches.

By using the mode of freely browsing the virtual bronchoscope, the doctor can conveniently and rapidly switch and browse any branch of the bronchial tree by the aid of the conventionally available operation mode. On the basis, because the number of branches at each branch of the bronchus is not too much and is not more than 5 generally, doctors can more easily judge the trend of the bronchus at the corresponding branch of the bronchus, so that corresponding naming marks can be completed on the bronchus below three levels in the browsing process from top to bottom, the operation of branch naming is greatly accelerated and simplified, important references are provided for positioning of surgeons in the operation and medical teaching, and training data can be provided for an algorithm based on deep learning.

The conception, the specific structure and the technical effects of the present invention will be further described with reference to the accompanying drawings to fully understand the objects, the features and the effects of the present invention.

Drawings

Figure 1 is the centerline of each bronchial segment. All centerlines in FIG. 1 begin with B, the numbers following B reflecting the branching order of the centerlines from the uppermost level to the lowermost level, respectively. Where B1 is the superior bronchial centerline, B11 and B12 are the left and right main bronchial centerlines and their parent centerlines are B1, and so on B111 and B112 are inferior centerlines of B11, and B121 and B122 are inferior centerlines of B12.

Fig. 2 is a schematic diagram showing a current path when viewing VB. The path marking line in the display of the figure points to the current VB viewing direction.

Fig. 3 is a display diagram showing switching to another path in VB browsing. The path-identifying lines in the graph display have been switched to another path direction by interaction.

FIG. 4 is a display and selection of a bronchial nametag template. The template in the figure shows the branch naming labels commonly used in clinic at present, for example, the branch naming labels are divided into right upper leaf bronchus (RUL) and right middle leaf bronchus (RML) from the lower part of the right main bronchus, and the right middle leaf bronchus can be divided into several segments of bronchus (RB4, RB5 and RB6), and the like. And displaying a corresponding named branch structure by using a tree list in the interface, and selecting any added bronchus name as a mark name of the currently browsed bronchus section when selecting the bronchus name. And can also mark the existing bronchus by newly establishing a new name.

Fig. 5 is a diagram showing named bronchial names marked during VB browsing. RB1a in the figure is the selected current bronchial naming marker, shown in VB at the start of the corresponding bronchial segment.

FIG. 6 is a flow chart of an embodiment of the present invention.

Fig. 7 is a block diagram of an embodiment of a system in accordance with the present invention.

Fig. 8 is a lung area maximum slice image of a chest CT image in a sagittal position.

Fig. 9 is a picture of the lung region segmentation of fig. 8 interpolated to a 300 x 400 fixed image size.

Fig. 10 is a template image in sagittal position created by N chest CT images.

Fig. 11 is an image of a slice of a chest CT image at an axial position, wherein the red box shows the central region of the image.

Fig. 12 is an external-skin air segmentation image in which red regions are external-skin air segmentation results.

Fig. 13 is a lung segmentation image in which a red region is a lung segmentation result.

Fig. 14(a) is an xyz voxel coordinate representation of a rectangular parallelepiped three-dimensional image. In this embodiment, there are 40 pixels in total on the z-coordinate; a total of 1600 pixels on the y-coordinate; there are 1408 pixels in total on the x-coordinate.

(39,1599,1407) indicating the pixel coordinate position of the pixel with z coordinate 39, y coordinate 1599, and x coordinate 1407; and the other coordinate values are analogized in the same way.

Fig. 14(b) is an image of a slice of a chest CT image at the axial position, in which the portion encircled by the red square frame is the selected main bronchus seed point.

Fig. 15(a) (b) (c) are images of a slice of a CT image of the breast at the axial, coronal and sagittal positions, respectively, with red regions showing the positions of the main bronchus in the axial, coronal and sagittal positions, respectively.

Fig. 16(a) (b) (c) are images of a slice of a CT image of the breast at the axial, coronal and sagittal positions, respectively, with red regions showing the locations of the main bronchus coarse segmentation optimization at the axial, coronal and sagittal positions, respectively.

Fig. 17(a), (b), and (c) are images of a layer of a CT image of the breast at the axial, coronal, and sagittal positions, wherein the red regions show the results of thresholding the peripheral bronchi and 2D connected domain analysis at the axial, coronal, and sagittal positions, respectively.

Fig. 18 is a 3D view of peripheral bronchial candidate bronchial sections connected into peripheral bronchial candidate bronchial segments, each represented by a color.

Fig. 19 is a 3D view of a candidate bronchial segment of peripheral bronchi connected to a long bronchial segment, wherein each long bronchial segment is represented by one color.

Fig. 20 is a 3D view of the connection of a peripheral bronchial segment to the main bronchus, with the red portion representing the main bronchus and the other colors representing the peripheral bronchial segment.

In each of fig. 18 to 20, the dotted straight lines indicate the extending direction of the xyz axis of the 3D view.

Fig. 21 is a schematic structural diagram of relations between cross-sectional directions, connection lines between central points, included angles, and the like in the process of extracting candidate bronchial segments of peripheral bronchi. Wherein, the included angle alpha represents the included angle between the section 1 direction and the connecting line of the central points; the angle beta represents the angle between the direction of the cross-section 2 and the line connecting the center points.

Fig. 22 is a frame diagram of the bronchial tree extraction module.

Detailed Description

In order to make the technical means, the characteristics, the purposes and the functions of the invention easy to understand, the invention is further described with reference to the specific drawings. However, the present invention is not limited to the following embodiments.

It should be understood that the structures, ratios, sizes, and the like shown in the drawings and described in the specification are only used for matching with the disclosure of the specification, so as to be understood and read by those skilled in the art, and are not used to limit the conditions under which the present invention can be implemented, so that the present invention has no technical significance, and any structural modification, ratio relationship change, or size adjustment should still fall within the scope of the present invention without affecting the efficacy and the achievable purpose of the present invention.

Embodiment 1a method for freely browsing VB and a method for marking

CT (computed tomography) imaging equipment scans to generate CT images.

1. And processing the CT image. Performing lung segmentation on a chest CT image, performing main bronchus segmentation and peripheral bronchus segmentation, and finally connecting the main bronchus segmentation and the peripheral bronchus segmentation to form integral bronchus segmentation to obtain a bronchus tree; pixel data of the bronchial tree is extracted, the pixel value of the extracted bronchial tree is set to 1, and the pixel values of the rest part (i.e. the region of the non-bronchial tree) are set to 0. The extracted bronchial tree pixel data is the basis for subsequent processing.

2. Centerline extraction using a centerline extraction method of bronchial tree based data: centerline data for the total bronchi may be extracted from the tagged bronchial tree pixel data using various sophisticated centerline extraction methods that have been disclosed so far. (for example, centerlines of the centerline segmented Image can be extracted according to the method described in the documents T.C.Lee, R.L.Kashyap, and C.N.Chu.Building sketon models via 3-D media surface/axis three better algorithms computer Vision, Graphics, and Image Processing,56(6): 462-478, 1994. the generated centerlines are split into centerline data in units of each bronchial segment according to their hierarchical order.

3. The Marching cube algorithm is used to perform iso-surface extraction based on the pixel values of previously marked bronchial pixel data, thereby generating triangularly tiled three-dimensional model data, which is the basis for virtual bronchoscope display (i.e., rendering).

4. Finding and merging from each end bronchus upwards using previously generated centerline coordinate data for each segment of the bronchus, a unique, unique path from the entrance of the bronchus to the distal location of each bronchus can be obtained. Each path is a series of points represented based on three-dimensional spatial coordinates (x, y, z).

5. When Virtual Bronchoscope (VB) browsing is performed, any path is taken as an initial path, and three-dimensional playing browsing is performed from the position of a main airway. The VB image can be played from the initial position of the path to the end along the path by using a mouse wheel, keyboard keys or other operation modes, and during playing, besides displaying the VB image, the indication direction of the current path needs to be displayed, as shown in fig. 2. And at the bifurcation part close to the next path, using a mouse or a rocker or other operation modes to perform corresponding three-dimensional visual angle transformation, and when the direction of the visual angle transformation and the corresponding bifurcation direction are smaller than a certain threshold value (15 degrees) and the current visual angle transformation is larger than a certain angle (20 degrees), searching for a path which accords with the current visual angle from the generated paths. The specific method comprises the following steps: if the current bronchus position is B11, searching the path which is in accordance with B11 as the superior node and has the smallest angle difference with the current view angle as the current new path. After the path is determined, the current browsing path is automatically switched to the corresponding branch direction, and the current path display is updated to a path along the corresponding branch direction, as shown in fig. 3. When the user uses a mouse wheel, a keyboard key or other modes to continue browsing the VB forward, the played image of the VB is carried out along a new path, and the user can change the current bronchial browsing path and browse the bronchial browsing path expected by the user. When the next bronchial bifurcation is encountered, reference is made to the previous approach. When the user uses a mouse, a joystick or other modes to perform path rollback, the VB still performs rollback display along the current path.

6. Repeat step 5 until the user can navigate to all desired bronchial end paths.

7. When the browser is in the current VB browsing mode, a branch mark can be directly selected from the preset branch template for naming the bronchial segment at the current browsing position, as shown in FIG. 4. After selection, the naming mark is directly assigned to the currently browsed bronchial segment and recorded, and in combination with the method in step 6, all the browsed bronchial segments can be named and marked. The next time a VB view is taken, the named tagged bronchial segment can automatically display its named tag when viewed close to the bronchus, see fig. 5.

In this embodiment, the obtaining of the bronchial tree in step 1 by performing lung segmentation on the CT image of the chest, performing main bronchus segmentation and peripheral bronchus segmentation, and finally connecting the main bronchus segmentation and the peripheral bronchus segmentation to form an integrated bronchus segmentation includes the following steps:

s1 lung segmentation

S11 two-dimensional lung template image creation

S12, realizing lung segmentation

S13, breast CT image judgment

S2 main bronchus segmentation

S21, selecting main bronchus seed points

S22, main bronchus rough segmentation-self-adaptive supervision region growing

S23, optimizing main bronchus rough segmentation result

S3 peripheral bronchus segmentation

S31 extraction of candidate bronchus section of peripheral bronchus

S32 extraction of candidate bronchial segments of peripheral bronchus

S33 connection between candidate bronchial segments of peripheral bronchus

S4 bronchial segmentation connecting main bronchial segmentation and peripheral bronchial segmentation into a whole

S41, judging whether there is overlapping pixel

S42, processing the case that there is no overlapped pixel in the step S41

S43, processing the overlapping ratio < 50% in the step S43

The following is a detailed description:

s1 lung segmentation

S11 two-dimensional lung template image creation

In the embodiment, the data lung segmentation golden standard is obtained from 50 chest CT images by combining automatic segmentation and a doctor manual correction mode. The slice image with the largest area of the lung segments on the sagittal site was extracted (see fig. 8), and the 2D images were interpolated to the template image size (size 300 × 400, interlamellar spacing 1.0mm × 1.0mm), see fig. 9. And traversing the interpolated images, and counting the probability that each pixel on the template belongs to the lung, wherein the calculation formula is as follows:

probability of a pixel belonging to the lung, i.e. number of images of which a pixel belongs to the lung in 50 CT images/50

Pixels with a probability greater than 50% are set to 1, and the other pixels are set to 0. The template image results are shown in FIG. 10.

S12, realizing lung segmentation

Traversing 12 side pixels of the input cuboid three-dimensional image, selecting pixels with the density of less than-800 HU as seed points, and performing 3D self-adaptive region growth. The cuboid three-dimensional image is a CT scanning image. The method comprises the following specific steps:

and taking a variable T as the upper limit of a region growth threshold, taking an initial value as-800 HU, and starting 3D self-adaptive region growth from the neighborhood of the seed point.

Judging whether the area growing pixel falls on the central area of the image axis (a square area with the axis image central point as the center and the side length of 100 pixels, see fig. 11, the red frame area represents the image central area), if so, indicating that the current area growing threshold is larger than the human skin threshold, making T equal to T-1 (i.e. returning to the previous threshold), and then jumping to the next step. If not, judging if T>400HU, jump to the next step, if T<-400, let T ═ T +1, then jump to the previous step

And taking the T value obtained in the step two as an upper threshold limit, performing 3D adaptive region growing again, dividing the region growing result as air outside the skin (see figure 12), and setting the pixel value obtained by corresponding region growing in the original cuboid three-dimensional image as 0 HU.

Searching pixel points with the pixel density less than-800 HU in the central area of the image axial position as seed points, and simultaneously taking the T value obtained in the step two as the upper limit of a threshold value to carry out 3D self-adaptive area growth

Performing connectivity analysis on the pixel points obtained by the region growth in the last step, and extracting a connected region with the largest volume, wherein the region is a lung segmentation result to obtainTo the lung segmentation image (see fig. 13).

S13, breast CT image judgment

The layer with the largest candidate lung segmentation area is found at the lung segmentation image vector position obtained in step S12, and the layer image is extracted and interpolated to the template image size. Traversing the interpolation image and the two-dimensional lung template image, if the pixel belongs to the lung on the template image and also belongs to the lung on the interpolated image, the pixel is considered as an overlapping pixel. All overlapped pixels are counted, and the overlapping degree of the interpolation image (set as I1) and the template image (set as I2) is calculated, wherein the calculation formula is as follows:

overlap 2/(number of pixels of the I1 image belonging to the lung + number of pixels of the I2 image belonging to the lung)

The number of overlapping pixels refers to the number of pixels belonging to the lung on both the I1 map and the I2 map.

If the overlap degree is more than 50%, the extracted maximum connected region is considered as lung segmentation. Otherwise, the incoming image is not considered to be a chest CT image, and the following steps are not continued.

S2 main bronchus segmentation

S21, selecting main bronchus seed points

The coordinates of each pixel on the image can be represented by (x, y, z), see fig. 14 (a).

The method comprises the following specific steps:

i. traversing pixels of the lung segmentation image in the central area of the axial position (see fig. 11), and selecting a pixel point with the maximum coordinate z as an initial point from all pixels belonging to the lung. If a plurality of pixels have the same coordinate z and the coordinate z is the maximum, one of the pixels is taken as an initial point.

And ii, setting the speed of segmenting the pixels of the lung to be 1.0, and calculating the distance between each pixel point and the initial point by using a fast marching algorithm (fast marching).

Starting from the layer at the initial point, performing 2D communication domain analysis layer by layer along the decreasing z-direction. Until a layer is found that meets the following condition: 1) a plurality of connected regions; 2) obtaining the distance S from each connected region to an initial point; s by traversing the pixels in the connected region, get them toAccumulating initial point distances, and dividing the initial point distances by the number of pixels in a connected region to obtain the initial point distances; wherein the maximum distance S_maxAt a minimum distance S_minThe difference is greater than 50 mm. And taking all pixel points on the connected region with the minimum initial point distance as main bronchus seed points.

The bronchial seed point selection results are shown in FIG. 14 (b).

S22, main bronchus rough segmentation-3D self-adaptive supervised region growing

The method comprises the following specific steps:

i. 3D region growth was performed starting from the main trachea seed point, taking variable H (initial value of H-1000 HU) as the upper threshold.

And if the volume of the region growing pixels is smaller than the upper limit of the volume of the normal human bronchus, making H equal to H +1, jumping back to the first step, and if not, continuing to execute the next step.

And iii, extracting the central line of the region growing pixel, establishing a bronchial tree structure, judging that the section of the bronchial tubes is leaked if the number of the sub-bronchial tubes is more than 3, or the radius of the sub-bronchial tubes is more than 20% of that of the parent bronchial tubes, or the included angle between the parent bronchial tubes and the sub-bronchial tubes is less than 90 degrees, and setting the section of the bronchial tubes and the sub-bronchial tubes thereof as 0. And if the execution times of the third step is less than 3, jumping to the second step, and if not, continuing to execute the next step.

And iv, roughly dividing the area growing pixel with the leakage removed finally as a main gas pipe.

The results of the rough segmentation of the main bronchus are shown in fig. 15.

S23 main bronchus rough segmentation result optimization

The main bronchus is roughly divided from the real tracheal wall by a certain distance from the image, if the division result is not attached to the tracheal wall, the calculation of the radius of the bronchus is inaccurate, and the quantitative analysis of the bronchus is interfered. The specific optimization steps are as follows:

i. and extracting the central line of the roughly divided main trachea, and calculating the radius corresponding to each point on the central line.

Constructing a segment of a cylinder along the centerline to surround the segmented bronchus according to the radius of the centerline point, the radius of the cylinder may be 1.5 times the current radius of the bronchus.

And iii, in each section of cylinder, roughly dividing the main trachea into seed points, taking a variable K (the initial value is the minimum pixel value of non-main trachea division in the neighborhood) as the upper threshold, and performing 3D adaptive region growth.

if the area growing pixel point does not exceed the current cylinder range, recording the average pixel density of the area growing pixel point and the minimum pixel density in the neighborhood, taking the difference value of the area growing pixel point and the minimum pixel density as the score of the growing result, and simultaneously making K equal to K +1 and jumping back to the previous step. And if the region growing pixel point is outside the current cylinder, executing the next step.

And v, obtaining the highest region growing result as the optimization of the main bronchus rough segmentation result of the section.

The result of the main bronchus rough segmentation is optimized as shown in fig. 16.

S3 peripheral bronchus segmentation

S31 extraction of candidate bronchus section of peripheral bronchus

Peripheral bronchial lumens have low contrast, and a single threshold or even multiple thresholds may not be able to distinguish all peripheral bronchi and lungs well. But for any peripheral airway cross-section in the axial, coronal or sagittal orientation, a threshold is usually found to distinguish the bronchi and lungs in that cross-section. By utilizing the characteristic, the candidate cross section of the peripheral bronchus can be extracted, and the specific steps are as follows:

i. the variable F (initial value-1000 HU) is taken as the upper threshold, and the chest CT image is subjected to threshold segmentation.

Performing 2D connected domain analysis on the segmentation results at the axis, corona or sagittal positions, and recording the average pixel density and the neighborhood minimum pixel density of these connected regions. And taking the interpolation of the two as the score of the connected region. And if the F is less than-500 HU, making F equal to F +1, jumping back to the previous step, and otherwise, executing the next step.

For each connected region, finding a set of adjacent connected regions for that connected region on the front and back layers. And obtaining a set of adjacent front and back connected region combinations through permutation and combination. And solving a straight line fitting for each adjacent connected domain combination, so that the sum of the distances from all pixel points of the connected domain combination to the straight line is minimum. In the present invention, the combination of connected regions and the set of connected regions do not mean one. Assuming that the current connected region (0) has two adjacent connected regions 1, 2 on the previous layer and two connected regions 3, 4 on the next layer, 1, 2, 3, 4 are called as the set of adjacent connected regions. 1, 0, 3; 1, 0, 4; 2, 0, 3; the four 2, 0, 4 are connected region combinations, which together are called a set of connected region combinations.

Calculating the overlapping degree of each adjacent connected region combination in the direction of the fitted straight line and the median of the connected region score. And taking the intermediate connected region as a candidate bronchus section of the peripheral bronchus, taking the product of the combination overlapping degree and the median of the scores of the connected regions as the score of the candidate bronchus section of the peripheral bronchus, and taking the direction of the fitted straight line as the orientation of the peripheral bronchus on the candidate section.

v. for the set of peripheral bronchial candidate sections with overlapping relationship, only the peripheral bronchial candidate section with the highest score is retained.

The results of the peripheral bronchial candidate cross-section extraction are shown in fig. 17.

S32 peripheral bronchus candidate bronchial segment extraction

Connecting the peripheral bronchus candidate bronchus sections to form a candidate bronchus section, wherein the connection needs to meet the following conditions:

i. the cross sections are adjacent.

Angle of cross-sectional direction less than 60 degrees.

Each cross-sectional direction makes an angle of less than 60 degrees with respect to a line connecting the center points of the two cross-sections (see fig. 21).

When multiple sections are connected, the line connecting their center points cannot be retracted.

v. connected cross-sections, having at least one pixel density, all of which comprise.

The overlap ratio between the front and rear sections of the current section is smaller than the overlap ratio between the front and rear sections of the current section.

And connecting the peripheral bronchus candidate bronchus sections which meet the connection conditions to form a section of candidate bronchus section.

The extraction result of the candidate bronchial segment of the peripheral bronchus is shown in fig. 18.

S33 peripheral bronchus candidate bronchial intersegment connection

There are two manifestations of a candidate bronchial segment:

i. completely in a certain level of bronchus.

Spanning multiple bronchi.

There are also two ways of connecting candidate bronchial segments:

connecting in a certain level of branch trachea.

v. connecting the adjacent two-stage bronchi.

The specific connection steps are as follows:

i. if any two candidate bronchial segments cross the multi-stage bronchi, firstly, the two candidate bronchial segments are disconnected at the branch point between two stages, and the connection between the sub candidate bronchial segment and the other bronchial segment is respectively obtained.

And ii, calculating the distance between the port cross section of the two candidate bronchial segments (including the sub-candidate bronchial segment) and the port cross section of the other candidate bronchial segment, and taking the minimum distance as the distance between the two candidate bronchial segments and the connection position.

And iii, judging whether the two candidate bronchial sections are connected in the tube or between the tubes, if the two candidate bronchial sections are connected in the bronchial tube, meeting the following conditions:

the included angle between the candidate bronchial segments is less than 60 degrees

The coincidence rate of the cross sections to be connected in the direction of the line connecting the centers of gravity of the cross sections is more than 60%. The calculation method of the section gravity center is as follows: the coordinate sum of all pixels on the cross section divided by the number of pixels is taken as the center of gravity of the cross section.

Carrying out 2D self-adaptive region growth from one section to the other section layer by layer, wherein the coincidence rate of each section grown and the sections of the front layer and the rear layer in the gravity center connecting line direction is more than 60 percent

If the two adjacent bronchus are connected, the following conditions are required to be met:

the included angle between the candidate bronchial segments of the peripheral bronchus is less than 90 degrees

The coincidence rate of the cross sections to be connected in the direction of the line connecting the centers of gravity of the cross sections is more than 50%

Carrying out 2D self-adaptive region growth from one section to the other section layer by layer, wherein the coincidence rate of each section grown and the sections of the front layer and the rear layer in the gravity center connecting line direction is more than 50 percent

And connecting the short candidate bronchial segments according to the above conditions to form a long candidate bronchial segment.

The results of the connection between the candidate bronchial segments of peripheral bronchi are shown in fig. 19.

The specific connection steps are as follows:

The final overall bronchial segmentation result, which is composed of the connection of the peripheral bronchus and the main bronchus, is shown in fig. 20.

Embodiment 2 another method of freely browsing VB and method of marking

In this embodiment, when the unique paths are obtained in step 4, naming marks are directly performed on the unique paths; after the naming mark, the naming mark is directly endowed to the currently browsed bronchial segment and recorded; the next time a VB view is taken, the named bronchial segment that has been named can automatically display its named marker when it is viewed close to the bronchus. Without selecting a branch marker from the preset branch template for naming the bronchial segment at the current browsing position in step 7. Otherwise, the procedure was as in example 1.

More or less coincident parts exist by adopting the unique paths (such as B1-B11 in B1-B11-B111 and B1-B11-B112 are coincident parts), and all paths are taken as the basis of subsequent path switching. As shown in fig. 1, B1, B11, B12, B111, B112, B12, B121, and B122 are the centerlines of the bronchial segments, and can be combined into several different paths: B1-B11-B111, B1-B11-B112, B1-B12-B121, B1-B12-B122, each path being a series of points represented based on three-dimensional spatial coordinates (x, y, z).

Embodiment 3 a system for freely browsing VB and a system for marking freely browsing VB

The system for freely browsing VB and the system for freely browsing VB for marking (see fig. 7) in the embodiment both include a bronchial tree extraction module, which is used for performing lung segmentation on a chest CT image, performing main bronchus segmentation and peripheral bronchus segmentation, and finally connecting the main bronchus segmentation and the peripheral bronchus segmentation to form an integral bronchial segmentation, so as to obtain a bronchial tree; setting pixel values of the bronchial tree to 1 and pixel values of regions of the non-bronchial tree to 0;

a virtual bronchoscope rendering and path switching module for virtual bronchoscope rendering and path switching: rendering the virtual bronchoscope according to the three-dimensional model data generated by the three-dimensional data generation module; when virtual bronchoscope browsing is carried out, three-dimensional playing browsing is carried out from the position of a main bronchus, any path connected with the main bronchus is used as an initial path, and a mouse, a keyboard, a touch screen or a remote rod is used, so that a VB image can be played from the initial position of the path to the tail end along the path; when playing, besides displaying VB images, displaying the indication direction of the current path; performing corresponding three-dimensional visual angle transformation at a bifurcation part close to the next path by using a mouse, a keyboard, a touch screen or a remote lever, and searching a new path which accords with the current visual angle from the generated paths when the direction of the visual angle transformation and the corresponding bifurcation direction are less than a threshold value of 15 degrees and the current visual angle transformation is greater than an angle 20; after a new path is determined, the current browsing path can be automatically switched to the corresponding bifurcation direction, and meanwhile, the current path display is updated to be a path along the corresponding bifurcation direction, so that browsing is carried out along the bronchial path expected by the user; when the next bronchus bifurcation is met, the path switching is repeated;

the backspacing display module is used for backspacing a path by using a mouse, a keyboard, a touch screen or a remote rod so that VB is backspacing displayed along the current path;

the bronchus branch manual marking module is used for directly selecting a branch mark from a preset bronchus naming branch template to name and mark the bronchus section of the current browsing position when the bronchus branch manual marking module is in the current VB browsing mode; after selection, the naming mark is directly endowed to the currently browsed bronchial segment and recorded, and the naming mark can be carried out on all browsed bronchial segments by combining a virtual bronchoscope rendering and path switching module; the next time a VB view is taken, the named bronchial segment that has been named can automatically display its named marker when it is viewed close to the bronchus.

In this embodiment, the bronchial tree extraction module includes the following modules, as shown in fig. 22.

1. The single-layer sagittal lung template image generation module is used for extracting the lung area maximum layer images of the N chest CT images on the sagittal position to obtain N sagittal lung area maximum layer images; interpolating the N sagittal lung area maximum layer images to a fixed image size, and creating to obtain a template image; counting the probability that each pixel on the template image belongs to the lung, wherein the probability is obtained by calculating according to formula 1:

2. the chest CT image judging module comprises a lung segmentation module and a segmentation result and sagittal lung template comparison module.

And 2.1 a lung segmentation module used for carrying out lung segmentation on the input image. Lung segmentation was performed specifically as follows: the input image is a cuboid three-dimensional image; selecting pixels with pixel density less than-800 HU from 12 edges of the input cuboid three-dimensional image as air segmentation seed points outside the skin, and performing 3D adaptive region growth; the self-adaptation in the 3D self-adaptation region growing means that the upper limit variable T of a region growing threshold is gradually increased, and whether region growing pixels appear in the central region of each layer of the axis position of the cuboid three-dimensional image is detected; if the current region growing threshold value is larger than the human skin threshold value, returning to the previous threshold value, re-growing the region, taking the region growing result as the air outside the skin for segmentation, and setting the corresponding pixel on the original cuboid three-dimensional image as 0 HU; searching pixels with pixel density of less than-800 HU in the central area of each layer of the axis position of the cuboid three-dimensional image as lung segmentation seed points to carry out 3D self-adaptive area growth, wherein the upper limit of the threshold is the threshold of the last 3D area growth in air segmentation outside the skin; and performing connectivity analysis on the region growing result of the 3D self-adaptive region growing of the lung segmentation seed points, and extracting a connected region with the largest volume as lung segmentation to obtain a lung segmentation image.

2.2 the segmentation result and sagittal lung template comparing module is used for extracting the lung area maximum layer on the sagittal lung from the lung segmentation image obtained by the lung segmentation module, and interpolating the layer image to a fixed image size, and recording the layer image as an I1 image; recording the template image created by the I1 image and the single-layer sagittal lung template image generation module as an I2 image, comparing, and determining that the input image is a chest CT image when the overlapping degree is more than 50%, wherein the input image can be used for extracting the bronchial tree; otherwise, the input image is not the chest CT image and cannot be used for extracting the bronchial tree; the degree of overlap is calculated by equation 2:

3. And the main bronchus segmentation module is used for extracting and obtaining main bronchus segmentation from lung segmentation. The main bronchus segmentation module includes:

3.1 a main bronchus seed point selecting module, which is used for traversing pixels in the central area of each layer of the axial position of the lung segmentation image and selecting a pixel point with the maximum coordinate z as an initial point; if a plurality of pixel points with the maximum coordinate z exist at the same time, taking one of the pixel points as an initial point; calculating the distance between each pixel point and the initial point by using a fast traveling algorithm; starting from the layer where the initial point is located, and performing 2D connected domain analysis layer by layer along the z reducing direction; until a layer is found that meets the following condition:

I. a plurality of connected regions;

taking all pixel points on the communication area with the minimum distance from the initial point as main bronchus seed points;

3.2, a main bronchus rough segmentation module, which is used for obtaining the main bronchus rough segmentation by adopting 3D self-adaptive supervision area growth from the seed point of the main bronchus;

3.3, a main bronchus rough segmentation result optimizing module, which is used for constructing a section of cylinder by extracting the bronchus central line of the main bronchus rough segmentation to roughly divide and surround the main bronchus, performing 3D self-adaptive region growth, recording the average pixel density and the neighborhood minimum pixel density of each growth result, and selecting the region growth result with the maximum difference as the optimization of the main bronchus rough segmentation; and (4) optimizing each section of the main bronchus rough segmentation to complete the result optimization of the main bronchus rough segmentation.

4. And the peripheral bronchus segmentation module is used for extracting and obtaining peripheral bronchus segmentation from the to-be-segmented lung. The peripheral bronchial segmentation module includes:

4.1 peripheral bronchus candidate section extraction module, for taking variable F as the upper threshold limit of pixel density, the initial value is-1000 HU, F traverses each integer value in the range of [ -1000HU, -500HU ], using this value as the upper threshold limit, and carrying out threshold segmentation on the chest CT image; 2D connected domain analysis is carried out on the segmentation result on the axial position, the coronal position or the sagittal position of the chest CT image, the average pixel density and the neighborhood minimum pixel density of the connected regions are recorded, and the interpolation values of the connected regions are taken as the scores of the corresponding connected regions; if F is less than-500 HU, making F equal to F +1, repeatedly executing threshold segmentation on the chest CT image by using the value as the upper threshold limit; 2D connected domain analysis is carried out on the segmentation result on the axial position, the coronal position or the sagittal position of the chest CT image, the average pixel density and the neighborhood minimum pixel density of the connected regions are recorded, the interpolation of the connected regions is taken as the score operation of the corresponding connected region, otherwise, the adjacent connected region set of the connected region is searched on the front layer and the back layer of each connected region, and the set of the adjacent front and back connected region combination is obtained through arrangement and combination; calculating a fitting straight line for each adjacent connected region combination, so that the sum of the distances from all pixel points in the connected region combination to the fitting straight line is minimum; calculating the overlapping degree of each adjacent connected region combination in the direction of the fitted straight line and the median of the scores of the connected regions; taking the middle connected region as a candidate bronchus section of the peripheral bronchus, taking the product of the combination overlapping degree and the median of the score of the connected region as the score of the candidate bronchus section of the peripheral bronchus, and taking the direction of a fitting straight line as the direction of the peripheral bronchus on the candidate section, namely the section direction of the candidate bronchus section of the peripheral bronchus; for the peripheral bronchus candidate bronchus section sets with overlapping relation, only the peripheral bronchus candidate bronchus section with the highest score is reserved.

4.2 a peripheral bronchus candidate bronchus section extracting module, which is used for connecting peripheral bronchus candidate bronchus sections meeting the connection condition in the obtained peripheral bronchus candidate bronchus sections to form a peripheral bronchus candidate bronchus section of a section; the connection conditions are as follows:

i. peripheral bronchial candidate bronchial sections are adjacent;

the ratio of overlap between two peripheral bronchial candidate cross-sections before and after the current peripheral bronchial candidate cross-section is smaller than the ratio of overlap between the current peripheral bronchial candidate cross-section and the respective front and rear peripheral bronchial candidate cross-sections.

4.3 a peripheral bronchus candidate bronchus section connecting module, configured to connect any two segments of peripheral bronchus candidate bronchus sections, wherein if a peripheral bronchus candidate bronchus section spans multiple levels of bronchus, the peripheral bronchus candidate bronchus section is first disconnected into multiple segments of sub-peripheral bronchus candidate bronchus sections at a bifurcation between two levels, and the sub-peripheral bronchus candidate bronchus sections are respectively connected with another peripheral bronchus candidate bronchus section; calculating the distance between the port section of the peripheral bronchial candidate bronchial segment and the port section of another peripheral bronchial candidate bronchial segment, and taking the minimum distance as the distance between the two peripheral bronchial candidate bronchial segments and the connection position; cutting off whether the two peripheral bronchus candidate bronchus sections are connected in a certain stage of bronchus or connected between two adjacent stages of bronchus;

5. And the peripheral bronchus and main bronchus connecting module is used for connecting the extracted main bronchus segmentation with the peripheral bronchus segmentation to form a bronchus tree. The peripheral bronchial and main bronchial connection modules are used to form the bronchial tree, in particular by:

judging whether overlapping pixels exist: if the peripheral bronchial candidate bronchial segment of the peripheral bronchial segmentation and the main bronchus segmentation have overlapped pixels, the peripheral bronchial candidate bronchial segment can be directly connected with the main bronchus; if the peripheral bronchial candidate bronchial segment and the main bronchus are segmented without overlapping pixels, performing 2D region growth layer by layer on the main bronchus from the cross section of the port of the peripheral bronchial candidate bronchial segment, which is closer to the main bronchus;

the case of no overlapping pixels is processed: if the overlapping ratio of the cross section of the growing area and the cross section of the previous layer is more than or equal to 50%, the candidate bronchial segment of the peripheral bronchus is directly connected with the main bronchus; if the overlapping ratio of the cross section of the growing area and the cross section of the previous layer is less than 50%, the candidate bronchial section of the peripheral bronchus cannot be directly connected with the main bronchus;

the case of overlap ratio < 50% was treated: if the length of the peripheral bronchial candidate is less than 10mm, discarding; otherwise, a cylinder is constructed to connect the peripheral bronchial candidate segment and the main bronchus according to the peripheral bronchial candidate bronchial heading and its end section.

Embodiment 4 Another System for freely browsing VB and System for marking freely browsing VB

The same procedure as in example 3 was repeated, except that the manual labeling module for bronchial branches was used.

The manual marking module for the bronchial branches in the embodiment is used for naming and marking the central line of each bronchial segment when the unique path generating module obtains a unique path from an inlet of a bronchus to the tip position of each bronchus; after the naming mark, the naming mark is directly endowed to the currently browsed bronchial segment and recorded; the next time a VB view is taken, the named bronchial segment that has been named can automatically display its named marker when it is viewed close to the bronchus.

Embodiment 5A computer-readable storage Medium

The computer-readable storage medium described in this embodiment has stored thereon a computer program, which when executed by a processor implements the steps of the method described in embodiment 1 or 2.

The foregoing detailed description of the preferred embodiments of the invention has been presented. It should be understood that numerous modifications and variations could be devised by those skilled in the art in light of the present teachings without departing from the inventive concepts. Therefore, the technical solutions available to those skilled in the art through logic analysis, reasoning and limited experiments based on the prior art according to the concept of the present invention should be within the scope of protection defined by the claims.

Claims

1. A method for freely browsing VB, comprising:

step 1, performing lung segmentation on a chest CT image, performing main bronchus segmentation and peripheral bronchus segmentation, and finally connecting the main bronchus segmentation and the peripheral bronchus segmentation to form integral bronchus segmentation so as to obtain a bronchus tree; setting the pixel value of the bronchial tree to be 1, and setting the pixel value of the area which is not the bronchial tree to be 0;

2. A method for freely browsing VB according to claim 1, further comprising step 6 of performing path rollback using a mouse, keyboard, touch screen or joystick, so that VB is displayed in rollback along the current path.

3. A method for freely browsing VB according to claim 1, wherein γ is 15 degrees; delta is 20 degrees.

4. A method for freely browsing VB according to claim 1, wherein the specific method for searching for a new path that matches the current view in the generated paths in step 5 is: and searching a path which is in accordance with the current position of the bronchus and has the minimum angle difference with the current view angle in the paths which are used as the superior nodes, and using the path as a new path of the current view angle.

5. A method for marking free browsing VB, characterized by comprising the method for marking free browsing VB according to any one of claims 1-4; further comprising:

when the browser is in the current VB browsing mode, a branch mark can be directly selected from a preset bronchus naming branch template to name and mark the bronchus section of the current browsing position; after selection, the naming mark is directly assigned to the currently browsed bronchial segment and recorded, and in combination with the step 5, the naming mark can be performed on all browsed bronchial segments; the next time a VB view is taken, the named bronchial segment that has been named can automatically display its named marker when it is viewed close to the bronchus.

6. A method for marking free browsing VB, characterized by comprising the method for marking free browsing VB according to any one of claims 1-4; further comprising:

naming the centerlines of the bronchial segments when the unique path from the entrance of the bronchi to the distal position of each bronchus is obtained in step 4; after the naming mark, the naming mark is directly endowed to the currently browsed bronchial segment and recorded; the next time a VB view is taken, the named bronchial segment that has been named can automatically display its named marker when it is viewed close to the bronchus.

7. A system for free browsing of VB, comprising:

the bronchial tree extraction module is used for performing lung segmentation on the chest CT image, performing main bronchus segmentation and peripheral bronchus segmentation, and finally connecting the main bronchus segmentation and the peripheral bronchus segmentation to form integral bronchial segmentation so as to obtain a bronchial tree; setting the pixel value of the bronchial tree to be 1, and setting the pixel value of the area which is not the bronchial tree to be 0;

the centerline extraction module is used for extracting a centerline based on the bronchial tree to obtain a bronchial centerline tree model; the centerline coordinate data of each bronchus is composed of a series of points expressed based on three-dimensional space coordinates;

the unique path generating module is used for upwards searching and combining from each tail end bronchus according to the bronchus centerline tree model to obtain a unique path from an inlet of the bronchus to the end position of each bronchus;

a virtual bronchoscope rendering and path switching module for virtual bronchoscope rendering and path switching: rendering the virtual bronchoscope according to the three-dimensional model data generated by the three-dimensional data generation module; when virtual bronchoscope browsing is carried out, three-dimensional playing browsing is carried out from the position of a main bronchus, any path connected with the main bronchus is used as an initial path, and a mouse, a keyboard, a touch screen or a remote rod is used, so that a VB image can be played from the initial position of the path to the tail end along the path; when playing, besides displaying VB images, displaying the indication direction of the current path; at the bifurcation part close to the next path, using a mouse, a keyboard, a touch screen or a remote lever to carry out corresponding three-dimensional visual angle transformation, and when the direction of the visual angle transformation and the corresponding bifurcation direction are less than a threshold value gamma and the current visual angle transformation is greater than an angle delta, searching a new path which accords with the current visual angle in the generated paths; after a new path is determined, the current browsing path can be automatically switched to the corresponding bifurcation direction, and meanwhile, the current path display is updated to be a path along the corresponding bifurcation direction, so that browsing is carried out along the bronchial path expected by the user; the path switch is repeated when the next bronchial bifurcation is encountered.

8. A system for freely browsing VB according to claim 7, further comprising: and the backspacing display module is used for backspacing the path by using a mouse, a keyboard, a touch screen or a remote rod so that the VB is backspacing displayed along the current path.

9. A system for freely browsing VB according to claim 7, wherein γ is 15 degrees; delta is 20 degrees.

10. The system for freely browsing VB according to claim 7, wherein the virtual bronchoscope rendering and path switching module searches for a new path that matches the current view angle among the generated paths by: and searching a path which is in accordance with the current position of the bronchus and has the minimum angle difference with the current view angle in the paths which are used as the superior nodes, and using the path as a new path of the current view angle.

11. A system for marking a free browsing VB, comprising the system for marking a free browsing VB according to any one of claims 7 to 10; the system also comprises a bronchus branch manual marking module which is used for directly selecting a branch mark from a preset bronchus naming branch template to name and mark the bronchus section at the current browsing position when the system is in the current VB browsing mode; after selection, naming marks are directly assigned to the currently browsed bronchial segments and recorded, and all the browsed bronchial segments can be named and marked by combining the virtual bronchoscope rendering and path switching module; the next time a VB view is taken, the named bronchial segment that has been named can automatically display its named marker when it is viewed close to the bronchus.

12. A system for marking a free browsing VB, comprising the system for marking a free browsing VB according to any one of claims 7 to 10; the system also comprises a bronchial branch manual marking module which is used for naming and marking the central line of each bronchial segment when the unique path generation module obtains the unique path from the entrance of the bronchus to the position of each bronchial tip; after the naming mark, the naming mark is directly endowed to the currently browsed bronchial segment and recorded; the next time a VB view is taken, the named bronchial segment that has been named can automatically display its named marker when it is viewed close to the bronchus.

13. A computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, which computer program, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 6.