EP3975863A1 - Passive-ultrsound-sensor-based initialization for image-based device segmentantion - Google Patents
Passive-ultrsound-sensor-based initialization for image-based device segmentantionInfo
- Publication number
- EP3975863A1 EP3975863A1 EP20725568.8A EP20725568A EP3975863A1 EP 3975863 A1 EP3975863 A1 EP 3975863A1 EP 20725568 A EP20725568 A EP 20725568A EP 3975863 A1 EP3975863 A1 EP 3975863A1
- Authority
- EP
- European Patent Office
- Prior art keywords
- medical device
- interventional medical
- imagery
- shape
- location
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 claims abstract description 118
- 230000008569 process Effects 0.000 claims abstract description 73
- 238000012545 processing Methods 0.000 claims abstract description 42
- 230000015654 memory Effects 0.000 claims abstract description 33
- 238000002604 ultrasonography Methods 0.000 claims description 145
- 238000001914 filtration Methods 0.000 claims description 29
- 239000000523 sample Substances 0.000 claims description 24
- 238000012285 ultrasound imaging Methods 0.000 claims description 21
- 238000004590 computer program Methods 0.000 claims description 12
- 230000011218 segmentation Effects 0.000 description 27
- 238000012800 visualization Methods 0.000 description 14
- 238000003384 imaging method Methods 0.000 description 10
- 230000008439 repair process Effects 0.000 description 5
- 239000000463 material Substances 0.000 description 4
- 210000001519 tissue Anatomy 0.000 description 4
- 230000006870 function Effects 0.000 description 3
- 238000004519 manufacturing process Methods 0.000 description 3
- 230000002792 vascular Effects 0.000 description 3
- 210000003484 anatomy Anatomy 0.000 description 2
- 230000003190 augmentative effect Effects 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000013152 interventional procedure Methods 0.000 description 2
- 238000005259 measurement Methods 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 238000001356 surgical procedure Methods 0.000 description 2
- 239000002033 PVDF binder Substances 0.000 description 1
- 208000031481 Pathologic Constriction Diseases 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 238000005452 bending Methods 0.000 description 1
- 238000001574 biopsy Methods 0.000 description 1
- 210000000988 bone and bone Anatomy 0.000 description 1
- 230000000747 cardiac effect Effects 0.000 description 1
- 230000001684 chronic effect Effects 0.000 description 1
- 229920001577 copolymer Polymers 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 238000010304 firing Methods 0.000 description 1
- 239000007943 implant Substances 0.000 description 1
- 230000008676 import Effects 0.000 description 1
- 230000001788 irregular Effects 0.000 description 1
- 239000003550 marker Substances 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 210000004115 mitral valve Anatomy 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 229920002981 polyvinylidene fluoride Polymers 0.000 description 1
- 208000037804 stenosis Diseases 0.000 description 1
- 230000036262 stenosis Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B8/00—Diagnosis using ultrasonic, sonic or infrasonic waves
- A61B8/08—Detecting organic movements or changes, e.g. tumours, cysts, swellings
- A61B8/0833—Detecting organic movements or changes, e.g. tumours, cysts, swellings involving detecting or locating foreign bodies or organic structures
- A61B8/0841—Detecting organic movements or changes, e.g. tumours, cysts, swellings involving detecting or locating foreign bodies or organic structures for locating instruments
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B8/00—Diagnosis using ultrasonic, sonic or infrasonic waves
- A61B8/42—Details of probe positioning or probe attachment to the patient
- A61B8/4245—Details of probe positioning or probe attachment to the patient involving determining the position of the probe, e.g. with respect to an external reference frame or to the patient
- A61B8/4254—Details of probe positioning or probe attachment to the patient involving determining the position of the probe, e.g. with respect to an external reference frame or to the patient using sensors mounted on the probe
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B8/00—Diagnosis using ultrasonic, sonic or infrasonic waves
- A61B8/48—Diagnostic techniques
- A61B8/483—Diagnostic techniques involving the acquisition of a 3D volume of data
Definitions
- An ultrasound tracking technology estimates the position of a passive ultrasound sensor (e.g., PZT, PVDF, copolymer or other piezoelectric material) in the field of view (FOV) of a diagnostic ultrasound B-mode image by analyzing the signal received by the passive ultrasound sensor as imaging beams from an ultrasound probe sweep the field of view.
- a passive ultrasound sensor is an acoustic pressure sensor, and such a passive ultrasound sensor may be used to determine a location of an interventional medical device to which it is attached.
- Time-of-flight measurements provide the axial/radial distance of the passive ultrasound sensor from an imaging array of the ultrasound probe, while amplitude measurements and knowledge of the direct beam firing sequence provide the lateral / angular position of the passive ultrasound sensor.
- FIG. 1 illustrates a known system for tracking an interventional medical device using a passive ultrasound sensor.
- an ultrasound probe 102 emits an imaging beam 103 that sweeps across a passive ultrasound sensor 104 on a tip of an interventional medical device 105.
- An image of tissue 107 is fed back by the ultrasound probe 102.
- a location of the passive ultrasound sensor 104 on the tip of the interventional medical device 105 is provided as a tip location 108 upon determination by a signal processing algorithm.
- the tip location 108 is overlaid on the image of tissue 107 as an overlay image 109.
- the image of tissue 107, the tip location 108, and the overlay image 109 are all displayed on a display 100.
- the known technology for passive ultrasound sensors provides the location of the passive ultrasound sensor 104 but not the shape of the interventional medical device 105. In many clinical situations, such as for example cardiac and vascular interventions, it may be advantageous to determine the shape of the interventional medical device 105.
- a controller for determining a shape of an interventional medical device in an interventional medical procedure based on a location of the interventional medical device includes a memory that stores instructions and a processor that executes the instructions.
- the instructions When executed by the processor, the instructions cause a system that includes the controller to implement a process that includes obtaining the location of the interventional medical device and obtaining imagery of a volume that includes the interventional medical device.
- the process implemented when the processor executes the instructions also includes applying, based on the location of a point on the interventional medical device, image processing to the imagery to identify the interventional medical device including the shape of the interventional medical device.
- the process implemented when the processor executes the instructions further includes segmenting the interventional medical device to obtain a segmented representation of the interventional medical device.
- the segmented representation of the interventional medical device is overlaid on the imagery.
- a tangible non-transitory computer readable storage medium stores a computer program.
- the computer program When executed by a processor, the computer program causes a system that includes the tangible non-transitory computer readable storage medium to perform a process for determining a shape of an interventional medical device in an interventional medical procedure based on a location of the interventional medical device.
- the process performed when the processor executes the computer program from the tangible non-transitory computer readable storage medium includes obtaining the location of at least one point on the interventional medical device and obtaining imagery of a volume that includes the interventional medical device.
- the process performed when the computer program is executed by a processor also includes applying, based on the location of the interventional medical device, image processing to the imagery to identify the interventional medical device including the shape of the interventional medical device.
- the process performed when the computer program is executed by a processor further includes segmenting the interventional medical device to obtain a segmented representation of the interventional medical device.
- the segmented representation of the interventional medical device is overlaid on the imagery.
- a system for determining a shape of an interventional medical device in an interventional medical procedure based on a location of a passive ultrasound sensor located using an ultrasound imaging probe includes an ultrasound imaging probe, a passive ultrasound sensor, and a controller.
- the ultrasound imaging probe emits beams during the interventional medical procedure.
- the passive ultrasound sensor is fixed to the interventional medical device during the interventional medical procedure.
- the controller includes a memory that stores instructions and a processor that executes the instructions. When executed by the processor, the instructions cause the system to implement a process that includes obtaining the location of the passive ultrasound sensor based on emission of a beam from the ultrasound imaging probe and obtaining imagery of a volume that includes the interventional medical device and the passive ultrasound sensor.
- the process implemented when the processor executes the instructions also includes applying, based on the location of the passive ultrasound sensor, image processing to the imagery to identify the interventional medical device including the shape of the interventional medical device and location of the interventional medical device.
- the process implemented when the processor executes the instructions further includes segmenting the interventional medical device to obtain a segmented representation of the interventional medical device.
- the segmented representation of the interventional medical device is overlaid on the imagery together with the location of the passive ultrasound sensor.
- FIG. 1 illustrates a known system for tracking an interventional medical device using a passive ultrasound sensor.
- FIG. 2 illustrates a system for passive-ultrasound-sensor-based initialization for image-based device segmentation, in accordance with a representative embodiment.
- FIG. 3 illustrates a process for passive-ultrasound-sensor-based initialization for image-based device segmentation, in accordance with a representative embodiment.
- FIG. 4 illustrates a visualization progression for passive-ultrasound-sensor- based initialization for image-based device segmentation, in accordance with a representative embodiment, in accordance with a representative embodiment.
- FIG. 5 illustrates another process for passive-ultrasound-sensor-based initialization for image-based device segmentation, in accordance with a representative embodiment.
- FIG. 6 illustrates another process for passive-ultrasound-sensor-based initialization for image-based device segmentation, in accordance with a representative embodiment.
- FIG. 7 illustrates a set of 2-dimensional or X-plane images used for optimizing a view for passive-ultrasound-sensor-based initialization for image-based device
- FIG. 8 illustrates a visualization of a mesh of an interventional medical device overlaid in an ultrasound volume in passive-ultrasound-sensor-based initialization for image- based device segmentation, in accordance with a representative embodiment.
- the shape of the interventional medical device can be obtained readily using the location of the passive ultrasound sensor.
- the shape can be quantified using the system that identifies the location of the passive ultrasound sensor.
- the location of the passive ultrasound sensor can be used to initialize image processing algorithms that use, for example, spatial filtering or cross correlation with a known shape, to determine the shape of the device.
- a mesh of the interventional medical device can be generated and overlaid to enhance visualization.
- FIG. 2 illustrates a system for passive-ultrasound-sensor-based initialization for image-based device segmentation, in accordance with a representative embodiment.
- an ultrasound system 200 includes an ultrasound imaging probe 210, a controller 220, a console 290, an interventional medical device 201 and a passive ultrasound sensor SI.
- the ultrasound system 200 is representative of a system that includes a controller 220 used to obtain a location of an interventional medical device and imagery of a volume that include the interventional medical device.
- the ultrasound system 200 is representative of a system that includes a controller 220 for applying image processing to the imagery based on the location of the interventional medical device 201, and for segmenting the interventional medical device to obtain a segmented representation of the interventional medical device 201.
- other types of imaging systems may be used to perform these features described herein, and still fall within the spirit and scope of the present disclosure.
- a system as described herein does not have to be an ultrasound system.
- the controller 220 includes a memory 221 that stores instructions and a processor 222 that executes the instructions.
- a controller 220 as described herein may be distributed among multiple devices that each include a combination of memory and processor to perform one or more characteristic functions attributed to the controller 220 herein.
- the console 290 includes a memory 291 that stores instructions and a processor 292 that executes the instructions.
- the console 290 also includes a monitor 295 and a touch panel 296.
- the memory 291 and the processor 292 may be considered a sensor unit that determines the location of the passive ultrasound sensor SI and provides the location of the passive ultrasound sensor SI to the controller 220.
- another combination of a memory and a processor may be used to receive the voltage readings from the passive ultrasound sensor SI and the timing of beams from the ultrasound imaging probe 210 and determine the location of the passive ultrasound sensor S 1 to provide to the controller 220.
- a shape of an interventional medical device 201 can be quantified by locating the tip of the interventional medical device 201 (or another location on the interventional medical device 201) using a passive ultrasound sensor SI.
- the tip of the interventional medical device 201 is located using the passive ultrasound sensor SI in either a two-dimensional or a three-dimensional ultrasound space.
- image processing techniques such as spatial filtering, are applied to the ultrasound image to enhance structures that are potentially identifiable as the body of the interventional medical device 201.
- the interventional medical device 201 may be segmented and overlaid on the image based on the most prominent device-like structures that appear near the known location of the passive ultrasound sensor S 1.
- a processor 222 or a processor 292 for a controller is tangible and non- transitory.
- the term“non-transitory” is to be interpreted not as an eternal characteristic of a state, but as a characteristic of a state that will last for a period.
- the term “non-transitory” specifically disavows fleeting characteristics such as characteristics of a carrier wave or signal or other forms that exist only transitorily in any place at any time.
- a processor is an article of manufacture and/or a machine component.
- a processor 222 for a controller 220 is configured to execute software instructions to perform functions as described in the various embodiments herein.
- a processor 22 for a controller 220 may be a general-purpose processor or may be part of an application specific integrated circuit (ASIC).
- ASIC application specific integrated circuit
- a processor 222 for a controller may also be a microprocessor, a microcomputer, a processor chip, a controller, a microcontroller, a digital signal processor (DSP), a state machine, or a programmable logic device.
- a processor 222 for a controller may also be a logical circuit, including a programmable gate array (PGA) such as a field programmable gate array
- a processor 222 for a controller may be a central processing unit (CPU), a graphics processing unit (GPU), or both. Additionally, any processor described herein may include multiple processors, parallel processors, or both. Multiple processors may be included in, or coupled to, a single device or multiple devices.
- A“processor” as used herein encompasses an electronic component which is able to execute a program or machine executable instruction. References to the computing device comprising“a processor” should be interpreted as possibly containing more than one processor or processing core. The processor may for instance be a multi-core processor. A processor may also refer to a collection of processors within a single computer system or distributed amongst multiple computer systems. The term computing device should also be interpreted to possibly refer to a collection or network of computing devices each including a processor or processors. Many programs have instructions performed by multiple processors that may be within the same computing device or which may even be distributed across multiple computing devices.
- Memories such as the memory 221 or the memory 291 described herein are tangible storage mediums that can store data and executable instructions and are non- transitory during the time instructions are stored therein.
- the term“non- transitory” is to be interpreted not as an eternal characteristic of a state, but as a characteristic of a state that will last for a period.
- the term“non-transitory” specifically disavows fleeting characteristics such as characteristics of a carrier wave or signal or other forms that exist only transitorily in any place at any time.
- a memory described herein is an article of manufacture and/or machine component.
- Memories described herein are computer-readable mediums from which data and executable instructions can be read by a computer.
- Memories as described herein may be random access memory (RAM), read only memory (ROM), flash memory, electrically programmable read only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, a hard disk, a removable disk, tape, compact disk read only memory (CD-ROM), digital versatile disk (DVD), floppy disk, blu- ray disk, or any other form of storage medium known in the art. Memories may be volatile or non-volatile, secure and/or encrypted, unsecure and/or unencrypted.“Memory” is an example of a computer-readable storage medium. Computer memory is any memory which is directly accessible to a processor. Examples of computer memory include, but are not limited to RAM memory, registers, and register files. References to“computer memory” or“memory” should be interpreted as possibly being multiple memories. The memory may for instance be multiple memories within the same computer system. The memory may also be multiple memories distributed amongst multiple computer systems or computing devices.
- RAM random access memory
- ultrasound system 200 For convenience, reference to features of the ultrasound system 200 will be used throughout the present disclosure in and for other embodiments for the sake of consistency. However, as noted above the ultrasound system 200 is only an example of a system that may perform the functions and functionality described herein.
- FIG. 3 illustrates a process for passive-ultrasound-sensor-based initialization for image-based device segmentation, in accordance with a representative embodiment.
- the process of FIG. 3 begins by emitting a beam from an ultrasound imaging probe.
- the ultrasound imaging probe 210 in FIG. 2 may emit the beam as part of a sequence of beams at S310.
- the location of the passive ultrasound sensor S 1 may be determined in a predetermined three-dimensional coordinate system with an origin set for the ultrasound imagery.
- Registration may be performed to align the three-dimensional coordinate system for the ultrasound imagery with other three- dimensional coordinate systems, such as for the interventional medical device 201 including for imagery from the viewpoint of the interventional medical device when the interventional medical device 201 is an imaging device such as an endoscope.
- Registration may be performed by aligning landmarks in imagery in two different underlying coordinate systems in order to import one coordinate system to the other by image transformation.
- Registration may also be performed to align 2D or 3D ultrasound imagery to a 2D or projection imaging space, for example by identifying common landmarks in each imaging space or identifying the pose of the ultrasound transducer in another imaging space.
- the process of FIG. 3 next includes obtaining imagery of the ultrasound volume that includes the interventional medical device 201 and the passive ultrasound sensor S 1.
- the imagery may be ultrasound imagery resulting from the ultrasound beams emitted by the ultrasound imaging probe 210 during the interventional medical procedure.
- the process of FIG. 3 includes applying image processing to the imagery based on the location of the passive ultrasound sensor SI to identify the
- the image processing at S340 may be applied based on a location of a point on the interventional medical device 201 or based on a location of at least one point on the interventional medical device 201.
- the constraints used at S340 may include one or more of characteristics of pixel intensity, of relative pixel location, of predetermined shape(s), and/or of dimensional characteristic(s). As an example, pixel intensity of pixels corresponding to locations of the interventional medical device 201 may be higher than for pixel intensity of pixels
- the search may be limited to pixels within a predetermined range measured in distance, pixels or something else, from the pixels corresponding to the location of the passive ultrasound sensor SI.
- the image processing may involve filtering the pixels to eliminate pixels based on predetermined characteristics, so that the remaining elements of the original image better show the interventional medical device 201. That is, elements of the device shape (i.e., of the interventional medical device 201) remain after the filtering.
- the image processing therefore intentionally discards representations of shapes such as anatomical features, but not necessarily based on the shape insofar as the discarding may be based on the pixel intensity or pixel location and so on.
- the characteristics used for the image filtering may include a predetermined shape selected from, for example, a library of predetermined shapes corresponding to different interventional medical devices with different shapes.
- the characteristics used for the image filtering may include a dimensional characteristic of a predetermined shape, such as a minimum or maximum length, width, height, diameter, radius, cross-sectional area, curvature, and so on.
- a wire as the interventional medical device 201 may have a very small cross-section, and the search for the wire may look for an end with an area or diameter less than a threshold corresponding to the very small cross-section.
- the process of FIG. 3 next includes reconstructing and segmenting a reconstruction of a shape of an interventional medical device 201 or a predetermined shape of the interventional medical derive to obtain a segmented representation of the interventional medical device 201.
- the predetermined device shape may be used as a predetermined constraint so that the predetermined device shape is used as a candidate for the shape of the interventional medical device 201.
- the elements of the device shape i.e., of the
- segmentation is a representation of the surface of structures such as anatomical features and/or interventional medical devices such as the interventional medical device 20 land consists for example of a set of points in three-dimensional (3-D) coordinates on the surfaces of the structure, and triangular plane segments defined by connecting neighboring groups of three points, such that the entire structure is covered by a mesh of non intersecting triangular planes.
- a three-dimensional model of the structure is obtained by segmenting. Segmenting as described herein for S350 and similar operations in other embodiments may involve performing segmentation on an interventional medical device 201, anatomy structures, and/or other structures present in a three-dimensional ultrasound volume.
- the process of FIG. 3 includes overlaying the segmented representation of the interventional medical device 201 on imagery together with the location of the passive ultrasound sensor S 1.
- FIG. 3 illustrates a visualization progression for passive-ultrasound-sensor- based initialization for image-based device segmentation, in accordance with a representative embodiment, in accordance with a representative embodiment.
- FIG. 4 the progression of FIG. 4 shows an interventional medical device 201 tip location.
- FIG. 4 shows the effects or results of image processing.
- the effect is much reduced detail of the overall imagery, but much greater detail of the interventional medical device 201.
- the progression of FIG. 4 shows the interventional medical device 201 overlaid on the original image. That is, the interventional medical device 201 identified and highlighted in visualization B may be overlaid in visualization C.
- the visualization of the interventional medical device 201 from visualization B may be augmented such as by filling in missing pixels in order to entirely populate the full expected shape according to a model retrieved from a library of models. Therefore, the image processing used to reduce the detail of the overall imagery for visualization B may also include augmenting to enhance the detail of the interventional medical device.
- the shape of the interventional medical device 201 is obtained from image processing that is initialized based on the location of the passive ultrasound sensor SI.
- the tip or other location of the interventional medical device 201 is located in the 3D volume using tracking of the passive ultrasound sensor S 1.
- Image processing techniques are applied to, in the embodiment of FIG. 4, make tube-like structures appear more clearly.
- the highest intensity tube-like structure near the location of the passive ultrasound sensor S 1 may be assumed to be the body of the interventional medical device 201, which is then overlaid on the original image.
- a process that involves the progression of FIG. 4 may be used for an image processing algorithm applied in a three-dimensional volume to locate and enhance the visualization of the interventional medical device 201.
- the two-dimensional imaging plane from an ultrasound imaging probe 210 may be used to sweep the three- dimensional space that includes the interventional medical device 201.
- the resultant imagery may be analyzed subsequently to obtain a three-dimensional volumetric model of the space and/or the interventional medical device 201 in the space.
- the three-dimensional volumetric model of the space and/or the interventional medical device 201 may then be segmented.
- an initial two-dimensional plane may be set so that it includes a passive ultrasound sensor, and then the two-dimensional plane is incrementally rotated to roughly pivot around the passive ultrasound sensor.
- Two-dimensional image processing such as filtering and clustering may be performed on each frame in the rotational sweep to identify portions of the interventional medical device 201.
- the view of the interventional medical device 201 in each plane may be analyzed to determine which plane from the sweep provides the best display of the device, such as the longest section of the device, or shows the most distal portion near the passive ultrasound sensor. The segmentation of the interventional medical device is then displayed.
- FIG. 5 illustrates another process for passive-ultrasound-sensor-based initialization for image-based device segmentation, in accordance with a representative embodiment.
- the process of FIG. 5 starts with obtaining a location of a passive ultrasound sensor S 1 in an ultrasound volume.
- a search for the interventional medical device 201 in imagery may start at the location of the passive ultrasound sensor S 1.
- a search for the constraints may be a search for regions in the imagery that satisfy predetermined constraints.
- a region that satisfied predetermined constraints may be within a specified radius or distance from the passive ultrasound sensor and may include characteristics such one or more pixels with a pixel intensity higher than a predetermined threshold.
- the passive ultrasound sensor S 1 may include characteristics such one or more pixels with a pixel intensity higher than a predetermined threshold.
- predetermined constraints may be for a particular shape that is part of the shape of the interventional medical device 201, such as a comer or set of comers, one or more angles in the profile of the interventional medical device 201 that would appear in ultrasound imagery, and other types of constraints.
- the constraints may be predetermined constraints, and may vary based on the type of interventional medical device 201, the type of surgery, and characteristics of the subject of the interventional medical procedure including anatomical characteristics
- a process performed by a controller 220 or the ultrasound system 200 that includes the controller 220 may include filtering the imagery from an ultrasound imaging system to eliminate representations of a subject of the interventional medical procedure.
- filtering may be performed to eliminate representations of tissue, bone and other anatomical features of the subject of the interventional medical procedure, since the object sought in the image processing is the interventional medical device 201.
- the process of FIG. 5 includes searching for regions of maximum intensity surrounding the location of the passive ultrasound sensor and identifying points with the maximum intensity.
- the maximum intensity may be a pixel intensity above a
- predetermined threshold or a relative intensity greater than a predetermined threshold of intensities of nearby pixels.
- the process of FIG. 5 next includes connecting neighboring points.
- the neighboring points may be for pixels with intensities over a threshold and within a predetermined distance from the location of the passive ultrasound sensor SI.
- the process of FIG. 5 ends with overlaying the shape of the interventional medical device 201 on the ultrasound imagery.
- the overlaid interventional medical device 201 is a representation of the interventional medical device 201, and may be superimposed on the ultrasound imagery, and highlighted such as by a highlighted outline.
- the process of FIG. 5 and other embodiments herein can be used for many types of interventional procedures.
- the features herein can be used for monitoring shape of an interventional medical device 201 such as a wire during stenosis or occlusion crossings to detect buckling of the interventional medical device 201.
- the features herein can be used to detect progression of an interventional medical device 201 such as a wire with respect to the vessel to check if the interventional medical device 201 has exited the vessel wall.
- FIG. 6 illustrates another process for passive-ultrasound-sensor-based initialization for image-based device segmentation, in accordance with a representative embodiment.
- the process of FIG. 6 starts at S610 with obtaining a location of a passive ultrasound sensor in an ultrasound volume.
- the process of FIG. 6 includes setting filter constraints and identifying an interventional medical device 201 model.
- the filter constraints may be predetermined constraints or may be constraints that are dynamically set for each different interventional medical procedure.
- the filter constraints may be applied in order to identify regions that satisfy predetermined constraints, so that the image processing to be performed is only performed at the identified regions or at least starts at the identified regions.
- the process of FIG. 6 next includes fixing a tip of an identified interventional medical device 201 model to a location of the passive ultrasound sensor in the ultrasound volume.
- the process of FIG. 6 includes determining an interventional medical device 201 optimal angle relative to a location of the passive ultrasound sensor SI in the ultrasound volume. The angle may be used to pose the interventional medical device 201 starting from the location ultrasound sensor SI and aligned in the direction of the angle from the passive ultrasound sensor S 1.
- the process of FIG. 6 concludes with overlaying the shape of the interventional medical device 201 on the ultrasound surgery.
- a mesh resulting from segmenting a shape of a predetermined structure as the interventional medical device 201 may be aligned based on the process at S640, and then placed so that the representation of the comer or other extremity of the interventional medical device 201 overlaps the representation of the passive ultrasound sensor SI in or on the passive ultrasound imagery.
- FIG. 7 illustrates a set of 2-dimensional or X-plane images used for optimizing a view for passive-ultrasound-sensor-based initialization for image-based device
- FIG. 7 five different 2-dimensional or X-plane images are labelled A, B, C, D and E, and each shows a view in which the interventional medical device 201 varies due to the differing viewpoints.
- the most complete view is that of 2-dimensional or X-plane image "C", which corresponds to the view with the greatest amount of detail of the interventional medical device 201.
- imagery from two-dimensional or X-plane views can be viewed to see which provides the best detail of the interventional medical device 201.
- the imagery from FIG. 7 may be obtained before or after segmentation of the interventional medical device 201, and then used for example to identify or confirm the best placement, orientation and pose of the interventional medical device 201 on the underlying three- dimensional ultrasound volume.
- an initial two-dimensional plane may be set so that it includes a passive ultrasound sensor, and then the two-dimensional plane is incrementally rotated to roughly pivot around the passive ultrasound sensor.
- location is determined by a mechanism other than passive ultrasound sensors, the location of a point on an
- interventional medical device 201 may still be used as the starting point of the sweep.
- Two- dimensional image processing such as filtering and clustering may be performed on each frame in the rotational sweep to identify portions of the interventional medical device 201.
- the view of the interventional medical device 201 in each plane can be analyzed to determine which plane from the sweep provides the best display of the device, such as the longest section of the device, or shows the most distal portion near the passive ultrasound sensor. The segmentation of the interventional medical device is then displayed.
- FIG. 8 illustrates a visualization of a mesh of an interventional medical device 201 overlaid in an ultrasound volume in passive-ultrasound-sensor-based initialization for image-based device segmentation, in accordance with a representative embodiment.
- the interventional medical device 201 may be a SHD device which is segmented into a mesh and overlaid in a 3 -dimensional ultrasound volume.
- the shape of the interventional medical device 201 is known, and may be rigid, and can be fit to the ultrasound imagery of the 3-dimensional ultrasound volume.
- the location of the passive ultrasound sensor is used to initialize the search volume so as to identify the proper orientation, pose and overall placement of the interventional medical device 201 in FIG. 8.
- the interventional medical device 201 may be rigid, such as in the case of a transeptal puncture needle or mitral repair device.
- Image processing techniques can be used to search for the specific shape of the device in the region near the passive ultrasound sensor.
- the mesh of the known device can then be overlaid on the two-dimensional or three-dimensional ultrasound image as shown in FIG. 8.
- the overlaid mesh of the known device may be superimposed on the ultrasound imagery, and highlighted such as by color, brightness, or other visual characteristics to make the superimposed mesh distinctive in the combined image.
- passive-ultrasound-sensor-based initialization for image-based device segmentation enables identification of an interventional medical device 201 in ultrasound imagery, and placement of a model of the interventional medical device 201 in or on the ultrasound imagery.
- the passive-ultrasound-sensor based initialization for image- based segmentation can be used in myriad ways such as to ensure that an interventional medical device 201 is being deployed in a correct pose and along a correct path, such as to detect irregular an path during an interventional medical procedure (e.g., septal puncture or chronic total occlusion crossing), and/or to quantify an interventional medical device 201 in three-dimensions during an interventional medical procedure.
- passive-ultrasound-sensor-based initialization for image-based device segmentation has been described with reference to particular means, materials and embodiments, passive-ultrasound-sensor-based initialization for image-based device segmentation is not intended to be limited to the particulars disclosed; rather passive- ultrasound-sensor-based initialization for image-based device segmentation extends to all functionally equivalent structures, methods, and uses such as are within the scope of the appended claims.
- examples above describe use of features herein for structural heart repair or peripheral vascular intervention.
- Other practical applications of the features herein may include detecting bending of an interventional medical device 201 such as a needle during a deep tissue biopsy.
- Other practical applications of the features herein may include providing a reliable in-body two-dimensional projection fiducial for registration between ultrasound and/to X-ray.
- Example 1 A controller (220) for determining a shape of an interventional medical device (201) in an interventional medical procedure based on a location of the interventional medical device (201), comprising:
- a memory (221) that stores instructions
- a processor that executes the instructions, wherein, when executed by the processor, the instructions cause a system that includes the controller (220) to implement a process that includes:
- Example 2 The controller (220) of Example 1, wherein the process implemented by the system further comprises:
- segmented representation of the interventional medical device is overlaid on the imagery together with the location of the passive ultrasound sensor.
- Example 3 The controller (220) of Example 2, wherein the process implemented by the system further comprises:
- predetermined constraints include characteristics of pixel intensity, and pixel location relative to the location of the passive ultrasound sensor.
- Example 4 The controller (220) of Example 2, wherein the process implemented by the system further comprises:
- the predetermined constraints include at least one predetermined shape used as a candidate for the shape of the interventional medical device.
- Example 5 The controller (220) of Example 2, wherein the process implemented by the system further comprises:
- Example 6 The controller (220) of Example 1, wherein the process implemented by the system further comprises:
- Example 7 The controller (220) of Example 6, wherein the segmenting is performed on the reconstruction of the shape of the interventional medical device, and the segmented representation of the interventional medical device comprises a segmented representation of the reconstruction of the shape of the interventional medical device.
- Example 8 The controller (220) of Example 1, wherein the segmenting is performed on a predetermined shape based on the image processing, and the segmented representation of the interventional medical device comprises a segmented representation of the predetermined shape of the interventional medical device so that the segmented representation of the predetermined shape is overlaid on the imagery.
- Example 9 A tangible non-transitory computer readable storage medium that stores a computer program, the computer program, when executed by a processor, causing a system that includes the tangible non-transitory computer readable storage medium to perform a process for determining a shape of an interventional medical device in an interventional medical procedure based on a location of the interventional medical device, the process performed when the processor (222) executes the computer program from the tangible non- transitory computer readable storage medium comprising:
- segmenting the interventional medical device to obtain a segmented representation of the interventional medical device, wherein the segmented representation of the interventional medical device is overlaid (S360) on the imagery.
- Example 10 The tangible non-transitory computer readable storage medium of Example 9, wherein the process implemented by the system further comprises:
- Example 11 The tangible non-transitory computer readable storage medium of Example 10, wherein the process implemented by the system further comprises:
- predetermined constraints include characteristics of pixel intensity, and pixel location relative to the location of the passive ultrasound sensor
- segmented representation of the interventional medical device is overlaid on the imagery together with the location of the passive ultrasound sensor.
- Example 12 The tangible non-transitory computer readable storage medium of Example 10, wherein the process implemented by the system further comprises:
- Example 13 The tangible non-transitory computer readable storage medium of Example 10, wherein the process implemented by the system further comprises:
- predetermined constraints include at least one dimensional characteristic of a predetermined shape.
- Example 14 The tangible non-transitory computer readable storage medium of Example 9, wherein the process implemented by the system further comprises:
- Example 15 The tangible non-transitory computer readable storage medium of Example 14,
- segmenting is performed on the reconstruction of the shape of the shape of the interventional medical device
- segmented representation of the interventional medical device comprises a segmented representation of the reconstruction of the shape of the interventional medical device.
- Example 16 The tangible non-transitory computer readable storage medium of Example 9, wherein the segmenting is performed on a predetermined shape based on the image processing, and the segmented representation of the interventional medical device comprises a segmented representation of the predetermined shape of the interventional medical device so that the segmented representation of the predetermined shape is overlaid on the imagery.
- Example 17 A system (200) for determining a shape of an interventional medical device (201) in an interventional medical procedure based on a location of a passive ultrasound sensor (SI) located using an ultrasound imaging probe (210), comprising: an ultrasound imaging probe (210) that emits beams during the interventional medical procedure;
- SI passive ultrasound sensor
- SI passive ultrasound sensor
- a controller comprising a memory (221) that stores instructions and a processor (222) that executes the instructions, wherein, when executed by the processor (222), the instructions cause the system to implement a process that includes:
- segmenting the interventional medical device to obtain a segmented representation of the interventional medical device, wherein the segmented representation of the interventional medical device is overlaid (S360) on the imagery together with the location of the passive ultrasound sensor.
- Example 18 The system of Example 17, further comprising:
- a sensor unit (291/292) that determines the location of the passive ultrasound sensor and provides the location of the passive ultrasound sensor to the controller (220) ; and a display (295) that displays the segmented representation of the interventional medical device based on the segmenting by the controller (220) and that displays the location of the passive ultrasound sensor determined by the sensor unit.
- Example 19 The system of Example 17, wherein the process implemented by the system further comprises:
- Example 20 The system of Example 17, wherein the process implemented by the system further comprises: filtering (S340) the imagery to eliminate representations of a subject of the interventional medical procedure in which the interventional medical device is inserted, wherein elements of the shape remain in the imagery after the filtering, and
- invention merely for convenience and without intending to voluntarily limit the scope of this application to any particular invention or inventive concept.
- inventive concept merely for convenience and without intending to voluntarily limit the scope of this application to any particular invention or inventive concept.
- specific embodiments have been illustrated and described herein, it should be appreciated that any subsequent arrangement designed to achieve the same or similar purpose may be substituted for the specific embodiments shown.
- This disclosure is intended to cover any and all subsequent adaptations or variations of various embodiments. Combinations of the above embodiments, and other embodiments not specifically described herein, will be apparent to those of skill in the art upon reviewing the description.
Landscapes
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Surgery (AREA)
- Engineering & Computer Science (AREA)
- Medical Informatics (AREA)
- General Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Heart & Thoracic Surgery (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Molecular Biology (AREA)
- Animal Behavior & Ethology (AREA)
- Veterinary Medicine (AREA)
- Public Health (AREA)
- Physics & Mathematics (AREA)
- Biophysics (AREA)
- Pathology (AREA)
- Radiology & Medical Imaging (AREA)
- Robotics (AREA)
- Ultra Sonic Daignosis Equipment (AREA)
Abstract
Description
Claims
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201962855013P | 2019-05-31 | 2019-05-31 | |
EP19189347.8A EP3771435A1 (en) | 2019-07-31 | 2019-07-31 | Passive-ultrasound-sensor-based initialization for image-based device segmentation |
PCT/EP2020/063878 WO2020239514A1 (en) | 2019-05-31 | 2020-05-19 | Passive-ultrsound-sensor-based initialization for image-based device segmentantion |
Publications (1)
Publication Number | Publication Date |
---|---|
EP3975863A1 true EP3975863A1 (en) | 2022-04-06 |
Family
ID=70681869
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP20725568.8A Pending EP3975863A1 (en) | 2019-05-31 | 2020-05-19 | Passive-ultrsound-sensor-based initialization for image-based device segmentantion |
Country Status (5)
Country | Link |
---|---|
US (1) | US20220218302A1 (en) |
EP (1) | EP3975863A1 (en) |
JP (1) | JP2022534916A (en) |
CN (1) | CN113950294A (en) |
WO (1) | WO2020239514A1 (en) |
Family Cites Families (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10398393B2 (en) * | 2007-10-02 | 2019-09-03 | Stryker European Holdings I, Llc | Dynamic reference method and system for interventional procedures |
CN103313661B (en) * | 2011-01-13 | 2016-05-18 | 皇家飞利浦电子股份有限公司 | In three-D ultrasonic, conduit is visual |
US8795178B2 (en) * | 2012-12-31 | 2014-08-05 | General Electric Company | Ultrasound imaging system and method for identifying data from a shadow region |
EP2996587B1 (en) * | 2013-03-28 | 2022-08-24 | Koninklijke Philips N.V. | Instrument localization in guided high dose rate brachytherapy |
WO2015092582A1 (en) * | 2013-12-20 | 2015-06-25 | Koninklijke Philips N.V. | Automatic ultrasound beam steering and needle artifact suppression |
WO2015092628A1 (en) * | 2013-12-20 | 2015-06-25 | Koninklijke Philips N.V. | Ultrasound imaging systems and methods for tracking locations of an invasive medical device |
WO2016009350A1 (en) * | 2014-07-16 | 2016-01-21 | Koninklijke Philips N.V. | Intelligent real-time tool and anatomy visualization in 3d imaging workflows for interventional procedures |
US11259774B2 (en) * | 2014-12-01 | 2022-03-01 | Koninklijke Philips N.V. | Registration of optical shape sensing tool |
EP3229698B1 (en) * | 2014-12-09 | 2023-07-05 | Koninklijke Philips N.V. | Single-modality-based visual distinguishing of medical intervention device from tissue |
EP3600067A1 (en) * | 2017-03-30 | 2020-02-05 | Koninklijke Philips N.V. | Oss foreshortening detection systems |
WO2018234230A1 (en) * | 2017-06-19 | 2018-12-27 | Koninklijke Philips N.V. | Interleaved imaging and tracking sequences for ultrasound-based instrument tracking |
-
2020
- 2020-05-19 JP JP2021570310A patent/JP2022534916A/en active Pending
- 2020-05-19 US US17/614,697 patent/US20220218302A1/en active Pending
- 2020-05-19 CN CN202080040310.XA patent/CN113950294A/en active Pending
- 2020-05-19 EP EP20725568.8A patent/EP3975863A1/en active Pending
- 2020-05-19 WO PCT/EP2020/063878 patent/WO2020239514A1/en active Search and Examination
Also Published As
Publication number | Publication date |
---|---|
JP2022534916A (en) | 2022-08-04 |
WO2020239514A1 (en) | 2020-12-03 |
CN113950294A (en) | 2022-01-18 |
US20220218302A1 (en) | 2022-07-14 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
JP6395995B2 (en) | Medical video processing method and apparatus | |
ES2718543T3 (en) | System and procedure for navigation based on merged images with late marker placement | |
US8867808B2 (en) | Information processing apparatus, information processing method, program, and storage medium | |
JP4758355B2 (en) | System for guiding medical equipment into a patient's body | |
US8861822B2 (en) | Systems and methods for enhanced imaging of objects within an image | |
JP4750429B2 (en) | Image display device | |
EP2807978A1 (en) | Method and system for 3D acquisition of ultrasound images | |
US9936896B2 (en) | Active system and method for imaging with an intra-patient probe | |
US20160317118A1 (en) | Automatic ultrasound beam steering and needle artifact suppression | |
EP1727471A1 (en) | System for guiding a medical instrument in a patient body | |
CN107106128B (en) | Ultrasound imaging apparatus and method for segmenting an anatomical target | |
JP2007296362A (en) | Enhanced function ultrasound image display | |
KR20140032810A (en) | Method and appartus of maching medical images | |
JP6393698B2 (en) | Data display and processing algorithms for 3D imaging systems | |
US11183295B2 (en) | Medical image processing apparatus and medical image processing method which are for medical navigation device | |
US20110295120A1 (en) | 3d ultrasound apparatus and method for operating the same | |
EP3832599A1 (en) | Device for providing 3d image registration and method therefor | |
US9990725B2 (en) | Medical image processing apparatus and medical image registration method using virtual reference point for registering images | |
CN106456084A (en) | Ultrasound imaging apparatus | |
US7376254B2 (en) | Method for surface-contouring of a three-dimensional image | |
US20120078101A1 (en) | Ultrasound system for displaying slice of object and method thereof | |
US20130170724A1 (en) | Method of generating elasticity image and elasticity image generating apparatus | |
US20220218302A1 (en) | Passive-ultrasound-sensor-based initialization for image-based device segmentation | |
EP3771435A1 (en) | Passive-ultrasound-sensor-based initialization for image-based device segmentation | |
US20220022967A1 (en) | Image-based device tracking |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: UNKNOWN |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE |
|
PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE |
|
17P | Request for examination filed |
Effective date: 20220103 |
|
AK | Designated contracting states |
Kind code of ref document: A1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
DAV | Request for validation of the european patent (deleted) | ||
DAX | Request for extension of the european patent (deleted) | ||
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: EXAMINATION IS IN PROGRESS |
|
17Q | First examination report despatched |
Effective date: 20231005 |