WO2013038284A1

WO2013038284A1 - Generating a three-dimensional model from an object of interest

Info

Publication number: WO2013038284A1
Application number: PCT/IB2012/054368
Authority: WO
Inventors: Klaus Erhard; Michael Grass; Tim Nielsen; Thomas Buelow
Original assignee: Koninklijke Philips Electronics N.V.; Philips Intellectual Property & Standards Gmbh
Priority date: 2011-09-13
Filing date: 2012-08-27
Publication date: 2013-03-21

Abstract

For generating a three-dimensional model (60) from an object of interest optical tomography raw data acquired from the object and X-ray image data acquired from the object (50) is used to select a reference model of the object (18) from a plurality of reference models based on the X-ray image data and the optical tomography raw data. The three- dimensional model is generated from the reference model by adjusting the reference model to the X-ray image data.

Description

GENERATING A THREE-DIMENSIONAL MODEL FROM AN OBJECT OF INTEREST

FIELD OF THE INVENTION

The invention relates to the field of imaging. In particular, the invention relates to a device, a method, a computer program, a computer readable medium and a system for generating a three-dimensional model from an object of interest.

BACKGROUND OF THE INVENTION

Usually, for breast cancer screening an X-ray mammography image of a breast is evaluated for suspicious areas that may indicate a tumor in the tissue of the breast.

For acquiring the X-ray mammography image, the breast may be compressed and flattened between two compression paddles and an X-ray detector arrangement with a digital detector may radiograph the breast, may generate X-ray image data and the X-ray image data may be displayed on the screen of a workstation connected to the X-ray detector. A physician then may investigate and evaluate the displayed X-ray image.

For further information about the tissue of the breast, additionally a tomography of the breast may be acquired. In this context it is known to use optical tomography, which causes no exposure to possible dangerous radiation. In optical tomography, visual or infrared light is radiated onto the breast and the scattered or passed light is detected from different directions with an optical detector. As in X-ray tomography, from the raw data of the optical detector a three-dimensional representation of the object of investigation is generated.

Due to strong scattering of visual or infrared light inside the breast, the optical tomography only may provide a very diffuse image.

SUMMARY OF THE INVENTION

It may be an object of the invention to generate a three-dimensional representation of an interior of an object of interest, wherein the representation has a high resolution and the object is only exposed to a low amount of dangerous radiation.

This object is achieved by the subject-matter of the independent claims. Further exemplary embodiments are evident from the dependent claims and the following description.

A first aspect of the invention relates to a device, for example a workstation, for generating a three-dimensional model from an object of interest.

According to an embodiment of the invention, the device comprises an interface for receiving optical tomography raw data acquired from the object and for receiving X-ray image data acquired from the object, a selector for selecting a reference model of the object from a plurality of reference models based on the X-ray image data and the optical tomography raw data and a generator for generating the three-dimensional model from the reference model by adjusting the reference model to the X-ray image data.

A first aspect of the invention relates to a method for generating a three- dimensional model from an object of interest. The object of interest may by a human breast.

The above mentioned device may be adapted to execute the method.

According to an embodiment of the invention, the method comprises the steps of: receiving optical tomography raw data acquired from the object; receiving X-ray image data acquired from the object; selecting a reference model of the object from a plurality of reference models based on the X-ray image data and/or the optical tomography raw data; and generating the three-dimensional model from the reference model by adjusting the reference model to the X-ray image data.

The X-ray image data may be a mammography of a breast.

The three-dimensional model may be adjusted by parameterizing the reference model with the aid of the X-ray image data. By individual parameterization of a reference model of a breast, a realistic three-dimensional distribution (a three-dimensional model) of the glandular tissue of the breast may be estimated from one or more two dimensional mammograms (i.e. the X-ray image data). This estimated tissue distribution can be used as input to an optical reconstruction algorithm.

A further aspect of the invention relates to a system for generating a three- dimensional model, which comprises an X-ray detector arrangement adapted for acquiring X- ray image data, an optical detector arrangement adapted for acquiring optical tomography raw data; wherein the system is adapted to carry out the method as described in the above and in the following.

The system may be or may be part of a breast cancer screening device, which combines X-ray mammography and optical tomography in a single system.

The method and the device may be used in the context of breast cancer screening as part of the system, for example as reconstruction unit of a combined

Mammography-DOT diffuse optical tomography system.

Alternatively, the method and the device may be used as part of the mammography analysis chain in a Viewing/CAD workstation in order to improve CAD results.

It has to be understood that features of the method as described in the above and in the following may be features of the device and the system as described in the above and in the following.

Further aspects of the invention are a computer program for generating a three- dimensional model of the object , which, when being executed by a processor, is adapted to carry out the method steps and a computer-readable medium, in which such a computer program is stored, for example a software for the simulation of mammograms from the breast model and/or algorithms for reconstruction of optical tomography data that can benefit from an estimate of the three-dimensional breast tissue distribution.

The computer programmay be executed on a processor of the above mentioned device or system and/or may be stored and/or loaded in a memory of the device or system.

It has to be noted that a computer-readable medium may be a floppy disk, a hard disk, an USB (Universal Serial Bus) storage device, a RAM (Random Access Memory), a ROM (Read Only Memory) and an EPROM (Erasable Programmable Read Only Memory). A computer readable medium may also be a data communication network, e.g. the Internet, which allows downloading a program code.

These and other aspects of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS

Below, embodiments of the present invention are described in more detail with reference to the attached drawings.

Fig. 1 shows a schematic view of a system according to an embodiment of the invention.

Fig.2 shows a diagram showing data types and a data flow according to an embodiment of the invention.

Fig. 3 shows a flow diagram for a method according to an embodiment of the invention.

In principle, identical parts are provided with the same reference symbols in the figures.

DETAILED DESCRIPTION OF EMBODIMENTS

Fig. 1 shows a system 10 comprising a workstation 12 and a detector arrangement 14. The detector arrangement 14 comprises two compression paddles 16 which may receive and compress a human breast 18 to be examined by the system 10. In general, the object 18 may be any kind of object that may be examined with X-ray and optical tomography imaging, for example other kinds of animal and human parts.

The detector arrangement 14 comprises an optical detector arrangement 20 and an X-ray detector arrangement 22 that are situated around the compression paddles such that X-ray image data and optical image data may be generated from the object 18.

The optical detector arrangement 20 comprises a light source 24 and an optical detector 26. The light source 24 may irradiate visible or infrared radiation. The optical detector 26 may detect the radiation from different directions and may generate tomography raw data of the breast 18.

The X-ray detector arrangement 22 comprises an X-ray source 28 for generation X-rays and an X-ray detector 30 for detecting the X-rays and generating X-ray image data of the breast 18. With such a detector arrangement 22, the system 10 may be a full field digital mammography system 10.

The workstation 12 or device 12 may control the detector arrangement 14.

According to an embodiment of the invention, the device 12 is adapted to control the X-ray detector arrangement 22.

According to an embodiment of the invention, the device 12 is adapted to control the optical detector arrangement 20.

However, it is possible that the workstation 12 only receives the generated data from the detector arrangement 14.

Furthermore, the workstation 12 may comprise a display for displaying the generated image data.

The device 12 and the system 10 are adapted for generating a three- dimensional model from the breast 18 (the object of interest) as will be explained in detail in the following.

The device 12 may comprise a processor 32 that executes a computer program that performs the steps of such a method. The processor 32 may comprise an interface 32 for receiving data, a selector 32 for selecting data from a database 34 and a generator 32for generating image data.

According to an embodiment of the invention, the device 12 comprises an interface 32 for receiving optical tomography raw data 52 acquired from the object 18 and for receiving X-ray image data 50 acquired from the object 18, a selector 32 for selecting a reference model 58 of the object 18 from a plurality of reference models based on the X-ray image data 50 and the optical tomography raw data 52 and a generator 32 for generating the three-dimensional model 60 from the reference model 58 by adjusting the reference model 58 to the X-ray image data 50.

Fig.2 shows a diagram showing data types and a data flow in the system 10. The data types/data structures described in the following may be stored in a memory of the device 12 and may be processed by the processor 32 of the device 12.

According to an embodiment of the invention, the system 10 comprises an X- ray detector arrangement 22 adapted for acquiring X-ray image data 50.

According to an embodiment of the invention, the system 10 comprises an optical detector arrangement 20 adapted for acquiring optical tomography raw data 52.

The X-ray image data 50 and/or the optical tomography raw data 52 from the detector arrangement 14 may be received in the device 12 and may be used for generating a coarse model 54 of the object 18. The X-ray image data 50 may represent a two-dimensional view of the object 18. For example, the X-ray image data 50 may comprise a mammography. The optical tomography data 52 may represent a plurality of two-dimensional views of the object 18 and may comprise information about the three-dimensional composition of the object 18.

The coarse model 54 may comprise a surface model of the object 18, i.e. a model of the outer contour of the object 18. For determining the coarse model 54 and in particular the surface model, an elastic deformation model 56 may be used. The elastic deformation model may be stored in a database 34 in the device 12. The coarse model 54 may be a three-dimensional model of the object 18.

In general, a model may be any type of data that comprises information of the (optional three-dimensional) structural composition of the object 18. With the aid of the coarse model 54 and further parameters, a reference model 58 may be determined from the database 34. The database 34 may store a plurality of predefined reference models 58, one of which may be selected. A reference model 58 may be a three-dimensional model of the object 18.

According to an embodiment of the invention, the device 12 comprises a database 34 for storing a plurality of reference models 58.

The data of the coarse model 54 is then used to adjust the reference model 58 in such a way that a three-dimensional model 60 is generated from the reference model 58 that may be a good approximation of the real composition of the object 18. This may be done by projecting the reference model 58 to a two-dimensional projection 59, comparing the two- dimensional projection with the X-ray image data 50 and adjusting the three-dimensional model 60 based on the result of the comparison.

In particular, the three-dimensional model 60 may be a parametric in-silico model 60 of a breast 18 and may represent realistically shaped tissue components, which may comprise skin, glandular tissue, duct system, adipose tissue and/or ligaments.

The three-dimensional model 60 may comprise X-ray attenuation coefficients, the coarse model 54 and/or the elastic deformation model 56.

The three dimensional model 60 may be used in the end, to support the generation of three-dimensional tomography data 62 from the optical tomography raw data 52. The three-dimensional tomography data 62 may comprise a three-dimensional

representation of the object 18. From the three-dimensional tomography data 62 three- dimensional views of the object 18 and two-dimensional slices through the object 18 may be derived.

The three-dimensional tomography data 62 may be used for examination of the object 18. For example, a physician may view slices through the breast 18 displayed on the device 12.

According to an embodiment of the invention, the three-dimensional model 60 and/or the three-dimensional tomography data 62 may be displayed by the device 12, for example by the display of the device 12.

It may be possible that a computer program, stored in a memory of the workstation 12, is running on the workstation 12 which processes these data and which executes the method as described in the above and in the following.

Fig. 3 shows a flow diagram for a method for generating a three-dimensional model 60 from an object 18.

In step S10, the X-ray detector arrangement 22 generates X-ray image data 50, which is received in the device 12.

According to an embodiment of the invention, the method comprises the step of receiving X-ray image data 50 acquired from the object 50.

In step S12, the optical detector arrangement 20 generates optical tomography data 52, which is received in the device 12.

According to an embodiment of the invention, the method comprises the step of receiving optical tomography raw data 52 acquired from the object 18.

In step S14, the device 12 generates a coarse model 54 of the object 18.

The coarse model 54 may comprise a surface model of the object 18.

From the X-ray image data 50, a two-dimensional contour line of the object 18 may be determined, for example with a Canny edge filter or with a Gaussian mixture model. Then a distance map may be computed with respect to the detemiined breast contour. The distance map may indicate the distance of a point of the X-ray image data 50 to the contour line of the object 18.

The surface model, which may be a three-dimensional breast contour model, may be finally generated from the two-dimensional contour line and a height value, which is computed from the normalized distance map. The height value may be the distance of the paddles 16. At a specific distance from the contour line it may be assumed that the object 18 is touching both paddles. Between the contour line and the specific distance, the three- dimensional surface of the object 18 may be modeled with semi-circles.

The system 10 and in particular the detector arrangement 14 may be arranged such that the optical tomography raw data 52 and the X-ray image data 50 are acquired without changing the position of the object 18 between the two scans, i.e. the respective detection processes of the two detector arrangements 20, 22. In such a case, the surface contour of the object 18 nearly does not change between the scans and it may be assumed that the surface contour of the X-ray image data 50 is equal to the surface contour of the optical tomography data 52.

The coarse model 54 may comprise a (coarse) density model of the object 18.

In particular, the percentage of fibro-glandular tissue in the object 18 may be estimated via automated object density estimation.

For each pixel of the X-ray image data 50 inside the two dimensional object contour, a separation of the line integral value (the attenuation value at the pixel) μπώ_ίΐυτβ into a contribution from adipose and glandular tissue may be computed as

^mi "f .ir* * - · - z ! o?-^* ' ' ' " r^{! *} ^jlarsd ΪΡ )) ^~"~ ^glanc ^ gland f P )

with the known pixel dependent object height H(p), computed from the coarse 5 surface model 54, and known attenuation values

of adipose and μ_§1&η£ι of glandular tissue, for example generated by calibration of the system 10. Hence, for each pixel an local object density estimate d(p) may be computed as

d(p) = ¾_a!3d(p)/H{p3

as well as an overall object density estimate 5, given by

Π d = ^» C d(p dA

U Sjyeass ares

According to an embodiment of the invention, the method comprises the step of estimating a coarse model 54 of the object 18 from the X-ray image data 50 and/or the optical tomography data 52.

According to an embodiment of the invention, the coarse model 54 comprises a 5 surface model of the object 18.

According to an embodiment of the invention, the coarse model 54 comprises a density model generated from the X-ray image data 50. The density model may comprise an estimate of the local and/or the overall object density.

In step SI 6, the device 12 determines a reference model 58.

0 The coarse model 54, in particular the surface model from step S14 and the density model from step S16 may be used to initialize a reference model 58, which may be a parametric in-silico breast model 58, comprising realistically shaped tissue components (for example skin, glandular tissue/duct system, adipose tissue, ligaments). The reference model 58 may further comprise X-ray attenuation coefficients and an elastic deformation model 56.5 An initial reference model may be selected from a database 34 according to the estimated overall density value ^ and the corresponding view position (for example LCC "left cranio-caudal", RCC "right cranio-caudal", LMLO "left medio-lateral oblique", RMLO "right medio-lateral oblique") of the X-ray detector arrangement 22. The view position may influence the compression of the object 18 and the corresponding reference model. Additional0 features such as age and total breast volume may be used for refining the selection of a

reference model 58.

The database 34 of reference models 58 may be generated for example with a breast phantom generator, for example the different reference models may be modelled with an interactive modelling software.

It is also possible that the database 34 is generated from magnetic resonance imaging breast examinations. The imaging data of the different examinations may be first segmented in glandular and adipose tissue and may then be categorized with respect to the before mentioned features. Then the segmented breast volumes may be mapped to a semi- ellipsoidal reference breast shape with a volume given by the reference class. Finally, all breast imaging data within one reference class may be summed and normalized to create one final reference model 58 (probability map) for each reference class.

According to an embodiment of the invention, the method comprises the step of selecting a reference model 58 of the object 18 from a plurality of reference models based on the coarse model 54 which is based on the X-ray image data 50 and/or the optical tomography raw data 52.

According to an embodiment of the invention, the plurality of reference models is stored in a database 34.

According to an embodiment of the invention, the reference model 58 is retrieved from the database 34 based on selection parameters comprising an estimated overall density ^ ; the overall object volume and/or an view direction of the X-ray image data 50.

In step SI 8, the reference model 58 may be mapped to the surface model 54, thus generating a deformed reference model 58. The compression force and the distance of the compression paddles 16 may also be taken into account during the modelling.

In the following steps S20 to S24, the reference model 58 is adjusted to the X- ray image data 50, thus becoming the three-dimensional model 60. An iterative optimization process may be used during which the model parameters may be adapted in order to minimize the difference between the real mammogram and the (simulated/estimated) three-dimensional model 60.

The three-dimensional model 60 may be generated from the reference model 58 by adjusting parameters of the reference model 58, for example, the relationship between tissue components.

In step S20, the device 12 projects the reference model 58 (and in further iteration steps the three dimensional model 60, i.e. the iteratively adjusted reference model 58) to a two dimensional projection 59. For example, from the reference breast model 58 with its current parameterization a forward mammography projection (i.e. a projection from three dimensions to two dimensions) is determined.

According to an embodiment of the invention, the method comprises the step of projecting the (iteratively adjusted) reference model 58 to two-dimensional projection data 59.

In step S22, the device 12 compares the two dimensional projection data 59 with the X-ray image data 50.

A difference measure between the X-ray image data 50 and the projection data 59, generated from the reference model 58, may be computed, for example via sum of squared differences. In particular, the adjusting process may be controlled via geometry parameters (e.g. control points of surface mesh) in an iterative procedure by minimizing the sum of squared differences between the actual object data (the X-ray image data 50) and the projected simulated object data (the projection data 59) generated from the (deformed) reference model 58.

According to an embodiment of the invention, the method comprises the step of comparing the two-dimensional projection data 59 with the X-ray image data 50.

According to an embodiment of the invention, a difference measure between the two-dimensional projection data 59 and the X-ray image data 50 is calculated for comparing the two-dimensional projection data 59 and the X-ray image data 50.

In step S24, the device 12 adjusts parameters of the (perhaps already iteratively adjusted) reference model 58.

According to an embodiment of the invention, the method comprises the step of adjusting the reference model 58 based on the comparison of the two-dimensional projection data 59 and the X-ray image data 50.

According to an embodiment of the invention, the reference model 58 is adjusted by modifying the two-dimensional projection data 59 and minimizing the difference measure.

The parameters (for example the relationships between tissue components, the tissue distribution) of the reference model 58 may be modified such that the difference measure is minimized. Here, inner energy terms may be applied to the reference model 58 in order to keep the basic shape of the breast and its components inside realistic ranges.

The (deformed) reference model 58 may be used to generate the three- dimensional model 62 by weighted back-projection of the line integral values into the

(deformed) reference model 58. The weights may be given by the normalized probability map of the (deformed) reference model 58. According to an embodiment of the invention, the method comprises the step of generating the three-dimensional model 60 from the reference model 58 by adjusting the reference model 58 to the X-ray image data 50.

In step S26, the optical tomography raw data 52 are transformed into three- dimensional tomography data 62 with the aid of the three-dimensional model 60.

Since the three-dimensional model 60 may assure a realistic distribution of the different tissue components, the three-dimensional distribution of the fibro-glandular tissue at the optimized parameter state of the three-dimensional model 60 may be used as input estimate for the optical data reconstruction of the tomography data.

An iterative reconstruction of optical tomography data 52 may benefit from initialization with an estimate of the three-dimensional tissue distribution within the breast, i.e. the three-dimensional model 60.

According to an embodiment of the invention, the method comprises the step of generating three-dimensional tomography image data 62 from the three-dimensional model 60 and the optical tomography image raw data 52.

In other words, the method may also be a method of reconstruction three- dimensional tomography image data 62.

In an alternative embodiment of the method, the object 18 may be repositioned between the acquisition of the mammogram and the optical tomography data, i.e. between steps S10 and S12.

In this case the reference model 58, and in particular the estimated three- dimensional distribution of the fibro-glandular tissue (i.e. the three-dimensional model 60) may need to be deformed in order to match the positioning of the object 18 during the optical data acquisition in step S12.

In step S14, in order to do so, the surface model 54 may be reconstructed in both positioning states (for example by using estimation techniques purely based on the X-ray data 50 / optical data 52 which is available and/or by additional use of fiducial markers, structured light, time-of-flight range scanners, or equivalent methods).

According to an embodiment of the invention, a first surface model of the object 18 is estimated from the X-ray image data 50 and a second surface model of the object 18 is estimated from the optical tomography data 52.

In step SI 8, an elastic deformation model 56 is used to estimate the deformation of the inner breast tissue (namely of the fibro-glandular tissue) based on the known two surface shapes.

According to an embodiment of the invention, the first surface model is mapped to the second surface model with an elastic deformation model of the object 18.

The deformed reference model 58 may be used to enhance the optical tomography reconstruction process as described in the above.

While the invention has been illustrated and described in detail in the drawings and foregoing description, such illustration and description are to be considered illustrative or exemplary and not restrictive; the invention is not limited to the disclosed embodiments. Other variations to the disclosed embodiments can be understood and effected by those skilled in the art and practising the claimed invention, from a study of the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. A single processor or controller or other unit may fulfil the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage. Any reference symbol in the claims should not be construed as limiting the scope.

Claims

CLAIMS:

1. A device (12) for generating a three-dimensional model (60) from an object of interest (18), the device (12) comprising:

an interface (32) for receiving optical tomography raw data (52) acquired from the object (18) and for receiving X-ray image data (50) acquired from the object (18);

a selector (32) for selecting a reference model (58) of the object (18) from a plurality of reference models based on the X-ray image data (50) and the optical tomography raw data (52);

a generator (32) for generating the three-dimensional model (60) from the reference model (58) by adjusting the reference model (58) to the X-ray image data (50).

2. A method for generating a three-dimensional model (60) from an object of interest (18), the method comprising the steps of:

receiving optical tomography raw data (52) acquired from the object (18); receiving X-ray image data (50) acquired from the object (18); selecting a reference model (58) of the object (18) from a plurality of reference models based on the X-ray image data (50) and the optical tomography raw data (52);

generating the three-dimensional model (60) from the reference model (58) by adjusting the reference model (58) to the X-ray image data (50).

3. The method of claim 1, further comprising the steps of:

projecting the reference model (58) to two-dimensional projection data (59); comparing the two-dimensional projection data (59) with the X-ray image data

(50);

adjusting the reference model (58) based on the comparison of the two- dimensional projection data and the X-ray image data (50).

4. The method of claim 3,

wherein a difference measure between the two-dimensional projection data (59) and the X-ray image data (50) is calculated for comparing the two-dimensional projection data (59) and the X-ray image data (50);

wherein the reference model (58) is adjusted by modifying the two-dimensional projection data (59) and minimizing the difference measure.

5. The method of claim 3 or 4,

wherein the reference model (58) is adjusted by back-projecting modified two- dimensional projection data (59).

6. The method of one of the preceding claims, further comprising the steps of:

estimating a coarse model (54) of the object (18) from the X-ray image data (50) and/or the optical tomography data (52).

7. The method of claim 6,

wherein the coarse model (54) comprises a surface model of the object (18).

8. The method of claim 6 or 7,

wherein the coarse model (54) comprises a density model generated from the X-ray image data (50).

9. The method of one of the preceding claims,

wherein a first surface model of the object (18) is estimated from the X-ray image data (50);

wherein a second surface model of the object (18) is estimated from the optical tomography data (52);

wherein the first surface model is mapped to the second surface model with an elastic deformation model of the object (18).

10. The method of one of the preceding claims,

wherein the plurality of reference models are stored in an database (34);

wherein the reference model (58) is retrieved from the database (34) based on selection parameters comprising an estimated overall density; the overall object volume and/or an view direction of the X-ray image data (50).

11. The method of one of the preceding claims, further comprising the step of: generating three-dimensional tomography image data (62) from the three- dimensional model (60) and the optical tomography image raw data (52).

12. A computer program for generating a three-dimensional model from an object of interest, which, when being executed by a processor, is adapted to carry out the steps of the method of one of the claims 2 to 11.

13. A computer-readable medium, in which a computer program according to claim

12 is stored.

14. The device (12) of claim 1, further comprising:

a database (34) for storing a plurality of reference models.

15. A system (10) for generating a three-dimensional model, comprising:

an X-ray detector arrangement (22) adapted for acquiring X-ray image data (50); an optical detector arrangement (20) adapted for acquiring optical tomography raw data (52); and

a device (12) adapted to execute the method of one of claims 2 to 11.