CN118318276A

CN118318276A - System and method for contouring medical images and reviewing existing contours

Info

Publication number: CN118318276A
Application number: CN202280076569.9A
Authority: CN
Inventors: M·J·古丁; P·卢尼; D·波克罗欧
Original assignee: Mirada Medical Co ltd
Current assignee: Mirada Medical Co ltd
Priority date: 2021-10-07
Filing date: 2022-09-28
Publication date: 2024-07-09

Abstract

A method and system for contouring medical images in a contouring system are described. The method comprises the following steps: providing at least one medical image to be outlined by a user; determining that a user has initiated contouring of a structure on the medical image; the contouring system determines the structure being or about to be contoured; in response to the determination of the structure, the system displays a guideline for contouring of the determined structure; and drawing the outline of the determined structure according to the displayed guidance. Methods and systems for reviewing contours on contoured medical images are also described.

Description

System and method for contouring medical images and reviewing existing contours

Technical Field

The present invention relates to the field of medical imaging and medical image processing, in particular to contouring of anatomical structures on medical images or examination of pre-existing contours on medical images, and in particular in the field of radiation therapy treatment planning.

Background

In many cases, a clinician needs to outline anatomical structures on medical images. For example, in radiation therapy treatment planning, organs, tumors, and target areas are typically delineated by a radiation oncologist on medical images of a patient. Patient medical images are typically acquired using Computed Tomography (CT) scanning, but other imaging modalities may also be acquired, such as Magnetic Resonance Imaging (MRI), positron Emission Tomography (PET), or Single Photon Emission Computed Tomography (SPECT). These medical images typically comprise a 2D image stack of a cross-section of the patient, as well as a 3D volumetric image. In preparation for treatment, organs At Risk (OAR) need to be delineated on the "planning" image in order to preserve healthy tissue. The planning software may then calculate a treatment plan that maximizes the radiation dose to the target volume and tumor while minimizing the dose to surrounding healthy tissue. Such a delineation process is referred to as contour rendering, i.e. a term that may be used to indicate a delineation process in 2D or 3D that defines the boundaries of anatomical structures on medical images. Similarly, the term automatic contour drawing may be used to indicate an automatically generated contour on a medical image.

Manual gold standard contouring of a patient medical image by a human operator is time consuming and subject to variability in delineating anatomical structures on the medical image [1]. This variability is due to intra-and inter-operator variations and variations between different institutions or departments. To reduce this variability, organizations and specialized organizations have developed contouring guidelines (such as [2 ]) and atlases (such as [3 ]). Guidelines for use have been found to reduce variability between observers [4].

The guideline takes the form of a text description detailing how the contours of the structure should be drawn, while in this case the atlas is a "gold standard" set of contours drawn on an example patient image. Typically, the guidelines and/or atlases are formed by consensus among experts.

Although clinical teams receive training and education in guidelines, changes may reoccur over time if staff becomes self-full/amnestic [5] or the guideline is updated. Accordingly, peer review is often suggested as a way to examine and alert staff members of the contouring guidelines. This process is time consuming and does not account for (unintentional) inter-institution variations.

Some technical solutions have been proposed to help comply with guidelines. One approach is to use an automatic contouring method to generate contours from a system that should reproduce the guideline. Although automatic contouring has been shown to reduce inter-clinician variability [6], guidelines are still required to be known to correct any automatic contouring errors to comply with these guidelines [5]. Another approach is to make these guidelines (e.g., [7 ]) more readily available to the clinician at the time of contouring. But this approach may interrupt the clinician's workflow and require them to find the appropriate reference information at work. That is, the guideline may specify a series of details of different structures, and when a clinician draws a contour of a particular structure on a medical image, they will need to find relevant text in the guideline to check its contour drawing, even if the correct guideline set is presented. Furthermore, different institutions or geographical areas may define different treatment planning schemes, and thus may have different contouring guidelines, and such solutions are not suitable for local deployment.

[1]Brouwer CL,Steenbakkers RJ,van den Heuvel E,Duppen JC,Navran A,Bijl HP,Chouvalova O,Burlage FR,Meertens H,Langendijk JA,van't Veld AA.3D variation in delineation of head and neck organs at risk.Radiation Oncology.2012 12 Months of the year; 7 (1): 1-0.

[2]Brouwer CL,Steenbakkers RJ,Bourhis J,Budach W,Grau C,Grégoire V,Van Herk M,Lee A,Maingon P,Nutting C,O'Sullivan B.CT-based delineation of organs at risk in the head and neck region:DAHANCA,EORTC,GORTEC,HKNPCSG,NCIC CTG,NCRI,NRG Oncology and TROG consensus guidelines.Radiotherapy and Oncology.2015 10 Months and 1 day of the year; 117 (1): 83-90.

[3]https://www.nrgoncology.org/About-Us/Center-for-Innovation-in-Radiation-Oncology/Head-and-Neck/Head-and-Neck-Atlases. Access date: 05/10/2021

[4]Sun KY,Hall WH,Mathai M,Dublin AB,Gupta V,Purdy JA,Chen AM.Validating the RTOG-endorsed brachial plexus contouring atlas:an evaluation of reproducibility among patients treated by intensity-modulated radiotherapy for head-and-neck cancer.International Journal of Radiation Oncology*Biology*Physics.2012 1 Day 3 months; 82 (3): 1060-4.

[5]Brouwer CL,Boukerroui D,Oliveira J,Looney P,Steenbakkers RJ,Langendijk JA,Both S,Gooding MJ.Assessment of manual adjustment performed in clinical practice following deep learning contouring for head and neck organs at risk in radiotherapy.Physics and imaging in radiation oncology.2020 10 Months and 1 day of the year; 16:54-60.

[6]Vinod SK,Min M,Jameson MG,Holloway LC.A review of interventions to reduce inter-observer variability in volume delineation in radiation oncology.Journal of medical imaging and radiation oncology.2016 6 Months of the year; 60 (3): 393-406.

[7] Https:// anatom-e.com/verify/access date: 05/10/2021

Disclosure of Invention

Accordingly, the present invention has solved the following problem(s);

There is a need for a system and method to facilitate the display of relevant guidance (guideline and/or example atlas images) regarding the specific structure being outlined during the silhouette rendering of one or more new contours on a medical image or during the review or editing of one or more pre-existing contours on a medical image to assist a clinical staff member in outlining anatomical structures. Such systems and methods are described and disclosed.

According to the present invention, there is provided a method for contouring a medical image in a contouring system, comprising the steps of: providing at least one medical image to be outlined by a user; determining that a user has initiated contouring of a structure on the medical image; the contouring system determines the structure being or about to be contoured; in response to the determination of the structure, the system displays a guideline for contouring of the determined structure; and drawing the outline of the determined structure according to the displayed guidance.

In an alternative embodiment of the present invention, there is also provided a method for reviewing contours on a contoured medical image, comprising the steps of: providing at least one medical image annotated with one or more pre-existing contours; displaying at least one medical image annotated with one or more pre-existing contours; determining that the user has initiated review of a pre-existing contour on the medical image; determining a structure associated with a pre-existing profile that the user is reviewing; in response to the determination of the structure, a guidance for contouring of the determined structure is displayed.

Preferably, the method further comprises the step of completing the review of the pre-existing profile or editing the pre-existing profile according to the displayed guidance. Further preferably, completing the review of the pre-existing profile includes providing feedback regarding the profile. In an embodiment of the invention, the feedback is provided as comments or notes about the outline.

In a preferred embodiment, the method further comprises the step of outputting an censored or user initiated profile associated with the image.

Preferably, the determination that the user has initiated the contour mapping or review is determined when the user starts to map or edit the contour of the structure on the at least one medical image.

In a preferred embodiment of the invention, the determination to edit or review the pre-existing contour is done using eye tracking hardware and/or software to identify the location where the user gazes at the at least one medical image.

Preferably, the guidance for contouring is at least one of: a clinical guideline for contouring the determined structure or at least one atlas image displaying at least one example contour of the determined structure.

It is further preferred that the structure being reviewed or outlined is determined by finding a mapping between at least one medical image and at least one atlas image. Preferably, the mapping is done using deformable image registration. In a preferred embodiment of the invention, the image registration comprises one or more of the following: deformable registration, affine registration, or rigid registration.

In an embodiment of the invention, the method further comprises predicting the structure being outlined using machine learning. Preferably, the structure being censored or profiled is determined by comparison to a profile generated using automatic profiling based on machine learning. In an embodiment of the invention, machine learning based automatic contouring uses one of the following: convolutional neural networks, random forest methods, and support vector machines.

Preferably, the determination of the structure on the medical image being examined or outlined is done using eye tracking hardware and/or software to identify the location where the user gazes at the at least one medical image.

In an embodiment of the invention, the user profiling the determined structure is one of manual, semi-automatic or automatic profiling. Preferably, the contouring system uses the contour name of the structure being contoured to determine the structure being contoured or inspected. Further preferably, the contour name is provided to a lookup table such that the name is associated with a standardized naming convention.

In an embodiment of the invention, the steps of the method are repeated such that a plurality of structures in at least one medical image are profiled.

Preferably, the pre-existing contours displayed to the user are manual or semi-automatic contours from a previous contouring session, or contours that are automatically generated.

In a preferred embodiment of the invention, the at least one medical image is one of a CT scan, an MRI scan, a PET scan or an ultrasound scan. Further preferably, the at least one medical image is one of a 2D image, a 3D image or a time series of medical images.

According to the present invention there is also provided a system for contouring at least one medical image, comprising: a display for displaying at least one medical image to be outlined by a user; a processor for determining that a user has initiated contouring of a structure on the medical image; the processor determines a structure being profiled; in response to the determination of the structure, the system displays to a user guidance for the contouring of the determined structure; such that the determined structure may be contoured in accordance with the displayed guidance.

There is also provided, in accordance with another embodiment of the present invention, a system for reviewing or editing contours on a contoured medical image, including: providing at least one medical image annotated with a pre-existing contour; a display for displaying at least one medical image annotated with a pre-existing contour; a processor for determining that a user has initiated review of a pre-existing contour on the image, the processor determining a structure of the pre-existing contour being reviewed; in response to the determination of the structure, the system displays a guideline for contouring of the determined structure; to allow for the completion of a profile review or editing of an existing profile in accordance with the displayed guidance.

Drawings

Further details, aspects and embodiments of the invention will be described, by way of example only, with reference to the accompanying drawings. In the drawings, like reference numbers may be used to identify identical or functionally similar elements. Elements in the figures are illustrated for simplicity and clarity and have not necessarily been drawn to scale.

FIG. 1a is a flow chart illustrating a method according to an embodiment of the invention;

FIG. 1b is a flow chart illustrating a method according to an alternative embodiment of the invention;

FIG. 2a is a flow chart illustrating a method according to another alternative embodiment of the invention;

FIG. 2b is a flow chart illustrating a method according to another alternative embodiment of the invention;

FIG. 3 is an example user interface according to an embodiment of the invention;

Fig. 4 illustrates a simplified block diagram of an example of a medical imaging system.

Detailed Description

The present invention is a system and method for enabling efficient contouring or delineation of structures on medical images, or for reviewing existing contours, and actively displaying relevant guidance information to assist a user during contouring or during review of existing contours. Preferably, the instruction information for the user is at least one of a clinical guideline for contouring a determined structure on the medical image or at least one atlas image generated from the clinical contouring guideline showing at least one example contour of the determined structure on the medical image.

Referring now to fig. 4, a simplified block diagram of an example of a medical imaging system 400 is illustrated, the medical imaging system 400 being arranged such that medical images can be displayed to a user for silhouette rendering or for reviewing existing contours. In a preferred embodiment of the invention, the system comprises a display for displaying at least one medical image, wherein the medical image is to be outlined or has been annotated with a pre-existing contour to be examined. In the illustrated example, the medical imaging system 400 comprises one or more user terminals 401 (e.g. comprising a workstation or the like) arranged to access medical images stored in, for example, a database 402 or other data storage. In embodiments of the invention, these may provide at least one medical image annotated with a pre-existing contour, or a medical image to be contoured. In the example shown, a single database 402 is illustrated.

It will be appreciated that the user terminal 401 may be arranged to access medical images from more than one data storage means. Further, in the example shown, the database 402 is illustrated as being external to the user terminal 401. It will be appreciated that the user terminal 401 may equally be arranged to access medical images stored locally on one or more internal storage elements within a local storage module shown at 410, such as a memory element shown at 403 or a disc element shown at 409.

In an embodiment of the invention, for contouring a medical image, the workstation may comprise a processor for determining that a user has initiated contouring of a structure on the medical image; the processor determines a structure being profiled; in response to the determination of the structure, the system displays to a user guidance for the contouring of the determined structure; such that the determined structure may be contoured in accordance with the displayed guidance.

In an alternative embodiment of the invention, for reviewing pre-existing contours on medical images, the processor determines that the user has initiated review of pre-existing contours on the images, the processor also determining the structure of the pre-existing contours being reviewed; in response to the determination of the structure, the system displays a guideline for contouring of the determined structure; to allow for the completion of a profile review or editing of an existing profile in accordance with the displayed guidance.

The user terminal 401 further includes one or more signal processing modules, such as the signal processing module shown generally at 404. The signal processing module(s) are arranged to execute computer program code stored, for example, within the local storage module 410. In the example shown, the signal processing module(s) 404 are arranged to execute computer program code comprising one or more of the automation components shown in the example; the component 405 automatically detects relevant contouring guidelines or guides for the structure being contoured. The signal processing module 404 in the illustrated example is further arranged to execute computer program code comprising one or more image display components 406; the signal processing module 404 in the illustrated example is further arranged to execute computer program code comprising one or more guideline or guide display components 412; in response to the user interacting with the image and outline display 406, the guide display component(s) 412 are arranged to display relevant portion(s) or example(s) of the guide for outline drawing as determined by 405, for example on the display screen 407 or the like, to the user.

The instructions selected by 405 and displayed by 412 may be loaded from database 402, memory 403, or disk 409. The medical imaging system 400 may also include one or more user input devices, such as shown generally at 408, to enable a user to interact with computer program code or the like executing on the signal processing module(s) 404. After interaction, the user instructs the system to save the results back to the local storage 410 or database 402, as indicated by the bi-directional arrow 411.

Fig. 1a illustrates one embodiment of the present invention. At step 101, the system loads at least one medical image (which may be a 2D or 3D, or a time series of medical images, wherein the time series is a series of 2D or 3D images taken in a single scan session, wherein, for example, each image may represent a different phase of the respiratory or cardiac cycle) to be outlined by the user. Preferably, the medical image is a CT image, but other images such as MRI, PET, SPECT or ultrasound images are also possible. The loading process may be user initiated or may be automated. The medical images may be provided from a database or from a file system or an external storage device, such as a disk. Such a system is expected to have normal viewing capabilities of the medical image visualization tool and the user may choose to change views. At some point, after the medical image has been loaded, it is determined that the user initiates a silhouette rendering process of the medical image using manual, semi-automatic or automatic silhouette rendering tools available in the system at step 102. Tools commonly provided by such systems include point-based polygon rendering tools, hand-drawing pen-like tools, and brush-like region filling tools. Preferably, the determination that the user has initiated the contour mapping or the review of the contour is determined when the user starts to map or edit the contour of the structure on the at least one medical image. This may be accomplished, for example, by tracking cursor movement on the display screen. In an alternative embodiment of the invention, the determination that the contour is being generated, edited or reviewed is accomplished using eye tracking hardware and/or software to identify the location where the user gazes at the at least one medical image. Once contouring is initiated, at step 103, the system determines which anatomical structure on the medical image is or is to be contoured by the user. Preferably, this determination of the structure on the medical image being outlined is done automatically by the system. Examples of methods for performing this operation are described in more detail below, but examples may be using the contour name of the contour currently being reviewed or generated (as provided by the user), and a look-up table or database for correlating the user-specified structure name with a standard naming convention (e.g., [8 ]), so that the structure being contoured is indicated by the user.

After determining which structure in the medical image the user is contouring, the system displays instructional information to the user regarding the contours of the structure being contouring. As described above, the guidance for the user is at least one of a clinical guideline for contouring the determined structure or at least one atlas image generated from the clinical contouring guideline showing at least one example contour of the determined structure on the medical image. The structure may then be contoured in accordance with the displayed guidance. Preferably, the instructional information is one or more relevant portions of the guideline displayed to the user via the system user interface at step 104. In a preferred embodiment of the invention, the display of the directions is an interactive display based on user behavior when performing the contouring. This interactivity reduces the time required to find the appropriate guide or chapter of the guide. This is in contrast to previous contouring systems and methods in which, for example, the image to be contoured is displayed on one screen and the general guideline is displayed on another screen, where the user must manually scroll through the displayed guideline to reach the appropriate guideline for the structure currently being contoured.

In some embodiments of the invention, the instructional information may be an atlas. This is illustrated in fig. 1b, where numbered items corresponding to fig. 1a are equivalent, the display of the guideline 104 is replaced with a graphical display of one or more example cases and outlines (atlases) for the same structure.

For both embodiments in fig. 1a and 1b, the user may then review the guidance (one or more guidelines and/or atlases) while continuing to contour structures on the medical image to assist the user in the contouring process. Then, at step 105, the user completes the contouring of the structure. The process of contouring the structure (steps 102-105) may be repeated for a plurality of different structures on the medical image, thus contouring the plurality of structures in at least one medical image until the user is satisfied at step 106. If all structures have been outlined, the workflow proceeds to step 107. If there are some structures that still have to be outlined, the workflow returns to step 102. In this way, the system may be able to update the guidance of each iteration and will only display the guidance of the contour that is currently being generated, not the guidance of the previous contour that has been completed. Then, at step 107, the system will save, store or output the one or more depictions created by the user for further use in the clinical workflow. The process may also be repeated for other medical images so that the user may contour a series of images in a single contouring session.

In an alternative embodiment of the invention, as shown in fig. 2a, the method proceeds in a substantially similar manner. In this embodiment of the invention, one or more pre-existing contours on the medical image may be edited or reviewed, or one or more new contours may be added to the medical image that already has a pre-existing contour. This is in contrast to the previous embodiment as depicted in fig. 1, in which the initial medical image does not have any contours, and the contours are generated by the system user. At step 201, the system loads at least one medical image (which may be a 2D or 3D, or a time series of medical images). For the embodiment of fig. 1, the medical image is preferably a CT image, but a different imaging method is also possible, such as MRI, PET, SPECT or ultrasound. The system also loads and displays one or more pre-existing contours or depictions on the anatomical structure visible in the medical image, as indicated at step 202, before performing any further contouring. One or more pre-existing contours may then be reviewed to conform to the particular contour's guidance and/or edited to better conform to the guidance. These existing one or more depictions may be from a manual silhouette of a previous silhouette rendering session performed by a user (whether the same user or a different user), or from an automatic silhouette rendering system. Then, at step 203, the user initiates review or contouring of the contours of the medical image, and the systems and methods determine that the user has initiated contouring on the medical image. In an embodiment of the invention, when the user starts to draw or edit the contour of the structure on the at least one medical image, it is determined that the user has initiated an examination of the contour or a determination of the contour drawing. In an alternative embodiment of the invention, the determination of the pre-existing contour to be edited or reviewed is done using eye tracking hardware and/or software to identify the location where the user gazes at the at least one medical image.

Such contouring of the determined structure on the medical image by the user may be manual, semi-automatic or automatic contouring of the image with pre-existing contours. The contouring may be a de novo depiction of a new structure on the medical image with one or more existing contours, or an edit of one of the one or more pre-existing depictions that have been displayed on the medical image. The system and method then determines which anatomical structure on the medical image is being contoured and/or has a contour that is reviewed by the user at step 204. Preferably, this is done automatically by the system. In determining which structure in the medical image is being outlined, the system may display instructional information for the structure being outlined to the system user at step 205. Preferably, the instruction information is one or more relevant parts of the instruction provided to the user, preferably via a user interface of the system. The user interface may be on the screen on which contouring is taking place, or on a separate screen that is another part of the user interface. In determining the structure on the medical image being outlined, the system may preferably load and display an atlas at step 206, which shows an example outline of the structure that has been drawn according to the guideline. Steps 205 and 206 may both be performed or only one of these steps may be performed. Further, step 206 may be performed independently of step 205 being performed—thus step 206 may be an optional step within the workflow indicated in fig. 2. As with the previously described workflow, once the user has completed the contouring at step 207, the system may preferably store, save or output one or more contours created by the user at step 208 for further use in the clinical workflow. In some embodiments of the invention, this phase of saving/storing the output may be done at the time of performing the contour drawing, such that only a partial depiction of the structure being contour drawn is initially saved for further use, which continues as the contour drawing continues until a complete contour is saved. The output of the profile is typically in a format commonly used by the medical device community, such as the DICOM format. DICOM is also an image transmission protocol so data can be transmitted to other devices such as a Treatment Planning System (TPS) or a Picture Archiving and Communication System (PACS). Alternatively, the profile (partial or complete) may be stored to a local or network file system. In an embodiment of the invention, the steps of the method of inspecting a pre-existing contour as described above are repeated such that a plurality of pre-existing contours on the medical image are inspected.

In an alternative embodiment of the invention, as shown in fig. 2b, the method of the invention proceeds in a substantially similar manner. In this embodiment of the invention, one or more pre-existing contours on the medical image may be reviewed relative to the guidance of contour rendering of the structure on the medical image. At step 209, the system loads at least one medical image (which may be a 2D or 3D, or a time series of medical images). For the embodiment of fig. 1, the medical image is preferably a CT image, but a different imaging method is also possible, such as MRI, PET, SPECT or ultrasound. The system also loads and displays one or more pre-existing contours or depictions on the anatomical structure visible in the medical image, as indicated by step 210, before performing any review of the pre-existing contours. These existing depictions on the medical image displayed to the user may be from a manual or semi-automatic contouring session performed by the user (whether the same user or a different user), or from an automatic contouring system. Preferably, the system displays at least one medical image annotated with one or more pre-existing contours, wherein the pre-existing contours are to be reviewed. Then, at step 211, it is determined that the user has initiated an inspection of at least one of the pre-existing contours of the medical image. In an embodiment of the invention, the determination of the pre-existing contour to be edited or reviewed is done using eye tracking hardware and/or software to identify the location where the user gazes at the at least one medical image.

The system and method determines that the user has initiated an examination of a pre-existing contour on the medical image, and then, at step 212, determines an anatomical structure associated with the pre-existing contour on the medical image that the user is examining. Preferably, the structural determination is accomplished automatically by the system and method.

After determining which structure in the medical image the pre-existing contours being reviewed are related to, the system may display instructional information for the contouring of that particular structure to the system user at step 213. Preferably, the instruction information is one or more relevant parts of the instruction provided to the user, preferably via a user interface of the system. After determining the structure on the medical image being outlined, the system may preferably load and display an atlas at step 214, which shows the structure being drawn according to the instructional information. Steps 213 and 214 may be both performed or only one of them may be performed. The system then reviews the one or more contours and provides feedback regarding the contours at step 215. In an embodiment of the invention, the method may further comprise the step of completing the review of the pre-existing profile or editing the pre-existing profile according to the displayed guidance.

Completing the review of the pre-existing profile may also include providing feedback regarding the profile. The review or feedback may take the form of comments added to the image or outline or comments written for the outline. The comments and notes may be recorded in the same system or may be recorded separately in separate systems. As with the previously described workflow, once the user has completed review of the pre-existing contours at step 216, the system stores, saves or outputs contour annotations or comments created by the user for further use in the clinical workflow at step 217. The method may further comprise the step of outputting an inspected contour associated with the image. The output of such a review of the profile is typically in a format commonly used by the medical device community, such as the DICOM structured report format, but such a system may use proprietary formats. The comments or reports generated by the user regarding the outline at step 216 may be recorded in a document format such as an MS Word or PDF document. This may be stored within the oncology information system. The review at step 215 may also be performed verbally by the user interpreting or giving feedback to another user regarding the guideline deviation.

In the head-on contouring method or pre-existing contour review as described above, it is determined that the user has initiated a determination of contour mapping or contour review when the user begins to map or edit contours of structures on at least one medical image.

The step of detecting the correct structure and linking it to the guidance for contouring and/or contour inspection is the core of the present invention.

The structure in the medical image being profiled or having a pre-existing profile being reviewed by the user may be determined in a number of ways. Preferably, the structure will be automatically determined. The examples given below are provided as possible embodiments of the invention but are not to be considered as exhaustive.

In an embodiment of the invention, the contouring system uses the contour name of the structure being contoured to determine the structure being contoured or inspected. The method for determining the structure being profiled includes:

When the user interface prompts the user to specify a contour name for the contour currently being reviewed or generated, a lookup table or database may be used to correlate the user-specified structure name to a standard naming convention (e.g., [8 ]), so that the structure being contoured is indicated by the user. Similarly, when automatic contour drawing of a structure on a medical image is used as a starting point for contour drawing, the name of the structure will be specified by the automatic contour drawing system, and a similar look-up table may be applied.

In various embodiments of the present invention, the structure being reviewed or outlined may be determined by finding a mapping between at least one medical image and at least one atlas image. In various embodiments of the present invention, mapping is accomplished using image registration. Preferably, an image registration [9] technique can be used to find a mapping between the atlas image and the patient medical image. Image registration is the process of determining alignment or spatial correspondence (also referred to as mapping between two or more images). Differences within images to be aligned to different extents may be corrected using one or more of rigid, affine, and deformable image registration methods. In rigid alignment, translation and rotation of the image may be performed. In affine alignment, clipping and scaling of images may be performed in addition to translation and rotation. In deformable alignment, translation of various points within the image can be performed. Using the mapping and as the mapping of the contours proceeds, the location of the medical image on the patient medical image from which the user performed the contouring is mapped to a corresponding location on the atlas image. By finding the structure closest to the contour being rendered on the patient medical image in the existing set of structures defined on the atlas image, spatial correspondence may be used to determine the structure in the patient medical image being contour rendered. The concept of closest structure may be quantified, for example, using a scoring system based on a measure of contour similarity between the user drawn contours and the atlas contour set. For example, dice similarity coefficients measuring spatial overlap between the drawn contours and the atlas contour set may be used to estimate the closest structure with the largest overlap. Metrics such as the average surface distance between the drawn contours and the atlas contour set may be used to estimate the closest structure with the average minimum spatial distance. The closest structure [10] may also be selected by comparing the centroids of the two contours or any combination of voting schemes in the contour or shape similarity measure. Note that the user profile may not be complete, so a similarity comparison may be performed on a fully drawn profile or a partial region of the atlas profile.

In a preferred embodiment of the present invention, machine learning, both for contour generation and for contour inspection, can be used to predict the structure being profiled or the contour being inspected. Preferably, the structure being censored or profiled is determined by comparison to a profile generated using automatic profiling based on machine learning. The machine learning based method may preferably be used to determine the organ being contoured or having a contour to be reviewed from the position in the image that the user is contoured/reviewing [11]. One such method is to apply machine learning based automatic contour drawing to an image (with or without displaying the result) as a way to determine the similarity or proximity of a user drawn contour to the automatic contour. Alternatively, a machine learning method may be used to map the appearance of the image being drawn by the user directly to the classification of the organ being contoured. For example, information from a given image block (patch) of a medical image is used with information about the structure to which the image block relates to train a classifier to predict the structure of the given image block. The classifier may then be used to predict the structure given the medical image blocks surrounding the location on the medical image that the user is contouring.

A similar approach may be taken by using a "dictionary" of structural image appearances. Image similarity measurements (such as those typically used for image registration) are used to compare blocks around the location on the medical image where the user is conducting contouring or reviewing an existing contour to each block within the atlas image. Each block within the atlas image is assigned a structure(s) that it can represent. The structure on the medical image being outlined or the structure on which the user is reviewing the pre-existing contours may be determined by finding which atlas image block is most similar to the block around the location on the medical image on which the user is outlining or reviewing the pre-existing contours, and thus assigning the structure(s) from the atlas. Where multiple structures may be assigned to a block, a comparison of the multiple blocks may be used to determine the most likely candidate structure.

Alternatively, the partially outlined structure being outlined by the user may be used with a machine learning classifier to determine the structure on the medical image being outlined. For example, a set of partial contours may be used with information about the structure they will become when completed to train a classifier to predict the structure of a given partial structure. The classifier can then be used to predict a structure given its partial user profile.

Various machine learning methods known to those skilled in the art and that may be used for machine learning based automatic contouring may include, but are not limited to: convolutional neural networks, random forest methods, and support vector machines.

The relative location and proximity of the partially drawn user contours on the medical image to other previously identified structures may be used in association with demographic data from previous patient cases to identify the structure on the medical image that is being contoured. For example, the heart is known to exist between two lungs, above the liver. Thus, if the user has previously outlined the lung and liver on the medical image and is now outlined on the medical image between the lung and liver above as shown on the medical image, the system may identify the new structure on the medical image as the heart.

Such a method of determining structures on medical images being outlined or reviewed may be applied singly or in combination. The determination of the structure on the medical image may be done automatically. A ranking of possible structures on the medical image may also be created such that structures determined to be most likely to be reviewed or outlined by the user are ranked first. A guide or atlas for the first ranked structure may then preferably be automatically displayed to the user. A simple interface may be provided to the user to reject the structure if this is not what they are considering, and then the next most likely candidate guideline or atlas may be displayed.

After determining that a contour or structure with a pre-existing contour being reviewed in the medical image is being outlined, appropriate guidelines (preferably guidelines and/or atlases) are displayed to the user via a user interface or some other display means. The database may be used to store relationships between the identified structures and the instructional information. The database may store the instructional information for retrieval (such as instructional data) directly, or for storing files and locations of information within files so that the system can retrieve the appropriate files and display the instructional information in the corresponding structure. Such databases may be preconfigured, configured by a user or system administrator, or automatically configured from a remote service.

Mapping the guidance (guideline and atlas) information into such a database may be performed manually as an initial configuration, or may be automatically populated using the natural language process of published guidelines for detecting relevant paragraphs. Such natural language processing may take into account anatomical names used in the guideline, as well as format analysis such as a title, to determine the structure in question, rather than the appearance of one structure being referred to as defining a silhouette rendering of another structure.

An image registration between the medical image being reviewed or outlined and the atlas image may be calculated. Such image registration may be used to determine equivalent locations within the atlas image as the user navigates through the medical image. For example, the location of a user's "brush" in a medical image used during silhouette rendering may be mapped to an atlas image via image registration. The atlas image and the contour may then be displayed by this location on the atlas image. This enables the user to interactively see the appropriate image slices of the atlas image as they do the silhouette rendering.

Displaying a guide (guideline and/or atlas) to the user is still helpful in cases where the user is reviewing previously generated contours to account for editing, but the user has not yet begun editing pre-existing contours. In this case, no physical input by the user to the system (such as movement of a "brush" tool for silhouette rendering) can be used to determine which silhouettes the user wishes to consider and which guidelines will be displayed appropriately. In this case, the structure of interest may be determined automatically, preferably by identifying the location where the user gazes at the at least one medical image using eye tracking hardware and/or software connected to the system. When a user gazes on a patient image, their gaze location, together with the above-described method, may be used to determine the structure under consideration, and related guidance (guideline and/or atlas) may be displayed to the user via the system user interface.

An example user interface is illustrated in fig. 3. A standard application menu item is displayed at 301 and an application toolbar is displayed at 302. These items are expected to occur in such a system, but are not necessary for the invention. The user may use menus and toolbars to change the interaction pattern between the visualization and the drawing. The patient medical image is displayed in a visualization pane 303. In this stylized example, a number of structures are shown by way of outline: patient skin-304, right lung-305, left lung-306, and heart-307. The small circle indicated as 308 represents the user's cursor position as if the user were in the process of tracing the heart. In this figure, the system has determined that the user is contouring the heart on the medical image and has displayed appropriate guidance for compliance at 309 and specific guidance for contouring the heart at 310. The user interface also displays an atlas image of the patient and the identified anatomy, i.e., heart (at 315), in pane 311. The atlas image shows the patient at 312, the right lung at 313, the left lung at 314, and the heart at 315. Upon detecting that the user is contouring the heart, the atlas will show the contours of the heart as drawn according to the guideline. As shown, the interface is located on a single screen, but it may also work on different partitioned screens.

It is also helpful when the system displays an appropriately structured atlas to the user and the patient and atlas image are 3D medical images, the atlas displaying the same slice position. This may be accomplished by linking the position in the patient image indicated by the user mouse or using eye tracking hardware or software to position navigation in the set of images, thereby calculating the spatial correspondence between the two using deformable image registration. Further, the view relative to the patient axis may be linked between the atlas and the patient to ensure that the same cross-section through the patient is displayed.

Accordingly, the disclosed invention solves the problem of ensuring that a clinician contouring an anatomical structure on a medical image or reviewing a pre-existing contour on a medical image has readily accessible relevant guideline (guideline or atlas) information for that structure during the contouring or contouring process.

[8]Mayo CS,Moran JM,Bosch W,Xiao Y,McNutt T,Popple R,Michalski J,Feng M,Marks LB,Fuller CD,Yorke E.American Association of Physicists in Medicine Task Group 263:standardizing nomenclatures in radiation oncology.International Journal of Radiation Oncology*Biology*Physics.2018 3 Months 15 days; 100 (4):1057-6 6.

[9] Maintz JA, viergever MA. A survey of MEDICAL IMAGE registration. MEDICAL IMAGE ANALYSIS. 3 months 1 1998; 2 (1): 1-36.

[10]Veltkamp,Remco.(2001).Shape matching:Similarity measures and algorithms.Proceedings-International Conference on Shape Modeling and Applications,SMI 2001.188-197.10.1109/SMA.2001.923389.

[11]Yang J,Veeraraghavan H,Armato III SG,Farahani K,Kirby JS,Kalpathy-Kramer J,van Elmpt W,Dekker A,Han X,Feng X,Aljabar P.Autosegmentation for thoracic radiation treatment planning:a grand challenge at AAPM 2017.Medical physics.2018 10 Months of the year; 45 (10):4568-81.

Examples of the invention may be applied to any or all of the following: a Picture Archiving and Communication System (PACS), an advanced visualization workstation, an imaging acquisition workstation, a web-based or cloud-based medical information and imaging system, a radiation therapy Treatment Planning System (TPS), a radiation therapy linear accelerator console, a radiation therapy proton beam console.

The invention has been described with reference to the accompanying drawings. It should be appreciated that the invention is not limited to the specific examples described herein and shown in the drawings. Furthermore, because the illustrated embodiments of the present invention may be implemented, for the most part, using electronic components and circuits known to those skilled in the art, details will not be explained in any greater extent than that considered necessary as illustrated above, for the understanding and appreciation of the underlying concepts of the present invention and in order not to obfuscate or distract from the teachings of the present invention.

The invention may be implemented in a computer program for running on a computer system, comprising at least code portions for performing the steps of a method according to the invention or enabling a programmable device such as a computer system to carry out the functions of an apparatus or system according to the invention.

A computer program is a list of instructions such as a particular application program and/or operating system. The computer program may for example comprise one or more of the following: subroutines, functions, processes, object methods, object implementations, executable applications, applets, servlets, source code, object code, shared libraries/dynamic load libraries, and/or other sequences of instructions designed for execution on a computer system. Thus, some examples describe a non-transitory computer program product having executable program code stored therein for automatic contouring of cone beam CT images.

The computer program may be stored internally on a tangible and non-transitory computer readable storage medium or transmitted to the computer system via a computer readable transmission medium. All or some of the computer program may be provided on computer readable media permanently, removably or remotely coupled to an information processing system. Tangible and non-transitory computer readable media may include, for example, but are not limited to, any number of the following media: magnetic storage media including disks and tape storage media; optical storage media such as optical disc media (e.g., CD ROM, CD R, etc.) and digital video disc storage media; nonvolatile memory storage media including semiconductor-based memory units such as FLASH memory, EEPROM, EPROM, ROM; a ferromagnetic digital memory; MRAM; volatile storage media include registers, buffers or caches, main memory, RAM, etc.

Computer processes typically include an executing (running) program or portion of a program, current program values and state information, and resources used by the operating system to manage the execution of the process. An Operating System (OS) is software that manages the sharing of computer resources and provides programmers with an interface for accessing those resources. The operating system processes system data and user input and responds by assigning tasks and internal system resources as services to users and programs of the system and managing them.

The computer system may, for example, include at least one processing unit, associated memory, and a plurality of input/output (I/O) devices. When executing the computer program, the computer system processes the information according to the computer program and generates resultant output information via the I/O device.

In the foregoing specification, the invention has been described with reference to specific examples of embodiments thereof. It will be evident that various modifications and changes may be made therein without departing from the scope of the invention as set forth in the appended claims, and that the claims are not limited to the specific examples described above.

Those skilled in the art will recognize that the boundaries between logic blocks are merely illustrative and that alternative embodiments may merge logic blocks or circuit elements or impose an alternate decomposition of functionality upon various logic blocks or circuit elements. Thus, it is to be understood that the architectures depicted herein are merely exemplary, and that in fact many other architectures can be implemented which perform the same function.

Any arrangement of components to achieve the same functionality is effectively "associated" such that the desired functionality is achieved. Thus, any two components herein combined to achieve a particular functionality can be seen as "associated with" each other such that the desired functionality is achieved, irrespective of architectures or intermedial components. Likewise, any two components so associated can also be viewed as being "operably connected," or "operably coupled," to each other to achieve the desired functionality.

Furthermore, those skilled in the art will recognize that boundaries between the above described operations are merely illustrative. The multiple operations may be combined into a single operation, the single operation may be distributed among additional operations, and the operations may be performed at least partially overlapping in time. Moreover, alternative embodiments may include multiple instances of a particular operation, and the order of operations may be altered in various other embodiments.

Other modifications, variations, and alternatives are possible. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense.

In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of other elements or steps than those listed in a claim. Furthermore, the terms "a" or "an," as used herein, are defined as one or more than one. Moreover, the use of introduced phrases such as "at least one" and "one or more" in a claim should not be construed to imply that the introduction of another claim element by the indefinite articles "a" or "an" limits any particular claim containing such introduced claim element to inventions containing only one such element, even if the same claim includes the introduced phrases "one or more" or "at least one" and indefinite articles such as "a" or "an". The same is true for the use of definite articles. Unless otherwise indicated, terms such as "first" and "second" are used to arbitrarily distinguish between the elements such terms describe. Thus, these terms are not necessarily intended to indicate temporal or other prioritization of such elements. The mere fact that certain measures are recited in mutually different claims does not indicate that a combination of these measures cannot be used to advantage.

Claims

1. A method for contouring a medical image in a contouring system, comprising the steps of:

Providing at least one medical image to be outlined by a user;

determining that a user has initiated contouring of a structure on the medical image;

The contour drawing system determines a structure being contoured;

in response to a determination of a structure being outlined, the system displays a guideline for the profiling of the determined structure;

and drawing the outline of the determined structure according to the displayed guidance.

2. A method for reviewing contours on a contoured medical image, comprising the steps of:

providing at least one medical image annotated with one or more pre-existing contours;

displaying the at least one medical image annotated with one or more pre-existing contours;

determining that the user has initiated review of a pre-existing contour on the medical image;

determining a structure associated with a pre-existing profile that the user is reviewing;

In response to the determination of the structure, a guidance for contouring of the determined structure is displayed.

3. The method of claim 2, further comprising the step of completing review of the pre-existing profile or editing the pre-existing profile in accordance with the displayed guidance.

4. A method as claimed in claim 3, wherein completing the review of the pre-existing profile comprises providing feedback about the profile.

5. The method of claim 4, wherein the feedback is provided as comments or notes about the outline.

6. A method as claimed in any preceding claim, further comprising the step of outputting an inspected profile or a user initiated profile associated with the image.

7. A method as claimed in any preceding claim, wherein when a user starts to draw or edit a contour of a structure on the at least one medical image, a determination is made that the user has initiated contour drawing or review of the contour.

8. The method of claim 2, wherein the determination of the pre-existing contour to be edited or reviewed is accomplished using eye tracking hardware and/or software to identify the location at which the user gazes at the at least one medical image.

9. A method according to any preceding claim, wherein the guidance for contouring is at least one of: a clinical guideline for contouring the determined structure or at least one atlas image displaying at least one example contour of the determined structure.

10. A method as claimed in any preceding claim, wherein the determination of the structure being examined or profiled is made by finding a mapping between the at least one medical image and at least one atlas image.

11. The method of claim 10, wherein the mapping is accomplished using image registration.

12. The method of claim 11, wherein the image registration comprises one or more of: deformable registration, affine registration, or rigid registration.

13. A method as claimed in any preceding claim, further comprising using machine learning to predict the structure being profiled.

14. The method of claim 13, wherein the determination of the structure being inspected or profiled is made by comparison to a profile generated using machine-learning based automatic profiling.

15. The method of claim 13, wherein the machine learning based automatic contouring uses one of: convolutional neural networks, random forest methods, and support vector machines.

16. The method of any preceding claim, wherein the determination of the structure on the medical image being inspected or outlined is done using eye tracking hardware and/or software to identify the location at which the user gazes at the at least one medical image.

17. A method as claimed in any preceding claim when dependent on claim 1, wherein the contouring of the determined structure by the user is one of manual, semi-automatic or automatic contouring.

18. The method of claim 17, wherein the contouring system uses the contour name of the structure being contoured to determine the structure being contoured or inspected.

19. The method of claim 18, wherein the profile name is provided to a lookup table such that the name is associated with a standardized naming convention.

20. A method as claimed in any preceding claim when dependent on claim 1, wherein the steps of the method are repeated such that a plurality of structures in the at least one medical image are profiled.

21. A method as claimed in any preceding claim when dependent on claim 2, wherein the pre-existing contours displayed to the user are manual contours from a previous contouring session or automatically generated contours.

22. A method as claimed in any preceding claim when dependent on claim 2, wherein the steps of the method are repeated such that a plurality of pre-existing contours on the medical image are reviewed.

23. The method of any preceding claim, wherein the at least one medical image is one of a CT scan, an MRI scan, a PET scan, a SPECT scan, or an ultrasound scan.

24. The method of any preceding claim, wherein the at least one medical image is one of a 2D image, a 3D image or a time series of medical images.

25. A system for contouring at least one medical image, comprising:

a display for displaying at least one medical image to be outlined by a user;

a processor for determining that a user has initiated contouring of a structure on the medical image;

the processor determining a structure being profiled;

In response to the determination of the structure, the system displays to a user guidance for the contouring of the determined structure; such that the determined structure may be contoured in accordance with the displayed guidance.

26. A system for reviewing or editing contours on a contoured medical image, comprising:

Providing at least one medical image annotated with a pre-existing contour;

a display for displaying at least one medical image annotated with a pre-existing contour;

a processor for determining that a user has initiated an inspection of a pre-existing contour on an image, the processor determining a structure of the pre-existing contour being inspected;

In response to the determination of the structure, the system displays a guide for the contouring of the determined structure; to allow for the completion of contour review or editing of existing contours in accordance with the displayed guidance.