WO2011063529A1

WO2011063529A1 - Verification tool and method

Info

Publication number: WO2011063529A1
Application number: PCT/CA2010/001897
Authority: WO
Inventors: David Koff; Nadine Koff; Greg Butler; Ian Maynard; Taras Vasylenko; Alex Pakka; Paul Pierre; Oleg Ovalchuk
Original assignee: Real Time Radiology Inc.
Priority date: 2009-11-25
Filing date: 2010-11-25
Publication date: 2011-06-03

Abstract

A verification tool for medical diagnoses enables and automates an objective blind peer review of diagnostic assessment and arbitration diagnostic assessment of medical images. The verification tool executes a method of identifying a reviewer (604); assigning the medical diagnostic image to the reviewer (612); receiving a second report from the reviewer (614); displaying the first report to the reviewer (616); and receiving a discrepancy analysis (618) from the reviewer based on comparison of the first and second reports. The discrepancy analysis is queried (620). The first report is issued if no discrepancy exists (624), If a discrepancy exists, the method further comprises the steps of identifying an arbitrator (704); displaying the medical diagnostic image, the first report and the second report to the arbitrator (708); and receiving an arbitration report from the arbitrator containing a discrepancy indicator. The first report and the arbitration report are issued if the discrepancy indicator agrees with the first report (71G). The second report and the arbitration report are issued if it agrees with the second report (710). The arbitration report is issued if it disagrees with both reports (722).

Description

VERIFICATION TOOL AND METHOD

Field of the Invention

[0001] The present invention relates generally to diagnostic imaging management and in particular to a system and method for verifying interpretations of diagnostic images.

Related US Application

[0002 ] This application claims the benefit of U.S. Provisional application No. 61/264,394 assigned to Real Time Radiology, assignee of the subject application, the contents of which are incorporated by reference.

Background of the Invention

[0003] Governmental and other medical oversight agencies mandate the review of diagnostic medical Imaging and diagnostic interpretations thereof in order to maintain quality control and integrity of diagnostic interpretations. In order to meet this mandate, diagnostic interpretations are reviewed with three objectives in mind: (1) to determine if current clinical findings differ significantly from findings documented in the original report, and if so, to immediately communicate to the patient's referring physician whenever clinical discrepancies are uncovered, (2) to positively report on instances of no clinically significant discrepancy so that there are definitive findings with respect to each exam and (3) to easily summarize and analyze the extent of differences across a body of findings.

[0004] Given the clinical and potential legal consequences of an incorrect diagnostic interpretation, the review process for diagnostic interpretation must meet rigorous standards, including the review of interpretations which is "blind" to the original radiologist's report, comparison of the original findings against the review findings and assessment of discrepancies, and provision for an arbitration review to arrive at a confirmed diagnosis.

fOOOSj Coordinating software products for diagnostic medical imaging and diagnostic interpreting services are known. Such coordinating software products are capahle of permitting access to pre-arranged schedules of available resources with workload to be done. These products permit access to pre-arranged schedules of available resources with job orders to be done. Image producers use such software products to allot image studies to prospective medical specialists. An example of such scheduling software is disclosed in U.S. Provisional Patent Application No. 61/284394, assigned to Real time Radiology, assignee of the subject application, the contents of which are incorporated by reference.

[0006] While the above-described diagnostic imaging management can be useful, improvements are of course desirable. For example, typically, such diagnostic imaging distribution and management tools do not offer any methods for verification of results of interpretation of diagnostic Imaging, do not offer methods to classify the differences in terms that are medically meaningful and do not offer methods to manage multi-stage verification steps.

[0007] It is therefore an object of the present invention to provide a system and method for the verification of interpretation of diagnostic images using diagnostic imaging management that overcomes the above deficiencies, _N

[0008] It is a further objective to provide a quality assurance tool for the verification ot medical diagnoses which enables and automates objective peer review and arbitration of diagnostic findings which is applicable to multiple clinical disciplines including radiology, cardiology, pathology, and dermatology.

[0009] More particularly it is an objective of the present invention to provide a system for scheduling of peer review and arbitration of interpretation of diagnostic images which will meet the objectives of conducting a review of interpretations which is "blind" to the original report, comparison of the original findings against the review findings and assessment of discrepancies, and provision for an arbitration review to arrive at a confirmed diagnosis.

Summary of the Invention

[00010] According to one aspect there is provided a method of verifying a first report of results from a first reader of a previously interpreted medical diagnostic image which medical diagnostic image was created by an image producer. The method comprises the steps of identifying a reviewer; displaying the medical diagnostic image to the reviewer; receiving a second report from the reviewer;

displaying the first report to the reviewer; and receiving a discrepancy analysis from the reviewer based on a comparison of the first report and the second report. The next step is querying the discrepancy analysis to determine whether a discrepancy exists between the first report and the second report. The first report is issued to the image producer in the event that no discrepancy exists. In the event that a discrepancy exists, the method further comprises the steps of identifying an arbitrator; displaying the medical diagnostic image, the first report and the second report to trie arbitrator; and receiving an arbitration report from the arbitrator containing a discrepancy indicator. The first report and the arbitration report are issued to the image producer in the event that the discrepancy indicator is an agreement with the first report. The second report and the arbitration report are issued to the image producer in the event that the discrepancy indicator is in agreement with the second report. The arbitration report is issued to the image producer in the event that the discrepancy indicator is in disagreement with the first report and is in disagreement with the second report.

[00011] The identifying of a reviewer further comprises the steps of

selecting from a weighted list of credentialed image readers one credentialed image reader; querying whether the selected one credentialed image reader has seen said medical diagnostic image; and in the event that the selected one credentialed image reader has not seen said diagnostic image, qualifying said selected one credentialed image reader as a reviewer.

[00012] The identifying of an arbitrator further comprises the steps of selecting from a weighted list of credentialed image readers one credentialed image reader; querying whether the selected one credentialed image reader has seen said medical diagnostic image; and in the event that the selected one credentialed image reader has not seen said diagnostic image, qualifying said selected one credentialed image reader as an arbitrator.

Brief Description of the Drawings

[00013] Embodiments will now be described more fully with reference to the accompanying drawings in which:

[000.14] Figure 1 is an exemplary block diagram of a system for coordinating and distributing service requests to service providers;

[00015] Figure 2 is a block diagram of an embodiment of the system of Figure

1;

[Oooiftj Hgure 3 is an entity relationship diagram of an embodiment of the components of Figure 1 ;

[00017] Figure 4 is an exemplary flow diagram showing the method used by the components of Figure 2: [00018] Figure 5 is a schematic diagram showing an exemplary relationship between the messaging layer and the components of the system of Figures 1 and 3;

[00019] Figure 6 is a flow chart showing the process of reviewing and verifying by the system of Figure 2;

[00020] Figure 7 is a flow chart showing the process of arbitration by the system of Figure 2;

[00021] Figure 8 is a screenshot of an exemplary verification process of Figure 6; and

[00022] Figure 9 is a screenshot of an exemplary arbitration process of Figure 7.

Detailed Description of the Embodiments

[00023] Turning to Figure 1 , a system for coordinating and distributing medical imaging service requests from image producers to medical diagnostic imaging specialist or image reading service providers is shown and is generally identified by reference numeral 100. The system in this embodiment shows a producer profile engine 102 accessing a database 105. The database 105 stores an image producer profile 106, an image reader profile 108, study daLa 110, a schedule 112. A workflow distribution engine 1 14 also accesses the database 105 and a reporting tool 116. A web-based user interface 118 also accesses the database 105.

[00024] In the embodiment of the system shown in Figure 1 , image producers are able to access a producer profile 106 through the Web UM 18. Image producers may include but are not limited to hospitals, clinics, laboratories, and other medical facilities capable of producing medical diagnostic images for interpretation by an image reading service providers, image reading service providers are usually radiologists but may include any medical specialist trained to interpret a medical image. The image reading service providers or image readers, in turn, access an image reader profile 108 through the Web Ul 118.

[00025] The producer profile engine 102 allows the image producer to store study data 110 on the databaee 105 for a particular study. A gtudy is a diagnostic medical image produced by an image producer to be read and interpreted by a medical specialist or image reading service provider. Study data 110 may include study history and study reports. Study history shows the image producer when a particular study was created, accessed, worked on, and by which medical specialist- Study reports are the written results of the study interpretation by an image reading service provider. Producers that supply studies for review have a "processing mode" attribute which defines workflow type: as "Normal" (or property is missed) for current study image viewing workflow; as "Review" for studies which are to be processed for review; and, as "Arbitration" for studies that should be processed for arbitration.

[00026] The Web Ul (118) allows image producers to provide information in the image producer profile 106 about their facilities, what modalities or imaging technologies they expect to be interpreted, a list of accredited image readers, and how they would like to be notified. An accredited image reader is an image reader whom the image producer has selected as an allowed image reader for that image producer's studies. An image producer profile 106 may also contain information such as but not limited to an identification number, location or physical address, volume of studies the image producer expects to send.

[00027] The Web Ul 118 allows image producers to specify the time periods for which they require image reader coverage and the volume of studies they expect to produce during this time period and store these requirements in the Schedule 112.

[00028] The image reader profile Web UM 18 allows image readers to schedule their workloads. The image reader profile Web Ul 118 also allows image readers to provide information in their image reader profile 108 about their expected capacity of work, modalities and subspecialties in which they specialize, image producers for whom they do not agree to provide service, and how they would like to be notified. An image reader's default expected capacity of work is specified in terms of number of units per hour for each specific study modality. For example, the value of zero for a specific study modality indicates that the image reader cannot provide interpretations for that modality,

[00029] The Web Ul 118 allows image reader to specify the time periods they are available to provide coverage, the volume of studies they are capable of interpreting and the particular image producers for which they are willing to provide coverage and store this information in the schedule 112.

[00030] The workload distribution engine 114 accesses study data 110 from the database and matches the availability of image readers with image producers' schedules 112 using capacity and credentialing rules, and thereby matches imaging producer coverage requirements to image reading service provider availability,

[00031] The Web Ul 118 displays lists of studies received on the system. Such lists include studies received but not interpreted, studies assigned to specific image readers, studies interpreted and studies cancelled. The Web Ul also provide functions to launch the Reporting Tool 116 for a particular study, to cancel a study, reserve a study or manually assign a study to a specific image reader. In addition, the Web Ul 118 provides the ability to display the results of a study interpretation created in the Reporting Tool 116

[00032] Figure 2 shows an exemplary schematic diagram of an embodiment of the system for distributing medical image reading service requests from image producers to image reading service providers. In Figure 2, third party industry standard picture archive and communications system (PACS) applications 202 communicate with a third party communication module 204, which is able to accept and discern different kinds of image studies. The third party communication module 204 is also able to accept other third party studies using other kinds of

communication protocols such as HL7. In this example, the third party

communication module 204 includes to receive and store DICOM studies, create notifications that a study has been received, issue DICOM study move commands and issue DICOM study queries. In this example, the third party communications module 204 includes a DICOM integration module 206 that may include but is not limited to such modules as a study notification module 208, a study routing module 210 and a prefetch module 212. The study notification module 208 creates messages upon receipt of a study and transmits those messages onto the messaging layer 214. The study routing module 210 receives study routing instruction messages from the messaging layer 210 and instructs the Third Party Communication module 204 to move the study to the instructed destination. The prefetch module 212 receives messages from the message Layer 214 on receipt of a study to query an industry standard Third Party Application (PACS) 204 about the availability of other related studies and retrieve studies which are relevant to the receive study.

[00033] In another embodiment, the third party communication module 204 communicates with a messaging layer 214 directly, bypassing the DICOM Integration Module 206. The messaging layer 214 receives information about a study from the third party communication module 204, and delivers a notification message that a study is available to the workload distribution engine 114.

[00034] The notification message typically contains information pulled from the DICOM headers of the study, which originates within a DICOM integration module 206. This information in the notification message may include but is not limited to the patient's ID, patient name, the image producer ID. the image producer's study identlfler, study creation date, ordering department, and number of images in the study.

[00035] In turn, the messaging layer 214 interacts with the workload distribution engine 114. In this example, the workload distribution engine 114 has a study profiling rules engine 216, a study routing module 218, a supply and demand planning module 220, a study distribution module 222, a results management module 224, and messaging delivery handler module 226. The study profiling rules engine 216 performs actions on the received study notifications from the messaging layer 214. Once these actions are performed, the study profiling rules engine 216 is able to determine if a particular study may be routed, what the priority of the study should be and if the study should be considered as an study to be interpreted (work item), a study for verification or a study for reference.

[U0036| The study routing module 218 uses information received as part of the study, such as producer ID and in certain embodiments the assigned image reader as determined by the Study Distribution Module 222 to create instruction messages to be passed via the Message Layer 214 to cause the Third Party Communications module 204 to move the study to a specific Third Party Application (PACS) 202 to be displayed. The supply and demand planning module 220 consumes Imaging producer coverage request information from the Schedule 112, image reader coverage availability information from the schedule 112, image reader credential information from the producer and reader profiles and matches producer requests to image reader availability and stores the agreement for service for each producer and reader intersection. The supply and demand planning module 220 create alerts through the scheduling module Ul 240 if the declared available image reader capacity Is Insufficient to provide interpretations tor the declared producer requests,

[00037] At specified intervals, the supply and demand planning module 220 provides notifications to image readers of upcoming coverage commitments. In one embodiment, these notrfications at specified intervals may be fulfilled through Internet calendar messages transmitted using the message delivery handler module 226 via e-mail, using the image reader's e-mail address stored in the image reader's profile 108. In an alternative embodiment, upcoming coverage commitment notifications are sent through periodic e-mail messages via the message delivery handler module 226 showing rolling upcoming coverage commitments in accordance with the periodic notification values stored in an image reader's profile 108 stored on the database 105. [00038] Wnen an order from an image producer is created, the order is passed to the study distribution module 222. The study distribution module 222 provides study distribution from multiple study image producers to multiple study consumers (readers). This distribution is based on a set of rules and weighting decisions such as the amount of outstanding work for each image reader, the priority of the study and service operator defined adjustment values.

[00039] The workload distribution engine 114 corresponds with a web user interface 238 which image producers, image readers and system administrators can access. The web user interface 238 may include a scheduling module Ul 240, the image producer profile Ul 241 , the image reader profile Ul 243 ,a reporting module Ul 242, a system admin Ul module, a study list Ul module 245, a verification Ut module 246 and a results tracking Ul module 247.

[00040] The scheduling module user interface provides a calendar oriented view for image producers to declare their coverage needs for specific time periods or for multiple recurring time periods. Image producers can change the expected study volume information stored in their profile 106 for each specific coverage request. A similar calendar oriented view is available for image readers to declare their coverage availability for specific periods or multiple recurrent periods. If the supply and demand planning module 220 determines that the sum of the available supply of image reader capacity is greater than or equal to the sum of the declared image producer demand, the user interface changes colour for the time period to indicate adequate coverage.

[00041] The image producer profile Ul 241 and the image reader profile Ul 243 provide screens for producers and reader to maintain their producer profile 326 and Image reader profile 324 information such as default volumes, contact information and in the case of the reader profile, subspecialty areas and excluded producer sites. The information entered into the user interface is stored in the producer profile 106 and image reader profile 108 respectively.

[00042] The study list Ul 245 provides lists and functions to display the studies available on the system for reporting, the status of .studies, the assignment history of a study, to which reader the study has been assigned and depending on the permissions of the user, the ability display the results of an interpretation or cancel a previously study designated for interpretation. Studies in the study list module are retrieved- from study data 110 [00043] The system admin Ul 244 provides input methods to create links between image producers and image reader e.g, accreditations 328, define producer profile engine 102 rules and parameters, define study distribution module 222 rules and parameters and create image producers ID'S 322 and image reader ID's 3?0.

[00044] The reporting module Ul 242 is launched from the study list Ul 245 to provide an input method to document the results of a study interpretation and to launch the appropriate diagnostic imaging viewer to display the image data associated with the study. Input forms can be different depending on data received in the study available message such as study description, modality and producer ID.

[00045] The verification Ul 246 module is launched from the study module for studies when the producer profile is set to review mode or arbitration mode. The verification module displays the original radiologist report for verification and depending on the current study status displays either an initial review quality classification input form Figure 6 or an arbitration review quality assurance input form Figure 7. The verification UI may also launch the reporting module Ul 242 depending on the verification workflow as defined in the Study Profile Rules Engine 216.

[00046] In a verification workflow, when the producer profile engine 102 is set to review mode or arbitration mode the Results Tracking Ul 247 displays the list of studies classified as requiring verification and allows a producer to monitor the outcomes of the verification interpretations and document the results of

communication of these outcomes. The results tracking Ul allows the image producer to document if the results have been communicated to referring physician and if patient follow-up has been marie Documentation includes time and date of the communications, transmission of results, person contacted and special notes of the communication. There can be <eiy or many communication documents for a particular result,

[00047] In one embodiment, information is acquired, transferred, or modified prior to the creation of an order for study by an image reader. The study profiling rules engine 216 references study profiling information stored on the database 105 for each image producer ID to determine rules and actions applicable for study available notifications received from that image producer. Rules and actions may be used singularly or in combination. In another embodiment, the study profiling rules engine 216 will retrieve the image producer ID information from the received study notification message sent from the messaging layer 214 and generate a study forward message to the DICOM integration module 206, which will cause the third party communications module 204 to forward the received study to a target interpretation PACS system defined in the producer profile.

[00048] In a further embodiment, the study profiling rules engine 216 references a list of original reports, supplied by image producer and imported into the Study Data 110 using standard database tools, to determine if the received study is to be considered a study for verification.

[00049] The study distribution module 222 matches image reader's availability with image producer schedules, credentiaiing, and workload rules to assign studies to image readers. When a study arrival notification has been received and a study enters the workload distribution engine 114, the study distribution module 222 assigns complexities to the study, sumo the complexities, and compares the amount of work assigned to each image reader. The complexities are derived from a combination of study fee revenue and estimated time to complete the study. An individual study's complexities may also be based on information derived from the received study, study description, study modality, and the body part that has been imaged in the study

[00050] In another embodiment, the DICOM integration module 206 initiates a query to identify the availability οΓ related studies stored In Third Party Application (PACS) 202. If there is a related study, the Pre-fetch module 212 will retrieve patient ID, study description, study modality, and sending PACS system information from the received study notification message as well as target PACS system information stored in the image producer profile 106. The study routing module 216 sends a command message to the pre-fetch module 212 of the DICOM Integration Module 206 to cause the third party communication 204 to issue a constrained DICOM Query message to the target PACS system. The received information is used by the prefetch module 212 to cause the third party communication 204 to issue a DICOM move command to the producer PACS. The producer PACS then sends the identified studies to a system defined related studies DICOM Application Entity Title. Upon receipt of each series of each related study, the DICOM integration module 206 will generate a "Study Available Event" and transmit this information using the messaging layer 214 to the study profiling rules engine 216 including the related studies logical address information (e.g. DICOM Application Entity Title). The study profiling rules engine 216 will use the related studies logical address information to determine that the retrieved study is for reference only and not create an order for interpretation. The Study profiling rules engine 216 will issue a "Study Forward" message tnrough the messaging layer 214 to the study routing module 210 which will cause the third party communication 204 to forward the study to a target

interpretation PACS system defined in the image producer profile stored data 106.

[00051] In a further embodiment, Instead of the DICOM integration module 206 soliciting related studies from the image producer, the image producer may send studies for interpretation and related Information together unsolicited. In this instance, the study profiling rules engine 216 references the time the study was created from the notification message to determine which of the studies sent is to be interpreted and which are the related information. The related information is typically studies which have been previously interpreted prior to the study to be read and interpreted. The etudy profiling rules engine 21 β will set the study status 332 to the value of "cancelled" for the related information. This can be manually converted to a valid study through a system admin Ul 244 by a system administrator if the study profiling rule incorrectly profiled the study. In turn, the system administrator can also manually cancel studies deemed to be valid studies if advised to do so by the image producer.

[00052] In a still further embodiment, the study profiling rules engine 216 may use the image producer ID in the image producer profile 106, the time the Study was received by the system, study priority, and the department of the image producer which has ordered the study to apply study priority rules to elevate or lower the priority of the study. This is discussed in detail in Figure 4.

[00053] Credentialing is the process of indicating that image readers can provide coverage, view study information and create interpretation reports for particular image producers. Associations are defined by the system administrator in the system admin Ul 244 and stored in the Accreditation table 328 in the database 105.

[00054] Figure 3 is an entity relationship diagram illustrating the relationship between components of the system in Figure 1. The Image Reader 320 has a unique ID. Associated to the Image Reader 320 is an Image Reader Profile 324 that includes suqh information as full name, contact details, skills and volume information. Image Reader(s) 320 creates availability commitments in the Schedule entity 330. Image Headers 320 are assigned to interpret a Study 332. A Study 332 has attributes such as a unique study identifier, study data such as producer ID, patient information, modality, study time and study description and workflow data such as indication to be interpreted (work item), status and priority. A Report 334 is created for Study 332 that is a work item. A Report 334 can have information such as signing image reader, version and specific interpretation content. Image Producer 322 has a unique ID. Associated to the Image Producer 322 is an Image Producer Profile 326 that includes such information as contact details, expected volume, and rules to be applied by the Producer Profile Engine 102 on receipt of a study from producer ID. Image Producer 322 can create coverage requests in the Scneduie 330. Image Producers 322 create Study(s) 332 to be transmitted to the system.

[00055] Figure 4 is a detailed flow chart of how the study distribution module 222 of the workload distribution engine 114 distributes studies. In step 402, a study is received by the workload distribution engine 114. In step 404, the workload distribution engine 114 checks the 3tudy priority as stored in the Study information 332 after applying rules in the producer profile engine 102.A priority study is a study which must be completed immediately. It will take precedence over other studies due to the urgency the results from the study are needed. In the producer profile engine 102, study priority may be determined by the time at which the study arrives. For example, an image producer may request that if a study is sent hy the image producer's facilities after midnight, the study is deemed a priority. Priority may also be determined by which department or which doctor requires the Study. For example, if the received study comes from an image producer's emergency department, it may be deemed a priority. An alternative way of determining priority may be based on the modality and body part with which the study is concerned. For example, an image producer may require that all magnetic resonance imaging (MRI) scans of the brain be deemed as priority studios.

[00056] In step 406, if the study is a high priority, the workload distribution engine 114 determines it the "hot seat" image readers are available to analyze and interpret the study. A hot seat image reader is an image reader who was the earliest to subscribe to coverage in the supply and demand planning module 220 for a specific image producer and who is logged into the service at the time that a priority study is received. For example, a first image reader has subscribed to work on studies from a first image producer on a particular day and time period and a second image reader has subscribed to the first and a second image producer on the same day and time period. If both the first and second image readers are logged in, then the first image reader will be the hot seat image reader for the first image producer while the second image reader is the hot seat image reader for the second image producer. The first reader will receive a study complexity adjustment to be applied to his complexity threshold for being hot seat reader. The first reader wilt receive an additional adjustment by the study distribution module for logging in at the specified time. If the first image reader is not logged on, then the second image reader is the hot seat image reader for both imago producers and the first reader will not be assigned adjustments to his complexity threshold.

[00057] in step 408, if a hot seat image reader is available and the study is high priority, the workload distribution engine 114 assigns the study without regard for readers' outstanding complexity and complexity thresholds. In step 410, if a hot seat image reader is not available, the workload distribution engine 114 determines if there are any active accredited image readers logged on to the system. If there is an active image reader logged on to the system, then the study distribution module 222 will assign the study to that active image reader in step 408. If no active image reader is available to read the priority study, then the study is held in a "free" state until an active image reader is available.

[00058] If in step 404, the study is not a priority study, then in step 414, the study distribution module 222 will determine if the study is reserved for a particular image reader. If the study is reserved for a particular image reader, then the workload distribution engine 114 will assign the study to that particular image reader in step 408.

[00059] A study is reserved for a particular image reader in one of several ways. In one method a study may become reserved for a particular image reader if a particular image reader has previously viewed medical images from a certain patient, a reservation may bo mode in automatically by the study distribution module 222 that any future image studies of that patient, within a defined time period should be sent to that image reader regardless of whether the image reader's thresholds have been met. When a new study arrives that is of a previous patient, as indicated in the study available message the study distribution module 222 matches the patient ID to previous studies and routes the new study to the image reader listed in the previous study. In another example, an image producer may consult with an image reader prior to performing the imaging examination. In Ihis oase, the image producer identifies the hot seat reader and his contact information through the Study List Ul 245 of the Web Ul 118. The image producer contacts the hot seat reader and communicates the patient ID of the patient that wilt be imaged to the hot seat radiologist. The hot seat reader may enter the patient ID into the image reader profile Ul 243 via the Web Ul 118. The study distribution module creates a reservatlon token for the patient and matches the patient token to the patient information received in the study notification message when the study eventually arrives. If the study is not reserved for a particular image reader, then in step 416, the workload distribution engine 114 loads a list of accredited image readers for the image producer that created the study. In step 418, the workload distribution engine 114 selects the fir st image reader In the list and then In step 420 determines if the image reader is able to read the study. An image reader is able to read the study if the image reader's profile 108 indicates that the image reader has experience in the modality of that study. An image reader's ability to read a study may also be affected by the image reader's subspecialties. For example if an image reader has a subspecialty in paediatric imaging, this would affect the image reader's ability to read a study of a geriatric patient. If the image reader is able to read the study, then in step 422, the study distribution module 222 applies a modality complexity adjustment to the study to define the studies complexity. Complexity adjustment is identified by apply the complexity weight associated with the any combination of the modality, body part examined and study description information included in the study available message, if in step 420, the image reader is not able to read the study, then that image reader will be removed from the list of image leaders in step 424.

[00060] In step 426, the study distribution module 222 determines if the image reader has an active schedule. An image reader has an active schedule if the image reader has subscribed to the image producer's coverage request for the allotted time slot that the study has been received. If the image reader has an active schedule, then in step 428, the study distribution module 222 applies o schedule adjustment to the image reader's workload outstanding workload complexity. If the image reader has been logged in longer than other readers and additional adjustment is applied to the reader's outstanding workload complexity. If image reader was the first to subscribe to the image producer's coverage request then a third adjustment may be applied. In step 430, the study distribution module 222 moves to the next image reader in the list and performs steps 420 to 428 on the next image reader.

[00061] Steps 420 to 428 repeat until in step 430 the workload distribution engine 114 comes to the end of the accredited image reader list. In step 432, the workload distribution engine 114 reorders the accredited image reader list in descending order of outstanding workload complexity. In step 434, the study distribution module 222 determines if the study has any pre-existing rules. These rules may include but are not limited to the settings in an image producer profile 106, the settings In the Image reader profile 108, the time interval, and study data. An image producer profile 106 may have a rule set for example, that all studies of a particular modality between certain hours are to be assigned to specific image readers regardless of their online status. An image reader profile 108 may have a rule set that declares the image reader's subspecialty or what body part images an image reader will accept. A time interval rule may include but Is not limited to such rules as studies between certain hours are to be read by a certain group of image readers. A study data rule may include but is not limited to such rules as studies of a particular modality may only be assigned to certain readers,

[00062] If rules exist, then in step 436, the study distribution module 222 applies the rules. In step 438, the study distribution module 222 removes candidate readers who do not fit within the rules, and then reorders the list with the remaining candidates in reverse or according to outstanding complexity. In step 440, the workload distribution engine 114 assigns the study to the image reader at the top of the list or the image reader with the least adjusted outstanding complexity that is within the outstanding complexity threshold. If rules do not exist in step 436, then the workload distribution engine 114 assigns the study to the image reader at the top of the list or the image reader with the least adjusted outstanding complexity that is within the outstanding complexity threshold in step 440. If no reader has an adjusted outstanding complexity below the outstanding complexity threshold set within the study distribution module, then the study is not assigned and the distribution service 222 begins again at step 416

[00063] Figure 5 is a sequence diagram depicting the message flow between image producers 301 Third Party Application 202 and image readers Reporting Module Ul 242 using Ihe system in Figure Z. Image producer 301 TTiird Party Application 202 sends a study to the Third Party Communications module 204 as indicated by arrow 602. The Third Party Communications module 204 sends a study arrived notification to the workload distribution engine 114 as indicated by arrow 604. The Third Party Communications module 204 sends a study available notification to the workload distribution engine ae indicated by arrow 60δ. The study available notification contains information for interpretation to the workload distribution engine 114 for determining to which image reader the study should be assigned. This information may include but is not limited to patient name, patient ID, study date, study description, body part examined, image producer name, image producer ID, referring physician. In arrow 610. a study forward notification is sent from the Third Party Communications module 204 to the Third Party Application 202. The study forward notification instructs the image producer interface to forward the study to a specific destination DICOM Application Entity Title. In arrow 612, the Third Party Communications Module 204 sends a query for prior relevant studies to th© Third Party Application 202 and instructs the Third Party Application 202 to send the resultant studies to a specific DICOM Application Entity Title. I he I hird Party Communications module 204 sends a prior study available event to the workload distribution engine 114 as indicated by arrow 614. This event passes information about regarding whether the study is a work item or for reference. Work items are studies which require an interpretation report to be created by an image reader. Reference items arc other imaging atudie3 for the same patient, similar bqdy parts and same or related modalities. Reference items are used by the image reader to determine if an item identified on the imaging study to be interpreted existed prior to the current study and if the identified item has changed in the interval between the acquisition of the reference study and the study to be interested. Reference items are commonly referred to as prior studies or simply Priors. The workload distribution engine 114 sends a set study priority command to the Study List Ul 245 (internally) as indicated in yrrow 616 or to the Third Party Application 202 as Indicated by arrow 616 as required. In this command, instructions are sent by the study profile engine 216 within the workload distribution engine 114 to set the study priority. This occurs when the study status is not initially sent with the study. The reporting module Ul 242 sends an image reader login event to the workload distribution engine 114 as indicated in arrow 618. A login event passes the login information of the image reader who has logged into the system, allowing the workload distribution engine 114 to know that that image reader is available to receive assignments. A study assignation command as demonstrated in arrow 620 is sent from the workload distribution engine 114 to the Third Party Application 202 if reporting is done external to the system or to the Study List Ul 242 if reporting is done internal to the system. This command indicates that the study should be available on a specific image reader's study list 242 or equivalent element of Third Party Application 202 system. A study retrieved event is sent from the reporting module Ul 242 to the workload distribution engine 114 as indicated in arrow 624. This event indicates that the study has been retrieved by the image reader for reading and interpretation. Arrow 626 indicates a study skipped event. This event signifies that the study has been closed but not interpreted The workload distribution engine 114 will consider the study available for reassignment. An on hold event Is sent from the reporting module Ul 242 to the workload distribution engine as indicated in arrow 628. This event informs the workload distribution engine 114 that the study does not have a final interpretation by the image reader, but the study is also not available for reassignment. A study read event is sent from the reporting module Ul 242 to the workload distribution engine 114 as indicated in arrow 630. This informs the workload distribution engine 114 that the interpretation of the study has been completed. Arrow 632 indicates a report delivery from the workload distribution engine 114 to the Third Party Communication 204. The report is further delivered to the image producer 301 Third Party Application 202 as indicated by arrow 634. An availability notification is sent from the reporting module UI242 to the workload distribution engine 114 as indicated by arrow 636, This notification informs the workload distribution engine 1 14 that the image reader continues to be available to interpret more studies. Arrow 638 indicates a logout event passed from the reporting module Ul 242 to the workload distribution engine 114. When the workload distribution engine 114 receives a notification that an image reader has logged out of the system, the workload distribution engine 114 will unassign studies and stop assigning now studies to that image reader, making those studies available to other logged in image readers.

[00064] Figure 6 is flow chart showing the process for reviewing and verifying of interpretation results. The first step is step is the receipt of a study to be reviewed 602. A study may enter this stage for multiple reasons. The original image viewing and assessment report created in the normal workflow by the system 100 and issued to the image producer may have been returned by the image producer for a review to double check the result. Alternatively, as part of the quality assurance function of the main system 100 studies are randomly selected for review as an integrity check on the image readers and the workflow process. A further alternative source from which review studies may be received is requests for review generated by the original study producer as part of a periodic quality assurance initiative. Once a study has been received for review 602, the review module then executes the task of finding a credentialed reader using the workload distribution engine 114 in step 604. 604. A potential reviewer Is then selected from the list of available readers. The potential reviewer is then filtered 606 to determine whether that person has already reviewed the particular study. If the person has previously viewed the study, a different potential reviewer is selected. The selection of the potential reviewer is then further flltered to determine whether his/her wortdoad is full 610. IF the selected potential reviewer has capacity to review the study, the study is assigned 612 to the reviewer to be viewed as a new study. This is a blind review, and the reviewer does not initially have access to the previous viewer's analysis of the study. The reviewer thon analyses the study and submits 614 a new report on the study using the reporting module ui 243. Once the new report has been submitted, the original report prepared by the previous viewer is displayed 616 to the reviewer. With the benefit of access to the previous viewer's report the reviewer then conducts an analysis of discrepancies between his/her report and that of the previous viewer. The reviewer then inputs 618 an indication of the level of discrepancy between the original report and the new report. The discrepancy input may be advantageously bo categorized on the following basis: No Discrepancy, meaning that the reviewer fully concurs with the original report; Partial discrepancy, meaning that there is no clinically significant difference which would in the opinion of the reviewer result in a change to patient management or well being; or Significant Discrepancy, meaning a clinically significant difference from the original report which would, in the opinion of the reviewer result in a change to patient management or well being. An input of Non- Diagnostic may also be input in situations where the study cannot be interpreted due to image quality. The review module notes the input and queries the input to determine 620 whether a discrepancy (whether partial or full), exists. If a discrepancy exists, the study is directed 622 to the Arbitration workflow 700 by sending a STUDY_AVAILABLE signal with additional fields including reported reader login, discrepancy type and comment to the study profiling rules engine 216. If there is no discrepancy, the reviewer's report is appended 624 to the study and transmitted to the Image producer as a completed review, and removed from the reviewer's work list.

[00065] Figure 7 is flow chart showing the process for arbitration of interpretation results embodied in the arbitration module workflow. The

STUDY_AVAtLABLE signal with additional fields including reported reader login, discrepancy type and comment is received 702 and logged. The study assignment process 704 is then commenced. A modified study assignment logic is employed to select a potential arbitration reader. The study cannot be assigned to the original reader or to the reviewer. Information about the reader and reviewer is avail be to the arbitration module from the STUDYJWAILABLE message. A filtering step 706 is conducted to eliminate as potential readers persons who have already viewed the study. A reader is selected who is credentialed to view the study and who has not previously reviewed the study, and it is placed into that arbitrator reader's work list. The reader then opens the assigned study in the Verification Ul 246, arbitration input form in step 712. When the arbitrator opens the assigned study in the Verification Ul, the study images along with the previous reports and the discrepancy information are displayed, A STUDY_RETREIVED message is generated to indicate that arbitration is underway, and a STUDY_SKIPPED message may be generated if the arbitrator closes the study without completing the arbitration. The arbitration module is targeted at finalizing whether a discrepancy call made at the review phase is truly a discrepancy. There can be several outcomes of the arbitration phase. The person assigned to conduct the arbitration mokes an assessment indication 710 and inputs details of the arbitration decision and any applicable comments regarding the diagnostic analysis of the images viewed. If the arbitrator indicates that the new review report is correct, a new report, created in the review phase is issued confirming the results of the review 712, and the discrepancy report is appended 714. If the arbitrator indicates that the original report was correct 416, the arbitrator creates a new report using the Report module Ul 242 indicating that the original report was correct and the report is issued 716 confirming the accuracy of the original report and a discrepancy report is appended 720. Alternatively, if the arbitrator has indicated that he/she does not agree with either the review report or with the original report, then a diagnostic arbitration report 722 is created containing the new diagnosis.

[00066] If there is a discrepancy, the image producer may contact a physician and log this contact attempt into the discrepancy journal or future reference using the results tracking ui 247. This step may occur multiple times with every attempt properly stored for audit purposes.

[00067] Although the embodiments described herein could apply to coordinating the distribution of medical diagnostic images for interpretation and verifying the results of interpretation, one of skill in the art will appreciate that these techniques could be applied to other interpretation and verifying disciplines.

Claims

What is claimed Is:

1. A method of verifying a first report of results from a first reader of a previously interpreted medical diagnostic image, said medical diagnostic image created by an image producer, the method comprising:

identifying a reviewer;

displaying the medical diagnostic image to the reviewer;

receiving a second report from the reviewer;

displaying the first report to the reviewer; and

receiving a discrepancy analysis from the reviewer based on a comparison of the first report and the second report.

2. The method of claim 1 , further comprising the step of querying the discrepancy analysis to determine whether a discrepancy exists between the first report and the second report.

3. The method of claim 2, further comprising the step of issuing the first report to the image producer in the event that no discrepancy exists.

4. The method of claim 2, wherein in the event that a discrepancy exists, the method further comprises:

identifying an arbitrator;

displaying the medical diagnostic image, the first report and the second report to the arbitrator; and

receiving an arbitration report from the arbitrator containing a discrepancy indicator.

5. The method of claim 4, further comprising the step of issuing the first report and the arbitration report to the image producer in the event that the discrepancy indicator is an agreement with the first report.

6. The method of claim 4, further comprising the step of issuing the second report and the arbitration report to the image producer in the event that the discrepancy indicator is an agreement with the second report.

7. The method of claim 4, further comprising the step of issuing the arbitration report to the image producer in the event that the discrepancy indicator is in disagreement with the first report and is in disagreement with the second report.

8. The method of claim 1 wherein the identifying of a reviewer further comprises: selecting from a weighted list of credentialed image readers one credentialed image reader;

querying whether the selected one credentialed image reader has seen said medical diagnostic image; and

in the event that the selected one credentialed image reader has not seen said diagnostic image, qualifying said selected one credentialed image reader as a reviewer.

9. The method of claim 4 wherein the identifying of an arbitrator further comprises:

Selecting from a weighted list of credentialed image readers one credentialed image reader;

querying whether the selected one credentialed Image reader nas seen said medical diagnostic image; and

in the event that the selected one credentialed image reader has not seen said diagnostic image, qualifying said selected one credentialed image reader as an arbitrator.