EP3998931A1 - Real-time deployment of machine learning systems - Google Patents
Real-time deployment of machine learning systemsInfo
- Publication number
- EP3998931A1 EP3998931A1 EP19937726.8A EP19937726A EP3998931A1 EP 3998931 A1 EP3998931 A1 EP 3998931A1 EP 19937726 A EP19937726 A EP 19937726A EP 3998931 A1 EP3998931 A1 EP 3998931A1
- Authority
- EP
- European Patent Office
- Prior art keywords
- display output
- video
- video frames
- objects
- frames
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
- A61B1/00002—Operational features of endoscopes
- A61B1/00004—Operational features of endoscopes characterised by electronic signal processing
- A61B1/00009—Operational features of endoscopes characterised by electronic signal processing of image signals during a use of endoscope
- A61B1/000094—Operational features of endoscopes characterised by electronic signal processing of image signals during a use of endoscope extracting biological structures
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
- A61B1/00002—Operational features of endoscopes
- A61B1/00004—Operational features of endoscopes characterised by electronic signal processing
- A61B1/00009—Operational features of endoscopes characterised by electronic signal processing of image signals during a use of endoscope
- A61B1/000096—Operational features of endoscopes characterised by electronic signal processing of image signals during a use of endoscope using artificial intelligence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
- G06F3/1423—Digital output to display device ; Cooperation and interconnection of the display device with other functional units controlling a plurality of local displays, e.g. CRT and flat panel display
- G06F3/1438—Digital output to display device ; Cooperation and interconnection of the display device with other functional units controlling a plurality of local displays, e.g. CRT and flat panel display using more than one graphics controller
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0012—Biomedical image inspection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/80—Recognising image objects characterised by unique random patterns
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H30/00—ICT specially adapted for the handling or processing of medical images
- G16H30/20—ICT specially adapted for the handling or processing of medical images for handling medical images, e.g. DICOM, HL7 or PACS
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H30/00—ICT specially adapted for the handling or processing of medical images
- G16H30/40—ICT specially adapted for the handling or processing of medical images for processing medical images, e.g. editing
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H40/00—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices
- G16H40/60—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices for the operation of medical equipment or devices
- G16H40/63—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices for the operation of medical equipment or devices for local operation
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H40/00—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices
- G16H40/60—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices for the operation of medical equipment or devices
- G16H40/67—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices for the operation of medical equipment or devices for remote operation
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
- G16H50/20—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for computer-aided diagnosis, e.g. based on medical expert systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10068—Endoscopic image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30004—Biomedical image processing
- G06T2207/30028—Colon; Small intestine
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V2201/00—Indexing scheme relating to image or video recognition or understanding
- G06V2201/03—Recognition of patterns in medical or anatomical images
- G06V2201/032—Recognition of patterns in medical or anatomical images of protuberances, polyps nodules, etc.
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G2360/00—Aspects of the architecture of display systems
- G09G2360/06—Use of more than one graphics processor to process data before displaying to one or more screens
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G2380/00—Specific applications
- G09G2380/08—Biomedical applications
Definitions
- Computer vision and machine learning networks are used to classify and identify objects in both digital images and videos.
- object classification a machine learning network is trained using a set of training data for classifying particular types of objects.
- object identification the machine learning network is used to recognize specific instances of one or more object types that may exist in an image.
- evaluating and inferring object types in real-time video data is often graphical processing unit (GPU) and central processing unit (CPU) intensive. Due to the intensive processing nature of real-time object inferencing of video data, in prior art embodiments of systems to infer images from video, a significant lag occurs in receiving, processing and rendering video output depicting imagery of the original video data and graphical indications of detected objects.
- the system provides a first window and second window displaying output data.
- the first window displays received video input data at a real-time frame rate of at least 50 frames per second.
- the second window displays graphical indications and/or textual information corresponding to objects detected in the video input data via an inferencing process using a machine learning network.
- the second window displays the graphical indications and/or textual information at a frame rate less than or equal to the frame rate of the first window.
- the second window overlays the first display window.
- the rendering of the first window and the inferencing of objects and subsequent display of graphical indications in the second window may be performed in a parallel or multi-threaded process.
- Parallel processing allows the system to display the received video input data at real-time frame rates of at least 50 frames per second, and also display the graphical indications in the second window at a frame rate that may be lower than or equal to the frame rate of the first window.
- one innovative aspect of the subject described in this specification can be embodied in systems, computer readable media, and methods that include operations for receiving video data from a video image capturing device.
- the received video data is converted into multiple video frames. These video frames are encoded into a particular color space format suitable for processing, as will be appreciated by a person skilled in the art such as those explained below.
- the system renders a first display output depicting imagery from the multiple encoded video frames.
- the system performs an inference on the video frames using a machine learning network in order to determine the occurrence of one or more objects in the video frames.
- the system renders a second display output depicting graphical information corresponding to the determined one or more objects from the multiple encoded video frames.
- the system then generates a composite display output including the imagery of the first display output overlaid with the graphical information of the second display output.
- FIG. 1 illustrates a block diagram of an example computer system utilized in real-time deployment of a machine learning network.
- FIG. 2 illustrates an example process for classification and identification of objects in video data.
- FIG. 3 illustrates a high-level block diagram of an example computer system utilized in real-time deployment of a machine learning network.
- FIG. 4 illustrates a low-level block diagram of an example computer system utilized in real-time deployment of a machine learning network.
- FIG. 5 illustrates a flowchart of an example process for evaluating video data to determine an external environmental state of a video monitoring device.
- FIG. 6 illustrates an example user interface of the computer system.
- FIG. 7 illustrates an example machine of the computer system.
- steps of the exemplary methods set forth in this exemplary patent can be performed in different orders than the order presented in this specification. Furthermore, some steps of the exemplary methods may be performed in parallel rather than being performed sequentially. Also, the steps of the exemplary methods may be performed in a network environment in which some steps are performed by different computers in the networked environment.
- a computer system may include a processor, a memory, and a non-transitory computer-readable medium.
- the memory and non-transitory medium may store instructions for performing methods and steps described herein.
- FIG. 1 illustrates a block diagram of an example computer system utilized in real-time deployment of a machine learning network.
- the computer system 100 includes a video conversion module 102, a computer vision module 104, a machine learning network module 106 and a graphical user interface module 108.
- the video conversion module 102 receives video data from an electronic device 110 capable of obtaining video data (e.g., a digital video camera).
- the video conversion module 102 converts the received video data into multiple video frames encoded into a particular color space format.
- the computer vision module 104 processes the multiple video frames and stores the multiple video frames onto a storage device 120.
- the storage device 120 may be a local or remote database or other type of storage cache.
- the computer vision module 104 converts the multiple video frames into an input to be processed by the machine learning network module 106.
- the machine learning network module 106 performs an inference on the multiple video frames and determines the occurrence of one or more object in the video frames.
- the graphical user interface module 108 renders a composite display output as a user interface 132 that includes a first and second display output.
- the first display output displays the encoded video frames at or near the frame rate of the encoded video frames.
- the second display output depicts graphical information corresponding to the determined one or more objects in the video frames.
- the second display output may depict the graphical information at a frame rate less than or equal to the frame rate of the first display output.
- the computer system 100 provides the composite display output to a monitoring device (e.g., an external and integral monitor.) The computer system 100 is further described in reference to Figure 7.
- FIG. 2 illustrates a flowchart of an example process for classifying and identifying objects in video data.
- the computer system 100 receives video data from a video capture device (block 210).
- the video capture device obtains real-time digital video using a digital video camera.
- the video capture device may be an external device connected to the computer system 100 either via a direct wired connection cable or via a wireless connection.
- the digital video camera may be configured as an integral component of the computer system 100.
- the real-time digital video is captured in high definition and has a resolution equal to or greater than 720p (1280x720 pixels).
- the computer system 100 converts the received video data into multiple video frames encoded into a particular color space format (block 220). Alternatively, the computer system 100 may receive digital video data that has already been encoded into a particular color space format. If so, the computer system 100 may forgo the conversion of the video data into an encoded color space format.
- An example suitable frame rate of the encoded video frames is a rate of 50 frames per second or greater.
- the computer system 100 renders a first display output depicting real-time imagery from the multiple encoded video frames (block 230).
- the system 100 may use a first group of one or more dedicated graphical processing units (GPUs) to render the first display output.
- the first display output displays the encoded video frames at or near the frame rate of the encoded video frames.
- the computer system 100 may display the video frames via the first display output at a high definition resolution of 1280x720 pixels or greater and at a high frame rate of 50 frames per second or higher.
- the computer system 100 performs an inference on the multiple video frames using a machine learning network (block 240) using a second group of one or more dedicated GPUs.
- the system 100 evaluates the encoded video frames through an inference process using a machine learning network that has been trained to classify one or more objects types in digital images.
- the machine learning inference process produces an overlay display having bounding polygon (e.g. bounding box) coordinates.
- bounding polygon e.g. bounding box
- the results of the inferencing i.e., the bounding polygon coordinates
- the system 100 may parallel process the rendering of the first and second display outputs using the first group of one or more GPUs, and the inferencing of video frames using the second group of one or more GPUs.
- the computer system 100 determines the occurrence of one or more objects in the multiple encoded video frames (block 250).
- the computer system 100 may use a trained machine learning network to identify the likelihood that a particular object of a certain class exists in a video frame.
- the machine learning network may assign a confidence level (e.g., a probability) that an object of a particular class occurs in a video frame.
- the computer system 100 generates a graphical indication around or about an identified object in the video frame indicating the location of the identified object.
- the computer system 100 may generate a graphical identifier, such as bounding perimeter (e.g., coordinates for an outlined rectangle), around or about an identified object.
- the computer system 100 may use other graphical indicators, such as a colored pixel group area, a change in the color or contrast of a pixel group area, or other graphical identifier indicating the occurrence of an identified object.
- the computer system 100 renders a second display output depicting graphical information corresponding to the determined one or more objects (block 260) in the video frames.
- the graphical information may include one or more of: the graphical indications, textual information identifying the type of object and/or textual information indicating the confidence that the object is of a certain type (e.g., a percentage or confidence score).
- the output values (e.g. graphical information) of object inferencing are stored in specific locations in memory of the first group of one or more GPUs.
- the computer system 100 may obtain information from the specific memory locations for every video frame.
- the system 100 then can overlay the graphical information for the second display output at the same frame rate as the first display output.
- the computer system 100 may render two windows at the same frame rate.
- the first window displays the real-time imagery obtained by the video capture device.
- the second window overlays the first window and displays the graphical information for the object inferencing.
- the computer system 100 then generates a composite display output including imagery of the first display output overlaid with the graphical information of the second display output (block 270).
- the first display output is rendered at or near the frame rate of the encoded video frames.
- the second display output includes the graphical information generated during the inferencing process using the trained machine learning network.
- the first display output is overlaid with the second display output. Overlaying the first display output with the second display output allows the computer system 100 to display real-time video with a high frame rate along with graphical indications of objects identified in the video by the machine learning network.
- the computer system 100 may provide the composite display output to an external monitoring device or a monitoring device integral to the computer system 100.
- the computer system 100 may stream the composite display output via a streaming media server to remote monitoring devices. For example, a user of a remote monitoring device may be able to view the procedure in real time via the Internet or other secured virtual private network.
- FIG. 3 illustrates an exemplary system utilized in real-time deployment of a machine learning network.
- a video endoscope and processor system 310 provides video to the computer system 330.
- the computer system 330 includes a video capture card and encoder 320 to encode video received from the endoscope and processor system 310.
- the computer system 330 includes a central processing unit (CPU) 332, two or more graphical processing units (GPUs) 334 and system memory 336.
- CPU central processing unit
- GPUs graphical processing units
- the computer system 330 executes various computer software modules 338, including: a module to receive video data from the endoscope and processor system; a module to convert the received video data into multiple video frames encoded into a particular color space format; a module to process the multiple video frames and store the multiple video frames onto a storage device; a module to convert the multiple video frames into an input to be processed by a machine learning network; a module to perform an inference on the multiple video frames and determine the occurrence of one or more objects in the video frames; and a module to renders a composite display output as a user interface that includes a first and second display output.
- the first display output displays real-time video obtained by the system 330.
- the second display output displays graphical indications and information about objects in the video identified by a machine learning network.
- the user interface generated by the system may be provided to an external monitoring device 340.
- the endoscope 310 unit includes a processor and other electronic circuitry to receive video imagery from an endoscope.
- the endoscope unit 310 may include one or more video outputs (e.g., ports or connectors): composite video, video graphics array (VGA), Bayonet Neill-Concelman (BNC), digital video interface (DVI), DVI-D (single or dual link), serial digital interface (SDI), HD-SDI, Dual Link HD-SDI, 3G-SDI, 6G-SDI, 12G-SDI, high definition multimedia interface (HDMI), mini HDMI, micro HDMI, DisplayPort, digital interface for video and audio (DiiVA), HDBaseT, CoaxPress, and/or mobile high-definition link (MHL).
- the video output of the endoscope unit is directly connected via wired cable to an input to the video capture card 320.
- the system 330 may use a video capture card 320 and encode the obtained video data into encoded video frames.
- An example suitable capture card allows receipt of a video input signal and conversion of the video input signal into encoded video frames of a particular color space format.
- FIG. 4 illustrates a process flow for classifying and identifying objects in video data in the context of an endoscopy procedure using the computer system described in FIG. 3.
- Endoscopy is a procedure used to evaluate or view inside a patient’s body.
- endoscopy may be used to diagnose diseases of the colon, stomach, esophagus, ear, nose, throat, heart, urinary tract, joints and abdomen.
- a physician inserts an endoscope into the patient’s body to obtain video imagery of the patient’s body.
- An endoscope 410 obtains video imagery (block 415) using a digital camera attached to the endoscope 410.
- An endoscope 410 typically includes a lighting source and a digital video camera attached near a probing end of the endoscope 410.
- the endoscope 410 is connected to a main endoscope unit 420 that controls the operation of the endoscope 410 and processes video imagery obtained by the endoscope 410 camera.
- the endoscope unit 420 outputs video imagery (block 425) to one or more video displays so that the examining physician may view the inside of the patient’s body during the endoscopy procedure.
- an endoscope 410 is a fiber-optic endoscope (also referred to as a fiber-scope) including a bundle of flexible-fiber optic cables coupled to the endoscope unit 420. Some of the fiber-optic cables are used to transmit a light source from the endoscope unit 420 to an examining end of the fiber-optic cables to illuminate a patient’s body cavity. Other fiber-optic cables include lenses at the examining end of the fiber-optic cables to capture imagery of the patient’s body cavity. The imagery is sent back via the fiber-optic cables to a receiving video camera to record the operation.
- a fiber-optic endoscope also referred to as a fiber-scope
- Some of the fiber-optic cables are used to transmit a light source from the endoscope unit 420 to an examining end of the fiber-optic cables to illuminate a patient’s body cavity.
- Other fiber-optic cables include lenses at the examining end of the fiber-optic cables to capture imagery of the patient’s body cavity. The imagery is sent
- a video output signal of the endoscope unit 420 is received by a video capture card 440 of the computer system 430.
- the video capture card 440 receives (e.g., captures) a video output signal (block 445) from the endoscope unit 420 by way of an SDI connector.
- the video capture card 440 converts the received video output signal into multiple encoded video frames.
- the received output signal may be encoded into various color space formats such as NV12, 1420, YV12, YUY2, YUYV, UYVY, UVYU, V308, IYU2, V408, RGB24, RGB32, V410, Y410 or Y42T.
- the multiple encoded video frames are stored into a memory cache of the video capture card 440.
- the multiple encoded video frames may be generated to correspond to a frame rate of 60 frames per second.
- the computer system 430 may be configured to generate a video frame rate of between 50 to 240 frames per second and/or generate a variable frame rate.
- the computer system 430 executes a computer vision module 460 for the identification and extraction of objects in the video frames.
- the computer vision module 460 is implemented using a first group of one or more CPUs.
- the computer vision module 460 may be implemented using the first or second group of GPUs, or a third group of one or more GPUs, or a combination thereof.
- OpenCV is used to perform these functions.
- OpenCV is an opensource real-time computer vision and machine learning software library.
- the computer vision module 460 obtains (e.g. captures) the stored video frame data (block 465) directly from the memory cache of the video capture card 440.
- the computer vision module 460 may retrieve the stored video frame data (block 465) through an Application Programming Interface (API) call to the video capture card 440 to directly transfer the data to the computer vision module 460 without transferring the data to RAM.
- API Application Programming Interface
- the computer vision module 460 may retrieve one or more video frames from the video capture card per retrieval call.
- the API is implemented by the video capture card 440.
- the first group of one or more CPUs implementing the computer vision module 460 performs a call to an API function implemented by the video capture card 440.
- the API function call may include a source address identifying the memory location in the video capture card 440 of the data to transfer, such as the stored video frame data (block 465), and an address indicating the location in the memory of the GPU to transfer the data.
- the video capture card 440 transfers the data from the source location to the destination location through an interconnect, such as a bus, a crossbar, network interconnect, or PCI Express.
- the CPU of computer system 430 is not involved in the transfer process.
- the computer vision module 460 may process the obtained video frame data and generate a video file. This process allows the computer system 430 to store video imagery of the endoscopy procedure.
- the computer system 430 may begin recording and storing video imagery received from the endoscope unit based on one or more events.
- the computer system 430 begins or stops recording video when the computer system 430 receives a specific command (such as a received user input to the computer system 430 to begin or stop recording.)
- the computer system 430 may begin storing the received video imagery when the computer system 430 determines that the endoscope unit begins transmitting video data to the computer system 430.
- the computer system 430 may begin recording video imagery received from the endoscope unit when the computer system 430 detects the start of an evaluation procedure.
- the computer system 430 may compare a first video frame to a second video frame and detect one or more changes in image characteristics, such as changes in contrast, hue, color, brightness or luminance. For example, as an endoscope is introduced to a patient’s body cavity the digital images in the video frames may change from a bright image to a darker image.
- the computer system 430 may detect the change in brightness as a begin procedure event and start recording the obtained video.
- the computer system 430 may detect the start of the evaluation procedure using a machine learning network to evaluate the video imagery and determine when the start of a procedure has likely begun.
- the computer system 430 may automatically stop recording when the computer system 430 detects that the endoscope is removed from the body cavity. Moreover, the computer system 430 may store a rolling period of video (e.g., a 30 second window), and include the rolling period video in the recorded video imagery.
- a rolling period of video e.g., a 30 second window
- the video file may be encrypted (block 435) by the computer system 430 and transmitted over a secure network to a cloud-based server (e.g., Amazon Simple Storage Service) 480 and stored (block 485) for later retrieval or analysis.
- a cloud-based server e.g., Amazon Simple Storage Service
- the computer vision module 460 converts the video frame data to a tensor object (block 467) for processing through a machine learning network.
- a tensor object comprises a vector or matrix of any number of dimensions.
- a tensor object may have dimension 0 (scalar), dimension 1 (vector), dimension 2 (2- dimensional matrix), or higher numbers of dimensions.
- the computer system 430 may process a video frame and convert the frame (e.g. an input image) into a tensor obj ect. The frame is converted into a tensor by first creating a new tensor object, and then mapping every pixel from that frame into the tensor.
- the computer system 430 uses a TensorFlow module 470 and its Object Detection API framework to perform an inference on the tensor object.
- TensorFlow is an opensource library for machine learning and numerical computation. In TensorFlow, computations involve tensors which are vectors or matrices of multi-dimensional arrays that represent various types of data. In TensorFlow, computation is approached as stateful dataflow graphs.
- the TensorFlow Object Detection API is an open source framework built on top of TensorFlow allowing the construction, training and deployment of object detection models.
- the computer system 430 uses one or more pre-trained machine learning networks and performs an inference process on the tensor objects (block 475).
- the machine learning networks may be trained to infer various object types or classes based on image features from a set image training data.
- the machine learning networks are trained to identify polyps based on a training set of digital images data including various images of polyps.
- the machine learning networks leam the features of the polyps and are able to identify the occurrence of a polyp in a digital image.
- Machine learning networks may be trained for different tasks such as identifying the location in the body of the endoscope, identifying whether a polyp is present, identifying the type of polyp, and so on.
- One or more of the machine learning networks may be run in parallel to increase processing bandwidth and complete inference more quickly.
- the TensorFlow module 470 produces results (block 477) indicating the likelihood that an input image is of a particular type of object.
- the TensorFlow module 470 outputs coordinate values associated with the detected object. For example, if a polyp is detected, a graphical indication such as a bounding polygon (e.g. a bounding box) with Cartesian coordinates describing a rectangle or square will be generated. The coordinate values will be outputted from Tensorflow module 470, and sent to Qt Framework 450 to graphically render the bounding polygon as an overlay.
- a bounding polygon e.g. a bounding box
- the computer system 430 executes a graphical user interface module 450 to generate a composite video output as a user interface.
- the composite video output may be displayed to an external monitoring device 490.
- the computer system 430 executes Qt software to manage and generate the composite video output.
- Qt software is an open-source toolkit for developing graphical user interfaces.
- the computer system 430 uses the Qt framework 450 to display two windows.
- the Qt framework 450 may be run on the first group of one or more CPUs, or a second group of or one or more CPUs that is different than the first group of CPUs running the computer vision module 460.
- the Qt framework may be run on the first group of GPUs, the second group of GPUs, or a third group of one or more GPUs, or a combination thereof.
- the Qt framework 450 obtains stored video frame data directly from the memory cache of the video capture card 440. The transfer may be performed without involving a CPU of computer system 430 and without the video frame data being stored in RAM of computer system 430.
- the Qt framework 450 may retrieve one or more video frames from the video capture card per retrieval call.
- the computer system 430 renders one window (i.e., a first display window) to display the multiple encoded video frames (block 455) of the received video.
- the computer system 430 uses another window (i.e. a second display window) to render an overlay window (block 457).
- the overlay window displays the multiple encoded video frames with graphical results generated from the processing of video frames via the machine learning network model.
- the API is implemented by the video capture card 440.
- the Qt framework 450 performs a call to an API function implemented by the video capture card 440.
- the API function call may include a source address identifying the memory location in the video capture card 440 of the data to transfer, such as one or more video frames, and an address indicating the location in the memory of the GPU to transfer the data.
- the video capture card 440 transfers the data from the source location to the destination location through an interconnect, such as a bus, a crossbar, network interconnect, or PCI Express.
- the first display window presents the multiple encoded video frames in real-time as they are received from the video capture card 440.
- the second display window presents a graphical overlay indicating identified objects and other textual and graphical information in the second display window.
- the first display window may present real-time imagery of an endoscopy procedure, such as a colonoscopy depicting imagery of a patient’s colon.
- the first display window presents the video obtained by the endoscope 410.
- the first display window may present other user interface information displayed by the endoscope unit 410 in the video.
- the second display window presents graphical and/or textual information laid over, in front of, or on top of the first display window.
- the first display window may be displayed at resolutions of 1080p (1920x1080 pixels), 2K (2048x1080 pixels), 1440p (2560x1440 pixels), 4K or 2160p (3840x2160 pixels) and/or 8K (7680x4320 pixels).
- FIG. 5 illustrates a flowchart of an example process for evaluating video data to determine an external environmental state of a video monitoring device.
- the computer system 100, 330, 430 may perform additional functions or processes based on evaluation of video frames.
- the computer system 100, 330, 430 receives the encoded video frames (block 510). For example, as an endoscope is inserted into a patient’s body cavity, the computer system 100, 330, 430 receives live video imagery from the endoscope unit and the computer system 100, 330, 430 converts the video data into encoded video frames.
- the computer system then evaluates the images of the encoded video frames (block 520).
- the computer system 100, 330, 430 may compare a first image to a subsequent second image, and/or compare the image characteristics of a particular video frame against a pre-determined set of required image characteristics.
- the computer system may determine an external environmental state of the video monitoring device (block 530). For example, in the context of a colonoscopy, the external environmental state of the endoscope may be evaluated by the computer system 100, 330, 430 to assess whether the colon is empty and clean enough to examine the colon for polyps and/or other abnormalities.
- the computer system 100, 300, 430 may evaluate via a machine learning network image characteristics (e.g., hue, color, brightness, contrast, luminance) of the video frames.
- the computer system 100, 330, 430 may evaluate the color of the colon to determine if the colon is prepared for the colonoscopy.
- a machine learning network may be trained on numerous digital images of colon preparation. The machine learning network may determine whether a colon is prepared by evaluating one or more video frames and determine a result, such as“1” for a good result, or“0” for a bad result. Based on the machine learning network, the system 100, 330, 430 may determine on a binary scale the quality of the colon preparation with 0 being the worst and 1 being the clearest. If the colon preparation is not of a sufficient quality, the system 100, 330, 430 may indicate via the second display output textual and/or graphical indications that the colon is not sufficiently prepared for the procedure.
- a machine learning network image characteristics e.g., hue, color, brightness, contrast, lumina
- the computer system 100, 330, 430 may perform one or more functions (block 540). For example, the computer system may forgo other inferencing of video images via the machine learning network if the computer system determines the environmental state is unsuitable (e.g., not prepared or clean enough) to identify objects in the video images. However, the computer system may perform other inferencing via the machine learning network if the external environmental state is determined to be suitable. For example, the computer system 100, 330, 430 may generate an indication or score of how well the colon is prepared. If the preparation score exceeds a pre determined threshold value, then the computer system may determine that the colon (i.e., the external environment) is suitable for other inferencing via the machine learning network. Additionally, the computer system 100, 330, 430 may generate a message or other user output indicating that the external environmental state is or is not suitable to perform a procedure (e.g., a colonoscopy).
- a procedure e.g., a colonoscopy
- the computer system 100, 330, 430 may generate an indication of the external environmental state of the video monitoring device (block 550).
- the computer system 100, 330, 430 may display an indication of the environmental state of the video monitoring device (block 560).
- the computer system 100, 330, 430 may provide an output indicating the external environmental state to the monitoring device.
- the user interface 600 illustrates a composite display of the first display window and the second display window.
- the user interface 600 includes a video display area 610 to display video from the first display window.
- the video display 610 area may be part of a sub-area of the user interface 600 or may include the entire display area of the user interface 600.
- the video display area 610 is a sub-area of the user interface 600.
- the video display area displays the video 620 of the first display window as described above.
- the user interface 600 includes a graphical information display area for displaying graphical information of the second display window. A portion of the graphical information display area is displayed over the video display area 610.
- a graphical indication e.g., a rectangular bounding polygon
- a detected object 630 is displayed over the video 620 in the video display area 610.
- the graphical information display area may also include other sub-areas of the user interface.
- the graphical information display area may include informational display areas 602, 604, 606, 608 that displays information corresponding to objects 630 identified in the video frames 620.
- the informational display area 602 may display information about the type or class of detected object 630.
- Informational display area 602 may indicate that the detected object 630 is a polyp.
- the informational display areas 602, 604, 606, 608 may be labeled with an identifier, such as alphanumeric characters, that correspond with labels on the graphical indications of the detected objects.
- the computer system 100, 330, 430 may have detected four objects in the video frames using the machine learning network as discussed above.
- the graphical information display area would include four graphical indications of objects (e.g., four rectangular bounding polygons). Each of the four graphical indications may be labeled 1, 2, 3, 4 and/or may be color coded.
- Each of the informational display areas 602, 604, 606, 608 would include information corresponding to a respective detected object.
- different object types may have been detected via the machine learning network. In this case, the graphical indications of the same type of object may be displayed with the same color, and other object types may be displayed in a different color.
- FIG. 7 illustrates an example machine of a computer system within which a set of instructions, for causing the machine to perform any one or more of the methodologies discussed herein, may be executed.
- the machine may be connected (e.g., networked) to other machines in a LAN, an intranet, an extranet, and/or the Internet.
- the machine may operate in the capacity of a server or a client machine in client- server network environment, as a peer machine in a peer-to-peer (or distributed) network environment, or as a server or a client machine in a cloud computing infrastructure or environment.
- the machine may be a personal computer (PC), a tablet PC, a set-top box (STB), a Personal Digital Assistant (PDA), a cellular telephone, a web appliance, a server, a network router, a switch or bridge, or any machine capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that machine.
- PC personal computer
- PDA Personal Digital Assistant
- STB set-top box
- STB set-top box
- a Personal Digital Assistant PDA
- a cellular telephone a web appliance
- server a server
- network router a network router
- switch or bridge any machine capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that machine.
- machine shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein.
- the example computer system 700 includes a processing device 702, a main memory 704 (e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM) such as synchronous DRAM (SDRAM) or Rambus DRAM (RDRAM), etc ), a static memory 706 (e.g., flash memory, static random access memory (SRAM), etc.), and a data storage device 718, which communicate with each other via a bus 730.
- main memory 704 e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM) such as synchronous DRAM (SDRAM) or Rambus DRAM (RDRAM), etc
- DRAM dynamic random access memory
- SDRAM synchronous DRAM
- RDRAM Rambus DRAM
- static memory 706 e.g., flash memory, static random access memory (SRAM), etc.
- SRAM static random access memory
- Processing device 702 represents one or more general-purpose processing devices such as a microprocessor, a central processing unit, or the like. More particularly, the processing device may be complex instruction set computing (CISC) microprocessor, reduced instruction set computing (RISC) microprocessor, very long instruction word (VLIW) microprocessor, or processor implementing other instruction sets, or processors implementing a combination of instruction sets. Processing device 702 may also be one or more special- purpose processing devices such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), network processor, or the like. The processing device 702 is configured to execute instructions 726 for performing the operations and steps discussed herein. In one embodiment, the computer system 700 may use a motherboard with PLX chip to optimize use of PCIe lanes of the motherboard.
- CISC complex instruction set computing
- RISC reduced instruction set computing
- VLIW very long instruction word
- ASIC application specific integrated circuit
- FPGA field programmable gate array
- DSP digital signal
- the computer system 700 may further include a network interface device 708 to communicate over the network 720.
- the computer system 700 also may include a video display unit 710 (e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)), an alphanumeric input device 712 (e.g., a keyboard), a cursor control device 714 (e.g., a mouse), a graphics processing unit 722, a signal generation device 716 (e.g., a speaker), video processing unit 728, and audio processing unit 732.
- a video display unit 710 e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)
- an alphanumeric input device 712 e.g., a keyboard
- a cursor control device 714 e.g., a mouse
- a graphics processing unit 722 e.g., a graphics processing unit 722
- signal generation device 716 e.g.,
- the data storage device 718 may include a machine-readable storage medium 724 (also known as a computer-readable medium) on which is stored one or more sets of instructions or software 726 embodying any one or more of the methodologies or functions described herein.
- the instructions 726 may also reside, completely or at least partially, within the main memory 704 and/or within the processing device 702 during execution thereof by the computer system 700, the main memory 704 and the processing device 702 also constituting machine-readable storage media.
- the instructions 726 include instructions to implement functionality corresponding to the components of a device to perform the disclosure herein.
- the machine-readable storage medium 724 is shown in an example implementation to be a single medium, the term“machine-readable storage medium” should be taken to include a single medium or multiple media (e.g., a centralized or distributed database, and/or associated caches and servers) that store the one or more sets of instructions.
- the term“machine-readable storage medium” shall also be taken to include any medium that is capable of storing or encoding a set of instructions for execution by the machine and that cause the machine to perform any one or more of the methodologies of the present disclosure.
- the term“machine- readable storage medium” shall accordingly be taken to include, but not be limited to, solid- state memories, optical media and magnetic media.
- the terms“engine” and“module”, as used herein, refer to logic embodied in hardware or firmware, or to a collection of software instructions, possibly having entry and exit points, written in a programming language, such as, for example, Java, Lua, C or C++.
- a software module may be compiled and linked into an executable program, installed in a dynamic link library, or may be written in an interpreted programming language such as, for example, BASIC, Perl, or Python. It will be appreciated that software modules may be callable from other modules or from themselves, and/or may be invoked in response to detected events or interrupts.
- Software modules configured for execution on computing devices may be provided on one or more computer readable media, such as compact discs, digital video discs, flash drives, or any other tangible media. Such software code may be stored, partially or fully, on a memory device of the executing computing device. Software instructions may be embedded in firmware, such as an EPROM. It will be further appreciated that hardware modules may be comprised of connected logic units, such as gates and flip-flops, and/or may be comprised of programmable units, such as programmable gate arrays or processors. The modules described herein are preferably implemented as software modules, but may be represented in hardware or firmware. Generally, the modules described herein refer to logical modules that may be combined with other modules or divided into sub-modules despite their physical organization or storage
- the present disclosure also relates to an apparatus for performing the operations herein.
- This apparatus may be specially constructed for the intended purposes, or it may comprise a general purpose computer selectively activated or reconfigured by a computer program stored in the computer.
- a computer program may be stored in a computer readable storage medium, such as, but not limited to, any type of disk including floppy disks, optical disks, CD-ROMs, and magnetic-optical disks, read-only memories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, or any type of media suitable for storing electronic instructions, each coupled to a computer system bus.
- the present disclosure may be provided as a computer program product, or software, that may include a machine-readable medium having stored thereon instructions, which may be used to program a computer system (or other electronic devices) to perform a process according to the present disclosure.
- a machine-readable medium includes any mechanism for storing information in a form readable by a machine (e.g., a computer).
- a machine-readable (e.g., computer-readable) medium includes a machine (e.g., a computer) readable storage medium such as a read only memory (“ROM”), random access memory (“RAM”), magnetic disk storage media, optical storage media, flash memory devices, etc.
Landscapes
- Engineering & Computer Science (AREA)
- Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Public Health (AREA)
- Biomedical Technology (AREA)
- General Health & Medical Sciences (AREA)
- Surgery (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Radiology & Medical Imaging (AREA)
- Physics & Mathematics (AREA)
- Pathology (AREA)
- Epidemiology (AREA)
- Primary Health Care (AREA)
- Veterinary Medicine (AREA)
- Molecular Biology (AREA)
- Heart & Thoracic Surgery (AREA)
- Optics & Photonics (AREA)
- Theoretical Computer Science (AREA)
- Animal Behavior & Ethology (AREA)
- Biophysics (AREA)
- Signal Processing (AREA)
- General Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Business, Economics & Management (AREA)
- General Business, Economics & Management (AREA)
- Quality & Reliability (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Multimedia (AREA)
- Computer Graphics (AREA)
- Human Computer Interaction (AREA)
- General Engineering & Computer Science (AREA)
- Image Analysis (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
Abstract
Description
Claims
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
PCT/US2019/041945 WO2021010985A1 (en) | 2019-07-16 | 2019-07-16 | Real-time deployment of machine learning systems |
Publications (2)
Publication Number | Publication Date |
---|---|
EP3998931A1 true EP3998931A1 (en) | 2022-05-25 |
EP3998931A4 EP3998931A4 (en) | 2023-04-19 |
Family
ID=74209864
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP19937726.8A Pending EP3998931A4 (en) | 2019-07-16 | 2019-07-16 | Real-time deployment of machine learning systems |
Country Status (3)
Country | Link |
---|---|
EP (1) | EP3998931A4 (en) |
JP (1) | JP7420916B2 (en) |
WO (1) | WO2021010985A1 (en) |
Families Citing this family (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10671934B1 (en) | 2019-07-16 | 2020-06-02 | DOCBOT, Inc. | Real-time deployment of machine learning systems |
US11423318B2 (en) | 2019-07-16 | 2022-08-23 | DOCBOT, Inc. | System and methods for aggregating features in video frames to improve accuracy of AI detection algorithms |
US11100373B1 (en) | 2020-11-02 | 2021-08-24 | DOCBOT, Inc. | Autonomous and continuously self-improving learning system |
KR102640314B1 (en) * | 2021-07-12 | 2024-02-23 | (주)휴톰 | Artificial intelligence surgery system amd method for controlling the same |
Family Cites Families (14)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7613335B2 (en) * | 2003-02-12 | 2009-11-03 | The University Of Iowa Research Foundation | Methods and devices useful for analyzing color medical images |
US20140031677A1 (en) * | 2012-01-20 | 2014-01-30 | Physical Sciences, Inc. | Apparatus and Method for Aiding Needle Biopsies |
US9715727B2 (en) | 2012-02-23 | 2017-07-25 | Smith & Nephew, Inc. | Video endoscopic system |
WO2014160510A2 (en) * | 2013-03-13 | 2014-10-02 | Massachusetts Institute Of Technology | Photometric stereo endoscopy |
US9354778B2 (en) * | 2013-12-06 | 2016-05-31 | Digimarc Corporation | Smartphone-based methods and systems |
US20150282749A1 (en) * | 2014-04-05 | 2015-10-08 | Surgisense Corporation | Apparatus, systems, and methods for mapping of tissue oxygenation |
US10169535B2 (en) * | 2015-01-16 | 2019-01-01 | The University Of Maryland, Baltimore County | Annotation of endoscopic video using gesture and voice commands |
CN108348143B (en) | 2015-10-30 | 2020-11-03 | 索尼公司 | Information processing apparatus, information processing method, and endoscope system |
KR101809673B1 (en) * | 2016-07-04 | 2018-01-18 | 주식회사 팝스라인 | Terminal and control method thereof |
JP6289762B1 (en) | 2016-12-26 | 2018-03-07 | 三菱電機株式会社 | Video processing apparatus, video processing method, video processing program, and video monitoring system |
JP6785181B2 (en) | 2017-04-12 | 2020-11-18 | 株式会社日立製作所 | Object recognition device, object recognition system, and object recognition method |
US20190066304A1 (en) * | 2017-08-31 | 2019-02-28 | Microsoft Technology Licensing, Llc | Real-time object segmentation in live camera mode |
JP6890184B2 (en) | 2017-09-15 | 2021-06-18 | 富士フイルム株式会社 | Medical image processing equipment and medical image processing program |
CN111295127B (en) | 2017-10-31 | 2022-10-25 | 富士胶片株式会社 | Examination support device, endoscope device, and recording medium |
-
2019
- 2019-07-16 JP JP2022502859A patent/JP7420916B2/en active Active
- 2019-07-16 WO PCT/US2019/041945 patent/WO2021010985A1/en unknown
- 2019-07-16 EP EP19937726.8A patent/EP3998931A4/en active Pending
Also Published As
Publication number | Publication date |
---|---|
EP3998931A4 (en) | 2023-04-19 |
JP2022541897A (en) | 2022-09-28 |
WO2021010985A1 (en) | 2021-01-21 |
JP7420916B2 (en) | 2024-01-23 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11694114B2 (en) | Real-time deployment of machine learning systems | |
JP7420916B2 (en) | Real-time deployment of machine learning systems | |
US10342410B2 (en) | Automated system for medical video recording and storage | |
AU2019431299B2 (en) | AI systems for detecting and sizing lesions | |
CN109640007B (en) | Artificial intelligence image sensing equipment | |
CN110599421B (en) | Model training method, video fuzzy frame conversion method, device and storage medium | |
JP2022554119A (en) | Endoscope image processing method, device, electronic device and computer program | |
WO2015118850A1 (en) | Endoscopic image diagnosis support system | |
WO2022073282A1 (en) | Motion recognition method based on feature interactive learning, and terminal device | |
WO2023207564A1 (en) | Endoscope advancing and retreating time determining method and device based on image recognition | |
JP2020013553A (en) | Information generating method and apparatus applicable to terminal device | |
CN111784668A (en) | Digestive endoscopy image automatic freezing method based on perceptual hash algorithm | |
WO2018012729A1 (en) | Display device and text recognition method for display device | |
US20220369920A1 (en) | Phase identification of endoscopy procedures | |
CN113706414A (en) | Training method of video optimization model and electronic equipment | |
WO2023125008A1 (en) | Artificial intelligence-based endoscope image processing method and apparatus, medium and device | |
KR20220078495A (en) | Method, apparatus and program to read lesion of small intestine based on capsule endoscopy image | |
JPWO2020054604A1 (en) | Information processing equipment, control methods, and programs | |
CN112261417B (en) | Video pushing method and system, equipment and readable storage medium | |
CN110772210B (en) | Diagnosis interaction system and method | |
US11120554B2 (en) | Image diagnosis apparatus, image diagnosis method, and program | |
Phillips et al. | Video capsule endoscopy: pushing the boundaries with software technology | |
CN113744266B (en) | Method and device for displaying focus detection frame, electronic equipment and storage medium | |
JP5662088B2 (en) | Remote visual inspection device | |
CN112766066A (en) | Method and system for processing and displaying dynamic video stream and static image |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE |
|
PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE |
|
17P | Request for examination filed |
Effective date: 20220212 |
|
AK | Designated contracting states |
Kind code of ref document: A1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
DAV | Request for validation of the european patent (deleted) | ||
DAX | Request for extension of the european patent (deleted) | ||
A4 | Supplementary search report drawn up and despatched |
Effective date: 20230322 |
|
RIC1 | Information provided on ipc code assigned before grant |
Ipc: G06F 3/14 20060101ALI20230316BHEP Ipc: G16H 30/40 20180101ALI20230316BHEP Ipc: G16H 30/20 20180101ALI20230316BHEP Ipc: G16H 30/00 20180101ALI20230316BHEP Ipc: A61B 8/00 20060101ALI20230316BHEP Ipc: A61B 1/05 20060101AFI20230316BHEP |
|
RAP1 | Party data changed (applicant data changed or rights of an application transferred) |
Owner name: SATISFAI HEALTH INC. |