EP3119318A1 - System and method for tissue contact detection and for auto-exposure and illumination control - Google Patents
System and method for tissue contact detection and for auto-exposure and illumination controlInfo
- Publication number
- EP3119318A1 EP3119318A1 EP15764341.2A EP15764341A EP3119318A1 EP 3119318 A1 EP3119318 A1 EP 3119318A1 EP 15764341 A EP15764341 A EP 15764341A EP 3119318 A1 EP3119318 A1 EP 3119318A1
- Authority
- EP
- European Patent Office
- Prior art keywords
- camera
- optical power
- output optical
- exposure time
- surgical system
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims description 169
- 238000005286 illumination Methods 0.000 title claims description 99
- 238000001514 detection method Methods 0.000 title claims description 29
- 230000003287 optical effect Effects 0.000 claims abstract description 219
- 229920006395 saturated elastomer Polymers 0.000 claims description 82
- 230000008859 change Effects 0.000 claims description 63
- 230000007423 decrease Effects 0.000 claims description 49
- 230000009467 reduction Effects 0.000 claims description 7
- 230000004044 response Effects 0.000 claims description 4
- 230000000451 tissue damage Effects 0.000 abstract description 9
- 231100000827 tissue damage Toxicity 0.000 abstract description 9
- 230000008569 process Effects 0.000 description 152
- 235000019557 luminance Nutrition 0.000 description 34
- 238000012545 processing Methods 0.000 description 21
- 238000010586 diagram Methods 0.000 description 14
- 230000003247 decreasing effect Effects 0.000 description 10
- 230000001360 synchronised effect Effects 0.000 description 8
- 230000002829 reductive effect Effects 0.000 description 7
- 238000013459 approach Methods 0.000 description 6
- 238000003384 imaging method Methods 0.000 description 5
- 230000000670 limiting effect Effects 0.000 description 5
- 238000012544 monitoring process Methods 0.000 description 5
- 238000004590 computer program Methods 0.000 description 4
- 230000026676 system process Effects 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 230000004888 barrier function Effects 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 230000001934 delay Effects 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000012886 linear function Methods 0.000 description 2
- 238000005259 measurement Methods 0.000 description 2
- 238000005096 rolling process Methods 0.000 description 2
- 229910052724 xenon Inorganic materials 0.000 description 2
- FHNFHKCVQCLJFQ-UHFFFAOYSA-N xenon atom Chemical compound [Xe] FHNFHKCVQCLJFQ-UHFFFAOYSA-N 0.000 description 2
- 230000003187 abdominal effect Effects 0.000 description 1
- 230000003213 activating effect Effects 0.000 description 1
- 210000003484 anatomy Anatomy 0.000 description 1
- 230000002238 attenuated effect Effects 0.000 description 1
- 230000003190 augmentative effect Effects 0.000 description 1
- 230000000740 bleeding effect Effects 0.000 description 1
- 210000004204 blood vessel Anatomy 0.000 description 1
- 238000005282 brightening Methods 0.000 description 1
- 239000011248 coating agent Substances 0.000 description 1
- 238000000576 coating method Methods 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 230000002401 inhibitory effect Effects 0.000 description 1
- 208000014674 injury Diseases 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000002372 labelling Methods 0.000 description 1
- QSHDDOUJBYECFT-UHFFFAOYSA-N mercury Chemical compound [Hg] QSHDDOUJBYECFT-UHFFFAOYSA-N 0.000 description 1
- 229910052753 mercury Inorganic materials 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 230000029058 respiratory gaseous exchange Effects 0.000 description 1
- 238000001356 surgical procedure Methods 0.000 description 1
- 230000008733 trauma Effects 0.000 description 1
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/30—Surgical robots
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
- A61B1/00002—Operational features of endoscopes
- A61B1/00004—Operational features of endoscopes characterised by electronic signal processing
- A61B1/00006—Operational features of endoscopes characterised by electronic signal processing of control signals
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
- A61B1/00002—Operational features of endoscopes
- A61B1/00004—Operational features of endoscopes characterised by electronic signal processing
- A61B1/00009—Operational features of endoscopes characterised by electronic signal processing of image signals during a use of endoscope
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
- A61B1/00002—Operational features of endoscopes
- A61B1/00043—Operational features of endoscopes provided with output arrangements
- A61B1/00055—Operational features of endoscopes provided with output arrangements for alerting the user
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
- A61B1/00002—Operational features of endoscopes
- A61B1/00062—Operational features of endoscopes provided with means for preventing overuse
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
- A61B1/00163—Optical arrangements
- A61B1/00165—Optical arrangements with light-conductive means, e.g. fibre optics
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
- A61B1/00163—Optical arrangements
- A61B1/00193—Optical arrangements adapted for stereoscopic vision
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
- A61B1/04—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor combined with photographic or television appliances
- A61B1/045—Control thereof
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
- A61B1/06—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor with illuminating arrangements
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
- A61B1/06—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor with illuminating arrangements
- A61B1/0655—Control therefor
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
- A61B90/30—Devices for illuminating a surgical field, the devices having an interrelation with other surgical devices or with a surgical procedure
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/30—Surgical robots
- A61B2034/301—Surgical robots for introducing or steering flexible instruments inserted into the body, e.g. catheters or endoscopes
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
- A61B90/06—Measuring instruments not otherwise provided for
- A61B2090/064—Measuring instruments not otherwise provided for for measuring force, pressure or mechanical tension
- A61B2090/065—Measuring instruments not otherwise provided for for measuring force, pressure or mechanical tension for measuring contact or contact pressure
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
- A61B90/08—Accessories or related features not otherwise provided for
- A61B2090/0801—Prevention of accidental cutting or pricking
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
- A61B90/08—Accessories or related features not otherwise provided for
- A61B2090/0801—Prevention of accidental cutting or pricking
- A61B2090/08021—Prevention of accidental cutting or pricking of the patient or his organs
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/30—Surgical robots
- A61B34/35—Surgical robots for telesurgery
Definitions
- aspects of this invention are related to endoscopic imaging and are more particularly related to tissue contact detection, displayed scene brightness control, camera auto-exposure control, and illumination control in a teleoperated surgical system.
- US 8,512,232 B2 describes introducing into a body an endoscope that provides a first level of illumination and also introducing into the body an illumination apparatus that provides a second level of illumination.
- the second level of illumination is greater than the first level of illumination and is used for imaging a large target area.
- the lower level of illumination from the endoscope is used for imaging a smaller target area.
- a teleoperated surgical system includes an illuminator, a camera, and a controller.
- the illuminator provides output optical power.
- the camera is configured to capture a scene including tissue.
- the controller is coupled to the illuminator and to the camera.
- the controller is configured to detect contact of an endoscope with tissue.
- the controller is configured to attenuate the output optical power following contact being detected.
- the controller includes a statistics module coupled to the camera.
- the statistics module receives the scene captured by the camera.
- the statistics module generates a brightness histogram for the captured scene and determines an overall brightness of the captured scene.
- a contact detection module in the controller is coupled to the statistics module.
- the contact detection module receives the overall brightness of the captured scene and receives a camera exposure time.
- the contact detection module is also coupled to the illuminator. The contact detection module detects contact of the tip of the endoscope with tissue.
- the second control loop increases the output optical power and decreases the camera exposure time if the camera exposure time is larger than a first exposure threshold.
- the second control loop decreases the output optical power and increases the camera exposure time if the camera exposure time is smaller than a second exposure threshold.
- the second control loop leaves the output optical power and the camera exposure time unchanged if the camera exposure time is between the first exposure threshold and the second exposure threshold.
- the first control loop and the second control loop are included in an auto-exposure module.
- a statistics module is coupled to the camera to receive the video stream and is coupled to the auto-exposure module.
- the statistics module is configured to create a brightness histogram for a frame in the video stream.
- a method of operating a teleoperated surgical system includes adjusting one or both of a video pipeline gain and a camera exposure time using an average brightness of a captured scene and a target brightness for a displayed scene.
- the method also configures output optical power of an illuminator and exposure time of a camera using the camera exposure time.
- the method reduces a number of saturated pixels in a second captured frame by reducing the target brightness.
- FIGs. 1A and IB are block diagrams of a teleoperated surgical systems that include a controller for controlling some or all of a number of saturated pixels in a scene, for controlling a video pipeline gain, for controlling an output optical power, for controlling a camera exposure time, and for detecting tissue contact.
- an endoscopic control system includes a controller that in turn includes camera control units 130L and 130R and an illumination controller 115.
- camera control units 130L and 130R include two control loops 131, 132.
- Fig. 1A The combination of control loops 131 and 132 is configured to maintain a target brightness of a displayed scene (the displayed scene being a scene in a captured frame) and is configured to maintain a minimum illuminator output.
- saturated pixels control loop 133 is configured to apply a range-constrained reduction to the target brightness.
- endoscope 101 (Figs. 1A and IB) continues to move toward tissue 103, eventually endoscope 101 contacts tissue 103. The contact is observed by detecting a change in output optical power level that results in little or no change in reflected luminance from tissue 103. When this situation is detected by camera control units 130L and 130R, the output optical power of illuminator 110 is reduced to a level that is safe for tissue contact.
- a target brightness is a brightness of scenes displayed on display unit 251.
- An initial target brightness sometimes referred to as the original target brightness, is set in teleoperated surgical system 200, e.g., a target brightness of 1500.
- a slide switch 266 is presented in a user interface presented on display unit 251, which permits the user to select a target brightness for the displayed scenes that is acceptable for the user.
- the initial target brightness was empirically determined based on feedback from users of teleoperated surgical systems such as the da Vinci teleoperated surgical system.
- the target brightness is a control parameter in control loops within auto-exposure module 232, as explained more completely below.
- control system implemented by auto- exposure module 232 controls the video pipeline gain 236 and camera exposure time 238.
- the change in overall brightness of the captured scene is a linear function of video pipeline gain 236 and camera exposure time 238.
- LIMIT SATURATED PIXELS process 441 minimizes over time the total screen area of display 251 that is consumed by saturated pixels. LIMIT SATURATED PIXELS process 441 controls ADJUSTED TARGET BRIGHTESS 444 to perform this minimization.
- Fig. 5A is a process flow diagram for one aspect of LIMIT SATURATED PIXELS process 441. Upon completion, LIMIT SATURATED PIXELS process 441 transfers to ADJUST GAIN AND EXPOSURE TIME process 442.
- ADJUST POWER AND EXPOSURE ADJUSTMENT process 443 controls the output optical power of illuminator 210 and the camera exposure time. If the camera exposure time is less than a second exposure threshold (see Fig. 6), the output optical power is decreased and a positive adjustment is applied to camera exposure time to compensate for the illumination change. If the camera exposure time is greater than a first exposure threshold (see Fig. 6), the output optical power of illuminator 210 is increased and a negative adjustment is applied to camera exposure time to compensate for the illumination change.
- ADJUST POWER AND EXPOSURE ADJUSTMENT process 443 determines whether a change of OUTPUT OPTICAL POWER 448 and a CAMERA EXPOSURE TIME ADJUSTMENT 447 are needed, and then returns processing to LIMIT SATURATED PIXELS process 441 for processing of the next frame.
- OUTPUT OPTICAL POWER 448 provides output optical power 237.
- Fig. 5C is a process flow diagram for one aspect of ADJUST POWER AND EXPOSURE ADJUSTMENT process 443.
- COUNT STATURATED PIXELS process 501 counts the number of saturated pixels 301 in histogram 300. In the example of Fig. 3, the count of saturated pixels in bin 255 is 41873.
- statistics module 231 If endoscope 201 is a stereoscopic endoscope, statistics module 231 generates a histogram for each of the left and right optical channels. If endoscope 201 is not a stereoscopic endoscope, statistics module 231 generates a histogram for the single optical channel. In one aspect, two bytes are used to represent the brightness and so the histograms have 512 bins along the x-axis.
- COUNT STATURATED PIXELS process 501 determines the number of saturated pixels, in one aspect, as the sum of the number of pixels in bin 511 of each of the left and right histograms.
- the saturated pixel threshold for the number of saturated pixels allowed in a scene was empirically determined.
- a teleoperated surgical system 200 was configured to vary the size of a saturated region in a scene displayed on display unit 251. Clinical trials with such a system were used to determine the threshold for the number saturated pixels and the bins used to count the number of saturated pixels so that a set of surgical tasks could be performed without users of the system labeling saturated regions as distracting or hindering accomplishment of the task.
- process 441 is four to five seconds, and the maximum reduction in target brightness, the range limit on target brightness, is thirty-five percent of the original target brightness 266.
- the range limit on target brightness is empirically determined. The range is determined ascertaining the lowest average brightness of displayed scenes that users identify as providing information that can be used in the surgical produce. The ratio of the lowest average brightness to the original target brightness is used to define the range limit on target brightness, e.g.,
- process 513 When processing transfers to INCREASE GAIN OR EXPOSURE TIME process 513, sometimes referred to process 513, the brightness of the captured image is too low. If the camera exposure time is not at a maximum camera exposure time Emax (see Fig. 6), process 513 increases the camera exposure time, which in turn increases the average frame brightness of subsequently captured frames. However, if the camera exposure time is at maximum camera exposure time Emax, the camera exposure time cannot be increased further. In this situation, process 513 increases the video pipeline gain.
- VIDEO PIPELINE GAIN 445 is greater than one, process 514 multiplies VIDEO PIPELINE GAIN 445 by the ratio of ADJUSTED TARGET
- PIPELINE GAIN 445 is at one and the remainder of the decrease is applied to
- Fig. 5C is a process flow diagram for one aspect of ADJUST POWER AND EXPOSURE ADJUSTMENT process 443. However, prior to considering Fig. 5C, parameters used in the processes of Fig. 5C are considered.
- the third control loop leaves the output optical power and the camera exposure time unchanged if the exposure time is between the first exposure threshold and the second exposure threshold.
- the maximum output optical power Pmax corresponds to an output optical power 800 milliwatts.
- the minimum output optical power Pmin corresponds to an output optical power of 400 milliwatts, the maximum camera exposure time Emax is 4096, and the minimum camera exposure time Emin is zero.
- the illumination change is synchronized with the compensating change in camera exposure time, such that the overall brightness of the video displayed to the surgeon remains about constant with a change in illumination. For example, a decrease in output optical power and an increase in camera exposure time must occur on the same video frame.
- about constant means constant to within tolerances of teleoperated surgical system 200.
- the processing of the video stream in teleoperated surgical system 200 has delays. Illumination control, video pipeline gain control, camera exposure control, video statistics collection and analysis are all at different pipeline delays. For example, at frame time tO, (Fig. 7) for a first frame 701 of the video stream, a frame is captured statistics are gathered for frame 701by statistics module 231 and a gain from a previously captured frame is written to the video pipeline.
- contact detection module 233 detects tissue contact, and attenuates the output optical power to a level that is safe for tissue contact.
- safe output optical power for tissue contact is greater than minimum output optical power. However, if tissue damage is dominated by conductive heat from contact between endoscope tip and tissue, safe output optical power for tissue contact could be below the minimum output optical power.
Landscapes
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Surgery (AREA)
- Engineering & Computer Science (AREA)
- Public Health (AREA)
- Biomedical Technology (AREA)
- Heart & Thoracic Surgery (AREA)
- Medical Informatics (AREA)
- Molecular Biology (AREA)
- Animal Behavior & Ethology (AREA)
- General Health & Medical Sciences (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Veterinary Medicine (AREA)
- Pathology (AREA)
- Optics & Photonics (AREA)
- Physics & Mathematics (AREA)
- Biophysics (AREA)
- Radiology & Medical Imaging (AREA)
- Signal Processing (AREA)
- Oral & Maxillofacial Surgery (AREA)
- Robotics (AREA)
- Endoscopes (AREA)
- Studio Devices (AREA)
- Circuit Arrangement For Electric Light Sources In General (AREA)
- Exposure Control For Cameras (AREA)
- Instruments For Viewing The Inside Of Hollow Bodies (AREA)
Abstract
Description
Claims
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
EP21154425.9A EP3834702A1 (en) | 2014-03-17 | 2015-03-17 | System and method for tissue contact detection and for auto-exposure and illumination control |
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201461954381P | 2014-03-17 | 2014-03-17 | |
US201461954336P | 2014-03-17 | 2014-03-17 | |
PCT/US2015/020892 WO2015142799A1 (en) | 2014-03-17 | 2015-03-17 | System and method for tissue contact detection and for auto-exposure and illumination control |
Related Child Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP21154425.9A Division EP3834702A1 (en) | 2014-03-17 | 2015-03-17 | System and method for tissue contact detection and for auto-exposure and illumination control |
EP21154425.9A Division-Into EP3834702A1 (en) | 2014-03-17 | 2015-03-17 | System and method for tissue contact detection and for auto-exposure and illumination control |
Publications (3)
Publication Number | Publication Date |
---|---|
EP3119318A1 true EP3119318A1 (en) | 2017-01-25 |
EP3119318A4 EP3119318A4 (en) | 2017-12-13 |
EP3119318B1 EP3119318B1 (en) | 2021-03-10 |
Family
ID=54145195
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP21154425.9A Pending EP3834702A1 (en) | 2014-03-17 | 2015-03-17 | System and method for tissue contact detection and for auto-exposure and illumination control |
EP15764341.2A Active EP3119318B1 (en) | 2014-03-17 | 2015-03-17 | System and method for tissue contact detection and for auto-exposure and illumination control |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP21154425.9A Pending EP3834702A1 (en) | 2014-03-17 | 2015-03-17 | System and method for tissue contact detection and for auto-exposure and illumination control |
Country Status (6)
Country | Link |
---|---|
US (3) | US10512512B2 (en) |
EP (2) | EP3834702A1 (en) |
JP (4) | JP6707033B2 (en) |
KR (4) | KR102524391B1 (en) |
CN (2) | CN106456276B (en) |
WO (1) | WO2015142799A1 (en) |
Families Citing this family (33)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN106456276B (en) * | 2014-03-17 | 2019-12-10 | 直观外科手术操作公司 | System and method for tissue contact detection and for automatic exposure and illumination control |
JP2017118296A (en) * | 2015-12-24 | 2017-06-29 | キヤノン株式会社 | Imaging apparatus, image processing apparatus, image processing method, image processing program, and storage medium |
WO2017141544A1 (en) * | 2016-02-16 | 2017-08-24 | ソニー株式会社 | Image processing device, image processing method, and program |
WO2018180339A1 (en) * | 2017-03-30 | 2018-10-04 | Hoya株式会社 | Electronic endoscope device |
US11268444B2 (en) * | 2017-05-18 | 2022-03-08 | Raytheon Technologies Corporation | Turbine cooling arrangement |
US10560629B2 (en) | 2017-05-23 | 2020-02-11 | Google Llc | Systems and methods for automatic exposure in high dynamic range video capture systems |
CN107610187B (en) * | 2017-09-19 | 2021-01-26 | 海信视像科技股份有限公司 | Control method and device for image acquisition of medical endoscopic device and endoscopic system |
US20200306000A1 (en) * | 2017-12-06 | 2020-10-01 | Sony Olympus Medical Solutions Inc. | Medical control apparatus and control method |
JP6982677B2 (en) * | 2018-02-28 | 2021-12-17 | オリンパス株式会社 | Subject observation system, light source device for endoscope, operation method of subject observation system, operation program of subject observation system |
JP7073145B2 (en) * | 2018-03-12 | 2022-05-23 | ソニー・オリンパスメディカルソリューションズ株式会社 | How to operate the medical dimming control device and the medical dimming control device |
JP6929459B2 (en) * | 2018-05-21 | 2021-09-01 | オリンパス株式会社 | Endoscope system, endoscope processor, endoscope system control method, and program |
US10638921B2 (en) * | 2018-07-20 | 2020-05-05 | Arthrex, Inc. | Medical imaging system |
US10979648B2 (en) * | 2018-11-07 | 2021-04-13 | Inuitive Ltd. | Stereoscopic apparatus and a method for determining exposure time |
CN113260297B (en) * | 2018-11-12 | 2024-06-07 | 奥林巴斯株式会社 | Endoscope device, light source device for endoscope, method for operating the same, and method for adjusting light source |
EP3709628B1 (en) * | 2019-03-14 | 2020-12-23 | Axis AB | Control of an illuminator |
JP7252023B2 (en) * | 2019-03-22 | 2023-04-04 | ソニー・オリンパスメディカルソリューションズ株式会社 | Image processing device and medical observation system |
CN110049257B (en) * | 2019-05-31 | 2021-09-21 | 影石创新科技股份有限公司 | Method for determining semi-synchronous exposure parameters and electronic device |
WO2021026158A1 (en) | 2019-08-05 | 2021-02-11 | GYRUS ACMI, INC., d/b/a Olympus Surgical Technologies America | Signal coordinated delivery of laser therapy |
DE102019128554B4 (en) * | 2019-10-22 | 2024-05-16 | Schölly Fiberoptic GmbH | Endoscopy method for improved display of a video signal and associated endoscopy system and computer program product |
US11576554B2 (en) * | 2019-12-20 | 2023-02-14 | Verily Life Sciences Llc | Method for adjusting an exposure of an endoscope |
CN111948798B (en) * | 2020-08-21 | 2022-04-01 | 上海微创医疗机器人(集团)股份有限公司 | Endoscope system and method for detecting contact of tip of endoscope with tissue |
JP2022070310A (en) * | 2020-10-27 | 2022-05-13 | Hoya株式会社 | Light source device and endoscope system |
JP2022076076A (en) * | 2020-11-09 | 2022-05-19 | Hoya株式会社 | Light source device and endoscope system |
US11375141B1 (en) | 2021-02-09 | 2022-06-28 | Arthrex, Inc. | Endoscopic camera region of interest autoexposure |
CN112911165B (en) * | 2021-03-02 | 2023-06-16 | 杭州海康慧影科技有限公司 | Endoscope exposure method, device and computer readable storage medium |
US11812938B2 (en) | 2021-03-31 | 2023-11-14 | Moon Surgical Sas | Co-manipulation surgical system having a coupling mechanism removeably attachable to surgical instruments |
US12042241B2 (en) | 2021-03-31 | 2024-07-23 | Moon Surgical Sas | Co-manipulation surgical system having automated preset robot arm configurations |
US11819302B2 (en) | 2021-03-31 | 2023-11-21 | Moon Surgical Sas | Co-manipulation surgical system having user guided stage control |
CA3212211A1 (en) | 2021-03-31 | 2022-10-06 | David Paul Noonan | Co-manipulation surgical system for use with surgical instruments for performing laparoscopic surgery |
US11832909B2 (en) | 2021-03-31 | 2023-12-05 | Moon Surgical Sas | Co-manipulation surgical system having actuatable setup joints |
US11844583B2 (en) | 2021-03-31 | 2023-12-19 | Moon Surgical Sas | Co-manipulation surgical system having an instrument centering mode for automatic scope movements |
US11986165B1 (en) | 2023-01-09 | 2024-05-21 | Moon Surgical Sas | Co-manipulation surgical system for use with surgical instruments for performing laparoscopic surgery while estimating hold force |
US11832910B1 (en) | 2023-01-09 | 2023-12-05 | Moon Surgical Sas | Co-manipulation surgical system having adaptive gravity compensation |
Family Cites Families (50)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPS5789842A (en) | 1980-11-25 | 1982-06-04 | Olympus Optical Co | Endoscope apparatus |
JP2511883B2 (en) * | 1986-06-18 | 1996-07-03 | 株式会社日立製作所 | Fuel cell |
US5469840A (en) * | 1991-12-10 | 1995-11-28 | Olympus Optical, Ltd. | Electromotive warping type endoscope with velocity control |
JPH06300976A (en) * | 1993-04-12 | 1994-10-28 | Fuji Photo Optical Co Ltd | Electronic endoscope device performing light quantity stop control |
US6080104A (en) * | 1995-05-16 | 2000-06-27 | Asahi Kogaku Kogyo Kabushiki Kaisha | Electronic endoscope system |
US5647368A (en) * | 1996-02-28 | 1997-07-15 | Xillix Technologies Corp. | Imaging system for detecting diseased tissue using native fluorsecence in the gastrointestinal and respiratory tract |
US6331181B1 (en) | 1998-12-08 | 2001-12-18 | Intuitive Surgical, Inc. | Surgical robotic tools, data architecture, and use |
JPH10192232A (en) | 1997-01-14 | 1998-07-28 | Olympus Optical Co Ltd | Endoscope unit |
JP4349689B2 (en) * | 1998-06-26 | 2009-10-21 | Hoya株式会社 | Electronic endoscope |
JP4031591B2 (en) * | 1999-03-19 | 2008-01-09 | フジノン株式会社 | Electronic endoscope apparatus having a zooming function |
US6425858B1 (en) * | 1999-03-19 | 2002-07-30 | Fuji Photo Optical Co., Ltd. | Electronic endoscope apparatus having magnification changing function |
JP3235832B2 (en) | 1999-08-10 | 2001-12-04 | オリンパス光学工業株式会社 | Endoscope device |
US6464633B1 (en) * | 1999-08-23 | 2002-10-15 | Olympus Optical Co., Ltd. | Light source device for endoscope using DMD |
JP2002078670A (en) | 2000-06-26 | 2002-03-19 | Fuji Photo Film Co Ltd | Fluorescent imaging system |
EP1467197A1 (en) * | 2000-06-26 | 2004-10-13 | Fuji Photo Film Co., Ltd. | Fluorescent image obtaining apparatus |
JP2002085345A (en) | 2000-07-11 | 2002-03-26 | Asahi Optical Co Ltd | Electronic endoscope |
JP4100875B2 (en) | 2001-03-26 | 2008-06-11 | フジノン株式会社 | Electronic endoscope device |
KR100411631B1 (en) | 2001-10-18 | 2003-12-18 | 주식회사 메디미르 | Fluorescence endoscope apparatus and a method for imaging tissue within a body using the same |
JP3938675B2 (en) * | 2001-10-26 | 2007-06-27 | オリンパス株式会社 | Light source device |
US6511422B1 (en) * | 2002-04-30 | 2003-01-28 | Karl Storz Imaging, Inc. | Method and apparatus for protection from high intensity light |
JP2005040400A (en) | 2003-07-23 | 2005-02-17 | Olympus Corp | Optical observation probe |
WO2005101980A2 (en) * | 2004-04-26 | 2005-11-03 | Given Imaging Ltd. | Device, system, and method of wide dynamic range imaging |
JP2006055503A (en) | 2004-08-23 | 2006-03-02 | Pentax Corp | Electronic endoscope apparatus |
DE102005011121B4 (en) * | 2004-10-14 | 2016-04-28 | Möller-Wedel GmbH & Co. KG | Method for optimized adjustment of the light output in the object plane in reflected-light microscopes |
WO2006132191A1 (en) | 2005-06-08 | 2006-12-14 | Olympus Medical Systems Corp. | Endoscope and image processing device |
JP4908003B2 (en) | 2006-01-30 | 2012-04-04 | 富士フイルム株式会社 | Electronic endoscope device |
JP2009537229A (en) * | 2006-05-19 | 2009-10-29 | マコ サージカル コーポレーション | Method and apparatus for controlling a haptic device |
JP5279181B2 (en) | 2006-10-23 | 2013-09-04 | Hoya株式会社 | Endoscope device with automatic light control processing function, automatic light control device for endoscope, program for endoscope device, and method for operating endoscope device |
US20080108871A1 (en) * | 2006-11-06 | 2008-05-08 | Mohr Catherine J | Vacuum stabilized overtube for endoscopic surgery |
US8498695B2 (en) * | 2006-12-22 | 2013-07-30 | Novadaq Technologies Inc. | Imaging system with a single color image sensor for simultaneous fluorescence and color video endoscopy |
CN102176855B (en) * | 2008-10-24 | 2013-09-18 | 奥林巴斯医疗株式会社 | Capsular endoscopic system |
EP2441374B1 (en) * | 2009-06-10 | 2016-11-16 | Olympus Corporation | Capsule endoscope device |
US8512232B2 (en) | 2009-09-08 | 2013-08-20 | Gyrus Acmi, Inc. | Endoscopic illumination system, assembly and methods for staged illumination of different target areas |
US9211058B2 (en) * | 2010-07-02 | 2015-12-15 | Intuitive Surgical Operations, Inc. | Method and system for fluorescent imaging with background surgical image composed of selective illumination spectra |
DE102010033423A1 (en) * | 2010-08-04 | 2012-02-09 | Karl Storz Gmbh & Co. Kg | Endoscope with adjustable viewing direction |
JP2012065690A (en) | 2010-09-21 | 2012-04-05 | Olympus Corp | Endoscope apparatus and method of controlling endoscope apparatus |
JP5653163B2 (en) | 2010-10-19 | 2015-01-14 | Hoya株式会社 | Endoscope device |
EP2514352B1 (en) * | 2010-10-26 | 2018-03-28 | Olympus Corporation | Endoscope |
JP5258869B2 (en) * | 2010-12-13 | 2013-08-07 | 富士フイルム株式会社 | Endoscope device |
US8698885B2 (en) * | 2011-02-14 | 2014-04-15 | Intuitive Surgical Operations, Inc. | Methods and apparatus for demosaicing images with highly correlated color channels |
WO2012127469A1 (en) | 2011-03-22 | 2012-09-27 | Given Imaging Ltd. | Systems and methods for synchronizing between an in-vivo device and a localization system |
JP5865606B2 (en) * | 2011-05-27 | 2016-02-17 | オリンパス株式会社 | Endoscope apparatus and method for operating endoscope apparatus |
US8878920B2 (en) * | 2011-07-12 | 2014-11-04 | Karl Storz Imaging, Inc. | Method and apparatus for protection from high intensity light |
JP5814698B2 (en) | 2011-08-25 | 2015-11-17 | オリンパス株式会社 | Automatic exposure control device, control device, endoscope device, and operation method of endoscope device |
CN102694981B (en) * | 2012-05-11 | 2014-10-29 | 中国科学院西安光学精密机械研究所 | Automatic exposure method for histogram equalization based on adaptive threshold segmentation |
US9277205B2 (en) * | 2012-05-14 | 2016-03-01 | Intuitive Surgical Operations, Inc. | Single-chip sensor multi-function imaging |
JP5623469B2 (en) * | 2012-07-06 | 2014-11-12 | 富士フイルム株式会社 | ENDOSCOPE SYSTEM, ENDOSCOPE SYSTEM PROCESSOR DEVICE, AND ENDOSCOPE CONTROL PROGRAM |
US9413976B2 (en) | 2012-08-08 | 2016-08-09 | Intuitive Surgical Operations, Inc. | Auto exposure of a camera in a surgical robot |
US20140052004A1 (en) * | 2012-08-15 | 2014-02-20 | Arthrex, Inc. | Endoscopic camera illumination system and method |
CN106456276B (en) * | 2014-03-17 | 2019-12-10 | 直观外科手术操作公司 | System and method for tissue contact detection and for automatic exposure and illumination control |
-
2015
- 2015-03-17 CN CN201580025492.2A patent/CN106456276B/en active Active
- 2015-03-17 JP JP2016558117A patent/JP6707033B2/en active Active
- 2015-03-17 WO PCT/US2015/020892 patent/WO2015142799A1/en active Application Filing
- 2015-03-17 EP EP21154425.9A patent/EP3834702A1/en active Pending
- 2015-03-17 KR KR1020227025574A patent/KR102524391B1/en active IP Right Grant
- 2015-03-17 US US15/126,960 patent/US10512512B2/en active Active
- 2015-03-17 CN CN201911105827.8A patent/CN110772205B/en active Active
- 2015-03-17 KR KR1020237013025A patent/KR102587513B1/en active IP Right Grant
- 2015-03-17 KR KR1020167028336A patent/KR102278806B1/en active IP Right Grant
- 2015-03-17 KR KR1020217021895A patent/KR102425786B1/en active IP Right Grant
- 2015-03-17 EP EP15764341.2A patent/EP3119318B1/en active Active
-
2019
- 2019-11-18 US US16/687,549 patent/US11331156B2/en active Active
-
2020
- 2020-05-19 JP JP2020087284A patent/JP6972233B2/en active Active
-
2021
- 2021-11-02 JP JP2021179417A patent/JP7376552B2/en active Active
-
2022
- 2022-03-30 US US17/709,001 patent/US20220218424A1/en active Pending
-
2023
- 2023-10-26 JP JP2023184032A patent/JP2023178470A/en active Pending
Also Published As
Publication number | Publication date |
---|---|
KR20220110596A (en) | 2022-08-08 |
KR102524391B1 (en) | 2023-04-24 |
WO2015142799A1 (en) | 2015-09-24 |
US20220218424A1 (en) | 2022-07-14 |
CN110772205B (en) | 2022-04-12 |
EP3119318A4 (en) | 2017-12-13 |
JP6707033B2 (en) | 2020-06-10 |
US20200078108A1 (en) | 2020-03-12 |
KR20160134739A (en) | 2016-11-23 |
EP3834702A1 (en) | 2021-06-16 |
JP6972233B2 (en) | 2021-11-24 |
JP2020171710A (en) | 2020-10-22 |
KR20210091357A (en) | 2021-07-21 |
JP2017510348A (en) | 2017-04-13 |
JP2023178470A (en) | 2023-12-14 |
US10512512B2 (en) | 2019-12-24 |
KR102425786B1 (en) | 2022-07-28 |
KR20230058176A (en) | 2023-05-02 |
KR102278806B1 (en) | 2021-07-19 |
EP3119318B1 (en) | 2021-03-10 |
US11331156B2 (en) | 2022-05-17 |
CN110772205A (en) | 2020-02-11 |
JP7376552B2 (en) | 2023-11-08 |
CN106456276B (en) | 2019-12-10 |
KR102587513B1 (en) | 2023-10-11 |
CN106456276A (en) | 2017-02-22 |
US20170095297A1 (en) | 2017-04-06 |
JP2022009921A (en) | 2022-01-14 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11331156B2 (en) | System and method for tissue contact detection and for auto-exposure and illumination control | |
US11759093B2 (en) | Surgical system including a non-white light general illuminator | |
US20230092807A1 (en) | Method for adjusting an exposure of an endoscope | |
JP5855795B2 (en) | Endoscope device | |
US11743596B1 (en) | Adaptive brightness non-uniformity correction in endoscope visualization | |
JPH10165363A (en) | Endoscopic image pickup signal processing device |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE |
|
PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE |
|
17P | Request for examination filed |
Effective date: 20161017 |
|
AK | Designated contracting states |
Kind code of ref document: A1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
AX | Request for extension of the european patent |
Extension state: BA ME |
|
DAV | Request for validation of the european patent (deleted) | ||
DAX | Request for extension of the european patent (deleted) | ||
A4 | Supplementary search report drawn up and despatched |
Effective date: 20171113 |
|
RIC1 | Information provided on ipc code assigned before grant |
Ipc: A61B 1/06 20060101ALI20171107BHEP Ipc: A61B 90/00 20160101AFI20171107BHEP |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: EXAMINATION IS IN PROGRESS |
|
17Q | First examination report despatched |
Effective date: 20200323 |
|
GRAP | Despatch of communication of intention to grant a patent |
Free format text: ORIGINAL CODE: EPIDOSNIGR1 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: GRANT OF PATENT IS INTENDED |
|
INTG | Intention to grant announced |
Effective date: 20200930 |
|
GRAS | Grant fee paid |
Free format text: ORIGINAL CODE: EPIDOSNIGR3 |
|
GRAA | (expected) grant |
Free format text: ORIGINAL CODE: 0009210 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE PATENT HAS BEEN GRANTED |
|
AK | Designated contracting states |
Kind code of ref document: B1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
REG | Reference to a national code |
Ref country code: GB Ref legal event code: FG4D |
|
RIN1 | Information on inventor provided before grant (corrected) |
Inventor name: RICHMOND, GEOFF Inventor name: HOFFMAN, BRIAN D. Inventor name: DICARLO, JEFFREY |
|
REG | Reference to a national code |
Ref country code: AT Ref legal event code: REF Ref document number: 1368950 Country of ref document: AT Kind code of ref document: T Effective date: 20210315 Ref country code: CH Ref legal event code: EP |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R096 Ref document number: 602015066674 Country of ref document: DE |
|
REG | Reference to a national code |
Ref country code: IE Ref legal event code: FG4D |
|
REG | Reference to a national code |
Ref country code: LT Ref legal event code: MG9D |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: LT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210310 Ref country code: FI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210310 Ref country code: GR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210611 Ref country code: HR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210310 Ref country code: BG Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210610 Ref country code: NO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210610 |
|
REG | Reference to a national code |
Ref country code: AT Ref legal event code: MK05 Ref document number: 1368950 Country of ref document: AT Kind code of ref document: T Effective date: 20210310 |
|
REG | Reference to a national code |
Ref country code: NL Ref legal event code: MP Effective date: 20210310 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: RS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210310 Ref country code: LV Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210310 Ref country code: SE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210310 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: NL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210310 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: EE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210310 Ref country code: CZ Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210310 Ref country code: AT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210310 Ref country code: SM Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210310 |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: PL |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: IS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210710 Ref country code: PL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210310 Ref country code: PT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210712 Ref country code: ES Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210310 Ref country code: RO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210310 Ref country code: SK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210310 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R097 Ref document number: 602015066674 Country of ref document: DE |
|
REG | Reference to a national code |
Ref country code: BE Ref legal event code: MM Effective date: 20210331 |
|
PLBE | No opposition filed within time limit |
Free format text: ORIGINAL CODE: 0009261 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: DK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210310 Ref country code: IE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20210317 Ref country code: MC Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210310 Ref country code: LU Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20210317 Ref country code: LI Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20210331 Ref country code: AL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210310 Ref country code: CH Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20210331 |
|
26N | No opposition filed |
Effective date: 20211213 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210310 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: IT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210310 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: IS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210710 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: BE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20210331 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: HU Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO Effective date: 20150317 |
|
P01 | Opt-out of the competence of the unified patent court (upc) registered |
Effective date: 20230510 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: CY Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210310 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210310 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: DE Payment date: 20240328 Year of fee payment: 10 Ref country code: GB Payment date: 20240319 Year of fee payment: 10 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: FR Payment date: 20240326 Year of fee payment: 10 |