CN113795862B - Depth of field synthesis system, camera and microscope - Google Patents

Depth of field synthesis system, camera and microscope Download PDF

Info

Publication number
CN113795862B
CN113795862B CN201980096048.8A CN201980096048A CN113795862B CN 113795862 B CN113795862 B CN 113795862B CN 201980096048 A CN201980096048 A CN 201980096048A CN 113795862 B CN113795862 B CN 113795862B
Authority
CN
China
Prior art keywords
depth
image
module
field
unit
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201980096048.8A
Other languages
Chinese (zh)
Other versions
CN113795862A (en
Inventor
庄堉
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shenzhen Sunsight Technologies Co ltd
Original Assignee
Shenzhen Sunsight Technologies Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shenzhen Sunsight Technologies Co ltd filed Critical Shenzhen Sunsight Technologies Co ltd
Publication of CN113795862A publication Critical patent/CN113795862A/en
Application granted granted Critical
Publication of CN113795862B publication Critical patent/CN113795862B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00Image enhancement or restoration
    • G06T5/50Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • GPHYSICS
    • G02OPTICS
    • G02BOPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B7/00Mountings, adjusting means, or light-tight connections, for optical elements
    • G02B7/28Systems for automatic generation of focusing signals
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/10Image acquisition modality
    • G06T2207/10141Special mode during image acquisition
    • G06T2207/10148Varying focus
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/20Special algorithmic details
    • G06T2207/20212Image combination
    • G06T2207/20221Image fusion; Image merging

Landscapes

  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Optics & Photonics (AREA)
  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Studio Devices (AREA)
  • Microscoopes, Condenser (AREA)

Abstract

The invention discloses a depth of field synthesis system, which comprises an image distance driving unit, a main control unit, a depth of field synthesis unit and an image transmission unit; the image distance driving unit changes the image distance of the object according to the control command of the main control unit, acquires images of different image distances of the object, and transmits the images to the main control unit; the main control unit processes the received image to form a data stream related to the image distance change, and the data stream is selected to be transmitted to the depth of field synthesizing unit to provide effective data and depth of field data for the depth of field synthesizing unit; or selecting to transmit the data stream to the image transmission unit; the depth of field synthesis unit realizes the depth of field synthesis of the image based on the data stream and transmits the depth of field synthesis to the image transmission unit; the image transmission unit outputs a data stream or a depth-of-field synthesized image. The invention can realize the optical motion required by the depth of field synthesis without changing the existing optical system, and collect images of different focal planes so as to meet the calculation requirement of a back-end system.

Description

Depth of field synthesis system, camera and microscope
Technical Field
The invention relates to the technical field of image processing, in particular to a depth of field synthesis system, a camera and a microscope.
Background
The traditional equipment with a depth of field synthesis system generally has 3 independent systems, mainly comprises an optical system, an imaging system and a depth of field synthesis algorithm system; the optical system is used for adjusting the object distance of optical imaging to change different focusing planes of a photographed object so as to acquire clear images (focusing images) of different heights of the photographed object. Imaging systems typically consist of a digital camera or image acquisition system for converting optical information into digital information. The depth of field synthesis algorithm system is composed of a computer or an image processing system with strong processing capacity, and is used for carrying out focus calculation on pictures with different focus planes and then synthesizing all the pictures into a clear picture. The whole device is a rather complex system, requiring close fitting of the individual systems, including optical interfaces, mechanical control, image data transmission, image data processing, etc.
The device requires that the optical system has an auto-focus or a focus-adjustable lens as a support, and normal operation of the device cannot be achieved without a drivable lens. While most industrial applications, or microscopy applications, use mechanical lenses that cannot be driven externally, all subject to conditions that make use of devices with depth of view synthesis systems. If the lens is changed into a lens with driving capability, the cost of the equipment is increased, the use threshold is also increased, and the lens cannot be really used in a large amount.
Disclosure of Invention
The invention provides a depth of field synthesis system, a camera and a microscope, which can realize optical movement required by depth of field synthesis under the condition that an optical system does not have a common lens capable of driving, such as fixed focus or zooming, a zoom microscope, a stereoscopic microscope and the like, and can acquire images of different focal planes under the condition that the existing optical system is not changed, so that the computing requirement of a back-end system is met.
The technical scheme of the invention is realized as follows:
A depth of field synthesis system comprises an image distance driving unit, a main control unit, a depth of field synthesis unit and an image transmission unit;
The image distance driving unit changes the image distance of the object according to the control command of the main control unit, acquires images of different image distances of the object, and transmits the images to the main control unit;
The main control unit processes the received image to form a data stream related to the image distance change, and the data stream is selected to be transmitted to the depth of field synthesizing unit to provide effective data and depth of field data for the depth of field synthesizing unit; or selecting to transmit a data stream to the image transmission unit;
The depth of field synthesis unit realizes the depth of field synthesis of the image based on the data stream and transmits the depth of field synthesis to the image transmission unit;
the image transmission unit outputs a data stream or a depth-of-field synthesized image.
As a preferred embodiment of the present invention, the image distance driving unit includes a sensor acquisition module, a motor driving module and a motor limiting module, where the motor driving module is fixedly connected with the sensor acquisition module, and the motor limiting module is fixed at an end position of the sensor acquisition module in a movement direction; the motor driving module receives a motor driving command of the main control unit, drives the sensor acquisition module to move, and adjusts the image distance; the sensor acquisition module acquires object images with different image distances; the motor limiting module monitors the position information of the sensor acquisition module, transmits the position information to the main control unit and generates depth information of object images corresponding to different image distances.
As a preferred embodiment of the present invention, the main control unit includes an image distance driving algorithm module, an image data preprocessing module and an image control module;
The image distance driving algorithm module converts the image distance variation parameters into motor driving parameters and sends the motor driving parameters to the motor driving module;
the image data preprocessing module performs preprocessing operation on the image acquired by the sensor acquisition module and transmits the image data preprocessing operation to the image control module;
and the image control module carries out parameter adjustment processing on the image after preprocessing operation.
As a preferred embodiment of the present invention, the depth of view synthesizing unit includes a depth of view control module and a depth of view synthesizing module,
The depth of field control module sets a motion interval corresponding to depth of field synthesis according to the depth of field synthesis requirement; or synthesizing a corresponding motion interval according to the focusing parameters and the self-learning depth of field; transmitting the motion interval and the motion step to the image distance driving algorithm module;
The depth of field synthesizing module collects images with different depth of field, calculates focusing parameters in each pixel point and the limiting area in the images, synthesizes the images according to the focusing parameters, selects images and depth information which have the largest focusing parameters and meet the conditions from all image data in the same position, synthesizes the images into one image until all the images are collected.
As a preferred embodiment of the present invention, the image transmission unit includes one or more of an HDMI interface, a DVI interface, a USB interface, and a network interface.
A camera comprises the depth of field synthesizing system.
A microscope comprising said depth of field synthesis system.
The invention has the beneficial effects that: on the premise that the optical system does not have a common lens capable of driving, such as fixed focus or zooming, a zoom microscope, a stereoscopic microscope and the like, the optical motion required by depth of field synthesis can be realized under the condition that the existing optical system is not changed, and images of different focal planes are acquired, so that the calculation requirement of a back-end system is met.
Drawings
In order to more clearly illustrate the embodiments of the invention or the technical solutions of the prior art, the drawings which are used in the description of the embodiments or the prior art will be briefly described, it being obvious that the drawings in the description below are only some embodiments of the invention, and that other drawings can be obtained according to these drawings without inventive faculty for a person skilled in the art.
FIG. 1 is a schematic diagram of the principle of image imaging;
fig. 2 is a schematic block diagram of a depth of field synthesis system according to the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
As shown in fig. 1, an object positioned in front of the lens forms an image after the lens, and the main plane of the lens is P; the focal point of the lens is F, F ', and the corresponding focal length is F, F'; the object is at a distance s from the main plane, and the image is at a distance s' from the main plane; equation 1 can be derived from the principle of gaussian imaging:
If the object is located at s, a clear real image can be acquired at the position at s', which is called the quasi-focal plane, and if the object is located away from the plane, the real image is called the out-of-focus plane; a clear image of the object can be obtained at the quasi-focal plane and a blurred image at the off-focal plane.
In a traditional camera, a focusing module of the lens is used for automatically adjusting the focus position of the lens without changing the image distance s', and the obtained object is clearly imaged at different s positions.
The general fixed focus lens and microscope can obtain clear imaging of the object under different distances from the lens by adjusting the distance from the main plane P of the lens to the object under the condition that the focal length and the image distance are not changed.
Both of the above conventional approaches are to condition the front-end optical parameters or positions to achieve image acquisition of different focal planes of the object, which has a limitation that the front-end requires a moving structure to achieve the above parameter changes. While most lenses are either not provided with this functionality or some lenses may be custom made with the above functionality, but at a significant cost.
From the above formula, it can be seen that if the image distance s' is adjusted without changing the positional relationship between the lens and the object to the main plane, the object distance s is changed, and a clear picture of the object in different planes is obtained. Therefore, in view of the above practical situation, the invention proposes a method for changing the image distance to realize the image acquisition of the depth of field synthesis on different focal planes of the object, and the adjusted motion structure is moved to the rear end of the optical system and integrated into the device of the invention, and all applicable lenses are not limited. That is, the invention acquires images of different planes of the photographed object by changing the image distance, and since the adjusted structure is moved into the device, all the front-end optical systems are not limited any more, so that the range of the depth of field synthesizing system can be wider.
As shown in fig. 2, the present invention provides a depth of field synthesis system, which includes an image distance driving unit, a main control unit, a depth of field synthesis unit and an image transmission unit;
The image distance driving unit changes the image distance of the object according to the control command of the main control unit, acquires images of different image distances of the object, and transmits the images to the main control unit;
The image distance driving unit comprises a sensor acquisition module, a motor driving module and a motor limiting module, wherein the motor driving module is fixedly connected with the sensor acquisition module, and the motor limiting module is fixed at the end position of the movement direction of the sensor acquisition module; the motor driving module receives a motor driving command of the main control unit and drives the sensor acquisition module to move, such as a rotation direction, a stroke and the like, so as to adjust an image distance; the sensor acquisition module acquires object images with different image distances, transmits image data to the main control unit at the rear end, is used as source data for image processing, and can also receive control signals transmitted by the main control unit to adjust parameters; the motor limiting module is mainly used for monitoring whether the sensor acquisition module reaches the maximum, minimum or original point position, so that the main control unit can monitor the position of the sensor acquisition module in real time and form depth information of an image. If the sensor acquisition module reaches the limit position, the motor limit module sends a signal to inform the main control unit, so that the motor limit position is monitored.
The main control unit processes the received image to form a data stream related to the image distance change, and the data stream is selected to be transmitted to the depth of field synthesizing unit to provide effective data and depth of field data for the depth of field synthesizing unit; or selecting to transmit the data stream to the image transmission unit;
The main control unit comprises an image distance driving algorithm module, an image data preprocessing module and an image control module; the image data transmitted from the front end is preprocessed, so that the image data can meet the picture or video of the display standard and is output to the display end. And meanwhile, the image control module is used for adjusting image parameters such as brightness, color gain, image contrast and the like of the image. And the position of the sensor acquisition module is adjusted according to the depth of field synthesis requirement to obtain image data of different focal planes of the photographed object, and the depth information transmitted by the motor limiting module is related to the image data photographed by each frame, so that the depth of field synthesis unit is convenient to use.
The image distance driving algorithm module converts the image distance variation parameters into motor driving parameters and sends the motor driving parameters to the motor driving module;
The image data preprocessing module performs preprocessing operation on the image acquired by the sensor acquisition module, wherein the preprocessing operation comprises data decoding, data conversion, noise reduction, color restoration and other operations, so that the image data accords with the standard of image display or video display, and the image data is transmitted to the image control module;
And the image control module carries out parameter adjustment processing on the image after the preprocessing operation. Such as adjusting brightness, contrast, color temperature, etc. of the image, so that the image can restore the reality under various environments.
The depth of field synthesis unit realizes the depth of field synthesis of the image based on the data stream and transmits the depth of field synthesis to the image transmission unit; the depth of field control module and the depth of field synthesis module are used for acquiring images with different depth of field, calculating focusing parameters in each pixel point and a limited area in the images, synthesizing the images according to the focusing parameters, selecting the image and depth information which have the largest focusing parameters and meet the conditions from all image data in the same position, and synthesizing the image and the depth information into one image until all the images are acquired.
The depth of field control module can manually set a motion section (a start area/an end area) corresponding to the depth of field synthesis according to the requirements of the depth of field synthesis; or synthesizing a corresponding motion interval according to the focusing parameters and the self-learning depth of field; transmitting the motion interval and the motion step to an image distance driving algorithm module;
The depth of field synthesis module is mainly to calculate the focusing parameters in each pixel and small area of each frame of image by collecting the image data and depth information processed by the front end image data preprocessing module, compare the data with the accumulated result of each frame, and if the focusing parameters of the current frame are larger than the accumulated focusing parameters and meet the limiting conditions of the area, replace the current focusing parameter data into the accumulated focusing parameter data and replace the current depth information into the accumulated depth data. After the next frame of image comes, the same data processing is carried out until all the image acquisition is completed. When the sensor board moves from the starting position to the ending position and all images are acquired and calculated, the module converts the accumulated data into a depth-of-field synthesized image and transmits the depth-of-field synthesized image to the image output unit.
The image transmission unit outputs a data stream or a depth-of-field synthesized image. The image transmission unit includes one or more of an HDMI interface, a DVI interface, a USB interface, and a network interface. Video stream data output from the front end is displayed on a display through DHMI interfaces or DVI interfaces, and meanwhile, data can be transmitted to a computer or other devices through interfaces such as USB, network and the like. The data after depth of field synthesis can also be transmitted to the equipment for use.
The invention also provides a camera comprising a depth of field synthesis system.
The invention also provides a microscope comprising a depth of field synthesis system.
The invention can realize the optical movement required by the depth of field synthesis and collect the images of different focal planes on the premise that the optical system does not have a common lens capable of driving, such as fixed focus or zooming, a zoom microscope, a stereoscopic microscope and the like, and can meet the calculation requirement of a back-end system under the condition that the existing optical system is not changed.
The foregoing description of the preferred embodiments of the invention is not intended to be limiting, but rather is intended to cover all modifications, equivalents, alternatives, and improvements that fall within the spirit and scope of the invention.

Claims (6)

1. A depth of field composition system, characterized by: the system comprises an image distance driving unit, a main control unit, a depth of field synthesizing unit and an image transmission unit;
The image distance driving unit changes the image distance of the object according to the control command of the main control unit, acquires images of different image distances of the object, and transmits the images to the main control unit;
The main control unit processes the received image to form a data stream related to the image distance change, and the data stream is selected to be transmitted to the depth of field synthesizing unit to provide effective data and depth of field data for the depth of field synthesizing unit; or selecting to transmit a data stream to the image transmission unit;
The depth of field synthesis unit realizes the depth of field synthesis of the image based on the data stream and transmits the depth of field synthesis to the image transmission unit; the depth of field synthesis unit comprises a depth of field synthesis module, wherein the depth of field synthesis module calculates the focusing parameters in each pixel and small area of each frame of image by collecting the image data and depth information processed by the front end image data preprocessing module, compares the focusing parameters with the accumulated result of each frame before, if the focusing parameters of the current frame are larger than the accumulated focusing parameters and meet the limiting conditions of the area, replaces the current focusing parameter data into the accumulated focusing parameter data, and replaces the current depth information into the accumulated depth data, after the next frame of image comes, the same data processing is carried out until all the images are collected, when the sensor plate moves from the starting position to the ending position, and after all the images are collected and calculated, the depth of field synthesis module converts the accumulated data into a depth of field synthesized image and transmits the depth of field synthesized image to the image output unit;
The image transmission unit outputs a data stream or an image after depth of field synthesis;
The image distance driving unit comprises a sensor acquisition module, a motor driving module and a motor limiting module, wherein the motor driving module is fixedly connected with the sensor acquisition module, and the motor limiting module is fixed at the end position of the sensor acquisition module in the movement direction; the motor driving module receives a motor driving command of the main control unit, drives the sensor acquisition module to move, and adjusts the image distance; the sensor acquisition module acquires object images with different image distances; the motor limiting module monitors the position information of the sensor acquisition module, transmits the position information to the main control unit and generates depth information of object images corresponding to different image distances; the motor limiting module is used for monitoring whether the sensor acquisition module reaches the maximum, minimum or origin position, so that the main control unit monitors the position of the sensor acquisition module in real time to form depth information of an image; if the sensor acquisition module reaches the limit position, the motor limit module sends a signal to inform the main control unit, so that the motor limit position is monitored.
2. A depth of field composition system according to claim 1, wherein: the main control unit comprises an image distance driving algorithm module, an image data preprocessing module and an image control module;
The image distance driving algorithm module converts the image distance variation parameters into motor driving parameters and sends the motor driving parameters to the motor driving module;
the image data preprocessing module performs preprocessing operation on the image acquired by the sensor acquisition module and transmits the image data preprocessing operation to the image control module;
and the image control module carries out parameter adjustment processing on the image after preprocessing operation.
3. A depth of field composition system according to claim 2, wherein: the depth of field synthesis unit comprises a depth of field control module;
The depth of field control module sets a motion interval corresponding to depth of field synthesis according to the depth of field synthesis requirement; or synthesizing a corresponding motion interval according to the focusing parameters and the self-learning depth of field; and transmitting the motion interval and the motion step to the image distance driving algorithm module.
4. A depth of field composition system according to any one of claims 1-3, wherein: the image transmission unit includes one or more of an HDMI interface, a DVI interface, a USB interface, and a network interface.
5. A camera comprising the depth of view synthesis system of any one of claims 1-3.
6. A microscope comprising a depth of field synthesis system as claimed in any one of claims 1 to 3.
CN201980096048.8A 2019-11-27 2019-11-27 Depth of field synthesis system, camera and microscope Active CN113795862B (en)

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2019/121127 WO2021102716A1 (en) 2019-11-27 2019-11-27 Depth-of-field synthesis system, camera, and microscope

Publications (2)

Publication Number Publication Date
CN113795862A CN113795862A (en) 2021-12-14
CN113795862B true CN113795862B (en) 2024-06-21

Family

ID=76129775

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201980096048.8A Active CN113795862B (en) 2019-11-27 2019-11-27 Depth of field synthesis system, camera and microscope

Country Status (2)

Country Link
CN (1) CN113795862B (en)
WO (1) WO2021102716A1 (en)

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5282045A (en) * 1990-04-27 1994-01-25 Hitachi, Ltd. Depth-of-field control apparatus and image pickup apparatus having the same therein
CN101233440A (en) * 2005-06-24 2008-07-30 诺基亚公司 Adaptive optical plane formation with rolling shutter

Family Cites Families (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101976673B (en) * 2010-09-29 2012-05-23 庄鸿 Image sensor chip
CN102314683B (en) * 2011-07-15 2013-01-16 清华大学 Computational imaging method and imaging system based on nonplanar image sensor
US9519972B2 (en) * 2013-03-13 2016-12-13 Kip Peli P1 Lp Systems and methods for synthesizing images from image data captured by an array camera using restricted depth of field depth maps in which depth estimation precision varies
CN104463817A (en) * 2013-09-12 2015-03-25 华为终端有限公司 Image processing method and device
CN104506771B (en) * 2014-12-18 2018-04-27 北京智谷睿拓技术服务有限公司 Image processing method and device
JP2017050605A (en) * 2015-08-31 2017-03-09 キヤノン株式会社 Image processing apparatus and method, imaging device, program
CN106210520B (en) * 2015-11-05 2019-03-19 杭州舜立光电科技有限公司 A kind of automatic focusing electronic eyepiece and system
US10270973B2 (en) * 2015-12-08 2019-04-23 Canon Kabushiki Kaisha Control device and imaging apparatus with a subject selectable mode with higher image blur correction
CN107493413A (en) * 2017-09-01 2017-12-19 佛山科学技术学院 A kind of big depth of field digital imaging method based on simple lens imaging
CN209265065U (en) * 2019-01-31 2019-08-16 周良云 A kind of adjustable microscope stand
CN109803133B (en) * 2019-03-15 2023-04-11 京东方科技集团股份有限公司 Image processing method and device and display device

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5282045A (en) * 1990-04-27 1994-01-25 Hitachi, Ltd. Depth-of-field control apparatus and image pickup apparatus having the same therein
CN101233440A (en) * 2005-06-24 2008-07-30 诺基亚公司 Adaptive optical plane formation with rolling shutter

Also Published As

Publication number Publication date
WO2021102716A1 (en) 2021-06-03
CN113795862A (en) 2021-12-14

Similar Documents

Publication Publication Date Title
JP5870264B2 (en) Imaging apparatus, imaging method, program, and integrated circuit
US20100265365A1 (en) Camera having image correction function, apparatus and image correction method
EP3320676B1 (en) Image capturing apparatus and method of operating the same
KR20100039430A (en) Image processor, image processing method, digital camera, and imaging apparatus
CN105527778A (en) Automatic focusing method for electric adjustable liquid lens
JP2011029905A (en) Imaging device, method and program
CN108024058B (en) Image blurs processing method, device, mobile terminal and storage medium
JP2010187225A (en) Imaging apparatus and imaging method
KR20070007437A (en) Auto focusing apparatus of camera and method of auto focusing
CN107872631B (en) Image shooting method and device based on double cameras and mobile terminal
CN103905735B (en) The mobile terminal and its dynamic for chasing after shooting function with dynamic chase after shooting method
CN106506939B (en) Image acquisition device and acquisition method
US20040036792A1 (en) Camera system and focus information display apparatus
CN201667683U (en) Large focal variation ratio automatic focusing all-in-one video camera
CN110753182B (en) Adjusting method and apparatus of image forming apparatus
CN112822371B (en) Image processing chip, application processing chip, data statistical system and method
CN107147848B (en) Automatic focusing method and real-time video acquisition system adopting same
JP2015211302A (en) Imaging apparatus and imaging system
TWI515503B (en) Automatic-focusing imaging capture device and imaging capture method
CN113795862B (en) Depth of field synthesis system, camera and microscope
JP2010156850A (en) Focus adjustment device and method
CN104717419A (en) Auto-focusing image capturing device and method
CN116208851A (en) Image processing method and related device
CN101867705A (en) Large-zoom ratio automatic focusing integrated camera
JP6006506B2 (en) Image processing apparatus, image processing method, program, and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant