WO2020138128A1 - Dispositif de traitement d'image, procédé de traitement d'image et programme - Google Patents

Dispositif de traitement d'image, procédé de traitement d'image et programme Download PDF

Info

Publication number
WO2020138128A1
WO2020138128A1 PCT/JP2019/050732 JP2019050732W WO2020138128A1 WO 2020138128 A1 WO2020138128 A1 WO 2020138128A1 JP 2019050732 W JP2019050732 W JP 2019050732W WO 2020138128 A1 WO2020138128 A1 WO 2020138128A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
unit
learned model
display
medical
Prior art date
Application number
PCT/JP2019/050732
Other languages
English (en)
Japanese (ja)
Inventor
牧平　朋之
弘樹 内田
律也 富田
Original Assignee
キヤノン株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from JP2019217331A external-priority patent/JP7341874B2/ja
Application filed by キヤノン株式会社 filed Critical キヤノン株式会社
Priority to CN201980086346.9A priority Critical patent/CN113226153A/zh
Publication of WO2020138128A1 publication Critical patent/WO2020138128A1/fr
Priority to US17/343,207 priority patent/US20210304363A1/en

Links

Images

Classifications

    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B3/00—Apparatus for testing the eyes; Instruments for examining the eyes
    • A61B3/10—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions

Definitions

  • the present invention relates to an image processing device, an image processing method, and a program.
  • OCT device As a method of non-destructively and non-invasively acquiring a tomographic image of a subject such as a living body, a device (OCT device) using optical coherence tomography (OCT) has been put into practical use.
  • OCT apparatus is widely used as an ophthalmic apparatus that acquires an image for ophthalmic diagnosis.
  • TD-OCT Time Domain OCT
  • the depth information of the subject is obtained by sequentially changing the position of the reference mirror.
  • SD-OCT Spectral Domain OCT
  • SS-OCT Sweep Source OCT
  • SD-OCT interference light that is caused to interfere with low coherence light is dispersed, and depth information is replaced with frequency information to be acquired.
  • SS-OCT interference light is acquired by using light whose wavelength has been previously dispersed using a wavelength swept light source.
  • SD-OCT and SS-OCT are also collectively called Fourier domain OCT (FD-OCT: Fourier Domain OCT).
  • a tomographic image based on the depth information of the subject can be acquired.
  • the acquired three-dimensional tomographic image is integrated in the depth direction and projected on a two-dimensional plane, whereby a front image of the measurement target can be generated.
  • images have been acquired a plurality of times and subjected to overlay processing. However, in such a case, it takes time to take a plurality of images.
  • Patent Document 1 discloses a technique of converting a previously acquired image into an image with higher resolution by an artificial intelligence engine in order to cope with rapid progress of medical technology and simple imaging in an emergency. .. According to such a technique, for example, an image acquired by taking less images can be converted into an image with higher resolution.
  • an image with high resolution may not be suitable for image diagnosis.
  • the object to be observed may not be properly grasped when there is a lot of noise or the contrast is low.
  • one of the objects of the present invention is to provide an image processing apparatus, an image processing method, and a program capable of generating an image more suitable for image diagnosis than ever before.
  • An image processing apparatus uses a learned model to generate a second medical image obtained by performing an image quality improvement process on the first medical image of the subject from the first medical image. And an image quality improving unit for generating an image quality, a comparing unit for comparing an analysis result obtained by analyzing the first medical image with an analysis result obtained by analyzing the second medical image, and the comparing unit.
  • a display control unit that controls the display of the display unit based on the comparison result.
  • 1 shows a schematic configuration of an OCT apparatus according to a first embodiment.
  • 1 shows a schematic configuration of a control unit according to the first embodiment.
  • 3 shows an example of teacher data according to the first embodiment.
  • 3 shows an example of teacher data according to the first embodiment.
  • An example of the structure of the learned model which concerns on Example 1 is shown.
  • 6 is a flowchart of a series of image processing according to the first embodiment.
  • An example of the report screen which switches and displays the image before and behind an image quality improvement process is shown.
  • An example of the report screen which switches and displays the image before and behind an image quality improvement process is shown.
  • An example of the report screen which displays the image before and behind an image quality improvement process side by side is shown.
  • An example of the report screen which displays simultaneously several images before an image quality improvement process is applied is shown.
  • FIG. 3 shows a schematic configuration of a control unit according to a second embodiment.
  • 9 is a flowchart of a series of image processing according to the second embodiment.
  • An example of changing the image quality improvement processing will be described.
  • An example of changing the image quality improvement processing will be described.
  • An example of the report screen which displays simultaneously several images before an image quality improvement process is applied is shown.
  • An example of the report screen which displays simultaneously the some image to which the image quality improvement process was applied is shown.
  • 9 is a flowchart of a series of image processing according to the third embodiment.
  • 7 shows a schematic configuration of a control unit according to a fourth embodiment.
  • 9 is a flowchart of a series of image processing according to the fourth embodiment.
  • An example of the report screen (display screen) which concerns on a modification is shown.
  • An example of the report screen (display screen) which concerns on a modification is shown.
  • An example of the report screen (display screen) which concerns on a modification is shown.
  • An example of the report screen (display screen) which concerns on a modification is shown.
  • An example of a configuration of a neural network used as a machine learning model according to Modification 12 is shown.
  • An example of a configuration of a neural network used as a machine learning model according to Modification 12 is shown.
  • An example of a configuration of a neural network used as a machine learning model according to Modification 12 is shown.
  • OCTA OCT Angiography
  • machine learning engine machine learning engine
  • OCTA is an angiography method that uses OCT and does not use a contrast agent.
  • three-dimensional motion contrast data acquired based on the depth information of the subject is integrated in the depth direction and projected on a two-dimensional plane to obtain an OCTA image (front blood vessel image or motion contrast front image). To generate.
  • the motion contrast data is data obtained by repeatedly photographing substantially the same portion of the subject and detecting a temporal change of the subject between the photographing.
  • substantially the same location refers to a location that is the same as an acceptable level for generating motion contrast data, and includes a location that is slightly deviated from the location that is exactly the same.
  • the motion contrast data is obtained, for example, by calculating the phase or vector of the complex OCT signal and the temporal change in intensity from the difference, ratio, correlation, or the like.
  • an image more suitable for image diagnosis than in the past is generated, and the authenticity of the tissue depicted on the image can be easily determined for such an image.
  • An image processing device is provided.
  • the image to be subjected to the image quality improvement processing is not limited to this, and may be a tomographic image, an En-Face image of brightness, or the like.
  • the En-Face image is a front image generated by projecting or integrating data within a predetermined depth range determined based on two reference planes on a two-dimensional plane in the three-dimensional data of the subject. Is.
  • the En-Face image includes, for example, a luminance En-Face image based on a luminance tomographic image and an OCTA image (motion contrast front image or motion contrast En-Face image) based on motion contrast data.
  • FIG. 1 shows a schematic configuration of an OCT apparatus according to this embodiment.
  • the OCT apparatus 1 is provided with an OCT imaging section 100, a control section (image processing apparatus) 200, an input section 260, and a display section 270.
  • the OCT imaging unit 100 includes an imaging optical system of the SD-OCT apparatus, and interferes with the return light from the eye E to which the measurement light is emitted via the scanning unit and the reference light corresponding to the measurement light. Based on the light, a signal including information on the tomographic image of the eye E (tomographic information) is acquired.
  • the OCT imaging unit 100 is provided with a light interference unit 110 and a scanning optical system 150.
  • the control unit 200 can control the OCT imaging unit 100, generate an image from a signal obtained from the OCT imaging unit 100 or another device (not shown), and process the generated/acquired image. ..
  • the display unit 270 is an arbitrary display such as an LCD display, and is a GUI for operating the OCT imaging unit 100 and the control unit 200, a generated image, an image subjected to arbitrary processing, and various information such as patient information. Can be displayed.
  • the input unit 260 is used to operate the control unit 200 by operating a GUI or inputting information.
  • the input unit 260 includes, for example, a mouse, a touch pad, a trackball, a touch panel display, a pointing device such as a stylus pen, a keyboard, and the like.
  • the display unit 270 and the input unit 260 can be integrally configured.
  • the OCT imaging unit 100, the control unit 200, the input unit 260, and the display unit 270 are separate elements in this embodiment, some or all of them may be integrally configured. Good.
  • a light source 111, a coupler 113, a collimating optical system 121, a dispersion compensating optical system 122, a reflection mirror 123, a lens 131, a diffraction grating 132, an imaging lens 133, and a line sensor 134 are provided in the light interference unit 110 of the OCT imaging unit 100. It is provided.
  • the light source 111 is a low coherence light source that emits near infrared light. The light emitted from the light source 111 propagates through the optical fiber 112a and enters the coupler 113 which is an optical branching unit.
  • the light incident on the coupler 113 is split into measurement light traveling toward the scanning optical system 150 side and reference light traveling toward the reference optical system side including the collimating optical system 121, the dispersion compensation optical system 122, and the reflection mirror 123.
  • the measurement light is incident on the optical fiber 112b and guided to the scanning optical system 150.
  • the reference light enters the optical fiber 112c and is guided to the reference optical system.
  • the reference light that has entered the optical fiber 112 c is emitted from the fiber end, enters the dispersion compensation optical system 122 via the collimating optical system 121, and is guided to the reflection mirror 123.
  • the reference light reflected by the reflection mirror 123 follows the optical path in the opposite direction and enters the optical fiber 112c again.
  • the dispersion compensating optical system 122 is for compensating the dispersion of the scanning optical system 150 and the optical system of the eye E to be inspected, and adjusting the dispersion of the measurement light and the reference light.
  • the reflecting mirror 123 is configured so that it can be driven in the optical axis direction of the reference light by a driving unit (not shown) controlled by the control unit 200, and the optical path length of the reference light is set with respect to the optical path length of the measurement light.
  • the relative optical path lengths of the reference light and the measurement light can be made to coincide with each other.
  • the scanning optical system 150 is an optical system configured to be movable relative to the subject's eye E.
  • the scanning optical system 150 is configured to be able to be driven in the front-back, up-down, left-right direction with respect to the eye axis of the eye E to be inspected by a driving unit (not shown) controlled by the control unit 200, and to perform alignment with the eye E to be inspected. It can be carried out.
  • the scanning optical system 150 may be configured to include the light source 111, the coupler 113, the reference optical system, and the like.
  • the scanning optical system 150 is provided with a collimating optical system 151, a scanning unit 152, and a lens 153.
  • the light emitted from the fiber end of the optical fiber 112 b is substantially collimated by the collimating optical system 151 and enters the scanning unit 152.
  • the scanning unit 152 has two galvanometer mirrors whose mirror surfaces are rotatable, one of which deflects light in a horizontal direction and the other of which deflects light in a vertical direction.
  • the scanning unit 152 deflects the incident light under the control of the control unit 200. Accordingly, the scanning unit 152 scans the measurement light on the fundus Er of the eye E in two directions, a main scanning direction perpendicular to the paper surface (X direction) and a sub-scanning direction in the paper surface direction (Y direction).
  • the main scanning direction and the sub scanning direction are not limited to the X direction and the Y direction, but are directions perpendicular to the depth direction (Z direction) of the eye E, and the main scanning direction and the sub scanning direction intersect with each other. Any direction is acceptable. Therefore, for example, the main scanning direction may be the Y direction or the sub scanning direction may be the X direction.
  • the measurement light scanned by the scanning unit 152 forms an illumination spot on the fundus Er of the eye E through the lens 153.
  • each illumination spot moves (scans) on the fundus Er of the eye E to be inspected.
  • the return light of the measurement light reflected/scattered from the fundus Er at the position of the illumination spot traces the optical path in the opposite direction, enters the optical fiber 112b, and returns to the coupler 113.
  • the reference light reflected by the reflecting mirror 123 and the return light of the measurement light from the fundus Er of the eye E are returned to the coupler 113 and interfere with each other to become interference light.
  • the interference light passes through the optical fiber 112d and is emitted to the lens 131.
  • the interference light is substantially collimated by the lens 131 and is incident on the diffraction grating 132.
  • the diffraction grating 132 has a periodic structure and disperses the input interference light.
  • the separated interference light is imaged on the line sensor 134 by the imaging lens 133 whose focus state can be changed.
  • the line sensor 134 outputs a signal according to the intensity of light emitted to each sensor unit to the control unit 200.
  • the control unit 200 can generate a tomographic image of the eye E to be inspected based on the interference signal output from the line sensor 134.
  • the control unit 200 can configure one B-scan image by collecting a plurality of A-scan images based on the interference signal acquired by the A-scan.
  • this B scan image is referred to as a two-dimensional tomographic image.
  • the galvanometer mirror of the scanning unit 152 can be finely driven in the sub-scanning direction orthogonal to the main scanning direction, and tomographic information at another location (adjacent scanning line) of the eye E can be acquired.
  • the control unit 200 can acquire a three-dimensional tomographic image in a predetermined range of the eye E by collecting a plurality of B scan images by repeating this operation.
  • FIG. 2 shows a schematic configuration of the control unit 200.
  • the control unit 200 is provided with an acquisition unit 210, an image processing unit 220, a drive control unit 230, a storage unit 240, and a display control unit 250.
  • the acquisition unit 210 can acquire the data of the output signal of the line sensor 134 corresponding to the interference signal of the eye E from the OCT imaging unit 100.
  • the data of the output signal acquired by the acquisition unit 210 may be an analog signal or a digital signal.
  • the control unit 200 can convert the analog signal into a digital signal.
  • the acquisition unit 210 can also acquire various images such as tomographic data generated by the image processing unit 220, a two-dimensional tomographic image, a three-dimensional tomographic image, a motion contrast image, and an En-Face image.
  • the tomographic data is data including information about a tomographic image of a subject, and is a signal obtained by performing a Fourier transform on an interference signal by OCT, a signal obtained by subjecting the signal to an arbitrary process, a tomographic image based on these, and the like. Is meant to include.
  • the acquisition unit 210 includes a shooting condition group of images to be image-processed (for example, shooting date, shooting region name, shooting region, shooting angle of view, shooting method, image resolution and gradation, image size of image, image filter). , And information about the image data format).
  • the shooting condition group is not limited to the exemplified ones. Further, the shooting condition group does not need to include all of the exemplified ones, and may include some of them.
  • the acquisition unit 210 acquires the imaging conditions of the OCT imaging unit 100 when the image was taken. Further, the acquisition unit 210 can also acquire the shooting condition group stored in the data structure forming the image according to the data format of the image. In addition, when the shooting condition is not stored in the data structure of the image, the acquisition unit 210 can also acquire the shooting information group including the shooting condition group from a storage device or the like separately storing the shooting condition.
  • the acquisition unit 210 can also acquire information for identifying the eye to be inspected, such as the subject identification number, from the input unit 260 or the like.
  • the acquisition unit 210 may acquire various data, various images, and various information from the storage unit 240 and other devices (not shown) connected to the control unit 200.
  • the acquisition unit 210 can store various acquired data and images in the storage unit 240.
  • the image processing unit 220 generates a tomographic image, an En-Face image, or the like from the data acquired by the acquisition unit 210 or the data stored in the storage unit 240, and performs image processing on the generated or acquired image. You can
  • the image processing unit 220 is provided with a tomographic image generation unit 221, a motion contrast generation unit 222, an En-Face image generation unit 223, and an image quality improvement unit 224.
  • the tomographic image generation unit 221 generates tomographic data by performing wave number conversion, Fourier transform, absolute value conversion (acquisition of amplitude), or the like on the data of the interference signal acquired by the acquisition unit 210, and based on the tomographic data, the tomographic image is generated.
  • a tomographic image of the optometry E can be generated.
  • the data of the interference signal acquired by the acquisition unit 210 may be data of the signal output from the line sensor 134, or acquired from a device (not shown) connected to the storage unit 240 or the control unit 200. It may be the data of the generated interference signal. Any known method may be adopted as the method of generating the tomographic image, and detailed description thereof will be omitted.
  • the tomographic image generation unit 221 can generate a three-dimensional tomographic image based on the generated tomographic images of a plurality of parts.
  • the tomographic image generation unit 221 can generate a three-dimensional tomographic image by arranging the tomographic images of a plurality of parts side by side in one coordinate system, for example.
  • the tomographic image generation unit 221 may generate a three-dimensional tomographic image based on tomographic images of a plurality of sites acquired from a device (not shown) connected to the storage unit 240 or the control unit 200.
  • the motion contrast generation unit 222 can generate a two-dimensional motion contrast image using a plurality of tomographic images obtained by photographing substantially the same place. Further, the motion contrast generation unit 222 can generate a three-dimensional motion contrast image by arranging the generated two-dimensional motion contrast images of the respective parts side by side in one coordinate system.
  • the motion contrast generation unit 222 generates a motion contrast image based on a decorrelation value between a plurality of tomographic images obtained by photographing substantially the same portion of the eye E to be inspected.
  • the motion contrast generation unit 222 acquires a plurality of aligned tomographic images with respect to a plurality of tomographic images obtained by photographing substantially the same positions where the photographing times are continuous with each other.
  • Various known methods can be used for alignment. For example, one of the plurality of tomographic images is selected as a reference image, the similarity with other tomographic images is calculated while changing the position and angle of the reference image, and the position of each tomographic image with respect to the reference image is calculated. The shift amount is calculated. Positioning of a plurality of tomographic images is performed by correcting each tomographic image based on the calculation result.
  • the alignment process may be performed by a component separate from the motion contrast generation unit 222. Further, the alignment method is not limited to this, and any known method may be used.
  • the motion contrast generation unit 222 calculates the decorrelation value for each of the two tomographic images of which the photographing times are continuous from each other of the plurality of tomographic images that have been aligned by using the following mathematical expression 1.
  • A(x, z) indicates the amplitude at the position (x, z) of the tomographic image A
  • B(x, z) indicates the amplitude at the same position (x, z) of the tomographic image B.
  • the resulting decorrelation value M(x,z) takes a value from 0 to 1, and becomes closer to 1 as the difference between the two amplitude values is larger.
  • the present embodiment has described the case of using a two-dimensional tomographic image of the XZ plane, a two-dimensional tomographic image of the YZ plane or the like may be used. In this case, the position (x, z) may be replaced with the position (y, z) or the like.
  • the decorrelation value may be obtained based on the luminance value of the tomographic image or may be obtained based on the value of the interference signal corresponding to the tomographic image.
  • the motion contrast generation unit 222 determines the pixel value of the motion contrast image based on the decorrelation value M(x, z) at each position (pixel position), and generates the motion contrast image. In addition, in the present embodiment, the motion contrast generation unit 222 calculates the decorrelation value for the tomographic images whose imaging times are continuous with each other, but the method of calculating the motion contrast data is not limited to this.
  • the two tomographic images for which the decorrelation value M is obtained need only have a shooting time within a predetermined time interval for each tomographic image corresponding to each other, and may not be continuous.
  • two tomographic images having an imaging interval longer than a normal specified time are extracted from the acquired tomographic images to calculate the decorrelation value. May be.
  • a variance value a value obtained by dividing the maximum value by the minimum value (maximum value/minimum value), or the like may be obtained.
  • the method of generating the motion contrast image is not limited to the above method, and any other known method may be used.
  • the En-Face image generation unit 223 can generate an En-Face image (OCTA image) that is a front image from the three-dimensional motion contrast image generated by the motion contrast generation unit 222. Specifically, the En-Face image generating unit 223 projects the three-dimensional motion contrast image on a two-dimensional plane based on, for example, two arbitrary reference planes in the depth direction (Z direction) of the eye E to be inspected. It is possible to generate an OCTA image that is a front image that has been processed. Further, the En-Face image generating unit 223 can also generate an En-Face image having the same brightness from the three-dimensional tomographic image generated by the tomographic image generating unit 221.
  • OCTA image En-Face image
  • the En-Face image generation unit 223 determines, for example, a representative value of pixel values in the depth direction at each position in the XY direction of a region surrounded by two reference planes, and sets the representative value as the representative value.
  • the pixel value at each position is determined based on this, and an En-Face image is generated.
  • the representative value includes a value such as an average value, a median value, or a maximum value of the pixel values within the range in the depth direction of the area surrounded by the two reference planes.
  • the reference plane may be a plane along the layer boundary of the slice of the eye E to be examined, or may be a plane.
  • the range in the depth direction between the reference planes for generating the En-Face image will be referred to as the En-Face image generation range.
  • the method of generating the En-Face image according to the present embodiment is an example, and the En-Face image generating unit 223 may generate the En-Face image using any known method.
  • the image quality improving unit 224 uses the learned model described later to generate a high quality OCTA image based on the OCTA image generated by the En-Face image generating unit 223. Further, the image quality improving unit 224, based on the tomographic image generated by the tomographic image generating unit 221, and the En-Face image of the luminance generated by the En-Face image generating unit 223, the high-quality tomographic image and the high-quality luminance The En-Face image may be generated.
  • the image quality improving unit 224 obtains not only the OCTA image captured by the OCT image capturing unit 100, but also the acquisition unit 210 from other devices (not shown) connected to the storage unit 240 or the control unit 200.
  • the image quality improving unit 224 may perform the image quality improving process for the three-dimensional motion contrast image and the three-dimensional tomographic image as well as the OCTA image and the tomographic image.
  • the drive control unit 230 can control the driving of the light source 111 of the OCT imaging unit 100, the scanning optical system 150, the scanning unit 152, the imaging lens 133, and other components connected to the control unit 200.
  • the storage unit 240 can store various data acquired by the acquisition unit 210, various images and data such as tomographic images and OCTA images generated and processed by the image processing unit 220, and the like.
  • the storage unit 240 also stores information about the subject's eye such as the subject's attributes (name, age, etc.) and measurement results (e.g., axial length and intraocular pressure) acquired using other test equipment, imaging parameters, and images.
  • the analysis parameter and the parameter set by the operator can be stored. Note that these images and information may be stored in an external storage device (not shown).
  • the storage unit 240 can also store a program or the like for executing the functions of the respective components of the control unit 200 by being executed by the processor.
  • the display control unit 250 can cause the display unit 270 to display various information acquired by the acquisition unit 210 and various images such as tomographic images, OCTA images, and three-dimensional motion contrast images generated and processed by the image processing unit 220. it can. In addition, the display control unit 250 can display information and the like input by the user on the display unit 270.
  • the control unit 200 may be configured using a general-purpose computer, for example.
  • the control unit 200 may be configured using a dedicated computer for the OCT apparatus 1.
  • the control unit 200 includes a storage medium including a CPU (Central Processing Unit), an MPU (Micro Processing Unit) (not shown), and a memory such as an optical disk and a ROM (Read Only Memory).
  • Each component other than the storage unit 240 of the control unit 200 may be configured by a software module executed by a processor such as a CPU or MPU. Further, each of the constituent elements may be configured by a circuit that performs a specific function such as an ASIC or an independent device.
  • the storage unit 240 may be configured by an arbitrary storage medium such as an optical disk or a memory, for example.
  • the control unit 200 may include one or more processors such as CPU and storage media such as ROM. Therefore, each component of the control unit 200 functions when at least one or more processors and at least one storage medium are connected and at least one or more processors execute a program stored in at least one or more storage media. May be configured to do so.
  • the processor is not limited to the CPU or MPU, and may be a GPU (Graphics Processing Unit) or the like.
  • the learned model according to the present embodiment generates and outputs an image that has been subjected to the image quality improvement processing based on the input image according to the learning tendency.
  • the image quality improving process in this specification refers to converting an input image into an image having a more suitable image quality for image diagnosis, and a high quality image means an image converted to an image having an image quality more suitable for image diagnosis.
  • the content of the image quality suitable for image diagnosis depends on what is desired to be diagnosed by various image diagnosis. Therefore, although it cannot be said unequivocally, for example, the image quality suitable for image diagnosis is low in noise, has high contrast, shows the subject in colors and gradations that are easy to observe, and has a large image size. Includes image quality such as high resolution. Further, it is possible to include an image quality in which an object or gradation which does not actually exist and which is drawn in the process of image generation is removed from the image.
  • the learned model is a model obtained by performing training (learning) on a machine learning model according to an arbitrary machine learning algorithm such as deep learning in advance using appropriate teacher data (learning data). However, it is assumed that the learned model does not perform any further learning, but can perform additional learning.
  • the teacher data is composed of one or more pairs of input data and output data pairs.
  • a pair of input data and output data is composed of an OCTA image and an OCTA image obtained by performing a superimposing process such as averaging on a plurality of OCTA images including the OCTA image.
  • the superimposed image subjected to the superimposing processing is a high-quality image suitable for image diagnosis because the pixels commonly drawn in the original image group are emphasized.
  • the generated high-quality image is a high-contrast image in which the difference between the low-luminance region and the high-luminance region is clear as a result of the commonly drawn pixels being emphasized.
  • random noise that occurs each time the image is captured can be reduced, or a region that was not drawn well in the original image at a certain point can be interpolated by another original image group.
  • the pairs that do not contribute to high image quality can be removed from the teacher data.
  • the high-quality image that is the output data forming the pair of teacher data has an image quality not suitable for image diagnosis
  • the image output by the learned model learned using the teacher data is also not suitable for image diagnosis.
  • the image quality will end up. Therefore, it is possible to reduce the possibility that the learned model will generate an image with an image quality not suitable for image diagnosis by removing from the teacher data a pair whose output data has an image quality not suitable for image diagnosis.
  • the learned model learned using the teacher data has an image that has a brightness distribution significantly different from that of the low-quality image and is not suitable for image diagnosis. It may be output. Therefore, a pair of input data and output data having a large difference in average luminance or luminance distribution can be removed from the teacher data.
  • the learned model learned using the teacher data sets the imaged object at a structure and position significantly different from the low-quality image. There is a possibility of outputting an image that is not suitable for the drawn image diagnosis. Therefore, a pair of input data and output data, which differ greatly in the structure or position of the imaged object to be drawn, can be removed from the teacher data.
  • the image quality improving unit 224 can improve the contrast and noise by the superimposing process when an OCTA image acquired by one imaging (inspection) is input. It is possible to generate a high-quality OCTA image that has been reduced. Therefore, the image quality improving unit 224 can generate a high quality image suitable for image diagnosis based on the low quality image which is the input image.
  • An image during learning will be described.
  • An image group forming a pair group of the OCTA image 301 and the high-quality OCTA image 302 forming the teacher data is created by a rectangular area image having a fixed image size corresponding to the positional relationship. Creation of the image will be described with reference to FIGS. 3A and 3B.
  • one of the pair groups forming the teacher data is the OCTA image 301 and the high quality OCTA image 302
  • the entire OCTA image 301 is used as input data and the entire high-quality OCTA image 302 is used as output data to form a pair.
  • a pair of input data and output data is formed by the whole of each image, but the pair is not limited to this.
  • a pair may be formed by using the rectangular area image 311 of the OCTA image 301 as input data and the rectangular area image 321 that is the corresponding imaging area in the OCTA image 302 as output data.
  • the scan range (shooting angle of view) and the scan density (the number of A scans and the number of B scans) are normalized to make the image sizes uniform, and the rectangular area sizes at the time of learning can be made uniform.
  • the rectangular area images shown in FIGS. 3A and 3B are examples of rectangular area sizes when learning separately.
  • the number of rectangular areas can be set to one in the example shown in FIG. 3A and can be set to a plurality in the example shown in FIG. 3B.
  • the rectangular area image 312 of the OCTA image 301 may be used as input data
  • the rectangular area image 322 that is the corresponding shooting area in the high quality OCTA image 302 may be used as output data to form a pair. it can.
  • a pair of different rectangular area images can be created from each pair of OCTA images and high-quality OCTA images.
  • the original OCTA image and the high-quality OCTA image by creating a large number of pairs of rectangular area images while changing the position of the area to different coordinates, it is possible to enhance the pair group forming the teacher data. ..
  • the rectangular areas are discretely shown, but the original OCTA image and the high-quality OCTA image should be divided into a continuous rectangular area image group having a constant image size without a gap.
  • the original OCTA image and the high-quality OCTA image may be divided into rectangular area image groups corresponding to each other at random positions. As described above, by selecting an image of a smaller area as a pair of input data and output data as a rectangular area, a large amount of pair data is generated from the OCTA image 301 and the high quality OCTA image 302 that form the original pair. it can. Therefore, the time required for training the machine learning model can be shortened.
  • FIG. 4 shows an example of the configuration 401 of the learned model used by the image quality improving unit 224.
  • the learned model shown in FIG. 4 is composed of a plurality of layer groups that are responsible for processing the input value group and outputting it.
  • the types of layers included in the learned model configuration 401 include a convolutional layer, a downsampling layer, an upsampling layer, and a merging layer.
  • the convolutional layer is a layer that performs convolution processing on the input value group according to parameters such as the set kernel size of the filter, the number of filters, the stride value, and the dilation value.
  • the number of dimensions of the kernel size of the filter may be changed according to the number of dimensions of the input image.
  • the down-sampling layer is a layer that performs processing to reduce the number of output value groups to less than the number of input value groups by thinning out or combining the input value groups. Specifically, for example, there is Max Pooling processing as such processing.
  • the upsampling layer is a layer that performs a process of increasing the number of output value groups more than the number of input value groups by duplicating the input value group or adding a value interpolated from the input value group. Specifically, as such processing, there is linear interpolation processing, for example.
  • the combining layer is a layer that inputs a value group such as an output value group of a certain layer or a pixel value group forming an image from a plurality of sources, and performs a process of combining and adding the value groups.
  • the parameters set in the convolutional layer group included in the configuration 401 shown in FIG. 4 for example, by setting the kernel size of the filter to 3 pixels wide, 3 pixels high, and 64 filters, a certain accuracy can be obtained. It is possible to improve the image quality of However, it should be noted that the degree of reproducibility of the training tendency from the teacher data to the output data may differ if the parameter settings for the layer group and the node group forming the neural network are different. In other words, in many cases, the appropriate parameter differs depending on the mode of implementation, and therefore it can be changed to a preferable value as necessary.
  • the characteristics of the CNN may be improved by changing the configuration of the CNN.
  • the better characteristics are, for example, that the accuracy of the image quality improvement processing is high, the time of the image quality improvement processing is short, the time required for training the machine learning model is short, and the like.
  • a batch normalization (Batch Normalization) layer or an activation layer that uses a normalized linear (ReLU: Rectifier Linear Unit) function is incorporated after the convolutional layer. Good.
  • data When data is input to the learned model of such a machine learning model, data according to the design of the machine learning model is output. For example, output data that is highly likely to correspond to the input data is output according to the tendency trained using the teacher data.
  • the learned model according to the present embodiment when the OCTA image 301 is input, a high quality OCTA image 302 is output according to the tendency of training using the teacher data.
  • the learned model When learning is performed by dividing the image area, the learned model outputs a rectangular area image that is a high-quality OCTA image corresponding to each rectangular area.
  • the image quality improving unit 224 first divides the OCTA image 301 that is the input image into rectangular area image groups based on the image size at the time of learning, and inputs the divided rectangular area image groups to the learned model. After that, the image quality improving unit 224 positions the rectangular area image groups, which are high-quality OCTA images output from the learned model, in the same positional relationship as the rectangular area image groups input to the learned model. Place and combine. As a result, the image quality improving unit 224 can generate a high quality OCTA image 302 corresponding to the input OCTA image 301.
  • FIG. 5 is a flowchart of a series of image processing according to this embodiment.
  • the acquisition unit 210 acquires a plurality of three-dimensional tomographic information obtained by imaging the subject's eye E a plurality of times.
  • the acquisition unit 210 may acquire the tomographic information of the eye E using the OCT imaging unit 100, or may acquire the tomographic information from the storage unit 240 or another device connected to the control unit 200.
  • the tomographic information of the eye E to be inspected is acquired using the OCT imaging unit 100.
  • the operator sits a patient as a subject in front of the scanning optical system 150, performs alignment, and inputs patient information and the like to the control unit 200, and then starts OCT imaging.
  • the drive control unit 230 of the control unit 200 drives the galvanometer mirror of the scanning unit 152, scans the substantially same location of the eye to be examined a plurality of times, and obtains a plurality of tomographic information (interference signals) at the substantially same location of the eye to be examined. ..
  • the drive control unit 230 finely drives the galvanometer mirror of the scanning unit 152 in the sub-scanning direction orthogonal to the main scanning direction, and acquires a plurality of tomographic information at another location (adjacent scanning lines) of the eye E to be inspected. To do.
  • the acquisition unit 210 acquires a plurality of three-dimensional tomographic information in a predetermined range of the eye E to be examined.
  • step S502 the tomographic image generation unit 221 generates a plurality of three-dimensional tomographic images based on the acquired plurality of three-dimensional tomographic information.
  • step S502 may be omitted.
  • the motion contrast generation unit 222 generates three-dimensional motion contrast data (three-dimensional motion contrast image) based on the plurality of three-dimensional tomographic images.
  • the motion contrast generation unit 222 may obtain a plurality of motion contrast data based on three or more tomographic images acquired at substantially the same location, and may generate an average value thereof as final motion contrast data. If the acquisition unit 210 acquires the three-dimensional motion contrast data from the storage unit 240 or another device connected to the control unit 200 in step S501, steps S502 and S503 may be omitted.
  • step S504 the En-Face image generation unit 223 generates an OCTA image for the three-dimensional motion contrast data, based on the instruction range of the operator or based on a predetermined En-Face image generation range.
  • steps S502 to S504 may be omitted.
  • the image quality improvement unit 224 uses the learned model to perform the image quality improvement process of the OCTA image.
  • the image quality improving unit 224 inputs the OCTA image into the learned model and generates a high quality OCTA image based on the output from the learned model.
  • the image quality improving unit 224 first divides the OCTA image, which is the input image, into rectangular region image groups based on the image size at the time of learning. Then, the divided rectangular area image group is input to the learned model. After that, the image quality improving unit 224 positions the rectangular area image groups, which are high-quality OCTA images output from the learned model, in the same positional relationship as the rectangular area image groups input to the learned model. By arranging them and combining them, a final high quality OCTA image is generated.
  • step S506 the display control unit 250 switches the high-quality OCTA image (second medical image) generated by the image quality improving unit 224 to the original OCTA image (first medical image) on the display unit 270. Display it.
  • a blood vessel that does not actually exist may be drawn on the OCTA image, or a blood vessel that originally exists may be erased.
  • the display control unit 250 switches the generated high-quality OCTA image to the original OCTA image on the display unit 270 so as to display the original OCTA image. It is possible to easily determine whether the blood vessel was also present in the image.
  • a series of image processing ends.
  • 6A and 6B show an example of a report screen for switching and displaying images before and after the image quality improvement process.
  • a tomographic image 611 and an OCTA image 601 before the image quality improvement processing are shown.
  • the report screen 600 shown in FIG. 6B shows a tomographic image 611 and an OCTA image 602 (high quality OCTA image) after the image quality improvement processing.
  • a pop-up menu 620 for selecting whether or not to perform the image quality improvement process is displayed. Is displayed.
  • the image quality improving unit 224 executes the image quality improving process on the OCTA image 601.
  • the display control unit 250 switches the OCTA image 601 displayed on the report screen 600 before the image quality improvement processing is performed to the OCTA image 602 after the image quality improvement processing is displayed. .. It is also possible to open the pop-up menu 620 by pressing the right mouse button again on the OCTA image 602 and switch to and display the OCTA image 601 before the image quality improvement processing.
  • the image switching method can be performed by any method other than the pop-up menu.
  • the image may be switched by a button (an example of the high image quality button) arranged on the report screen, a pull-down menu, a radio button, a check box, or a keyboard operation.
  • the images may be switched and displayed by operating the mouse wheel or touching the touch panel display.
  • the operator can arbitrarily switch and display the OCTA image 601 before the image quality improvement processing and the OCTA image 602 after the image quality improvement processing. Therefore, the operator can easily compare the OCTA images before and after the image quality improvement process, and can easily confirm the change in the OCTA image due to the image quality improvement process. Therefore, the operator can easily identify the blood vessels that do not actually exist in the OCTA image or the originally existing blood vessels disappear by the image quality improvement processing, and it is possible to easily identify them. The authenticity of the organization depicted in can be easily determined.
  • FIG. 7 shows an example of a report screen when the images before and after the image quality improvement process are displayed side by side.
  • an OCTA image 701 before the image quality improvement processing and an OCTA image 702 after the image quality improvement processing are displayed side by side.
  • the operator can easily compare the images before and after the image quality improvement process, and can easily confirm the change in the image due to the image quality improvement process. Therefore, the operator can easily identify a blood vessel that does not actually exist in the OCTA image by the image quality improvement processing, or even if the originally existing blood vessel disappears, the operator can easily identify it. The authenticity of the depicted tissue can be easily determined.
  • the display control unit 250 sets the transparency for at least one of the images before and after the image quality improvement process, and displays the images before and after the image quality improvement process on the display unit 270. It can be displayed overlaid.
  • the image quality improving unit 224 may perform the image quality improving process using the learned model not only on the OCTA image but also on the tomographic image and the En-Face image of the luminance.
  • the learned model may be one learned model learned using teacher data such as an OCTA image or a tomographic image, or a plurality of learned models learned for each image type.
  • the image quality improving unit 224 can use a learned model according to the type of the image for which the image quality improving process is performed.
  • the image quality improving unit 224 may perform the image quality improving process using the learned model for the three-dimensional motion contrast image and the three-dimensional tomographic image, and the learning data in this case can be prepared similarly to the above.
  • a tomographic image 711 before the image quality improving process and a tomographic image 712 after the image quality improving process are displayed side by side.
  • the display control unit 250 switches the tomographic image and the En-Face image of the brightness before and after the image quality improving process, such as the OCTA images before and after the image quality improving process illustrated in FIGS. 6A and 6B, and causes the display unit 270 to display the images. May be. Further, the display control unit 250 may cause the display unit 270 to display the tomographic images before and after the image quality improvement processing and the En-Face image of the brightness in an overlapping manner.
  • the operator can easily compare the images before and after the image quality improvement process, and can easily confirm the change in the image due to the image quality improvement process. Therefore, the operator can easily identify the tissue that does not actually exist in the image due to the image quality improvement process, or even if the tissue that originally exists disappears, the operator can easily identify it. The authenticity of the established organization can be easily determined.
  • the control unit 200 includes the image quality improvement unit 224 and the display control unit 250.
  • the image quality improving unit 224 uses the learned model to perform the second medical image in which at least one of noise reduction and contrast enhancement is performed from the first medical image of the eye to be inspected as compared with the first medical image. To generate.
  • the display control unit 250 switches the first medical image and the second medical image on the display unit 270 to display them side by side or overlap each other. Note that the display control unit 250 can switch the first medical image and the second medical image in accordance with an instruction from the operator and cause the display unit 270 to display the images.
  • control unit 200 can generate a high-quality image in which noise is reduced or contrast is emphasized from the original image. Therefore, the control unit 200 can generate an image more suitable for image diagnosis than before, such as a clearer image or an image in which a region to be observed or a lesion is emphasized.
  • the operator can easily compare the images before and after the image quality improving process, and can easily confirm the change of the image due to the image quality improving process. Therefore, the operator can easily identify the tissue that does not actually exist in the image due to the image quality improvement process, or even if the tissue that originally exists disappears, the operator can easily identify it. The authenticity of the established organization can be easily determined.
  • the superimposed image is used as the output data of the teacher data, but the teacher data is not limited to this.
  • a high-quality image obtained by performing the maximum posterior probability estimation process (MAP estimation process) on the original image group may be used as the output data of the teacher data.
  • MAP estimation process a high-quality image obtained by performing the maximum posterior probability estimation process (MAP estimation process) on the original image group.
  • MAP estimation processing a likelihood function is obtained from the probability density of each pixel value in a plurality of images, and a true signal value (pixel value) is estimated using the obtained likelihood function.
  • the high-quality image obtained by the MAP estimation process becomes a high-contrast image based on the pixel value close to the true signal value. Further, since the estimated signal value is obtained based on the probability density, noise that is randomly generated is reduced in the high-quality image obtained by the MAP estimation process. Therefore, by using the high-quality image obtained by the MAP estimation process as the teacher data, the learned model has high image quality suitable for image diagnosis, in which noise is reduced or high contrast is obtained from the input image. Images can be generated.
  • the method of generating the pair of input data and output data of the teacher data may be the same as the method of using the superimposed image as the teacher data.
  • a high-quality image obtained by applying a smoothing filter process to the original image may be used as the output data of the teacher data.
  • the learned model can generate a high-quality image with reduced random noise from the input image.
  • an image obtained by applying a gradation conversion process to the original image may be used as the output data of the teacher data.
  • the learned model can generate a high-quality image with contrast enhancement from the input image.
  • the method of generating the pair of input data and output data of the teacher data may be the same as the method of using the superimposed image as the teacher data.
  • the input data of the teacher data may be an image acquired from an imaging device having the same image quality tendency as the OCT imaging unit 100.
  • the output data of the teacher data may be a high-quality image obtained by high-cost processing such as the successive approximation method, or the subject corresponding to the input data has higher performance than the OCT imaging unit 100. It may be a high-quality image acquired by shooting with a shooting device. Furthermore, the output data may be a high-quality image acquired by performing noise reduction processing by a rule base based on the structure of the subject.
  • the noise reduction process can include, for example, a process of replacing a high-luminance pixel of only one pixel, which is apparently noise appearing in the low-luminance region, with an average value of neighboring low-luminance pixel values. .. Therefore, the learned model is an image captured by an image capturing device having a higher performance than the image capturing device used to capture the input image, or an image acquired in the image capturing process that requires more man-hours than the input image capturing process. It may be data.
  • the image quality improving unit 224 uses the learned model to generate a high-quality image in which noise is reduced or contrast is emphasized, the image quality improving process by the image quality improving unit 224 is described. Is not limited to this.
  • the image quality improving unit 224 is only required to be able to generate an image having an image quality more suitable for image diagnosis as described above by the image quality improving process.
  • any of the images before and after the image quality improvement process displayed side by side on the display unit 270 is displayed according to an instruction from the operator. It may be displayed enlarged. More specifically, for example, when the operator selects the OCTA image 701 on the report screen 700 shown in FIG. 7, the display control unit 250 can enlarge and display the OCTA image 701 on the report screen 700. Further, when the operator selects the OCTA image 702 after the image quality improvement processing, the display control unit 250 can enlarge and display the OCTA image 702 on the report screen 700. In this case, the operator can observe the image to be observed in more detail among the images before and after the image quality improvement process.
  • the control unit 200 sets the images displayed side by side to the image based on the changed generation range and the high image quality. You may change and display it in the converted image. More specifically, when the operator changes the generation range of the En-Face image via the input unit 260, the En-Face image generation unit 223 performs En before image quality improvement processing based on the changed generation range. -Generate Face image.
  • the image quality improving unit 224 uses the learned model to generate a high-quality En-Face image from the En-Face image newly generated by the En-Face image generating unit 223.
  • the display control unit 250 changes the En-Face images before and after the image quality improvement process displayed side by side on the display unit 270 to the newly generated En-Face images before and after the image quality improvement process and displays the En-Face images.
  • the operator can observe the En-Face images before and after the image quality improvement process based on the changed range in the depth direction while arbitrarily changing the range in the depth direction to be observed.
  • Modification 2 The first embodiment has described the example in which the image quality improvement processing is applied to the OCTA image, the tomographic image, and the like obtained by one imaging (inspection).
  • the image quality improving process using the learned model can be applied to a plurality of OCTA images or tomographic images obtained by a plurality of times of imaging (inspection).
  • FIGS. 8A and 8B a configuration will be described in which images to which image quality improvement processing using a learned model is applied are simultaneously displayed on a plurality of OCTA images, tomographic images, and the like.
  • FIG. 8A and 8B show an example of a time-series report screen for displaying a plurality of OCTA images acquired by photographing the same eye to be examined a plurality of times over time.
  • a plurality of OCTA images 801 before performing the image quality improvement processing are displayed side by side in time series.
  • the report screen 800 also includes a pop-up menu 820, and the operator can select whether or not to apply the image quality improvement process by operating the pop-up menu 820 via the input unit 260.
  • the image quality improving unit 224 applies the image quality improving process using the learned model to all the displayed OCTA images. Then, as shown in FIG. 8B, the display control unit 250 switches and displays the plurality of OCTA images 802 after the image quality improvement processing with the plurality of OCTA images 801 that have been displayed.
  • the display control unit 250 causes the plurality of OCTA images 801 before the image quality improvement processing to be displayed after the displayed image quality improvement processing.
  • the OCTA image 802 is switched and displayed.
  • an example is shown in which a plurality of OCTA images before and after the image quality improvement process using the learned model are simultaneously switched and displayed.
  • a plurality of tomographic images before and after the image quality improvement processing using the learned model, En-Face images of brightness, and the like may be simultaneously switched and displayed.
  • the operation method is not limited to the method using the pop-up menu 820, and buttons (an example of a high image quality button) arranged on the report screen, pull-down menus, radio buttons, check boxes, or keyboard, mouse wheel, and touch panel operations can be performed. Any operation method such as the above may be adopted.
  • the learned model outputs output data that is likely to correspond to the input data according to the learning tendency.
  • the learned model when learning is performed by using image groups having similar image quality tendencies as teacher data, outputs an image of which the image quality is more effectively improved with respect to the image having the similar tendency. be able to. Therefore, in the second embodiment, the image quality improvement processing is performed by a plurality of learned models learned by using the teacher data composed of the pair of groups that are grouped for each imaging condition such as an imaging region and each En-Face image generation range. As a result, the image quality improving process is performed more effectively.
  • the OCT apparatus according to the present embodiment will be described below with reference to FIGS. 9 and 10.
  • the configuration of the OCT apparatus according to the present embodiment is the same as that of the OCT apparatus 1 according to the first embodiment except for the control unit. The description is omitted.
  • the OCT apparatus according to the present embodiment will be described focusing on the difference from the OCT apparatus 1 according to the first embodiment.
  • FIG. 9 shows a schematic configuration of the control unit 900 according to this embodiment.
  • the configuration of the control unit 900 according to the present embodiment other than the image processing unit 920 and the selection unit 925 is the same as each configuration of the control unit 200 according to the first embodiment. Therefore, the same components as those shown in FIG. 2 are designated by the same reference numerals and the description thereof will be omitted.
  • the image processing unit 920 of the control unit 900 is provided with a selection unit 925 in addition to the tomographic image generation unit 221, the motion contrast generation unit 222, the En-Face image generation unit 223, and the image quality improvement unit 224.
  • the selecting unit 925 selects a learned model to be used by the image quality improving unit 224 from among a plurality of learned models based on the shooting conditions of the image to be subjected to the image quality improving process by the image quality improving unit 224 and the generation range of the En-Face image. Select.
  • the image quality improvement unit 224 uses the learned model selected by the selection unit 925 to perform image quality improvement processing on the target OCTA image, tomographic image, or the like, and generates a high-quality OCTA image or a high-quality tomographic image. ..
  • the learned model outputs the output data that is likely to correspond to the input data according to the learning tendency.
  • the learned model when learning is performed by using image groups having similar image quality tendencies as teacher data, outputs images with higher image quality more effectively with respect to the images having the similar tendency. be able to.
  • the present embodiment is configured with a pair of groups that are grouped for each shooting condition including the shooting region, shooting method, shooting region, shooting angle of view, scan density, and image resolution, and the generation range of the En-Face image. Prepare a plurality of trained models that have been trained using the teacher data.
  • a plurality of learned models such as a learned model in which an OCTA image in which the macular region is an imaging region is used as teacher data, and a model in which an OCTA image in which a teat is an imaging region is used as teacher data is learned.
  • a model in which the macula and the papilla are examples of the imaged region, and may include other imaged regions.
  • a learned model may be prepared in which the OCTA image for each specific imaging region in the imaging region such as the macula and the papilla is used as the teacher data.
  • the visualization of structures such as blood vessels and the like visualized in an OCTA image is greatly different between when the retina is photographed with a wide angle of view and low density and when the retina is photographed with a narrow angle of view and high density. Therefore, a learned model that has been learned may be prepared for each teacher data according to the shooting angle of view and the scan density.
  • the image capturing method there are image capturing methods such as SD-OCT and SS-OCT, and the image quality, the image capturing range, and the depth of depth in the depth direction are different due to the difference between these image capturing methods. Therefore, a learned model that has been learned may be prepared for each teacher data according to the shooting method.
  • an OCTA image in which blood vessels in all layers of the retina are extracted at once it is rare to generate an OCTA image in which blood vessels in all layers of the retina are extracted at once, and it is common to generate an OCTA image in which only blood vessels existing in a predetermined depth range are extracted. ..
  • an OCTA image in which blood vessels are extracted in each depth range is generated.
  • the form of the blood vessel depicted in the OCTA image greatly differs depending on the depth range.
  • the blood vessels visualized in the superficial layer of the retina form low density, thin and clear vascular networks, whereas the blood vessels depicted in the superficial layer of the choroid clearly identify individual blood vessels at high density. Things are difficult. Therefore, a learned model that has been learned for each teacher data according to the generation range of the En-Face image such as the OCTA image may be prepared.
  • the example in which the OCTA image is used as the teacher data has been described, but like the first embodiment, when performing the image quality improvement processing on the tomographic image, the En-Face image of the luminance, and the like, these images are used as the teacher data. be able to. In this case, a plurality of learned models that have been learned for each teacher data according to the shooting conditions of these images and the generation range of the En-Face image are prepared.
  • FIG. 10 is a flowchart of a series of image processing according to this embodiment. Note that description of the same processing as the series of image processing according to the first embodiment will be appropriately omitted.
  • the acquisition unit 210 acquires a plurality of three-dimensional tomographic information obtained by photographing the eye E to be inspected a plurality of times.
  • the acquisition unit 210 may acquire the tomographic information of the eye E using the OCT imaging unit 100, or may acquire the tomographic information from the storage unit 240 or another device connected to the control unit 200.
  • the acquisition unit 210 also acquires a group of imaging conditions related to tomographic information. Specifically, the acquisition unit 210 can acquire imaging conditions such as an imaging region and an imaging method when imaging the tomographic information. The acquisition unit 210 may acquire the imaging condition group stored in the data structure forming the data of the tomographic information according to the data format of the tomographic information. Further, when the imaging conditions are not stored in the data structure of the tomographic information, the acquisition unit 210 can acquire the imaging information group from a server, a database, or the like that stores a file that describes the imaging conditions. In addition, the acquisition unit 210 may estimate the imaging information group from the image based on the tomographic information by any known method.
  • the acquisition unit 210 acquires a group of imaging conditions related to the acquired images and data.
  • the acquisition unit 210 uses the acquisition conditions of the tomographic image. The group does not have to be acquired.
  • Steps S1002 to S1004 are the same as Steps S502 to S504 according to the first embodiment, a description thereof will be omitted.
  • the En-Face image generating unit 223 When the En-Face image generating unit 223 generates an OCTA image in step S1004, the process proceeds to step S1005.
  • the selecting unit 925 selects a learned model to be used by the image quality improving unit 224 based on the information of the shooting condition group and the generation range regarding the generated OCTA image and the teacher data regarding a plurality of learned models. More specifically, for example, when the imaging region of the OCTA image is the nipple portion, the selection unit 925 selects a learned model that has been trained using the OCTA image of the nipple portion as teacher data. Further, for example, when the generation range of the OCTA image is the shallow layer of the retina, the selection unit 925 selects a learned model that has been trained using the OCTA image having the shallow layer of the retina as the generation range for the training data. ..
  • the selection unit 925 sets the images having similar image quality as the teacher data even if the information of the shooting condition group or generation range of the generated OCTA image and the teacher data of the learned model does not completely match.
  • the learned model that has been learned may be selected.
  • the selection unit 925 may include a table in which the correspondence relationship between the imaging condition group and the generation range regarding the OCTA image and the learned model to be used is described.
  • step S1006 the image quality improving unit 224 uses the learned model selected by the selecting unit 925 to perform the image quality improving process on the OCTA image generated in step S1004 to generate a high quality OCTA image.
  • the method of generating a high-quality OCTA image is the same as step S505 according to the first embodiment, and thus the description thereof will be omitted.
  • step S1007 is the same as step S506 according to the first embodiment, a description thereof will be omitted.
  • step S1007 when the high quality OCTA image is displayed on the display unit 270, the series of image processing according to the present embodiment ends.
  • control unit 900 includes the selection unit 925 that selects the learned model used by the image quality improving unit 224 from the plurality of learned models.
  • the selecting unit 925 selects the learned model used by the image quality improving unit 224 based on the range in the depth direction for generating the OCTA image to be subjected to the image quality improving process.
  • the selection unit 925 can select the learned model based on the display site in the OCTA image to be subjected to the image quality improvement processing and the range in the depth direction for generating the OCTA image. Further, for example, the selection unit 925 is used by the image quality improvement unit 224 based on the imaging region including the display region in the OCTA image to be subjected to the image quality improvement process and the range in the depth direction for generating the OCTA image. You may select a completed model. Furthermore, for example, the selection unit 925 may select the learned model used by the image quality improvement unit 224 based on the shooting condition of the OCTA image to be subjected to the image quality improvement process.
  • control unit 900 performs the image quality improvement process by a plurality of learned models learned by using the teacher data configured by the paired groups grouped for each shooting condition and each En-Face image generation range.
  • the image quality improving process can be performed more effectively.
  • the selection unit 925 selects the learned model based on the imaging conditions such as the imaging region or the generation range of the OCTA image has been described, but the learned model based on conditions other than the above. May be changed.
  • the selection unit 925 determines, for example, according to the projection method (maximum intensity projection method or average value projection method) when generating an OCTA image or an En-Face image of brightness, and whether or not to remove artifacts caused by blood vessel shadows.
  • a trained model may be selected. In this case, it is possible to prepare a learned model that has been learned for each teacher data according to the projection method and the presence/absence of the artifact removal processing.
  • the selection unit 925 automatically selects an appropriate learned model according to the shooting conditions, the En-Face image generation range, and the like. On the other hand, there are cases where the operator desires to manually select the image quality improvement processing to be applied to the image. Therefore, the selection unit 925 may select the learned model according to the instruction of the operator.
  • the selection unit 925 may change the learned model and change the image quality improvement process applied to the image in accordance with the instruction of the operator.
  • 11A and 11B show an example of a report screen for switching and displaying images before and after the image quality improvement process.
  • the report screen 1100 shown in FIG. 11A shows a tomographic image 1111 and an OCTA image 1101 to which image quality improvement processing using the automatically selected learned model is applied.
  • the report screen 1100 shown in FIG. 11B shows the tomographic image 1111 and the OCTA image 1102 to which the image quality improving process using the learned model according to the instruction of the operator is applied.
  • the report screen 1100 shown in FIGS. 11A and 11B shows a process designation unit 1120 for changing the image quality improvement process applied to the OCTA image.
  • the OCTA image 1101 displayed on the report screen 1100 shown in FIG. 11A depicts a deep blood vessel (Deep Capillary) of the macula.
  • the image quality improvement process applied to the OCTA image using the learned model automatically selected by the selection unit 925 is suitable for the superficial blood vessel (RPC) of the papilla. Therefore, regarding the OCTA image 1101 displayed on the report screen 1100 shown in FIG. 11A, the image quality improvement processing applied to the OCTA image is not optimal for the blood vessel extracted in the OCTA image.
  • the selecting unit 925 changes the learned model used by the image quality improving unit 224 to a learned model learned using the OCTA image regarding the deep blood vessel of the macula as the teacher data in response to the selection instruction from the operator.
  • the image quality improving unit 224 performs the image quality improving process again on the OCTA image using the learned model changed by the selecting unit 925. As shown in FIG. 11B, the display control unit 250 causes the display unit 270 to display the high-quality OCTA image 1102 newly generated by the image quality improving unit 224.
  • the selecting unit 925 is configured to change the learned model according to the instruction of the operator, so that the operator can re-designate the appropriate image quality improvement process for the same OCTA image. it can. Further, the designation of the image quality improvement processing may be performed many times.
  • control unit 900 is configured so that the image quality improvement processing applied to the OCTA image can be manually changed.
  • control unit 900 may be configured to be able to manually change the image quality improvement process applied to a tomographic image, an En-Face image of brightness, and the like.
  • the report screens shown in FIGS. 11A and 11B have a mode in which images before and after the image quality improvement process are switched and displayed, but a report in a mode in which the images before and after the image quality improvement process are displayed side by side or displayed in an overlapping manner. It may be a screen.
  • the mode of the process designation unit 1120 is not limited to the modes shown in FIGS. 11A and 11B, and may be any mode capable of instructing the image quality improvement process or the learned model.
  • the types of image quality improvement processing shown in FIGS. 11A and 11B are examples, and other types of image quality improvement processing according to the teacher data for the learned model may be included.
  • a plurality of images to which the image quality improving process is applied may be displayed simultaneously. Further, at this time, it is possible to specify which image quality improvement process is to be applied.
  • An example of the report screen in this case is shown in FIGS. 12A and 12B.
  • the report screen 1200 shown in FIG. 12A shows an OCTA image 1201 before the image quality improvement processing.
  • the report screen 1200 shown in FIG. 12B shows the OCTA image 1202 to which the image quality improvement processing according to the operator's instruction is applied.
  • the report screen 1200 shown in FIGS. 12A and 12B shows a process designation unit 1220 for changing the image quality improvement process applied to the OCTA image.
  • the selection unit 925 selects the learned model according to the image quality improvement processing instructed by the processing designation unit 1220 as the learned model used by the image quality improvement unit 224.
  • the image quality improving unit 224 uses the learned model selected by the selecting unit 925 to perform the image quality improving process on the plurality of OCTA images 1201.
  • the display control unit 250 causes the generated plurality of high-quality OCTA images 1202 to be displayed at once on the report screen 1200 as shown in FIG. 12B.
  • the learned model may be selected/changed in accordance with the operator's instruction regarding the image quality improving process for the tomographic image, the En-Face image of luminance, and the like. ..
  • a plurality of images before and after the image quality improvement process may be displayed side by side on the report screen, or may be displayed in an overlapping manner. Also in this case, it is possible to display a plurality of images to which the image quality improving process according to the instruction from the operator is applied at once.
  • the image quality improving unit 224 automatically executes the image quality improving process after capturing the tomographic image and the OCTA image.
  • the image quality improving process performed by the image quality improving unit 224 using the learned model may take a long time. Further, it takes time for the motion contrast generation unit 222 to generate the motion contrast data and the En-Face image generation unit 223 to generate the OCTA image. Therefore, when an image is displayed after waiting for the image quality improvement processing to be completed after shooting, it may take a long time from shooting to display.
  • the imaging of the eye to be inspected using the OCT apparatus may fail due to blinking or unintentional movement of the eye to be inspected. Therefore, the convenience of the OCT apparatus can be improved by confirming the success or failure of imaging at an early stage. Therefore, in the third embodiment, the En-Face image and the OCTA image of the brightness based on the tomographic information obtained by photographing the eye to be inspected are displayed prior to the generation and the display of the high-quality OCTA image, so that the image can be obtained at an early stage.
  • the OCT device is configured so that the captured image can be confirmed.
  • the OCT apparatus according to this embodiment will be described below with reference to FIG. Since the configuration of the OCT apparatus according to the present embodiment is similar to that of the OCT apparatus 1 according to the first embodiment, the same reference numerals are used and the description is omitted. Hereinafter, the OCT apparatus according to the present embodiment will be described focusing on the difference from the OCT apparatus 1 according to the first embodiment.
  • FIG. 13 is a flowchart of a series of image processing according to this embodiment.
  • the acquisition unit 210 acquires a plurality of three-dimensional tomographic information by imaging the eye E with the OCT imaging unit 100.
  • step S1302 is the same as step S502 according to the first embodiment, a description thereof will be omitted.
  • the process proceeds to step S1303.
  • step S1303 the En-Face image generation unit 223 projects the three-dimensional tomographic image generated in step S1302 on a two-dimensional plane to generate a front image of the fundus (en-face image of brightness).
  • step S1304 the display control unit 250 causes the display unit 270 to display the En-Face image having the generated brightness.
  • step S1305 and S1306 are the same as steps S503 and S504 according to the first embodiment, a description thereof will be omitted.
  • step S1307 the display control unit 250 switches the OCTA image before the image quality improvement processing generated in step S1306 to the luminance En-Face image and causes the display unit 270 to display the image.
  • step S1308 similarly to step S505 according to the first embodiment, the image quality improving unit 224 performs the image quality improving process using the learned model on the OCTA image generated in step S1306 to obtain a high quality OCTA image.
  • step S1309 the display control unit 250 switches the generated high quality OCTA image to the OCTA image before the image quality improvement processing and causes the display unit 270 to display the OCTA image.
  • the display control unit 250 prior to the acquisition of the OCTA image by the acquisition unit 210, the luminance En ⁇ that is the front image generated based on the tomographic data in the depth direction of the eye to be inspected.
  • the Face image (third image) is displayed on the display unit 270.
  • the display control unit 250 switches the En-Face image of the displayed brightness to the OCTA image and causes the display unit 270 to display the OCTA image.
  • the display control unit 250 switches the displayed OCTA image to the high image quality OCTA image and displays it on the display unit 270 after the high image quality OCTA image is generated by the image quality improvement unit 224.
  • the operator can check the front image of the eye to be examined immediately after photographing, and can immediately judge the success or failure of photographing.
  • the operator since the OCTA image is displayed immediately after the OCTA image is generated, the operator determines at an early stage whether or not the plurality of three-dimensional tomographic information for generating the motion contrast data is appropriately acquired. You can judge.
  • the operator determines whether or not the photographing is successful at an early stage by displaying the tomographic image and the En-Face image of the brightness before the image quality improvement processing. be able to.
  • the motion contrast data generation process (step S1305) is started after the luminance En-Face image display process (step S1304), but the timing of the motion contrast data generation process is not limited to this. ..
  • the motion contrast generation unit 222 may start the generation process of the motion contrast data in parallel with the generation process (step S1303) of the luminance En-Face image and the display process (step S1304).
  • the image quality improving unit 224 may start the image quality improving process (step S1308) in parallel with the OCTA image display process (step S1307).
  • Example 4 The first embodiment has described the example in which the OCTA images before and after the image quality improvement process are switched and displayed. On the other hand, in the fourth embodiment, images before and after the image quality improvement process are compared.
  • the OCT apparatus according to this embodiment will be described below with reference to FIGS. 14 and 15.
  • the configuration of the OCT apparatus according to the present embodiment is the same as that of the OCT apparatus 1 according to the first embodiment except for the control unit. The description is omitted.
  • the OCT apparatus according to the present embodiment will be described focusing on the difference from the OCT apparatus 1 according to the first embodiment.
  • FIG. 14 shows a schematic configuration of the control unit 1400 according to this embodiment.
  • the configuration of the control unit 1400 according to the present embodiment is the same as that of the control unit 200 according to the first embodiment, except for the image processing unit 1420 and the comparison unit 1426. Therefore, the same components as those shown in FIG. 2 are designated by the same reference numerals and the description thereof will be omitted.
  • the image processing unit 1420 of the control unit 1400 is provided with a comparison unit 1426 in addition to the tomographic image generation unit 221, the motion contrast generation unit 222, the En-Face image generation unit 223, and the image quality improvement unit 224.
  • the comparison unit 1426 compares the image before the image quality improvement processing by the image quality improvement unit 224 (original image) with the image after the image quality improvement processing. More specifically, the comparison unit 1426 compares the images before and after the image quality improvement process, and calculates the difference between the pixel values at the corresponding pixel positions of the images before and after the image quality improvement process.
  • the comparison unit 1426 generates a color map image colored according to the magnitude of the difference value. For example, if the pixel value of the image after the image quality improvement processing is larger than that of the image before the image quality improvement processing, the warm color (yellow to orange to red) color tone is set to the pixel of the image after the image quality improvement processing. When the value is small, a cold (yellowish green to green to blue) color tone is used.
  • a cold color system on the color map image is the tissue restored (or newly created) by the image quality improvement processing.
  • the portion indicated by a cold color system on the color map image is noise (or tissue that has been erased) removed by the image quality improvement processing.
  • the color arrangement of the color map image is an example.
  • the color arrangement of the color map image is arbitrarily set according to the desired configuration, for example, the color arrangement of different color tones is performed according to the magnitude of the pixel value in the image after the image quality improvement processing with respect to the pixel value in the image before the image quality improvement processing. You may
  • the display control unit 250 can superimpose the color map image generated by the comparison unit 1426 on the image before the image quality improvement process or the image after the image quality improvement process and display it on the display unit 270.
  • step S1501 to S1505 are the same as steps S501 to S505 according to the first embodiment, and a description thereof will be omitted.
  • step S1505 when the high quality OCTA image is generated by the image quality improving unit 224, the process proceeds to step S1506.
  • step S1506 the comparison unit 1426 compares the OCTA image generated in step S1504 with the high-quality OCTA image generated in step S1505 to calculate the difference between the pixel values, and based on the difference between the pixel values. Generate a colormap image. Note that the comparison unit 1426 performs image comparison using another method such as a pixel value ratio or a correlation value between the images before and after the high image quality processing, instead of the difference between the pixel values before and after the high image quality processing. A color map image may be generated based on the result.
  • step S1507 the display control unit 250 superimposes the color map image on the image before the image quality improvement process or the image after the image quality improvement process and causes the display unit 270 to display the image.
  • the display control unit 250 can set the transparency of the color map so that the image on which the color map image is superimposed is not hidden, and display the color map image on the target image in an overlapping manner.
  • the display control unit 250 sets a high transparency in a portion of the color map image where the difference between the images before and after the image quality improvement process is small (the pixel value of the color map image is low), or places where the difference is equal to or less than a predetermined value.
  • the transparency may be set so that is completely transparent. By doing so, both the image displayed below the color map image and the color map image can be visually recognized well.
  • the comparison unit 1426 may generate the color map image including the transparency setting.
  • the control unit 1400 includes the comparison unit 1426 that compares the first medical image with the second medical image that has undergone the image quality improvement processing.
  • the comparison unit 1426 calculates a difference between the first medical image and the second medical image and generates a color map image that is color-coded based on the difference.
  • the display control unit 250 controls the display on the display unit 270 based on the comparison result by the comparison unit 1426. More specifically, the display control unit 250 superimposes the color map image on the first medical image or the second medical image and causes the display unit 270 to display the color map image.
  • the operator can more easily identify the tissue that does not actually exist in the image due to the image quality improving process or the tissue that originally exists disappears. It is possible to judge the authenticity of the organization more easily. Further, the operator can easily identify whether it is a newly drawn portion or an erased portion by the image quality improvement processing according to the color arrangement of the color map image.
  • the display control unit 250 can enable or disable the superimposed display of the color map image according to the instruction of the operator.
  • the on/off operation of the superimposed display of the color map image may be simultaneously applied to the plurality of images displayed on the display unit 270.
  • the comparison unit 1426 generates a color map image for each image before and after the corresponding image quality improvement processing, and the display control unit 250 causes the color map image to correspond to the image before the image quality improvement processing or the image after the image quality improvement processing. Can be superimposed and displayed.
  • the display control unit 250 may display the image before the image quality improvement process or the image after the image quality improvement process on the display unit 270 before displaying the color map image.
  • the same process can be performed when the image quality improving process is performed on a tomographic image, an En-Face image of brightness, and the like. Further, the comparison process and the color map display process according to the present embodiment can be applied to the OCT apparatus according to the second and third embodiments.
  • the comparison unit 1426 may compare the images before and after the image quality improvement process, and the display control unit 250 may display a warning on the display unit 270 according to the comparison result by the comparison unit 1426. More specifically, when the difference between the pixel values of the images before and after the image quality improvement process calculated by the comparison unit 1426 is larger than a predetermined value, the display control unit 250 causes the display unit 270 to display a warning. According to such a configuration, in the generated high-quality image, when the learned model generates a tissue that does not actually exist, or the tissue that originally exists is erased, The operator can be alerted.
  • the comparison between the difference and the predetermined value may be performed by the comparison unit 1426 or the display control unit 250. Further, instead of the difference, a statistical value such as an average value of the difference may be compared with the predetermined value.
  • the display control unit 250 may prevent the display unit 270 from displaying the image after the image quality improvement process when the difference between the images before and after the image quality improvement process is larger than a predetermined value.
  • a predetermined value in the generated high-quality image, when the learned model generates a tissue that does not actually exist, or the tissue that originally exists is erased, the high-quality image is generated. Misdiagnosis based on an image can be suppressed.
  • the comparison between the difference and the predetermined value may be performed by the comparison unit 1426 or the display control unit 250. Further, instead of the difference, a statistical value such as an average value of the difference may be compared with the predetermined value.
  • the comparison unit 1426 may compare the analysis results obtained by analyzing the images before and after the image quality improvement processing. More specifically, the comparison unit 1426 calculates, for each pixel position, the difference (degree of increase or decrease) in the analysis result obtained by analyzing the images before and after the image quality improvement process. For example, as shown in FIG. 16A, a difference image 1603 is acquired from the images before and after the image quality improvement processing (acquired image 1601, high-quality image 1602), and the difference image increases and decreases in the difference image. Information such as a location where the image is improved is displayed on the display unit 270 so that the user can easily understand the information by using different colors.
  • the difference image 1603 may be superimposed on at least one of the images (acquired image 1601 and high quality image 1602) before and after the image quality improvement processing.
  • a map display such as the evaluation map 1604 may be performed on a region having large change.
  • the display unit 270 may display an evaluation map showing the evaluation result in each area divided into a plurality of quadrants.
  • the number of quadrants may be, for example, two, four, eight, or the like as shown in the display area 1607, but is not limited to this.
  • the number of quadrants is two, for example, the area may be divided into upper and lower areas or the area may be divided into left and right areas.
  • the center of the quadrant may be a site of interest such as the macula and the optic papilla, but is not limited to this.
  • the operator may be allowed to specify an arbitrary position on the image after the image quality improvement process or the difference image as the center of the quadrant.
  • the coincidence rate of the result calculated as the difference between the analysis results obtained by analyzing the images before and after the image quality improvement process is digitized, and, for example, the evaluation result 1605 of the image quality improvement process, the evaluation of the image quality improvement process of FIG. 16B are performed.
  • the value 1615 can also be displayed.
  • the evaluation map 1604 may be displayed darker as the evaluation value is higher, and lighter as the evaluation value is lower.
  • the evaluation value may be displayed for each quadrant.
  • the analysis result is a value relating to at least one of the analysis parameters shown in the display area 1612 of FIG. 16B, and is, for example, a value relating to blood vessels (for example, blood vessel area density, blood vessel length density, blood vessel length), avascular It is at least one of a value related to a region (for example, circumference length, volume, area, and circularity) and a value related to edema region (a diseased region such as choroidal neovascularization) (for example, volume and area).
  • the analysis result may be, for example, a two-dimensional map (analysis map) of values related to at least one of the analysis parameters.
  • the display control unit 250 may display the comparison result of the comparison unit 1426 (result of comparing the analysis results) on the display unit 270. More specifically, as shown as the evaluation map 1604 in FIG. 16A, the display control unit 250 may cause the display unit 270 to display, as the comparison result, a color map image colored according to the magnitude of the difference value. .. Further, as shown in the evaluation result 1605 of FIG. 16A, the display control unit 250 may cause the display unit 270 to display a warning when the difference between the analysis results is larger than a predetermined value.
  • the display control unit 250 may cause the display unit 270 to display an area in which the difference in the analysis result is larger than a predetermined value, so as to be distinguishable from other areas in which the difference in the analysis result is less than or equal to the predetermined value. Good. Further, the display control unit 250 may cause the display control unit 250 to display a warning on the display unit 270 when the number of pixels having a difference larger than a predetermined value is larger than another predetermined value. Also, some of these displays may be done simultaneously.
  • a statistical value such as the average value of the differences may be compared with the predetermined value.
  • the type of analysis for example, blood vessel density, specific layer thickness
  • various regions of interest for example, blood vessels, specific layers
  • the evaluation result of the region of interest can be obtained more effectively than when the images are directly compared and evaluated.
  • the above-described direct comparison of images and comparison of analysis results may be selectively executed according to the type of image, or either evaluation result may be selectively displayed after the display unit 270. You may make it display on. For example, in the case of an OCTA image, there is a possibility that the above-described artifact may be erroneously recognized as a blood vessel, and therefore the evaluation value obtained by comparing the analysis results may be selectively displayed on the display unit 270. Good.
  • the comparing unit 1426 performs an image quality improvement process on a plurality of images obtained by photographing substantially the same portion of the eye to be inspected at different times and a plurality of images obtained by the image quality improvement process using the plurality of images. You may compare before and after. More specifically, the comparison unit 1426 calculates the difference between the pixel values of the pixel positions corresponding to each other in the images before and after the image quality improvement processing in the plurality of images. At this time, the display control unit 250 may display the comparison result (difference) by the comparison unit 1426 on the display unit 270 in the plurality of images. Thereby, the operator may select any one of the plurality of images in consideration of each comparison result (difference).
  • the display control unit 250 may cause the display unit 270 to display a statistical value such as an average value of a plurality of comparison results (differences) corresponding to a plurality of images.
  • the comparison unit 1426 may compare one image obtained by superimposing a plurality of images (averaging) and an image obtained by performing the image quality improvement process using the one image.
  • the plurality of images are front images generated based on information in the depth direction range of the eye to be inspected, it is preferable that the depth direction ranges are common to each other. At this time, for example, if the depthwise range of one image is set according to an instruction from the operator, the depthwise range of the other image may be set.
  • Two OCTA images are acquired as in the first acquired image 1701 and the second acquired image 1703 in FIG. Images obtained by performing image quality improvement processing on each of them (in FIG. 17, referred to as a first high quality image and a second high quality image) are acquired.
  • a first high quality image and a second high quality image are acquired.
  • the image quality of a shadowless image such as the second acquired image 1703 is improved, the image quality is appropriately improved, and the second high quality image 1704 can be obtained.
  • the evaluation of the image quality improvement processing for each of the first acquired image 1701 and the second acquired image 1703 is provided as a numerical value, so that it is possible to support the doctor's judgment appropriately.
  • an image having an influence such as turbidity has a lower evaluation value, an image having a higher evaluation value may be preferentially displayed.
  • an image with a high evaluation value may be selectively displayed without displaying an image with a low evaluation value.
  • the image quality improving process may be performed after averaging the first acquired image 1701 and the second acquired image 1703.
  • Modification 7 Further, in the above-described various examples and modified examples, by performing the same evaluation by using a device having a plurality of image quality improving units (a plurality of learned models obtained by learning with different learning data), It is possible to provide doctors with various diagnostic information.
  • the plurality of image quality improving units perform the image quality improving process using the learned model obtained by learning with the teacher images selected by the main reading center, as in the second image quality improving process 1613 of FIG. 16B. It can be executed selectively. For example, the user selects one of a plurality of facilities (hospitals, research institutes, etc.) as the second image quality improvement processing 1613 different from the first image quality improvement processing prepared in advance, and thereby the first image quality improvement processing is performed.
  • first high quality image 1609 the image obtained by performing the improvement processing
  • second high quality image 1611 the image obtained by performing the second image quality improvement processing
  • the country name (race) is described together with the facility name, so that a gene-dependent disease and a unique fundus image (myopia, normal tension glaucoma, blood vessel running) are displayed.
  • the image quality can be improved appropriately.
  • the acquired image 1608 and the first high quality image 1610 (1609) are compared to obtain an evaluation result such as an evaluation value 1615, and then the acquired image 1608 and the second high quality image 1611 are compared. By comparing, the evaluation result like the evaluation value 1614 can be obtained.
  • an evaluation value serving as a reference for whether or not to use it as learning data for additional learning may be selectable. For example, when either 75% or 85% can be selected as the above criterion, 85% has higher quality of learning data, but there is a possibility that a large number of learning data cannot be obtained. Although 75% includes learning data of relatively low quality, there is a possibility that a large number of learning data can be obtained. Note that the selectable evaluation values are not limited to these, and may be selected from three or more.
  • the above-mentioned criterion may be selected for each of them. Further, an evaluation value prepared in advance may be used as a reference, or may not be added when the difference is large compared with the evaluation value set as the reference.
  • the display control unit 250 displays the image selected according to the instruction from the examiner from the high-quality image and the input image generated by the image quality improving unit 224. Can be displayed on. Further, the display control unit 250 may switch the display on the display unit 270 from the captured image (input image) to the high-quality image in response to an instruction from the examiner. That is, the display control unit 250 may change the display of the low-quality image to the display of the high-quality image according to the instruction from the examiner. Further, the display control unit 250 may change the display of the high quality image to the display of the low quality image in response to the instruction from the examiner.
  • the image quality improvement unit 224 executes the image quality improvement processing (input of the image to the image quality improvement engine) by the image quality improvement engine (learned model for image quality improvement) in response to an instruction from the examiner. Then, the display control unit 250 may cause the display unit 270 to display the high-quality image generated by the image quality improving unit 224.
  • the image quality improving engine automatically generates a high quality image based on the input image, and the display control unit 250 causes the image quality controller The high-quality image may be displayed on the display unit 270 in response to the instruction.
  • the image quality improvement engine includes a learned model that performs the above-described image quality improvement processing (image quality improvement processing).
  • the display control unit 250 may change the display of the analysis result of the low-quality image to the display of the analysis result of the high-quality image in response to the instruction from the examiner. Further, the display control unit 250 may change the display of the analysis result of the high-quality image to the display of the analysis result of the low-quality image according to the instruction from the examiner. Of course, the display control unit 250 may change the display of the analysis result of the low image quality image to the display of the low image quality image in response to the instruction from the examiner. Further, the display control unit 250 may change the display of the low image quality image to the display of the analysis result of the low image quality image in response to an instruction from the examiner.
  • the display control unit 250 may change the display of the analysis result of the high quality image to the display of the high quality image in response to the instruction from the examiner. Further, the display control unit 250 may change the display of the high-quality image to the display of the analysis result of the high-quality image in response to the instruction from the examiner.
  • the display control unit 250 may change the display of the analysis result of the low image quality image to the display of the analysis result of another type of the low image quality image in response to an instruction from the examiner. Further, the display control unit 250 may change the display of the analysis result of the high-quality image to the display of the analysis result of another type of the high-quality image according to the instruction from the examiner.
  • the analysis result of the high-quality image may be displayed by superimposing the analysis result of the high-quality image on the high-quality image with arbitrary transparency.
  • the analysis result of the low image quality image may be displayed by superimposing the analysis result of the low image quality image on the low image quality image with arbitrary transparency.
  • the display of the analysis result may be changed, for example, to a state in which the analysis result is superimposed on the displayed image with arbitrary transparency.
  • the change to the display of the analysis result may be, for example, a change to the display of an image (for example, a two-dimensional map) obtained by blending the analysis result and the image with arbitrary transparency.
  • the image relating to processing such as display, high image quality, and image analysis according to the present modification may be a tomographic image as well as an OCTA image (motion contrast front image).
  • a tomographic image by B-scan not only a tomographic image by B-scan, but also a different image such as an SLO (Scanning Laser Ophthalmoscope) image, a fundus photograph, or a fluorescent fundus photograph may be used.
  • the user interface for executing the high image quality processing is to instruct execution of the high image quality processing for a plurality of images of different types, and select an arbitrary image from the plurality of images of different types. There may be an instruction to execute the image quality enhancement process.
  • At least one tomographic image to be displayed may be displayed with high image quality.
  • a high-quality tomographic image may be displayed in the region where the OCTA front image is displayed.
  • the B-scan tomographic image is not limited to a luminance tomographic image, and may be a B-scan tomographic image obtained using motion contrast data. It should be noted that only one tomographic image having a high image quality and displayed may be displayed, or a plurality of tomographic images may be displayed. When only one image is displayed, a tomographic image obtained by, for example, a circle scan may be displayed with high image quality.
  • the tomographic images acquired at different positions in the sub-scanning direction may be displayed.
  • images in different scanning directions may be displayed respectively.
  • the image characteristics of a plurality of tomographic images obtained by cross scan and the like are often similar, so for example, using a common learned model obtained by learning these tomographic images as learning data, The image quality in the scanning direction may be improved.
  • a plurality of partially selected tomographic images (for example, two tomographic images at positions symmetrical to each other with respect to a reference line). ) May be displayed respectively.
  • a plurality of tomographic images obtained at different dates and times may be displayed on the display screen for follow-up observation, and an instruction to improve image quality and an analysis result (for example, the thickness of a specific layer) may be displayed. .. Further, the image quality enhancement process may be performed on the tomographic image based on the information stored in the database.
  • the displayed SLO fundus image may be displayed with high image quality.
  • the En-Face image with high brightness may be displayed with high image quality.
  • a plurality of SLO fundus images and En-Face images with a plurality of brightnesses obtained at different dates and times are displayed on the display screen for follow-up observation, and instructions for high image quality and analysis results (for example, a specific layer thickness) are displayed. Etc.) may be displayed.
  • the image quality enhancement processing may be performed on the SLO fundus image or the En-Face image of brightness based on the information stored in the database.
  • the display of the tomographic image, the SLO fundus image, and the brightness En-Face image is merely an example, and these images may be displayed in any manner depending on the desired configuration. Further, at least two or more of the OCTA front image, the tomographic image, the SLO fundus image, and the brightness En-Face image may be displayed with high image quality by a single instruction.
  • the image processing unit 220 in the various embodiments and modifications described above may be provided with an analysis unit (not shown) in addition to the image quality improvement unit 224 and the like.
  • the analysis unit performs image analysis on the high-quality tomographic image generated by the image quality improvement unit 224 based on the analysis condition set for each region.
  • the analysis conditions set for each region for example, layer extraction or blood vessel extraction is set in the retina region or choroid region, and detection of vitreous or vitreous detachment is set in the vitreous region. To be done.
  • the analysis conditions may be set in advance or may be set appropriately by the operator.
  • the analysis unit can perform the layer extraction for the region for which the analysis condition is set, and perform the layer thickness value measurement or the like for the extracted layer. Further, when the blood vessel extraction is set as the analysis condition, the analysis unit can perform the blood vessel extraction on the region for which the analysis condition is set and perform the blood vessel density measurement or the like on the extracted blood vessel. Furthermore, when the detection of the vitreous body or the separation of the vitreous body is set as the analysis condition, the analysis unit detects the vitreous body or the separation of the vitreous body in the region for which the analysis condition is set. After that, the analysis unit can quantify the detected vitreous body and the detachment of the vitreous body to obtain the thickness, width, area, volume, etc.
  • the analysis conditions are not limited to these, and may be set arbitrarily according to the desired configuration. For example, detection of the fibrous structure of the vitreous for the region of the vitreous part may be set. In this case, the analysis unit can quantify the detected fibrous structure of the vitreous and determine the thickness, width, area, volume, etc. of the fibrous structure. Further, the analysis process according to the analysis condition is not limited to the above process and may be arbitrarily set according to a desired configuration. Further, the display control unit 250 may display the result of the image analysis performed by the analysis unit on the display unit 270 together with the high-quality tomographic image or separately from the high-quality tomographic image.
  • the display control unit 250 in the above-described various embodiments and modifications may display analysis results such as the layer thickness of a desired layer and various blood vessel densities on the report screen of the display screen. Further, optic disc, macula, vascular region, nerve fiber bundle, vitreous region, macula region, choroid region, sclera region, lamina cribrosa region, retinal layer boundary, retinal layer boundary end, photoreceptor cell, blood cell, The value (distribution) of the parameter relating to the site of interest including at least one of a blood vessel wall, an inner blood vessel boundary, a blood vessel outer boundary, a ganglion cell, a corneal region, a corner region, and Schlemm's canal may be displayed as the analysis result.
  • the artifacts include, for example, a false image area caused by light absorption by a blood vessel area or the like, a projection artifact, a band-shaped artifact in the front image generated in the main scanning direction of the measurement light depending on the state of the eye to be inspected (movement, blinking, etc.). It may be. Further, the artifact may be any artifact area as long as it is randomly generated on the medical image of the predetermined region of the subject every time of photographing.
  • the value (distribution) of the parameter regarding the area including at least one of the various artifacts (missing area) as described above may be displayed as the analysis result.
  • the value (distribution) of a parameter relating to a region including at least one abnormal site such as drusen, new blood vessels, vitiligo (hard vitiligo), and pseudo drusen may be displayed as the analysis result.
  • the comparison result obtained by comparing the standard value or standard range obtained using the standard database with the analysis result may be displayed.
  • the analysis result may be displayed in an analysis map, a sector indicating a statistical value corresponding to each divided area, or the like.
  • the analysis result may be generated using a learned model (analysis result generation engine, learned model for generating analysis result) obtained by learning the analysis result of the medical image as learning data. ..
  • the learned model includes learning data including a medical image and an analysis result of the medical image, learning data including a medical image and an analysis result of a medical image of a different type from the medical image, and the like. It may be obtained from Further, the learned model is obtained by learning using learning data including input data in which a plurality of medical images of different types of predetermined regions are set, such as a luminance front image and a motion contrast front image. Good.
  • the luminance front image corresponds to the luminance En-Face image
  • the motion contrast front image corresponds to the OCTA En-Face image.
  • the analysis result obtained by using the high quality image generated by the learned model for high image quality may be displayed.
  • the learned model for high image quality is obtained by learning the learning data in which the first image is the input data and the second image having a higher image quality than the first image is the correct answer data. May be.
  • the second image is enhanced in contrast or reduced in noise by, for example, superimposing processing of the plurality of first images (for example, averaging processing of the plurality of first images obtained by alignment).
  • the image may be a high-quality image that has been processed.
  • the input data included in the learning data may be a high quality image generated by a learned model for high image quality, or a set of a low quality image and a high quality image.
  • the learning data is, for example, at least an analysis value obtained by analyzing the analysis area (for example, an average value or a median value), a table including the analysis value, an analysis map, a position of the analysis area such as a sector in the image, and the like.
  • the data including one piece may be data (labeled (annotated)) that is labeled (annotated) as correct answer data (learning with a teacher).
  • the analysis result obtained by the learned model for generating the analysis result may be displayed in response to the instruction from the examiner.
  • the display control unit 250 in the above-described various embodiments and modifications may display various diagnostic results such as glaucoma and age-related macular degeneration on the report screen of the display screen.
  • the diagnosis result may display the specified position of the abnormal part or the like on the image, or may display the state or the like of the abnormal part by characters or the like.
  • a classification result of abnormal parts for example, Curtin classification
  • information indicating the probability of each abnormal part for example, a numerical value indicating a ratio
  • diagnosis result information necessary for the doctor to confirm the diagnosis may be displayed as the diagnosis result.
  • advice such as additional photographing can be considered.
  • the diagnosis result may be information regarding the future medical care policy of the subject.
  • the diagnosis result includes, for example, the diagnosis name, the type and state (degree) of the lesion (abnormal site), the location of the lesion in the image, the location of the lesion with respect to the region of interest, findings (interpretation findings, etc.), and the basis of the diagnosis name (affirmation).
  • Medical support information the basis for denying the diagnosis name (negative medical support information), and the like.
  • a diagnosis result which is more probable than the diagnosis result such as the diagnosis name input in response to an instruction from the examiner may be displayed as the medical support information.
  • the types of medical images that can be the basis of the diagnosis result may be displayed in a distinguishable manner.
  • the diagnosis result may be generated using a learned model (diagnosis result generation engine, a learned model for generating a diagnosis result) obtained by learning the diagnosis result of the medical image as learning data. .. Further, the learned model is obtained by learning using learning data including a medical image and a diagnosis result of the medical image, and learning data including a medical image and a diagnosis result of a medical image of a different type from the medical image. It may be obtained. Further, the diagnosis result obtained by using the high quality image generated by the learned model for high image quality may be displayed.
  • a learned model diagnosis result generation engine, a learned model for generating a diagnosis result obtained by learning the diagnosis result of the medical image as learning data. .
  • the learned model is obtained by learning using learning data including a medical image and a diagnosis result of the medical image, and learning data including a medical image and a diagnosis result of a medical image of a different type from the medical image. It may be obtained. Further, the diagnosis result obtained by using the high quality image generated by the learned model for high image quality may be
  • the input data included in the learning data may be a high quality image generated by a learned model for high image quality, or a set of a low quality image and a high quality image.
  • the learning data includes, for example, the diagnosis name, the type and state (degree) of the lesion (abnormal site), the position of the lesion in the image, the position of the lesion with respect to the region of interest, findings (interpretation findings, etc.), and the basis of the diagnosis name (affirmation).
  • Input information is labeled (annotated) as correct answer data (for supervised learning) that includes at least one of the reason for denying the diagnosis name (negative medical support information), etc. It may be data.
  • the diagnosis result obtained by the learned model for generating the diagnosis result may be displayed in response to the instruction from the examiner.
  • the display control unit 250 in the above-described various embodiments and modified examples the object recognition result (object detection result) of the partial region such as the attention site, the artifact region, or the abnormal site described above on the report screen of the display screen.
  • the segmentation result may be displayed. At this time, for example, a rectangular frame or the like may be superimposed and displayed around the object on the image. Further, for example, a color or the like may be superimposed and displayed on the object in the image.
  • the object recognition result and the segmentation result are the learned model (object recognition engine, object recognition engine A trained model, a segmentation engine, and a trained model for segmentation) may be used.
  • the analysis result generation and the diagnosis result generation described above may be obtained by using the above-described object recognition result and segmentation result. For example, the analysis result generation and the diagnosis result generation may be performed on the attention site obtained by the object recognition and the segmentation processing.
  • an adversarial generation network GAN: GENERAL ADVERSALIAL NETWORKS
  • VAE VARIATIONAL AUTO-ENCODER
  • a DCGAN Deep Constitutional GAN including a generator obtained by learning generation of a tomographic image and a discriminator obtained by learning discrimination between a new tomographic image generated by the generator and a real frontal fundus image.
  • a machine learning model can be used as a machine learning model.
  • the discriminator encodes the input tomographic image into a latent variable, and the generator generates a new tomographic image based on the latent variable. Then, the difference between the input tomographic image and the new generated tomographic image can be extracted as the abnormal portion.
  • VAE the input tomographic image is encoded by an encoder to be a latent variable, and the latent variable is decoded by a decoder to generate a new tomographic image. After that, the difference between the input tomographic image and the generated new tomographic image can be extracted as the abnormal portion.
  • a tomographic image has been described as an example of the input data, a fundus image, a front image of the anterior eye, or the like may be used.
  • the image processing unit 220 may detect an abnormal portion by using a convolutional auto encoder (CAE: Conventional Auto-Encoder).
  • CAE convolutional auto encoder
  • CAE Conventional Auto-Encoder
  • the same image is learned as input data and output data during learning.
  • an image having no abnormal portion is output according to the learning tendency.
  • the difference between the image input to the CAE and the image output from the CAE can be extracted as the abnormal portion.
  • not only the tomographic image but also the fundus image, the front image of the anterior eye, etc. may be used as the input data.
  • the image processing unit 220 provides information regarding the difference between the medical image obtained using the adversarial generation network or the auto encoder and the medical image input to the adversarial generation network or the auto encoder as the information regarding the abnormal part. Can be generated as As a result, the image processing unit 220 can be expected to detect an abnormal part at high speed and with high accuracy.
  • the auto encoder includes VAE, CAE, and the like.
  • the image processing unit 220 provides information regarding a difference between a medical image obtained from various medical images using a hostile generation network or an auto encoder and a medical image input to the hostile generation network or the auto encoder. , Can be generated as information about the abnormal part.
  • the display control unit 250 relates to a difference between a medical image obtained from various medical images using a hostile generation network or an auto encoder and a medical image input to the hostile generation network or the auto encoder.
  • the information can be displayed on the display unit 270 as information about the abnormal part.
  • the learned models used in the various embodiments and modifications described above may be generated and prepared for each type of disease or each abnormal site.
  • the control unit 200 can select the learned model to be used for the processing in accordance with the input (instruction) of the type of disease of the eye to be inspected or the abnormal site from the operator.
  • the learned model prepared for each type of disease or abnormal site is not limited to the learned model used for detection of the retinal layer or generation of the region label image, and for example, an engine for image evaluation or analysis. It may be a trained model used in the engine of the above.
  • the control unit 200 may identify the type of disease of the eye to be inspected or the abnormal portion from the image by using a separately prepared learned model.
  • the control unit 200 can automatically select the learned model to be used for the above-mentioned processing based on the type of disease or the abnormal part identified by using the separately prepared learned model. ..
  • the learned model for identifying the type of disease or abnormal site of the eye to be inspected is tomographic image or fundus image, etc. as input data, and the learning data of the type of disease or abnormal site in these images as output data. You may learn using a pair.
  • a tomographic image, a fundus image, or the like may be used alone as the input data, or a combination thereof may be used as the input data.
  • the learned model for generating the diagnostic result may be a learned model obtained by learning with learning data including input data in which a plurality of medical images of different types of predetermined regions of the subject are set.
  • learning data including input data in which a plurality of medical images of different types of predetermined regions of the subject are set.
  • the input data included in the learning data for example, input data in which a motion contrast front image of the fundus and a luminance front image (or luminance tomographic image) are set can be considered.
  • input data included in the learning data for example, input data in which a tomographic image (B scan image) of the fundus and a color fundus image (or a fluorescent fundus image) are set is also considered.
  • the plurality of medical images of different types may be anything acquired by different modalities, different optical systems, different principles, or the like.
  • the learned model for generating the diagnosis result may be a learned model obtained by learning with learning data including input data in which a plurality of medical images of different parts of the subject are set.
  • the input data included in the learning data for example, input data in which a tomographic image of the fundus (B scan image) and a tomographic image of the anterior segment (B scan image) are set can be considered.
  • input data included in the learning data for example, input data including a set of a three-dimensional OCT image (three-dimensional tomographic image) of the macula of the fundus and a circle scan (or raster scan) tomographic image of the optic disc of the fundus, and the like. Can also be considered.
  • the input data included in the learning data may be a plurality of medical images of different parts and different types of the subject.
  • the input data included in the learning data may be, for example, input data in which a tomographic image of the anterior segment of the eye and a color fundus image are set.
  • the various learned models described above are learned models obtained by learning with learning data including input data in which a plurality of medical images of different imaging fields of view of a predetermined region of the subject are set.
  • the input data included in the learning data may be a combination of a plurality of medical images obtained by time-dividing a predetermined region into a plurality of regions, such as a panoramic image.
  • the processing result can be improved.
  • the examiner may be configured to select each position on the wide-angle image in which the abnormal portion is detected, and an enlarged image of the abnormal portion at the selected position may be displayed. ..
  • the input data included in the learning data may be input data in which a plurality of medical images of a predetermined part of the subject at different dates and times are set as a set.
  • the display screen on which at least one of the above-mentioned analysis result, diagnosis result, object recognition result, and segmentation result is displayed is not limited to the report screen.
  • a display screen is, for example, at least one display screen such as a shooting confirmation screen, a display screen for follow-up observation, and a preview screen for various adjustments before shooting (display screen on which various live moving images are displayed). May be displayed in.
  • the examiner can confirm the accurate result even immediately after the imaging.
  • a frame surrounding the recognized object may be displayed so as to be superimposed on the live moving image.
  • the display change between the low-quality image and the high-quality image described above may be, for example, the display change between the analysis result of the low-quality image and the analysis result of the high-quality image.
  • Machine learning includes, for example, deep learning including a multi-layer neural network.
  • a convolutional neural network CNN
  • a technology related to an auto encoder self-encoder
  • a technique related to back propagation error back propagation method
  • a method (dropout) of randomly inactivating each unit (each neuron) may be used.
  • the machine learning is not limited to deep learning, and may be any learning as long as it uses a model capable of extracting (expressing) the feature amount of learning data such as an image by learning.
  • the machine learning model refers to a learning model based on a machine learning algorithm such as deep learning.
  • the learned model is a model that is trained (learned) using appropriate learning data in advance with respect to a machine learning model by an arbitrary machine learning algorithm.
  • the learned model does not perform any further learning, but can perform additional learning.
  • the learning data is composed of a pair of input data and output data (correct answer data).
  • the learning data may be referred to as teacher data, or the correct answer data may be referred to as teacher data.
  • the GPU can perform an efficient operation by processing more data in parallel. For this reason, when learning is performed multiple times using a learning model such as deep learning, it is effective to perform processing by the GPU. Therefore, in the present modification, a GPU is used in addition to the CPU for the processing by the image processing unit 220, which is an example of a learning unit (not shown). Specifically, when a learning program including a learning model is executed, the CPU and the GPU work together to perform the learning. The processing of the learning unit may be calculated only by the CPU or GPU. Further, the processing unit (estimation unit) that executes the processing using the various learned models described above may use the GPU similarly to the learning unit. In addition, the learning unit may include an error detection unit and an update unit (not shown).
  • the error detection unit obtains an error between the correct data and the output data output from the output layer of the neural network according to the input data input to the input layer.
  • the error detection unit may calculate the error between the output data from the neural network and the correct answer data using the loss function.
  • the updating unit updates the connection weighting coefficient between the nodes of the neural network based on the error obtained by the error detecting unit so that the error becomes small.
  • the updating unit updates the coupling weighting coefficient and the like by using the error back propagation method, for example.
  • the error back-propagation method is a method of adjusting the coupling weighting coefficient between the nodes of each neural network so that the above error becomes small.
  • a machine learning model used for image quality improvement and segmentation there are a function of an encoder composed of a plurality of layers including a plurality of downsampling layers and a function of a decoder composed of a plurality of layers including a plurality of upsampling layers.
  • ambiguous position information (spatial information) in a plurality of layers configured as an encoder is converted into a same-dimensional layer (a layer corresponding to each other) in a plurality of layers configured as a decoder. ) Is used (for example, using a skip connection).
  • a machine learning model used for image quality enhancement, segmentation, etc. for example, FCN (Fully Concurrent Network), SegNet, or the like can be used.
  • a machine learning model for recognizing an object for each area may be used according to a desired configuration.
  • a machine learning model for object recognition for example, RCNN (Region CNN), fastRCNN, or fastRCNN can be used.
  • YOLO You Only Look Once
  • SSD Single Shot Detector, or Single Shot MultiBox Detector
  • the machine learning model may be, for example, a capsule network (CapsNet).
  • CapsNet capsule network
  • each unit is configured to output a scalar value, so that, for example, spatial information regarding a spatial positional relationship (relative position) between features in an image is obtained. It is configured to be reduced. As a result, for example, it is possible to perform learning such that the influence of local distortion of an image, parallel movement, or the like is reduced.
  • each unit is configured to output the spatial information as a vector, and thus is configured to hold the spatial information, for example. Thereby, for example, learning can be performed in consideration of the spatial positional relationship between the features in the image.
  • the high image quality engine (learned model for high image quality) may be a learned model obtained by additionally learning the learning data including at least one high image quality image generated by the high image quality engine. Good. At this time, whether or not to use the high-quality image as learning data for additional learning may be configured to be selectable by an instruction from the examiner. It should be noted that these configurations are applicable not only to the learned model for improving image quality, but also to the various learned models described above. In addition, a learned model for generating correct answer data for generating correct answer data such as labeling (annotation) may be used for generating correct answer data used for learning various learned models described above.
  • the learned model for generating correct answer data may be obtained by performing additional learning (sequentially) on correct answer data obtained by labeling (annotating) the examiner. That is, the learned model for generating correct answer data may be obtained by additionally learning the learning data in which the data before labeling is the input data and the data after the labeling is the output data. Further, in a plurality of consecutive frames such as a moving image, it is configured to correct the result of a frame determined to be low in accuracy, considering the results of object recognition and segmentation of the preceding and following frames. Good. At this time, the corrected result may be additionally learned as correct answer data in response to an instruction from the examiner.
  • a partial region for example, a target region, an artifact region, an abnormal region, etc.
  • a learned model for object recognition or a learned model for segmentation predetermined image processing can be performed on each detected area. For example, consider the case of detecting at least two regions of the vitreous region, the retina region, and the choroid region. In this case, when performing image processing such as contrast adjustment on the detected at least two areas, it is possible to perform adjustment suitable for each area by using different image processing parameters. By displaying the image adjusted for each area, the operator can more appropriately diagnose the disease or the like in each area.
  • the configuration using different image processing parameters for each detected region may be similarly applied to the region of the eye to be detected detected without using the learned model, for example.
  • the various learned models described above may be used for at least one frame of the live moving image in the preview screens in the various embodiments and modifications described above.
  • the learned model corresponding to each live moving image may be used.
  • the processing time can be shortened, so that the examiner can obtain highly accurate information before the start of imaging. For this reason, for example, failure of re-imaging can be reduced, so that accuracy and efficiency of diagnosis can be improved.
  • the plurality of live moving images may be, for example, a moving image of the anterior segment for alignment in the XYZ directions, and a front moving image of the fundus for focus adjustment and OCT focus adjustment of the fundus observation optical system. Further, the plurality of live moving images may be, for example, a tomographic moving image of the fundus for coherence gate adjustment of OCT (adjustment of optical path length difference between measurement optical path length and reference optical path length).
  • OCT adjustment of optical path length difference between measurement optical path length and reference optical path length
  • a value for example, a contrast value or an intensity value
  • a threshold value for example, various adjustments such as OCT focus adjustment may be performed such that the peak value is reached.
  • the OCT of the OCT is performed so that a predetermined retinal layer such as a vitreous region or RPE detected using a learned model for object recognition or a learned model for segmentation is at a predetermined position in the depth direction.
  • the coherence gate adjustment may be performed.
  • the image quality improving unit 224 can perform high image quality processing on the moving image using the learned model to generate a high quality moving image.
  • the drive control unit 230 in a state in which a high-quality moving image is displayed, sets the reflection mirror in the reference optical system so that a partial region such as a region of interest obtained by the segmentation process is located at a predetermined position in the display region. It is possible to drive and control an optical member such as 123 for changing the shooting range. In such a case, the drive control unit 230 can automatically perform the alignment process based on the highly accurate information so that the desired region becomes the predetermined position of the display region.
  • the optical member for changing the photographing range may be, for example, an optical member for adjusting the coherence gate position, and specifically, the reflection mirror 123 or the like in the reference optical system.
  • the coherence gate position can be adjusted by an optical member that changes the optical path length difference between the measurement optical path length and the reference optical path length, and the optical member is, for example, for changing the optical path length of the measurement light (not shown). It may be a mirror or the like.
  • the optical member that changes the shooting range may be, for example, a stage unit (not shown).
  • the drive control unit 230 has been described above such that a partial area such as an artifact area obtained by the segmentation process or the like is re-photographed (rescanned) in the middle of photographing or at the end of photographing in response to an instruction regarding the start of photographing.
  • the scanning unit may be drive-controlled. Further, for example, when the information (eg, a numerical value indicating a ratio) indicating the certainty of the object recognition result regarding the attention site exceeds a threshold value, each adjustment or the start of photographing may be automatically performed. Good.
  • each adjustment and the start of imaging can be executed according to the instruction from the examiner. It may be configured to change to a different state (release the execution prohibited state).
  • the moving image to which the various learned models described above can be applied is not limited to the live moving image, and may be, for example, a moving image stored (saved) in the storage unit 240.
  • a moving image obtained by aligning at least one frame of the fundus tomographic moving image stored (saved) in the storage unit 240 may be displayed on the display screen.
  • a reference frame may be selected based on the condition that the vitreous body exists on the frame as much as possible.
  • each frame is a tomographic image (B scan image) in the XZ direction.
  • a moving image in which another frame is aligned in the XZ direction with respect to the selected reference frame may be displayed on the display screen.
  • the high-quality images (high-quality frames) sequentially generated by the learned model for high image quality may be displayed continuously for at least one frame of the moving image.
  • the same method may be applied to the X-direction alignment method and the Z-direction (depth direction) alignment method, or all different methods may be used. May be applied. Further, the alignment in the same direction may be performed a plurality of times by different methods. For example, the precise alignment may be performed after performing the rough alignment. In addition, as a method of alignment, for example, alignment using a retinal layer boundary obtained by segmentation processing of a tomographic image (B scan image) (coarse in the Z direction) and a plurality of tomographic images obtained by dividing the tomographic image are performed.
  • Positioning precision in the X direction and Z direction using correlation information (similarity) between the region and the reference image, and one-dimensional projection image generated for each tomographic image (B scan image) are used (in the X direction ) Positioning (positioning in the X direction) using a two-dimensional front image.
  • it may be configured such that rough alignment is performed in pixel units and then precise alignment is performed in subpixel units.
  • the imaging target such as the retina of the subject's eye has not been successfully imaged. Therefore, since there is a large difference between the medical image input to the learned model and the medical image used as the learning data, a high quality image may not be obtained with high accuracy. Therefore, when the evaluation value such as the image quality evaluation of the tomographic image (B scan) exceeds the threshold value, the display of the high quality moving image (continuous display of high quality frames) may be automatically started. Further, when the evaluation value such as the image quality evaluation of the tomographic image (B scan) exceeds the threshold value, the image quality enhancement button may be changed to a state (active state) that can be designated by the examiner.
  • a state active state
  • a learned model for high image quality that is different for each shooting mode with different scanning patterns and the like is prepared, and a learned model for high image quality that corresponds to the selected shooting mode is selected. Good. Further, one learned model for image quality improvement obtained by learning the learning data including various medical images obtained in different photographing modes may be used.
  • the learned model after the execution of the additional learning is evaluated, and if there is no problem, the preliminary learned model may be replaced with the learned model after the execution of the additional learning. Further, if there is a problem, a preliminary trained model may be used.
  • a learned model for classification for classifying the high-quality image obtained by the learned model for high image quality with other types of images is used. It may be used.
  • the learned model for classification uses, for example, a plurality of images including a high-quality image and a low-quality image obtained by the learned model for high image quality as input data, and the types of these images are labeled (annotation).
  • the image type of the input data at the time of estimation (at the time of prediction) is displayed together with the information (for example, a numerical value indicating a ratio) indicating the certainty of each image type included in the correct answer data at the time of learning.
  • the information for example, a numerical value indicating a ratio
  • a superimposing process of a plurality of low-quality images for example, an averaging process of a plurality of low-quality images obtained by alignment, etc. Therefore, a high-quality image in which high contrast, noise reduction, etc. are performed may be included.
  • the learned model after the execution of the additional learning for example, the learned model after the execution of the additional learning and the learned model before the execution of the additional learning (preliminary learned model) are used, respectively.
  • the plurality of high quality images obtained from the images may be compared, or the analysis results of the high quality images may be compared.
  • the comparison result of the plurality of high-quality images (an example of change due to additional learning) or the comparison result of analysis results of the plurality of high-quality images (an example of change due to additional learning) is within a predetermined range. It may be determined whether or not, and the determination result may be displayed.
  • a learned model obtained by learning for each imaging region may be selectively used. Specifically, learning including a first learned model obtained by using learning data including a first imaged region (lung, eye to be examined, etc.) and a second imaged region different from the first imaged region A plurality of trained models including a second trained model obtained using the data can be prepared. Then, the control unit 200 may include a selection unit that selects one of the plurality of learned models. At this time, the control unit 200 may include a control unit that executes additional learning on the selected learned model. In response to an instruction from the operator, the control means searches for data in which the imaged region corresponding to the selected learned model and the imaged image of the imaged region are paired, and the data obtained by the search is used as learning data.
  • the learning can be performed as additional learning on the selected trained model.
  • the imaged part corresponding to the selected learned model may be acquired from the information in the header of the data or manually input by the examiner. Further, the data search may be performed via a network from a server or the like of an external facility such as a hospital or a research institute. This makes it possible to efficiently perform additional learning for each imaged region using the imaged image of the imaged region corresponding to the learned model.
  • the selection unit and the control unit may be configured by a software module executed by a processor such as the CPU or MPU of the control unit 200. Further, the selection unit and the control unit may be configured by a circuit that performs a specific function such as an ASIC or an independent device.
  • the learning data for additional learning is acquired from an external server such as the facility described above via a network, it is desired to reduce tampering and decrease in reliability due to system troubles during additional learning. Therefore, the validity of the learning data for additional learning may be detected by confirming the matching by a digital signature or hashing. Thereby, the learning data for additional learning can be protected. At this time, if the legitimacy of the learning data for additional learning cannot be detected as a result of checking the consistency by digital signature or hashing, a warning to that effect is given and additional learning by the learning data is performed. Absent.
  • the server may be in any form such as a cloud server, a fog server, an edge server, etc., regardless of the installation location.
  • the data protection by confirming the matching as described above can be applied not only to the learning data for additional learning but also to the data including the medical image.
  • the image management system may be configured such that transactions of data including medical images among servers of a plurality of facilities are managed by a distributed network. Further, the image management system may be configured to connect a plurality of blocks in which the transaction history and the hash value of the previous block are recorded together in time series. It should be noted that as a technique for confirming the consistency, a cipher that is difficult to calculate even if a quantum computer such as a quantum gate method is used (for example, lattice cipher, quantum cipher by quantum key distribution, etc.) is used. Good.
  • the instruction from the examiner may be an instruction by voice or the like, as well as a manual instruction (for example, an instruction using a user interface or the like).
  • a machine learning model including a voice recognition model obtained by machine learning (a voice recognition engine, a learned model for voice recognition) may be used.
  • the manual instruction may be an instruction by inputting characters using a keyboard or a touch panel.
  • a machine learning model including a character recognition model (character recognition engine, learned model for character recognition) obtained by machine learning may be used.
  • the instruction from the examiner may be an instruction by a gesture or the like.
  • a machine learning model including a gesture recognition model gesture recognition engine, learned model for gesture recognition
  • the instruction from the examiner may be a visual line detection result of the examiner on the display screen of the display unit 270.
  • the line-of-sight detection result may be, for example, a pupil detection result using a moving image of the examiner obtained by photographing the periphery of the display screen of the display unit 270.
  • the object recognition engine as described above may be used to detect the pupil from the moving image.
  • the instruction from the examiner may be an instruction by an electroencephalogram, a weak electric signal flowing through the body, or the like.
  • the learning data character data or voice data (waveform data) indicating an instruction to display the results of the processing of the various learned models as described above is used as the input data, and various learned data is obtained. It may be learning data in which the correct instruction data is an execution command for actually displaying the result of the model processing on the display unit.
  • the learning data for example, character data or voice data indicating a display instruction of a high-quality image obtained by a learned model for high image quality is used as input data, and a high-quality image display execution command and a high-quality image are displayed. It may be learning data in which an execution command for changing the image quality button to the active state is correct data.
  • the learning data may be anything as long as the instruction content and the execution instruction content indicated by the character data, the voice data, or the like correspond to each other.
  • voice data may be converted into character data by using an acoustic model or a language model.
  • the waveform data obtained by a plurality of microphones may be used to perform the process of reducing the noise data superimposed on the voice data.
  • it may be configured such that an instruction by a character or a voice or the like and an instruction by a mouse, a touch panel or the like can be selected according to an instruction from the examiner.
  • ON/OFF of an instruction by characters or voice may be configured to be selectable according to an instruction from an examiner.
  • the machine learning includes deep learning as described above, and a recursive neural network (RNN) can be used as at least a part of the multi-layer neural network, for example.
  • RNN recursive neural network
  • an RNN that is a neural network that handles time series information will be described with reference to FIGS. 18A and 18B.
  • FIG. 18A shows the structure of RNN which is a machine learning model.
  • the RNN 3520 has a loop structure in the network, inputs the data x t 3510 at time t, and outputs the data h t 3530. Since the RNN 3520 has a loop function in the network, it is possible to take over the state at the current time to the next state, so that time series information can be handled.
  • FIG. 18B shows an example of input/output of the parameter vector at time t.
  • the data x t 3510 includes N pieces of data (Params1 to ParamsN).
  • the data h t 3530 output from the RNN 3520 includes N pieces of data (Params 1 to ParamsN) corresponding to the input data.
  • the LSTM may be used.
  • the LSTM can learn long-term information by including a forgetting gate, an input gate, and an output gate.
  • FIG. 19A the structure of the LSTM is shown in FIG. 19A.
  • the information that the network takes over at the next time t is the internal state c t-1 of the network called a cell and the output data h t-1 .
  • the lowercase letters (c, h, x) in the figure represent vectors.
  • FIG. 19B shows details of the LSTM3540.
  • FG is a forget gate network
  • IG is an input gate network
  • OG is an output gate network
  • each is a sigmoid layer. Therefore, a vector in which each element has a value of 0 to 1 is output.
  • the forgetting gate network FG determines how much past information is retained, and the input gate network IG determines which value is updated.
  • the CU is a cell update candidate network and is an activation function tanh layer. This creates a new vector of candidate values that will be added to the cell.
  • the output gate network OG selects a cell candidate element and selects how much information is transmitted at the next time.
  • the LSTM model described above is a basic form, it is not limited to the network shown here.
  • the connection between networks may be changed.
  • QRNN Quasi Current Neural Network
  • the machine learning model is not limited to the neural network, and boosting, support vector machine or the like may be used.
  • a technology related to natural language processing for example, Sequence to Sequence
  • a dialogue engine a dialogue model, a learned model for dialogue that responds to the examiner with an output such as text or voice may be applied.
  • a learned model obtained by pre-learning document data by unsupervised learning may be used.
  • a learned model obtained by further performing transfer learning (or fine tuning) on a learned model obtained by pre-learning may be used.
  • transfer learning or fine tuning
  • BERT Bidirectional Encoder Representations from transforms
  • a model capable of extracting (expressing) a context (feature amount) by itself by predicting a specific word in a sentence from both left and right contexts may be applied.
  • a model capable of determining the relationship (continuity) between two sequences (sentences) in input time series data may be applied.
  • a model in which a Transform Encoder is used in a hidden layer and a vector sequence is input and output may be applied.
  • the instructions from the examiner to which this modification can be applied are for changing the display of various images and analysis results as described in the various embodiments and modifications described above, and for generating the En-Face image.
  • the instruction from the examiner to which this modification can be applied may be not only an instruction after imaging but also an instruction before imaging. For example, an instruction regarding various adjustments and an instruction regarding setting of various imaging conditions. Alternatively, it may be an instruction regarding the start of shooting. Further, the instruction from the examiner to which this modification can be applied may be an instruction regarding the change of the display screen (screen transition). ..
  • the high-quality image and the like may be stored in the storage unit 240 according to an instruction from the examiner.
  • any part of the file name for example, the first part, the last part
  • the displayed image is a high-quality image generated by processing using a learned model for high image quality.
  • a display indicating that there is may be displayed together with the high quality image.
  • the user can easily identify by the display that the displayed high-quality image is not the image itself obtained by shooting, and therefore, it is possible to reduce erroneous diagnosis and improve diagnosis efficiency.
  • the display indicating that the image is a high-quality image generated by the process using the learned model for high-quality image is a display that can distinguish the input image and the high-quality image generated by the process. Any form may be used.
  • not only the processing using the learned model for image quality improvement, but also the processing using the various learned models as described above are the results generated by the processing using the learned model of the type. An indication that there is may be displayed with the result.
  • the display screen such as the report screen may be stored in the storage unit 240 as image data in accordance with an instruction from the examiner.
  • the report screen is a single image in which a high-quality image and the like and a display indicating that these images are high-quality images generated by the processing using the learned model for high-quality image are arranged side by side in the storage unit. It may be stored in 240.
  • the learned model for high image quality learned by what learning data may be displayed on the display unit 270.
  • the display may include a description of the types of input data and correct answer data of the learning data, and an arbitrary display regarding correct answer data such as an imaging region included in the input data and correct answer data. It should be noted that not only the processing using the learned model for image quality improvement, but also the processing using the various learned models as described above is performed by what kind of learning data the learned model of that type learns.
  • a display indicating whether it is a display may be displayed on the display unit 270.
  • the information (for example, characters) indicating that the image is generated by the processing using the learned model for high image quality is configured to be displayed or saved in a state of being superimposed on the high quality image or the like. May be.
  • the place to be superimposed on the image may be any place as long as it does not overlap the region in which the attention site or the like to be imaged is displayed (for example, the edge of the image).
  • a non-overlapping area may be determined and superposed on the determined area.
  • the image quality enhancement button is set as the default state so that the image quality enhancement button is in the active state (image quality enhancement processing is on) as the initial display screen of the report screen
  • the high quality image A report image corresponding to a report screen including the above may be transmitted to the server.
  • the image quality enhancement button is set to the active state by default, at the end of the examination (for example, the shooting confirmation screen or preview screen was changed to the report screen in response to an instruction from the examiner).
  • the report image corresponding to the report screen including the high-quality image may be (automatically) transmitted to the server.
  • various settings in the default settings for example, a depth range for generating an En-Face image on the initial display screen of the report screen, presence/absence of analysis map superimposition, whether or not a high-quality image is displayed, a display screen for follow-up observation It may be configured such that the report image generated based on at least one setting such as whether or not) is transmitted to the server.
  • the storage unit 240 may be a network data server, cloud, database, or the like.
  • the display control of the display unit 270 may be executed through the storage unit 240, the data management medium, or the image management system.
  • the image management system is a device and a system for receiving and storing an image captured by the image capturing device or an image processed by the image capturing device. Further, the image management system may transmit an image in response to a request from a connected device, perform image processing on a stored image, or request an image processing request to another device. it can.
  • the image management system may include, for example, an image preservation communication system (PACS).
  • PACS image preservation communication system
  • the image management system includes a database that can store various information such as the information of the subject associated with the received image and the imaging time. Further, the image management system is connected to a network and can transmit/receive images, convert images, and transmit/receive various information associated with stored images in response to a request from another device. ..
  • information for identifying the content related to learning in the various embodiments and modifications described above may be associated with the image and the information. Thereby, for example, it is possible to easily identify whether or not the stored image is an image after processing using the learned model.
  • the image management system is configured to, when receiving such image data, confirm with the transmission source whether or not the received image data is data obtained by processing using the learned model. May be done.
  • the information to be linked may be learning model information (other processing, disease, device, reading center).
  • the stored information and the displayed information in this modification may be the learned image evaluation results (numerical values, evaluation contents, information such as additional learning) as described above.
  • the first type using the result for example, the analysis result, the diagnosis result, the object recognition result, the segmentation result
  • the image input to the learned model of the second type different from the first type may be generated from the image input to the learned model.
  • the generated image is highly likely to be an image suitable as an image to be processed by the second type learned model. Therefore, an image obtained by inputting the generated image to the second type of learned model (for example, a high-quality image, an image showing the analysis result of an analysis map, an image showing the object recognition result, a segmentation result The accuracy of the image shown) can be improved.
  • the various learned models as described above may be learned models obtained by learning the learning data including the two-dimensional medical image of the subject, or the three-dimensional medical model of the subject. It may be a learned model obtained by learning learning data including images.
  • the similar case image search may be performed using an external database stored in the server or the like, with the analysis result and the diagnosis result obtained by the processing of the learned model as described above as a search key.
  • a similar case image search engine similar case image search model, learned model for similar case image search
  • the control unit 200 searches for similar case images related to the medical image from various medical images using a learned model for searching similar case images (which is different from the learned model for improving image quality). It can be carried out.
  • the display control unit 250 can cause the display unit 270 to display the similar case images obtained from various medical images by using the learned model for searching similar case images.
  • the similar case image is, for example, an image having a feature amount similar to the feature amount of the medical image input to the learned model.
  • a plurality of similar case images may be searched, and a plurality of similar case images may be displayed so that the order in which the feature amounts are similar can be identified.
  • the learning model including the image selected according to the instruction from the examiner and the feature amount of the image is used to additionally learn the learned model for similar case image search. It may be configured to be performed.
  • the process of generating the motion contrast data in the above-described embodiment and modification is not limited to the configuration performed based on the brightness value of the tomographic image.
  • the above-described various processes are performed on the tomographic data including the interference signal acquired by the OCT imaging unit 100, the signal obtained by performing the Fourier transform on the interference signal, the signal obtained by performing an arbitrary process on the signal, and the tomographic image based on these signals. May be applied. Also in these cases, the same effect as that of the above configuration can be obtained.
  • the configuration of the OCT imaging unit 100 is not limited to the above configuration, and a part of the configuration included in the OCT imaging unit 100 may be separate from the OCT imaging unit 100.
  • the configuration of the Mach-Zehnder interferometer is used as the interference optical system of the OCT imaging unit 100, but the configuration of the interference optical system is not limited to this.
  • the interference optical system of the OCT apparatus 1 may have a Michelson interferometer configuration.
  • the configuration of the OCT device according to the present invention is not limited to this.
  • the present invention can be applied to any other type of OCT device such as a wavelength swept OCT (SS-OCT) device using a wavelength swept light source capable of sweeping the wavelength of emitted light.
  • SS-OCT wavelength swept OCT
  • the present invention can also be applied to a Line-OCT device (or an SS-Line-OCT device) using line light.
  • the present invention can also be applied to a Full Field-OCT device (or SS-Full Field-OCT device) using area light.
  • the acquisition unit 210 acquires the interference signal acquired by the OCT imaging unit 100, the three-dimensional tomographic image generated by the image processing unit 220, and the like.
  • the configuration in which the acquisition unit 210 acquires these signals and images is not limited to this.
  • the acquisition unit 210 may acquire these signals from a server or a photographing device that is connected to the control unit via LAN, WAN, the Internet, or the like.
  • the learned model can be provided in the control units 200, 900, 1400 which are image processing devices.
  • the learned model can be composed of, for example, a software module executed by a processor such as a CPU.
  • the learned model may be provided in another server or the like connected to the control units 200, 900, 1400.
  • the control units 200, 900, and 1400 can perform the image quality improvement process using the learned model by connecting to the server including the learned model via an arbitrary network such as the Internet.
  • the server provided with the learned model may be in any form such as a cloud server, fog server, or edge server.
  • the image processed by the image processing device or the image processing method according to the various embodiments and modifications described above includes a medical image acquired by using an arbitrary modality (imaging device, imaging method).
  • the medical image to be processed can include a medical image acquired by an arbitrary imaging device or the like, or an image created by the image processing device or the image processing method according to the above-described embodiments and modifications.
  • the medical image to be processed is an image of a predetermined part of the subject (subject), and the image of the predetermined part includes at least a part of the predetermined part of the subject.
  • the medical image may include other parts of the subject.
  • the medical image may be a still image or a moving image, and may be a monochrome image or a color image.
  • the medical image may be an image showing the structure (morphology) of a predetermined site or an image showing its function.
  • the image showing the function includes images showing blood flow dynamics (blood flow rate, blood flow velocity, etc.) such as an OCTA image, a Doppler OCT image, an fMRI image, and an ultrasonic Doppler image.
  • the predetermined part of the subject may be determined according to the imaging target, human eyes (eye to be examined), brain, lungs, intestines, heart, pancreas, kidneys, organs such as liver, head, chest, It includes any part such as legs and arms.
  • the medical image may be a tomographic image of the subject or a front image.
  • the front image is, for example, a front image of the fundus of the eye, a front image of the anterior segment of the eye, a fundus image taken by fluorescence imaging, or at least a part of a range (three-dimensional OCT data) acquired by OCT in the depth direction of the imaging target.
  • An En-Face image generated using the data of 1. is included.
  • the En-Face image is an OCTA En-Face image (motion contrast front image) generated using at least a partial range of data in the depth direction of the imaging target for the three-dimensional OCTA data (three-dimensional motion contrast data). ) Is OK.
  • the three-dimensional OCT data and the three-dimensional motion contrast data are examples of the three-dimensional medical image data.
  • the motion contrast data is data indicating a change between a plurality of volume data obtained by controlling the measurement light to be scanned a plurality of times in the same region (same position) of the eye to be inspected.
  • the volume data is composed of a plurality of tomographic images obtained at different positions.
  • the motion contrast data can be obtained as the volume data by obtaining the data indicating the change between the plurality of tomographic images obtained at the substantially same position at each of the different positions.
  • the motion contrast front image is also called an OCTA front image (OCTA En-Face image) relating to OCT angiography (OCTA) for measuring the movement of blood flow, and the motion contrast data is also called OCTA data.
  • the motion contrast data can be obtained, for example, as a decorrelation value between two tomographic images or corresponding interference signals, a variance value, or a value obtained by dividing the maximum value by the minimum value (maximum value/minimum value). , May be obtained by any known method.
  • the two tomographic images can be obtained, for example, by controlling so that the measurement light is scanned a plurality of times in the same region (same position) of the subject's eye.
  • the En-Face image is, for example, a front image generated by projecting data in the range between two layer boundaries in the XY directions.
  • the front image is the depth range of at least a part of the volume data (three-dimensional tomographic image) obtained by using optical interference, and is the data corresponding to the depth range determined based on the two reference planes. Is projected or integrated on a two-dimensional plane and generated.
  • the En-Face image is a front image generated by projecting, on the two-dimensional plane, data corresponding to the depth range determined based on the detected retinal layer in the volume data.
  • the representative value of the data within the depth range is set as the pixel value on the two-dimensional plane.
  • the representative value may include a value such as an average value, a median value, or a maximum value of the pixel values within the range in the depth direction of the area surrounded by the two reference planes.
  • the depth range related to the En-Face image is a range including a predetermined number of pixels in a deeper direction or a shallower direction with reference to one of two layer boundaries regarding the detected retinal layer, for example. Good.
  • the depth range related to the En-Face image may be, for example, a range that is changed (offset) in accordance with an operator's instruction from a range between two layer boundaries regarding the detected retinal layer. Good.
  • the image capturing device is a device for capturing an image used for diagnosis.
  • the imaging device detects, for example, a device that obtains an image of a predetermined region by irradiating a predetermined region of the subject with light, radiation such as X-rays, electromagnetic waves, or ultrasonic waves, or radiation emitted from a subject. It includes a device for obtaining an image of a predetermined portion by doing so.
  • the imaging apparatuses according to the various embodiments and modifications described above include at least an X-ray imaging apparatus, a CT apparatus, an MRI apparatus, a PET apparatus, a SPECT apparatus, an SLO apparatus, an OCT apparatus, an OCTA apparatus, a fundus. It includes a camera and an endoscope.
  • the OCT device may include a time domain OCT (TD-OCT) device and a Fourier domain OCT (FD-OCT) device.
  • the Fourier domain OCT device may include a spectral domain OCT (SD-OCT) device and a wavelength swept OCT (SS-OCT) device.
  • the OCT device may include a Doppler-OCT device.
  • the SLO device and the OCT device may include a wavefront compensation SLO (AO-SLO) device using a wavefront compensation optical system, a wavefront compensation OCT (AO-OCT) device, and the like.
  • AO-SLO wavefront compensation SLO
  • the SLO device or the OCT device may include a polarization SLO (PS-SLO) device or a polarization OCT (PS-OCT) device for visualizing information on the polarization phase difference or depolarization.
  • PS-SLO polarization SLO
  • PS-OCT polarization OCT
  • the SLO device and the OCT device may include a pathological microscope SLO device, a pathological microscope OCT device, and the like.
  • the SLO device and the OCT device may include a handheld SLO device and a handheld OCT device.
  • the SLO device and the OCT device may include a catheter SLO device, a catheter OCT device, and the like.
  • the present invention supplies software (program) that realizes one or more functions of the various embodiments and modifications described above to a system or apparatus via a network or a storage medium, and a computer (or CPU) of the system or apparatus. (Or MPU, etc.) can also be realized by the process of reading and executing the program.
  • a computer has one or more processors or circuits and may include separate computers or networks of separate processors or circuits for reading and executing computer-executable instructions.
  • the processor or circuit may include a central processing unit (CPU), a micro processing unit (MPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), or a field programmable gateway (FPGA). Also, the processor or circuit may include a digital signal processor (DSP), a data flow processor (DFP), or a neural processing unit (NPU).
  • CPU central processing unit
  • MPU micro processing unit
  • GPU graphics processing unit
  • ASIC application specific integrated circuit
  • FPGA field programmable gateway
  • DSP digital signal processor
  • DFP data flow processor
  • NPU neural processing unit

Landscapes

  • Life Sciences & Earth Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Biophysics (AREA)
  • Ophthalmology & Optometry (AREA)
  • Engineering & Computer Science (AREA)
  • Biomedical Technology (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Physics & Mathematics (AREA)
  • Molecular Biology (AREA)
  • Surgery (AREA)
  • Animal Behavior & Ethology (AREA)
  • General Health & Medical Sciences (AREA)
  • Public Health (AREA)
  • Veterinary Medicine (AREA)
  • Eye Examination Apparatus (AREA)

Abstract

Ce dispositif de traitement d'image comprend: une unité d'amélioration de la qualité d'image pour générer, à partir d'une première image médicale d'un sujet, une seconde image médicale obtenue par réalisation d'un traitement d'amélioration de la qualité d'image de la première image médicale, à l'aide d'un modèle pré-appris; une unité de comparaison pour comparer un résultat d'analyse obtenu par analyse de la première image médicale, et un résultat d'analyse obtenu par analyse de la seconde image médicale; et une unité de commande d'affichage pour commander l'affichage sur une unité d'affichage sur la base du résultat de la comparaison provenant de l'unité de comparaison.
PCT/JP2019/050732 2018-12-26 2019-12-25 Dispositif de traitement d'image, procédé de traitement d'image et programme WO2020138128A1 (fr)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN201980086346.9A CN113226153A (zh) 2018-12-26 2019-12-25 图像处理装置、图像处理方法和程序
US17/343,207 US20210304363A1 (en) 2018-12-26 2021-06-09 Image processing apparatus, image processing method and computer-readable medium

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
JP2018-243769 2018-12-26
JP2018243769 2018-12-26
JP2019-217331 2019-11-29
JP2019217331A JP7341874B2 (ja) 2018-12-26 2019-11-29 画像処理装置、画像処理方法、及びプログラム

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/343,207 Continuation US20210304363A1 (en) 2018-12-26 2021-06-09 Image processing apparatus, image processing method and computer-readable medium

Publications (1)

Publication Number Publication Date
WO2020138128A1 true WO2020138128A1 (fr) 2020-07-02

Family

ID=71126666

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2019/050732 WO2020138128A1 (fr) 2018-12-26 2019-12-25 Dispositif de traitement d'image, procédé de traitement d'image et programme

Country Status (1)

Country Link
WO (1) WO2020138128A1 (fr)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112053363A (zh) * 2020-08-19 2020-12-08 苏州超云生命智能产业研究院有限公司 视网膜血管分割方法、装置及模型构建方法
WO2022079550A1 (fr) * 2020-10-14 2022-04-21 King Abdullah University Of Science And Technology Système et procédé de pré-entraînement pour traitement de données sismiques à l'aide d'un apprentissage machine
US20220183551A1 (en) * 2020-12-16 2022-06-16 Canon Kabushiki Kaisha Optical coherence tomography apparatus, control method for optical coherence tomography apparatus, and computer readable storage medium

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2007280229A (ja) * 2006-04-11 2007-10-25 Fujifilm Corp 類似症例検索装置、類似症例検索方法およびそのプログラム
WO2014167935A1 (fr) * 2013-04-08 2014-10-16 株式会社 日立メディコ Tomodensitomètre à rayons x, dispositif d'opération de reconstruction et procédé d'opération de reconstruction
WO2018039380A1 (fr) * 2016-08-26 2018-03-01 Elekta, Inc. Systèmes et procédés de segmentation d'images à l'aide de réseaux neuronaux convolutionnels

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2007280229A (ja) * 2006-04-11 2007-10-25 Fujifilm Corp 類似症例検索装置、類似症例検索方法およびそのプログラム
WO2014167935A1 (fr) * 2013-04-08 2014-10-16 株式会社 日立メディコ Tomodensitomètre à rayons x, dispositif d'opération de reconstruction et procédé d'opération de reconstruction
WO2018039380A1 (fr) * 2016-08-26 2018-03-01 Elekta, Inc. Systèmes et procédés de segmentation d'images à l'aide de réseaux neuronaux convolutionnels

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
DUFAN WU; KYUNGSANG KIM; GEORGES EL FAKHRI; QUANZHENG LI: "Iterative low-dose CT reconstruction with priors trained by artificial neural network", IEEE TRANSACTIONS ON MEDICAL IMAGING, vol. 36, no. 12, 1 December 2017 (2017-12-01), pages 2479 - 2486, XP055547653, ISSN: 0278-0062, DOI: 10.1109/TMI.2017.2753138 *
HASEGAWA, REI: "3. ''PixelShine'', a noise reduction process using AI", INNERVISION, vol. 32, no. 7, 25 June 2017 (2017-06-25), JP, pages 31 - 34, XP009522252, ISSN: 0913-8919 *
HIGAKI, TOORU: "Verification of physical characteristics of DIP '' AiCE", INNERVISION, vol. 33, no. 12, 25 November 2018 (2018-11-25), pages 6 - 7, XP009522250, ISSN: 0913-8919 *

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112053363A (zh) * 2020-08-19 2020-12-08 苏州超云生命智能产业研究院有限公司 视网膜血管分割方法、装置及模型构建方法
CN112053363B (zh) * 2020-08-19 2023-12-15 苏州超云生命智能产业研究院有限公司 视网膜血管分割方法、装置及模型构建方法
WO2022079550A1 (fr) * 2020-10-14 2022-04-21 King Abdullah University Of Science And Technology Système et procédé de pré-entraînement pour traitement de données sismiques à l'aide d'un apprentissage machine
US20220183551A1 (en) * 2020-12-16 2022-06-16 Canon Kabushiki Kaisha Optical coherence tomography apparatus, control method for optical coherence tomography apparatus, and computer readable storage medium
US11819275B2 (en) * 2020-12-16 2023-11-21 Canon Kabushiki Kaisha Optical coherence tomography apparatus, control method for optical coherence tomography apparatus, and computer-readable storage medium

Similar Documents

Publication Publication Date Title
JP7229881B2 (ja) 医用画像処理装置、学習済モデル、医用画像処理方法及びプログラム
JP7341874B2 (ja) 画像処理装置、画像処理方法、及びプログラム
JP7250653B2 (ja) 画像処理装置、画像処理方法及びプログラム
US20210104313A1 (en) Medical image processing apparatus, medical image processing method and computer-readable medium
JP7269413B2 (ja) 医用画像処理装置、医用画像処理システム、医用画像処理方法及びプログラム
US11922601B2 (en) Medical image processing apparatus, medical image processing method and computer-readable medium
JP7305401B2 (ja) 画像処理装置、画像処理装置の作動方法、及びプログラム
US11887288B2 (en) Image processing apparatus, image processing method, and storage medium
US20220151483A1 (en) Ophthalmic apparatus, method for controlling ophthalmic apparatus, and computer-readable medium
WO2020202680A1 (fr) Dispositif et procédé de traitement d'informations
WO2020138128A1 (fr) Dispositif de traitement d'image, procédé de traitement d'image et programme
JP7332463B2 (ja) 制御装置、光干渉断層撮影装置、光干渉断層撮影装置の制御方法、及びプログラム
JP7362403B2 (ja) 画像処理装置および画像処理方法
JP2021164535A (ja) 画像処理装置、画像処理方法、及びプログラム
WO2020075719A1 (fr) Dispositif de traitement d'image, procédé de traitement d'image et programme
JP2022011912A (ja) 画像処理装置、画像処理方法、及びプログラム
JP7406901B2 (ja) 情報処理装置及び情報処理方法
JP2021069667A (ja) 画像処理装置、画像処理方法及びプログラム
JP7488934B2 (ja) 画像処理装置、画像処理装置の作動方法、及びプログラム
JP2022121202A (ja) 画像処理装置および画像処理方法
JP2023010308A (ja) 画像処理装置および画像処理方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19902632

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19902632

Country of ref document: EP

Kind code of ref document: A1