CN113412615B - Information processing terminal, recording medium, information processing system, and color correction method - Google Patents
Information processing terminal, recording medium, information processing system, and color correction method Download PDFInfo
- Publication number
- CN113412615B CN113412615B CN202080012931.7A CN202080012931A CN113412615B CN 113412615 B CN113412615 B CN 113412615B CN 202080012931 A CN202080012931 A CN 202080012931A CN 113412615 B CN113412615 B CN 113412615B
- Authority
- CN
- China
- Prior art keywords
- display
- display image
- image
- unit
- information processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000012937 correction Methods 0.000 title claims abstract description 126
- 230000010365 information processing Effects 0.000 title claims abstract description 113
- 238000000034 method Methods 0.000 title claims description 21
- 239000003086 colorant Substances 0.000 claims abstract description 41
- 238000000605 extraction Methods 0.000 claims abstract description 30
- 238000003384 imaging method Methods 0.000 claims abstract description 25
- 239000000284 extract Substances 0.000 claims abstract description 12
- 239000007787 solid Substances 0.000 claims description 15
- 238000004891 communication Methods 0.000 claims description 4
- 238000012545 processing Methods 0.000 description 43
- 238000010586 diagram Methods 0.000 description 15
- 210000001747 pupil Anatomy 0.000 description 9
- 235000002673 Dioscorea communis Nutrition 0.000 description 8
- 241000544230 Dioscorea communis Species 0.000 description 8
- 208000035753 Periorbital contusion Diseases 0.000 description 8
- 238000012986 modification Methods 0.000 description 7
- 230000004048 modification Effects 0.000 description 7
- 239000011324 bead Substances 0.000 description 2
- 210000004087 cornea Anatomy 0.000 description 2
- 239000011521 glass Substances 0.000 description 2
- 230000007774 longterm Effects 0.000 description 2
- 239000003550 marker Substances 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 238000012935 Averaging Methods 0.000 description 1
- 241000282326 Felis catus Species 0.000 description 1
- 241001465754 Metazoa Species 0.000 description 1
- 241001469893 Oxyzygonectes dovii Species 0.000 description 1
- 235000002194 Pithecellobium dulce Nutrition 0.000 description 1
- 235000007891 Pithecellobium lobatum Nutrition 0.000 description 1
- 244000134260 Pithecellobium unguis cati Species 0.000 description 1
- 230000004397 blinking Effects 0.000 description 1
- CLOMYZFHNHFSIQ-UHFFFAOYSA-N clonixin Chemical compound CC1=C(Cl)C=CC=C1NC1=NC=CC=C1C(O)=O CLOMYZFHNHFSIQ-UHFFFAOYSA-N 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 229910003460 diamond Inorganic materials 0.000 description 1
- 239000010432 diamond Substances 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000012805 post-processing Methods 0.000 description 1
- 238000002310 reflectometry Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/10—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from different wavelengths
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T1/00—General purpose image data processing
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N17/00—Diagnosis, testing or measuring for television systems or their details
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N17/00—Diagnosis, testing or measuring for television systems or their details
- H04N17/02—Diagnosis, testing or measuring for television systems or their details for colour television signals
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/70—Circuitry for compensating brightness variation in the scene
- H04N23/76—Circuitry for compensating brightness variation in the scene by influencing the image signals
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- General Health & Medical Sciences (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Image Processing (AREA)
- Color Television Image Signal Generators (AREA)
- Studio Devices (AREA)
Abstract
An information processing terminal having a display unit that displays a display image, and an imaging unit that can image the face of a user who is looking at the display unit, the information processing terminal comprising: a display control unit that causes the display unit to display a display image including 1 or more colors; a shooting control unit that causes the shooting unit to shoot in a state where the display image is displayed; a display image extraction unit that extracts reflected light of the display image from a captured image; and a color correction unit that performs color correction on the photographed image based on the extracted reflected light of the display image.
Description
Technical Field
The invention relates to an information processing terminal, a program, an information processing system, and a color correction method.
Background
When shooting is performed by using a camera or other imaging device, the color of the captured image changes according to the surrounding illumination environment, the setting of the imaging device, and the like. Therefore, in order to accurately reproduce colors from a captured image, correction of colors is required.
Conventionally, a technique is known that corrects colors by preparing a color chart (color chart) and photographing the color chart.
(patent document 1)
Prior art literature
Patent literature
Patent document 1: japanese patent laid-open No. 2005-045438
Disclosure of Invention
Problems to be solved by the invention
However, in the conventional art, in order to perform color correction, a color chart must be prepared. Further, since the photographing scene needs to be included in the color chart, it is necessary to prevent other objects from being reflected on a portion of the color chart, and the range in which photographing is possible is limited. In addition, the color chart itself is also reflected in the captured image.
An object of one embodiment of the present invention is to provide an information processing terminal capable of performing color correction on a photographed image without using a color chart.
Means for solving the problems
An information processing terminal is disclosed which has a display unit for displaying a display image, and an imaging unit capable of imaging the face of a user who is looking at the display unit, and comprises: a display control unit that causes the display unit to display a display image including 1 or more colors; a shooting control unit that causes the shooting unit to shoot in a state where the display image is displayed; a display image extraction unit that extracts reflected light of the display image from a captured image; and a color correction unit that performs color correction on the photographed image based on the extracted reflected light of the display image.
ADVANTAGEOUS EFFECTS OF INVENTION
The color correction can be performed on the image photographed using the information processing terminal without using a color chart.
Drawings
Fig. 1 is a front view of one example of an information processing terminal according to the present embodiment.
Fig. 2 is a hardware configuration diagram of one example of an information processing terminal according to the present embodiment.
Fig. 3 is a processing block diagram of one example of an information processing terminal according to the present embodiment.
Fig. 4 is a flowchart of one example of overall processing using the color correction method of the information processing terminal according to the present embodiment.
Fig. 5 is a flowchart of color correction image extraction processing in the information processing terminal according to the present embodiment.
Fig. 6 is a flowchart of correction coefficient calculation processing in the information processing terminal according to the present embodiment.
Fig. 7 is a flowchart of color correction processing in the information processing terminal according to the present embodiment.
Fig. 8 is a diagram showing one example of a display image displayed with the display section of the information processing terminal according to the present embodiment.
Fig. 9 is a diagram for explaining extraction of a color correction image of the information processing terminal according to the present embodiment.
Fig. 10 is a diagram for explaining a processing example of color correction processing performed by the information processing terminal according to the present embodiment.
Fig. 11 is a configuration diagram of one example of an information processing terminal and a server apparatus according to the present embodiment.
Fig. 12A is a diagram showing a modification of the display image of the information processing terminal according to the present embodiment.
Fig. 12B is a diagram showing a modification of the display image of the information processing terminal according to the present embodiment.
Fig. 12C is a diagram showing a modification of the display image of the information processing terminal according to the present embodiment.
Fig. 12D is a diagram showing a modification of the display image of the information processing terminal according to the present embodiment.
Detailed Description
The present embodiment will be described below with reference to the drawings.
< hardware Structure >
Information processing terminal 1>
Fig. 1 is a front view of an information processing terminal 1 according to the present embodiment. The information processing terminal 1 may be a communication terminal such as a mobile phone or a smart phone, or an information terminal such as a portable personal computer or a tablet computer. Here, the information processing terminal 1 according to the present embodiment will be described by taking a smart phone as an example. The information processing terminal 1 includes a display unit 100 and an imaging unit 200 on the front side. The display unit 100 is a so-called touch panel. The imaging unit 200 is a camera provided on the display unit 100 side of the information processing terminal 1, that is, a so-called built-in camera.
The information processing terminal 1 is implemented by, for example, an information processing terminal constituted by hardware shown in fig. 2. Fig. 2 is a hardware configuration diagram of one example of an information processing terminal according to the present embodiment.
The information processing terminal 1 of fig. 2 includes a CPU (Central Processing Unit: central processing unit) 10, a RAM (Random Access Memory: random access Memory) 20, a ROM (Read Only Memory) 30, an external I/F (Interface) 40, a recording medium 45, a wireless I/F50, a sound processing unit 60, a display unit 100, and an imaging unit 200. Each connected by bus B1.
The CPU10 is an arithmetic device that reads a program (application program) from a storage device such as the ROM30 or the recording medium 45 onto the RAM20 and executes the processing.
The RAM20 is one example of a volatile semiconductor memory (storage device) that temporarily stores programs (application programs) and the like.
The ROM30 is one example of a nonvolatile semiconductor memory (storage device) capable of storing a program (application program) or the like even when the power supply is turned off. The ROM30 stores programs such as a BIOS (Basic Input/Output System) executed at the time of startup, various settings such as OS (Operating System) settings and network settings.
The external I/F40 is an interface with an external device such as the recording medium 45. The information processing terminal 1 of fig. 2 is capable of reading and/or writing the recording medium 45 using the external I/F40.
The recording medium 45 includes an SD (Secure Digital) memory card, a USB (Universal Serial Bus: universal serial bus) memory, and the like.
The wireless I/F50 is an interface to be connected to an external network such as a wireless LAN (Local Area Network: local area network) or the internet in a wireless manner. The wireless I/F50 includes an antenna, and is connected to an external network using a mobile communication system such as LTE (Long Term Evolution: long term evolution) or a wireless communication system such as wireless LAN (Local Area Network: local area network) or Bluetooth (registered trademark). The information processing terminal 1 can communicate with an external server apparatus or the like by using the wireless I/F50.
The sound processing unit 60 is a device that performs input/output processing of sound and the like. The microphone 62 and the speaker 64 are connected to the sound processing unit 60, and sound is input from the microphone 62 and output from the speaker 64.
The display unit 100 is a display with a touch panel. The display unit 100 displays a display image, UI (User Interface), information, and the like, which will be described later. Further, by detecting a touch on the screen, an input such as an operation is performed.
The imaging unit 200 is a camera provided on the display unit 100 side of the information processing terminal 1, that is, a so-called built-in camera. Since the photographing section 200 is provided on the display section 100 side of the information processing terminal 1 as shown in fig. 1, the photographing section 200 can photograph, for example, the face of the user who looks at the display section 100.
The information processing terminal 1 according to the present embodiment can realize various processes described later by executing programs (application programs) in the above-described hardware configuration.
The information processing terminal 1 of the present embodiment performs color correction on the face image of the user captured by the capturing section 200 using the face image of the user. Specifically, the information processing terminal 1 displays a predetermined display image on the display unit 100. The imaging unit 200 of the information processing terminal 1 images the face of the user while the user looks at the display unit 100. The information processing terminal 1 calculates a correction coefficient of the color using the display image of the display unit 100 reflected in the captured face image of the user. And performing color correction of the photographed user face image using the correction coefficient.
< software Structure >
The software configuration of the present embodiment will be described with reference to the information processing terminal 1. Fig. 3 is a processing block diagram of one example of the information processing terminal 1 according to the present embodiment. The information processing terminal 1 includes a display control unit 310, a shooting control unit 320, a display image extraction unit 330, a correction coefficient calculation unit 340, a color correction unit 350, and a control unit 400.
The display control unit 310 controls to display a predetermined image (for example, a display image described later) on the display section 100 of the information processing terminal 1. The display section 100 displays a predetermined image under the control of the display control unit 310.
The imaging control unit 320 controls the imaging unit 200 of the information processing terminal 1 to perform imaging in a state where a predetermined image is displayed on the display unit 100 of the information processing terminal 1. The photographing section 200 performs photographing under the control of the photographing control unit 320.
The display image extracting unit 330 extracts, from the image (captured image) captured by the capturing unit 200, an image (display image) displayed by the display unit 100, which is reflected in the eyes of the user, particularly in a so-called black bead portion (iris and pupil portion) of the eyes.
The correction coefficient calculation unit 340 finds the chromaticity value of the image extracted by the display image extraction unit 330. Based on the obtained chromaticity value and the chromaticity value of the image (display image) displayed by the display unit 100, a correction coefficient of the color is calculated.
The color correction unit 350 corrects the color of the image captured by the capturing section 200 using the correction coefficient calculated by the correction coefficient calculation unit 340.
The control unit 400 controls the entirety of the information processing terminal 1.
< treatment >
< overall treatment >
The information processing terminal 1 according to the present embodiment performs processing in the order shown in fig. 4, for example. Fig. 4 is a flowchart showing one example of the overall processing of the information processing terminal 1 according to the present embodiment. In the present embodiment, the display image 500 is displayed, the image is captured by the imaging unit 200, a correction coefficient is calculated using the captured image, and the color of the captured image is corrected.
(step S10) when a program (application program) is executed by the user, the control unit 400 of the information processing terminal 1 controls in such a manner as to start the photographing section 200. The photographing part 200 is started under the control of the control unit 400.
The display control unit 310 of the information processing terminal 1 controls the display unit 100 to display the display image (step S20). The display unit 100 displays a display image under the control of the display control unit 310.
An example of a display image displayed by the display unit 100 will be described. Fig. 8 is a diagram showing one example of a display image displayed with the display section 100 of the information processing terminal 1 according to the present embodiment. The display image 500 is a 5-color pattern painted with white, red, green, blue, and gray colors everywhere. And displaying the red, green and blue colors by solid colors. In fig. 8, white, red, green, blue, and gray are shown with the letter W, R, G, B, GY attached. (the same applies to FIGS. 12A, 12B, 12C, and 12D.)
White and red are arranged in the 1 st row, green and blue are arranged in the 2 nd row, and gray is arranged in the 3 rd row of the display image 500. In the display image 500 of the present embodiment, the height of the 3 rd line is higher than the heights of the 1 st and 2 nd lines.
(step S30) the information processing terminal 1 instructs the user to face the display section 100. Specifically, the control unit 400 of the information processing terminal 1 instructs "please face to face on the screen" by voice from the speaker 64 through the voice processing unit 60. ".
If necessary, the information processing terminal 1 may instruct the user to take off the glasses or the contact lenses when instructing the user to face the display portion 100.
The imaging control unit 320 of the information processing terminal 1 (step S40) controls the imaging unit 200 to perform imaging. The imaging unit 200 performs imaging under the control of the imaging control unit 320. When the photographing section 200 performs photographing, the photographing control unit 320 is set to a state in which the display image 500 is displayed on the display section 100.
Regarding the timing at the time of shooting, for example, shooting may be performed when the user taps the display section 100. In addition, shooting can be performed when a user operates a button such as a shutter button. The countdown may be performed by the display of the display unit 100 or the sound from the speaker 64 of the sound processing unit 60, and the photographing may be performed at the end of the countdown. In any case, when shooting is performed by the shooting unit 200, the shooting control unit 320 is set to a state in which the display image 500 is displayed on the display unit 100.
It should be noted that the order of step S20 and step S30 may be reversed. That is, first, the information processing terminal 1 instructs the user to face the display unit 100. Then, the display control unit 310 of the information processing terminal 1 controls the display unit 100 to display the display image.
At this time, instead of "please face to the screen" being instructed by the sound processing section 60 with sound from the speaker 64. The display unit 100 may display "please face to screen" in text. ". In addition, an illustration, an icon, or the like may be displayed on the display portion 100 to instruct the user to face the display portion 100. The image photographed by the photographing part 200 may be displayed.
In addition, in the case where the order of step S20 and step S30 is reversed, the photographing control unit 320 may set the display control unit 310 to cause the display section 100 to display the display image 500 immediately before photographing, and not to display the display image 500 immediately after photographing. That is, the display unit 100 may display the display image 500 only for a short period (moment) of time at the shooting timing, like a flash of a camera.
The display image extraction unit 330 of the information processing terminal 1 performs color correction image extraction processing for extracting an image (display image) displayed by the display section 100, which is reflected in the eyes of the user, from the image (captured image) captured by the capturing section 200 (step S50). The image (display image) displayed on the display unit 100 reflected on the eyes of the user is an image of the reflected light reflected on the eyes of the user, which is captured in the display image. Regarding the color correction image extraction processing, details will be described in the following paragraphs.
The correction coefficient calculation unit 340 of the information processing terminal 1 obtains the chromaticity value of the image extracted by the display image extraction unit 330 (step S60). Based on the obtained chromaticity value and the chromaticity value of the image (display image) displayed by the display unit 100, a correction coefficient calculation process for calculating a correction coefficient of the color is performed. The correction coefficient calculation process will be described in detail in the following paragraphs.
The color correction unit 350 of the information processing terminal 1 (step S70) performs color correction processing for correcting the color of the image captured by the capturing section 200 using the correction coefficient calculated by the correction coefficient calculation unit 340. In the present embodiment, color correction is performed on a captured image. Regarding the color correction processing, details will be described in the following paragraphs.
The control unit 400 of the information processing terminal 1 (step S80) saves the image subjected to the color correction processing.
< S50: color correction image extraction processing >
The color correction image extraction process of step S50 will be described. Fig. 5 is a flowchart of one example of the color correction image extraction process. In the color correction image extraction process, a portion of the display image reflected by the eyes is extracted from the captured image. The process of extracting a portion of the display image reflected by the eyes in the color correction image extraction process is one example of a process of extracting reflected light of the display image.
The display image extraction unit 330 of the information processing terminal 1 (step S110) performs face recognition processing on the image (captured image) captured by the capturing section 200, and recognizes the face contained in the captured image. As for the face recognition processing, a known technique may be applied.
(step S120) the display image extraction unit 330 of the information processing terminal 1 extracts a part of the face from the result obtained by performing face recognition in step S110. Further, feature points of the face are extracted. In the display image extraction unit 330 of the present embodiment, a so-called black eye portion (portion of the iris and pupil) of the eye is extracted as a feature point.
It is considered that, when the display image extraction unit 330 of the information processing terminal 1 extracts feature points of a face, a so-called black eye portion (iris and pupil portion) of an eye may not be extracted. For example, a case where eyes are closed at the time of photographing due to blinking or the like, or a case where eyes or contact lenses are worn is considered. In the case where the so-called black eye portion (the portion of the iris and pupil) of the eye cannot be extracted as such, it is possible to return to step S20 again to perform shooting again.
(step S130) the display image extraction unit 330 of the information processing terminal 1 extracts the display image 505 reflected by the eye from the so-called black eye bead portion (iris and pupil portion) of the eye extracted in step S120, the display image 500 displayed by the display unit 100. Fig. 9 is a display image 505 showing reflection at a so-called black eye bead portion (portion of iris and pupil) of the right eye. In step S130, a display image 505 reflected on the so-called black eye portion (iris and pupil portion) of the right eye is extracted.
In the information processing terminal 1 of the present embodiment, the display image 505 is extracted from the so-called black eye portion (iris and pupil portion) of the eye, but the portion for extracting the display image 505 is not limited to the black eye portion (iris and pupil portion). As long as the color can be extracted, a display image 505 reflected at a so-called white eye portion of the eye may be extracted.
< S60: correction coefficient calculation processing >
The correction coefficient calculation process of step S60 will be described. Fig. 6 is a flowchart of one example of correction coefficient calculation processing. In the correction coefficient calculation process, correction coefficients necessary for performing color correction are calculated.
The correction coefficient calculation unit 340 of the information processing terminal 1 (step S210) acquires and displays each of the images 500 for the display image 505 reflected at the eyes extracted by the display image extraction unit 330RGB values of the color-corresponding portion. The display image 500 is composed of 5 colors of white, red, green, blue, and gray. Here, white, red, green, blue, and gray are respectively color 1, color 2, color 3, color 4, and color 5. Obtaining RGB values (R) of a portion of the display image 505 corresponding to a portion of the color i (i=1, …, 5) of the display image 500 i ，G i ，B i )。
The RGB values of the portion of the display image 505 corresponding to the portion of the color i of the display image 500 may be, for example, an average value of the RGB values of the portion of the display image 505 corresponding to the portion of the color i of the display image 500. Further, for example, the RGB values of the pixels at the center or the average value of the RGB values of the pixels in the vicinity of the center of the portion of the display image 505 corresponding to the portion of the color i of the display image 500 may be used.
The correction coefficient calculation unit 340 of the information processing terminal 1 (step S220) calculates the correction coefficient based on the RGB values (R) of the portion corresponding to the color i (i=1, …, 5) of the display image 500 acquired in step S210 i ，G i ，B i ) The XYZ value of the color i is calculated based on the mathematical formula 1. Will be based on RGB values (R i ，G i ，B i ) The calculated XYZ values are set as (X i ，Y i ，Z i )。
[ mathematics 1]
(step S230) the correction coefficient calculation unit 340 of the information processing terminal 1 uses the XYZ values (X) of the respective colors calculated in step S220 i ，Y i ，Z i ) The xy value (chromaticity value) is obtained based on the equation 2. X to be in XYZ value i 、Y i 、Z i The corresponding xy value is set as x i 、y i 。
[ math figure 2]
(step S240) the correction coefficient calculation unit 340 of the information processing terminal 1 obtains the correction coefficient Δx based on the equation 3 using the xy value of each color calculated in step S230 i 、Δy i . X0 is as follows i 、y0 i Is the xy value of color i of display image 500.
[ math 3]
Δx i ＝x i -x0 i
Δy i ＝y i -y0 i
In the information processing terminal 1 of the present embodiment, red, green, and blue are displayed in solid colors as the display image 500. That is, the red color of the display image 500 is only a red color component, i.e., (R, G, B) = (1, 0). Likewise, the green color of the display image 500 is only the green color component, i.e., (R, G, B) = (0, 1, 0), and the blue color of the display image 500 is only the blue color component, i.e., (R, G, B) = (0, 1). Therefore, xy values (x 0) of red, green, and blue of the display image 500 i ，y0 i ) Is a chromaticity value of a solid color.
< S70: color correction processing >
The color correction process of step S70 will be described. Fig. 7 is a flowchart of one example of the color correction process. The color correction process corrects the color of the image using the correction coefficient calculated in the correction coefficient calculation process.
The color correction unit 350 of the information processing terminal 1 acquires an image for correction (step S310). In the information processing terminal 1 of the present embodiment, it is the image captured in step S30. That is, it is an image used in obtaining a color correction coefficient.
The color correction unit 350 of the information processing terminal 1 divides the image for correction into a red image, a green image, and a blue image divided in RGB components, respectively (step S320). Here, the red image is described in detailDetails are described. For each pixel of the red image, that is, for each pixel of the image for correction, the R value is obtained based on the equation 1, and the Y value (Y r ). Let y value of pure red be y r0 . Let Δx be the correction value obtained for red of the display image 500 r 、Δy r . Based on equation 4, a correction value Δx in XYZ value space is obtained r 、ΔZ r . Next, based on equation 5, a correction value Δr for each pixel in the RGB space is obtained r 、ΔG r 、ΔB r . Subtracting the calculated correction value DeltaR for each pixel in RGB space from each pixel value of the red image r 、ΔG r 、ΔB r . Through the above processing, each pixel of the red image is corrected. The same process is also performed for the green image and the blue image. Then, the corrected red, blue, and green images are combined by, for example, addition or the like, to generate a final corrected image.
[ mathematics 4]
[ math 5]
In the present embodiment, the correction is performed using the correction coefficient obtained for each color in the correction of the color, but for example, the correction may be performed using the same correction coefficient for all the colors by averaging the correction coefficients obtained. In addition, correction may be performed on only a part of the colors. In the present embodiment, the correction coefficients Δx, Δy are converted into correction values in RGB space to perform correction, but it is also possible to convert pixel values into xy space for an image to be corrected and perform correction using the correction coefficients Δx, Δy in the xy space.
The display image reflected by the eye is an image in which light from the display portion is reflected by the cornea or lens of the eye. Thus, the intensity may fluctuate widely depending on the reflectivity of the cornea or lens of the eye. In the present embodiment, by using the chromaticity value, the influence on the fluctuation of the intensity can be reduced to perform the correction of the color.
With the information processing terminal 1 of the present embodiment, color correction can be performed using the information processing terminal 1 without preparing a color chart. In addition, since the color chart is not used, there is no need to secure a space for photographing the color chart, and the color chart itself is not mapped into the photographed image.
Since the display image 505 that is displayed on the eyes is small in the image after the color correction, there is generally no problem even if the display image 505 is displayed as it is. However, if the display image 505 is intended, a modification may be made to set the display image 505 to a color that is the same as the surrounding color.
< processing example >
The results after the actual processing will be described. Fig. 10 is a diagram for explaining a processing example of color correction processing performed by the information processing terminal 1 according to the present embodiment.
Fig. 10 is a diagram of pure colors (red (R), green (G), blue (B)), as well as colors before correction (untreated) and colors after correction, plotted on CIE1976a×b. For 3 colors (solid colors) of red (R), green (G), and blue (B), red is drawn with a circular mark R0, green is drawn with a quadrangular mark G0, and blue is drawn with an inverted triangle mark B0. The intensity was changed for each solid color and the drawing was performed a plurality of times. The farther from the origin, the stronger the intensity of each solid color.
The values calculated from the RGB values before correction are plotted with the diamond-shaped marks MR, MG, MR, and the values calculated from the RGB values after correction are plotted with the triangle-shaped marks CR, CG, CB.
In fig. 10, when the RGB values of the captured image are correct, for example, for red, the mark MR of the RGB value before correction is drawn on the mark R0 of the pure color of red or between the marks R0. In the case of red, the coordinates of a and b indicate that the red-colored component is included in the case where the red-colored component is located on the upper side of a line connecting the red-colored solid marks R0 and the blue-colored component is included in the case where the red-colored component is located on the lower side of the line.
Similarly, for example, for green, when the RGB values of the captured image are correct, the mark MG of the RGB value before correction is drawn on the mark G0 of the pure color of green or between the marks G0. In the case of green, on the a x b x coordinates, a component including red is indicated when it is located on the upper side of a line connecting green solid marks G0, and a component including blue is indicated when it is located on the lower side of the line.
Similarly, for example, in the case where the RGB values of the photographed image are correct for blue, the mark MB of the RGB value before correction is drawn on the mark B0 of the solid color of blue or between the marks B0. In the case of blue, on the a x B x coordinates, a component including red is indicated when it is located on the upper side of a line connecting blue solid marks B0, and a component including green is indicated when it is located on the lower side of the line.
The pre-correction diamond shaped markers MR, MG, MB of fig. 10 were analyzed. For blue, the mark MB is located almost on a line connecting the blue solid marks B0. Thus, it can be considered that: with respect to blue, the color is reproduced. For red, the marker MR is drawn to the right compared to the red, solid-colored marker R0. Therefore, it can be judged that: it is slightly blue in color. For green, it is drawn on the lower side compared to the pure color mark G0. Therefore, it can be judged that: it is slightly blue in color.
The labels CR, CG, CB of the corrected triangles of fig. 10 are analyzed. It can be seen that: the corrected results are drawn almost on the marks R0, G0, B0 of the solid colors or between the marks R0, G0, B0, respectively. It can be seen from this: each color is corrected to the correct color.
< modification >
Although the processing is performed by the information processing terminal 1 alone in the present embodiment, as shown in fig. 11, the processing may be performed by an information processing system in which the information processing terminal 1 and a server apparatus 5 (information processing apparatus) connected to the information processing terminal 1 via a network 4 so as to be capable of data communication are combined. For example, the constitution thereof may be as follows. First, the information processing terminal 1 performs the processing of steps S10 to S40 in fig. 4. Next, the image captured by the capturing unit 200 of the information processing terminal 1 is transmitted to the server device 5. The server device 5 performs the processing of steps S50 to S60. The correction coefficient calculated by the server device 5 is transmitted to the information processing terminal 1. Finally, the information processing terminal 1 performs the processing of steps S70 to S80.
In addition, the server device 5 may perform the processing of steps S50 to S70 and may transmit the color-corrected image to the information processing terminal 1. In this case, the image data may be transmitted not only to the information processing terminal 1 that performed the image capturing but also to another information processing terminal or a server device.
The display image displayed on the display unit 100 is not limited to the display image 500. For example, the arrangement of colors of the display image 500 may be replaced. For example. Green and blue may be arranged in row 1, white and red in row 2, and gray in row 3 of the display image 500. As described above, in the display image 500 of the present embodiment, since the 3 rd row is higher than the 1 st row and the 2 nd row, gray is desirably arranged in the 3 rd row, but other colors may be arranged.
The number of colors of the display image is not limited to 5 colors of the display image 500 as shown in fig. 8. The display image may contain 1 or more colors.
As shown in fig. 12A, 12B, 12C, and 12D, the arrangement of colors may be changed. For example, as shown in the display image 510 of fig. 12A, only 1 color may be displayed 1 time, and for example, the display images 510 of white (W), gray (GY), … may be sequentially displayed and photographed and repeated. In this case, the correction coefficient may be obtained for each display image 510, and color correction may be performed for any or all of the captured images.
In addition, processing of sequentially displaying and photographing the display images 510 of the respective colors can be performed by using the continuous shooting function of the camera. The order of the colors displayed may be changed as appropriate.
As shown in the display image 520 of fig. 12B, 1 color may be displayed in 1 line. Note that the arrangement of colors is not limited to the example shown in fig. 12B, but may be changed.
As shown in the display image 530 of fig. 12C, 1 color may be displayed in a plurality of gradation levels. For example, the gradation of the achromatic color may be changed so that the 1 st row is light gray (GY 1), the 2 nd row is gray (GY 2) darker than the 1 st row, and the 3 rd row is gray (GY 3) darker. The color is not limited to an achromatic color such as Gray (GY) as shown in the example of fig. 12C, and may be changed to another color.
As shown in the display image 540 of fig. 12D, the color region may not be displayed in all portions in the lateral direction of the display section 100, but may be represented in the middle as an elliptical shape. The shape is not limited to an ellipse, and may be a polygon. The background may be Gray (GY) or white (W). In addition, the color may not be displayed, i.e., it may be black.
The colors are not limited to white, red, green, blue, and gray, and cyan, magenta, yellow, and the like may be used. In addition, cyan, magenta, yellow, and the like may be included in the combination of colors.
By using an achromatic image as a display image, the white balance can be adjusted. By using an image of a solid color such as red, green, or blue as a display image, each color can be corrected. Therefore, it is desirable to combine achromatic colors and respective colors. The combination may be provided with all colors in the same display image, or may be displayed sequentially. For example, white (achromatic color) may be displayed first to adjust the white balance, and then pure colors such as red may be displayed to calculate correction coefficients for the respective colors.
The information processing terminal 1 may be applied as long as it has a camera on the display screen side, that is, a so-called built-in camera. For example, a mobile phone with a built-in camera, a portable personal computer (so-called notebook personal computer or desktop computer) with a built-in camera, or a tablet computer may be applied. The camera may be an external camera as long as it can shoot the display screen side.
In a commercially available smart phone, the color of the display portion may be different from the solid color. In this case, when the correction coefficient is calculated in the present embodiment, the influence of the color development of the display unit can be corrected in the post-processing based on the model information of the smartphone or the like.
Although the information processing terminal 1 of the present embodiment uses a still image for correction, the correction may be performed using a video. For example, when a user operates a button such as a shutter button, a video may be shot at a time before and after it.
In the information processing terminal 1 of the present embodiment, the correction coefficient is calculated using the captured image and the process for performing the color correction of the captured image is performed, but the calculated correction coefficient may be used to perform the color correction of an image other than the captured image.
Although the information processing terminal 1 of the present embodiment uses the display image reflected by the eyes, the place where the display image is reflected is not limited to the eyes as long as the reflected display image can be captured. For example, an image reflected by a mirror or glasses or the like may be used.
Although in the information processing terminal 1 of the present embodiment, a display image reflected by the eyes of the user (human) is used, a display image reflected at the eyes of an animal (for example, dog or cat) may be used.
The embodiments are described in detail above, but the above embodiments are exemplary in all respects, and are not limited to the above embodiments. Various modifications and changes may be made to the above-described embodiments within the scope of the gist of the embodiments of the present invention described in the claims.
The present application is based on claim priority from japanese patent application No. 2019-024573, filed to the japanese patent office as a basic application on day 2 and 14 of 2019, the entire contents of which are incorporated herein by reference.
Symbol description
1. Information processing terminal
100. Display unit
200. Image pickup unit
310. Display control unit
320. Shooting control unit
330. Display image extraction unit
340. Correction coefficient calculation unit
350. Color correction unit
Claims (7)
1. An information processing terminal having a display unit that displays a display image, and an imaging unit that can image the face of a user who is looking at the display unit, the information processing terminal comprising:
a display control unit that causes the display unit to display a display image including 1 or more colors;
a shooting control unit that causes the shooting unit to shoot in a state where the display image is displayed;
a display image extraction unit that extracts reflected light of the display image from a captured image; and
a color correction unit that performs color correction on the photographed image based on the extracted reflected light of the display image,
wherein the display image extraction unit extracts the display image reflected in the eyes of the user from the captured image as reflected light of the display image.
2. The information processing terminal according to claim 1, wherein,
the display control unit displays the display image including red, green and blue,
the color correction unit performs color correction on the display image displayed by the display unit based on the extracted chromaticity values of red, green, and blue and the chromaticity values of red, green, and blue of the solid color included in the reflected light of the display image.
3. The information processing terminal according to claim 1 or 2, wherein,
the display control unit displays the display image including achromatic colors,
the color correction unit performs color correction on the display image displayed by the display section based on chromaticity values of display image colors contained in the extracted reflected light of the display image.
4. The information processing terminal according to claim 1 or 2, wherein,
the display control unit causes the display section to display the display image including patterns of white, red, green, blue, and gray.
5. A recording medium having recorded thereon a program for causing an information processing terminal having a display section for displaying a display image and an imaging section capable of imaging a face of a user who is looking at the display section to function as:
a display control unit that causes the display unit to display a display image including 1 or more colors;
a shooting control unit that causes the shooting unit to shoot in a state where the display image is displayed;
a display image extraction unit that extracts reflected light of the display image from a captured image; and
a color correction unit that performs color correction on the photographed image based on the extracted reflected light of the display image,
wherein the display image extraction unit extracts the display image reflected in the eyes of the user from the captured image as reflected light of the display image.
6. An information processing system including an information processing terminal having a display unit that displays a display image and an imaging unit that can image a face of a user who looks at the display unit, and an information processing device connected to the information processing terminal via a network so as to be capable of data communication, the information processing system including:
a display control unit that causes the display unit to display a display image including 1 or more colors;
a shooting control unit that causes the shooting unit to shoot in a state where the display image is displayed;
a display image extraction unit that extracts reflected light of the display image from a captured image; and
a color correction unit that performs color correction on the photographed image based on the extracted reflected light of the display image,
wherein the display image extraction unit extracts the display image reflected in the eyes of the user from the captured image as reflected light of the display image.
7. A color correction method performed by an information processing terminal having a display section that displays a display image and an imaging section that can image a face of a user who looks at the display section, the color correction method comprising:
a display control step of causing the display unit to display a display image including 1 or more colors;
a shooting control step of causing the shooting unit to shoot in a state where the display image is displayed;
a display image extraction step of extracting reflected light of the display image from a captured image; and
a color correction step of performing color correction on the photographed image based on the extracted reflected light of the display image,
wherein, in the display image extraction step, the display image reflected in the eyes of the user is extracted from the captured image as reflected light of the display image.
Applications Claiming Priority (3)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| JP2019024573 | 2019-02-14 | ||
| JP2019-024573 | 2019-02-14 | ||
| PCT/JP2020/000967 WO2020166256A1 (en) | 2019-02-14 | 2020-01-15 | Information processing terminal, program, information processing system, and color correction method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113412615A CN113412615A (en) | 2021-09-17 |
| CN113412615B true CN113412615B (en) | 2024-04-05 |
Family
ID=72045293
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202080012931.7A Active CN113412615B (en) | 2019-02-14 | 2020-01-15 | Information processing terminal, recording medium, information processing system, and color correction method |
Country Status (3)
| Country | Link |
|---|---|
| JP (1) | JP7335283B2 (en) |
| CN (1) | CN113412615B (en) |
| WO (1) | WO2020166256A1 (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112911148B (en) * | 2021-01-28 | 2022-10-14 | 维沃移动通信有限公司 | Image processing method and device and electronic equipment |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2005124105A (en) * | 2003-10-20 | 2005-05-12 | Seiko Precision Inc | Projector and color correcting method thereof |
| JP2009200782A (en) * | 2008-02-21 | 2009-09-03 | Nikon Corp | Image output characteristic correction system |
| JP2010157988A (en) * | 2008-12-05 | 2010-07-15 | Astro Design Inc | Color evaluation method and color evaluation system |
| JP2017026943A (en) * | 2015-07-27 | 2017-02-02 | 株式会社東芝 | Image display device and image processing device |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20160370591A1 (en) * | 2014-12-27 | 2016-12-22 | Fove, Inc. | Head mounted display |
-
2020
- 2020-01-15 JP JP2020572129A patent/JP7335283B2/en active Active
- 2020-01-15 WO PCT/JP2020/000967 patent/WO2020166256A1/en active Application Filing
- 2020-01-15 CN CN202080012931.7A patent/CN113412615B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2005124105A (en) * | 2003-10-20 | 2005-05-12 | Seiko Precision Inc | Projector and color correcting method thereof |
| JP2009200782A (en) * | 2008-02-21 | 2009-09-03 | Nikon Corp | Image output characteristic correction system |
| JP2010157988A (en) * | 2008-12-05 | 2010-07-15 | Astro Design Inc | Color evaluation method and color evaluation system |
| JP2017026943A (en) * | 2015-07-27 | 2017-02-02 | 株式会社東芝 | Image display device and image processing device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113412615A (en) | 2021-09-17 |
| JP7335283B2 (en) | 2023-08-29 |
| WO2020166256A1 (en) | 2020-08-20 |
| JPWO2020166256A1 (en) | 2021-12-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20170318233A1 (en) | Noise Models for Image Processing | |
| US8106958B2 (en) | Image processing apparatus and image processing method and image capturing apparatus | |
| US9117410B2 (en) | Image display device and method | |
| US10200663B2 (en) | Image processing device, imaging device, image processing method, and program | |
| US11477383B2 (en) | Method for providing preview and electronic device for displaying preview | |
| KR102318013B1 (en) | Electronic device composing a plurality of images and method | |
| US20150063694A1 (en) | Techniques for combining images with varying brightness degrees | |
| US10348958B2 (en) | Image processing apparatus for performing predetermined processing on a captured image | |
| CN106416218B (en) | Image processing apparatus, photographic device, image processing method and storage medium | |
| CN110648296B (en) | Pupil color correction method, correction device, terminal device and storage medium | |
| US20020080251A1 (en) | Digital imaging device, image processing device, digital imaging method, and image processing program | |
| US10957092B2 (en) | Method and apparatus for distinguishing between objects | |
| JP2009033361A (en) | Color adjustment unit, image-forming device, and program | |
| KR20190096748A (en) | electronic device and method for correcting image using external electronic device | |
| JPWO2020162262A1 (en) | Drug identification system, drug identification device, drug identification method and program | |
| CN113412615B (en) | Information processing terminal, recording medium, information processing system, and color correction method | |
| WO2014098143A1 (en) | Image processing device, imaging device, image processing method, and image processing program | |
| CN107105167B (en) | Method and device for shooting picture during scanning question and terminal equipment | |
| CN105976343B (en) | Picture exposure correction method and device and intelligent equipment | |
| US11509797B2 (en) | Image processing apparatus, image processing method, and storage medium | |
| US9531919B2 (en) | Image processing apparatus, image processing method, and recording medium that color-convert an input image into an output image suitable for a color gamut of a printing apparatus | |
| KR20200114275A (en) | Electronic device and method for secure of personal information included in image | |
| JP2006148326A (en) | Imaging apparatus and method of controlling the same | |
| JP6260094B2 (en) | Image processing apparatus, image processing method, and program | |
| KR102627785B1 (en) | Photo Correction Device and Driving Method Thereof |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |