WO2023011348A1 - 检测方法及电子设备 - Google Patents
检测方法及电子设备 Download PDFInfo
- Publication number
- WO2023011348A1 WO2023011348A1 PCT/CN2022/108950 CN2022108950W WO2023011348A1 WO 2023011348 A1 WO2023011348 A1 WO 2023011348A1 CN 2022108950 W CN2022108950 W CN 2022108950W WO 2023011348 A1 WO2023011348 A1 WO 2023011348A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- image
- area
- sunscreen
- camera
- ultraviolet
- Prior art date
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 110
- 230000000475 sunscreen effect Effects 0.000 claims abstract description 412
- 239000000516 sunscreening agent Substances 0.000 claims abstract description 412
- 239000006071 cream Substances 0.000 claims abstract description 4
- 238000000034 method Methods 0.000 claims description 79
- 210000004709 eyebrow Anatomy 0.000 claims description 10
- 238000004590 computer program Methods 0.000 claims description 8
- 238000004458 analytical method Methods 0.000 abstract description 3
- 230000036561 sun exposure Effects 0.000 abstract 1
- 230000006870 function Effects 0.000 description 43
- 238000003384 imaging method Methods 0.000 description 40
- 230000037072 sun protection Effects 0.000 description 39
- 238000013461 design Methods 0.000 description 29
- 238000012545 processing Methods 0.000 description 29
- 238000010586 diagram Methods 0.000 description 27
- 230000008569 process Effects 0.000 description 27
- 238000004891 communication Methods 0.000 description 21
- 230000001815 facial effect Effects 0.000 description 18
- 230000000694 effects Effects 0.000 description 16
- 239000013598 vector Substances 0.000 description 15
- 238000004422 calculation algorithm Methods 0.000 description 13
- 238000007726 management method Methods 0.000 description 12
- 210000003128 head Anatomy 0.000 description 11
- 238000012549 training Methods 0.000 description 11
- 239000000284 extract Substances 0.000 description 7
- 238000004364 calculation method Methods 0.000 description 6
- 239000000463 material Substances 0.000 description 6
- 230000005236 sound signal Effects 0.000 description 6
- 239000003086 colorant Substances 0.000 description 5
- 238000001914 filtration Methods 0.000 description 5
- 238000010295 mobile communication Methods 0.000 description 5
- 238000013528 artificial neural network Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 4
- 210000000887 face Anatomy 0.000 description 4
- 238000002372 labelling Methods 0.000 description 4
- 230000005808 skin problem Effects 0.000 description 4
- 238000012360 testing method Methods 0.000 description 4
- NOQGZXFMHARMLW-UHFFFAOYSA-N Daminozide Chemical compound CN(C)NC(=O)CCC(O)=O NOQGZXFMHARMLW-UHFFFAOYSA-N 0.000 description 3
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 238000007781 pre-processing Methods 0.000 description 3
- VYPSYNLAJGMNEJ-UHFFFAOYSA-N Silicium dioxide Chemical compound O=[Si]=O VYPSYNLAJGMNEJ-UHFFFAOYSA-N 0.000 description 2
- 238000010521 absorption reaction Methods 0.000 description 2
- 230000009471 action Effects 0.000 description 2
- 238000009499 grossing Methods 0.000 description 2
- 230000033001 locomotion Effects 0.000 description 2
- 230000007774 longterm Effects 0.000 description 2
- 239000011159 matrix material Substances 0.000 description 2
- 238000010606 normalization Methods 0.000 description 2
- 238000013102 re-test Methods 0.000 description 2
- 230000009467 reduction Effects 0.000 description 2
- 230000036555 skin type Effects 0.000 description 2
- 239000000126 substance Substances 0.000 description 2
- 230000001960 triggered effect Effects 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 206010039792 Seborrhoea Diseases 0.000 description 1
- 230000001133 acceleration Effects 0.000 description 1
- 206010000496 acne Diseases 0.000 description 1
- 239000008186 active pharmaceutical agent Substances 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000003796 beauty Effects 0.000 description 1
- 238000013529 biological neural network Methods 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 210000000988 bone and bone Anatomy 0.000 description 1
- 210000004556 brain Anatomy 0.000 description 1
- 230000010267 cellular communication Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000019771 cognition Effects 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 239000012141 concentrate Substances 0.000 description 1
- 239000000470 constituent Substances 0.000 description 1
- 239000013256 coordination polymer Substances 0.000 description 1
- 239000002537 cosmetic Substances 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 238000010348 incorporation Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 210000002569 neuron Anatomy 0.000 description 1
- 230000037312 oily skin Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000003825 pressing Methods 0.000 description 1
- 230000002265 prevention Effects 0.000 description 1
- 239000000047 product Substances 0.000 description 1
- 230000005855 radiation Effects 0.000 description 1
- 238000009877 rendering Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000037307 sensitive skin Effects 0.000 description 1
- 230000037384 skin absorption Effects 0.000 description 1
- 231100000274 skin absorption Toxicity 0.000 description 1
- 230000003238 somatosensory effect Effects 0.000 description 1
- 230000006641 stabilisation Effects 0.000 description 1
- 238000011105 stabilization Methods 0.000 description 1
- 239000013589 supplement Substances 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 230000037303 wrinkles Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/136—Segmentation; Edge detection involving thresholding
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/56—Extraction of image or video features relating to colour
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
Definitions
- the present application relates to the field of electronic technology, in particular to a detection method and electronic equipment.
- Skin problems can include blackheads, spots, wrinkles, etc.
- skin problems that can be caused by ultraviolet radiation.
- chemical sun protection measures users can apply sunscreen on the parts that need sun protection, so that part of the ultraviolet rays can be absorbed and the damage of ultraviolet rays to the parts where the sunscreen is applied can be reduced.
- an embodiment of the present application provides a detection method.
- the detection method can, but is not limited to, detect sunscreen application.
- the electronic equipment has an ultraviolet camera, respectively collects ultraviolet images and color images through the ultraviolet camera and the RGB camera, and matches the first region in the identified ultraviolet image to the corresponding color image, improving User experience.
- the present application provides a detection method, which is applied to an electronic device or a device (such as a chip) capable of realizing or matching the functions of an electronic device.
- the electronic device includes a first camera and a second camera ;
- the first camera is a color camera
- the second camera is an ultraviolet camera
- the method includes:
- the first image is collected by the first camera
- the second image is collected by the second camera, wherein the color image includes the first content.
- the first content may be the object to be detected collected by the color image.
- the object to be detected is an object to be detected for application of the sunscreen.
- the objects to be detected include parts of the human body, such as the face, hands, arms and other parts that need sun protection.
- Objects to be detected may also include other objects, for example, objects such as vehicles that need to be isolated from ultraviolet rays for sun protection.
- the electronic device can determine the application of sunscreen based on the ultraviolet information recorded in the ultraviolet image, so as to determine the area related to the application of sunscreen on the ultraviolet image, such as first area. Afterwards, the electronic device determines a second area corresponding to the first area on the color image, and displays a user interface, the user interface includes first content and first information for indicating the second area, that is, marking sunscreen in the color image Apply to the second area where the condition is relevant. In this way, the user can observe the second area related to the application of sunscreen through the color image.
- the first content includes a first face;
- the second image includes a second face, and the first face and the second face belong to the same user;
- Determining the first region on the second image specifically includes:
- Determining a second area corresponding to the first area on the first image specifically includes:
- a second area corresponding to the first area is determined.
- the first area is any one or more of the following areas: an area where sunscreen is applied, an area where sunscreen is not applied, an area where sunscreen is applied unevenly, and the thickness of sunscreen application For areas greater than the first threshold, apply sunscreen to areas less than the second threshold.
- the electronic device can accurately detect one or more of the above-mentioned areas of the ultraviolet image, so that it can provide users with rich sun protection reminders and provide refined guidance for users on sun protection.
- the method further includes:
- determining a second area corresponding to the first area on the first image includes:
- the second condition includes: the third image includes a human face, and the angle of the human face included in the third image relative to the electronic device is within a preset angle range.
- the joint analysis of sunscreen application information by collecting ultraviolet images from different angles can improve the accuracy of detection and provide refined guidance to users on sunscreen.
- collecting the first image through the first camera and collecting the second image through the second camera includes: at the same time, collecting the first image through the first camera and collecting the second image through the second camera second image.
- UV images can be used to detect sunscreen application.
- Color images can be used to detect faces, and can also be used to identify the angle of a face relative to an electronic device.
- the first camera and the second camera capture images synchronously, which can make the angle of the face relative to the electronic device in the images captured by the first camera and the second camera same.
- the electronic device can deduce the angle of the human face in the ultraviolet image relative to the electronic device according to the angle of the human face in the color image relative to the electronic device, so as to determine the usability of the ultraviolet image.
- the first image is collected by the first camera
- the second image is collected by the second camera, including:
- the first condition includes: the fifth image includes a human face.
- the electronic device first collects a color image through a color camera, and when it is determined that the color image includes a human face, that is, there is an object to be detected for sunscreen application, the electronic device then collects an ultraviolet image through the ultraviolet camera (for detecting sunscreen application), and collect color images through a color camera to prevent power consumption problems caused by turning on the UV camera earlier.
- the electronic device further includes an ultraviolet supplementary light
- collecting the second image through the second camera includes: turning on the ultraviolet supplementary light, and collecting the second image through the second camera.
- the method further includes:
- Capture a second image through a second camera including:
- the ultraviolet supplementary light is turned on, and the second image is collected by the second camera.
- the ultraviolet supplementary light is turned on only when the ambient ultraviolet intensity is low, so the damage of the ultraviolet supplementary light to the human body can be reduced.
- the method further includes: displaying the second information, and the second information is used to prompt the user to adjust the angle of the human face relative to the electronic device.
- the electronic device can collect ultraviolet images from multiple angles, and then conduct joint analysis of sunscreen application information through ultraviolet images from different angles, improving the accuracy of detection.
- determining the first area on the second face of the second image includes:
- the first area is determined in the fourth area of the second image based on the pixel values of the pixels contained in the fourth area.
- determining the first area in the fourth area of the second image according to the pixel values of the pixels contained in the fourth area includes:
- the target sub-area whose area is greater than or equal to the fourth threshold value among the plurality of target sub-areas is determined as the first area, and the target sub-area
- the third condition is met, and the third condition includes: the pixel mean value of the target sub-region is less than the fifth threshold, and the pixel value of the pixels contained in the target sub-region is less than the sixth threshold.
- the mobile phone can exclude small areas such as moles from the candidate areas for sunscreen application, thereby reducing the probability of areas such as moles being misjudged as areas for applying sunscreen, and improving detection accuracy.
- the method before determining the first region on the second image, the method further includes:
- a second human face in the second image is determined.
- the mobile phone can accurately identify the face in the color image, and match the corresponding area of the recognized face to the ultraviolet image.
- the third area is any one or more of the following areas: an area where sunscreen is applied, an area where sunscreen is not applied, an area where sunscreen is applied unevenly, and the thickness of sunscreen application For areas greater than the first threshold, apply sunscreen to areas less than the second threshold.
- the fourth area does not include areas corresponding to eyes, eyebrows, and mouth in the human face.
- the fourth zone is usually the one that doesn't need sunscreen.
- the fourth area is determined from the face of the ultraviolet image, and the first area is determined in the fourth area. Since the pixel values of areas such as eyes, eyebrows, and mouth can not be calculated, the amount of calculation can be reduced, and the detection efficiency of the electronic device can be improved.
- the first information is also used to indicate the uniformity of sunscreen application and/or the thickness of sunscreen application and/or whether to apply sunscreen in the second area. In this way, the user can know the accurate sunscreen application situation through the first information displayed on the electronic device, which is convenient for sunscreen guidance.
- an electronic device in a second aspect, includes a processor, a display screen, a first camera, and a second camera; wherein, the first camera is a color camera, and the second camera is an ultraviolet camera;
- the first camera is used to collect the first image
- the second camera is used to collect a second image, wherein the first image contains the first content
- a processor configured to determine a first area on the second image; determine a second area corresponding to the first area on the first image according to the first area;
- the display screen is used to display a user interface, the user interface includes first content and first information, and the first information is used to indicate the second area.
- the first content includes a first face
- the second image includes a second face, and the first face and the second face belong to the same user
- Determining the first region on the second image specifically includes:
- Determining a second area corresponding to the first area on the first image specifically includes:
- a second area corresponding to the first area is determined.
- the first area is any one or more of the following areas: the area where sunscreen is applied, the area where sunscreen is not applied, the area where sunscreen is applied unevenly, and the thickness of sunscreen application For areas greater than the first threshold, apply sunscreen to areas less than the second threshold.
- the first camera is also used to collect a third image
- the second camera is used to collect a fourth image when the third image satisfies the second condition
- a processor configured to determine a third region on the fourth image
- determining a second area corresponding to the first area on the first image includes:
- the second condition includes: the third image includes a human face, and the angle of the human face included in the third image relative to the electronic device is within a preset angle range.
- the first camera is used to collect the first image
- the second camera is used to collect the second image, including: at the same moment, the first camera collects the first image and the second The camera collects the second image.
- the first camera is used to collect the first image
- the second camera is used to collect the second image, including:
- the first camera is used to collect the fifth image; when the fifth image satisfies the first condition, the first image is collected, and the second camera is used to collect the second image;
- the first condition includes: the fifth image includes a human face.
- the electronic device further includes an ultraviolet fill light
- the second camera is used to capture the second image, including: turning on the ultraviolet fill light, and the second camera captures the second image.
- the processor is also used to determine the ambient ultraviolet intensity
- the second camera is used to collect the second image, including:
- the ultraviolet supplementary light is turned on, and the second camera collects the second image.
- the display screen is further used to display second information, and the second information is used to prompt the user to adjust the angle of the human face relative to the electronic device.
- determining the first area on the second face of the second image includes:
- the first area is determined in the fourth area of the second image based on the pixel values of the pixels contained in the fourth area.
- determining the first area in the fourth area of the second image according to the pixel values of the pixels included in the fourth area includes:
- the target sub-area whose area is greater than or equal to the fourth threshold value among the plurality of target sub-areas is determined as the first area, and the target sub-area
- the third condition is met, and the third condition includes: the pixel mean value of the target sub-region is less than the fifth threshold, and the pixel value of the pixels contained in the target sub-region is less than the sixth threshold.
- the processor is further configured to determine a second human face in the second image according to the first human face in the first image.
- the third area is any one or more of the following areas: an area where sunscreen is applied, an area where sunscreen is not applied, an area where sunscreen is applied unevenly, and the thickness of sunscreen application For areas greater than the first threshold, apply sunscreen to areas less than the second threshold.
- the fourth area does not include areas corresponding to eyes, eyebrows, and mouth in the human face.
- the fourth zone is usually the one that doesn't need sunscreen.
- the first information is also used to indicate the uniformity of sunscreen application and/or the thickness of sunscreen application and/or whether to apply sunscreen in the second area.
- the present application provides an electronic device, which has a function of implementing the detection method in any one of the foregoing aspects and any possible implementation manner.
- This function may be implemented by hardware, or may be implemented by executing corresponding software on the hardware.
- the hardware or software includes one or more modules corresponding to the above functions.
- the present application provides a computer-readable storage medium, including computer instructions.
- the computer instructions When the computer instructions are run on an electronic device, the electronic device executes any one of any aspect and any one of its possible implementations. detection method.
- the present application provides a computer program product, which, when the computer program product is run on an electronic device, causes the electronic device to execute any detection method according to any aspect and any possible implementation manner thereof.
- a circuit system in a sixth aspect, includes a processing circuit, and the processing circuit is configured to execute the detection method in any one of the above aspects and any possible implementation manners thereof.
- the embodiment of the present application provides a chip system, including at least one processor and at least one interface circuit.
- at least one processor executes the detection method in any one of the above aspects and any possible implementation manners.
- Fig. 1 is a schematic diagram of the principle of ultraviolet imaging provided by the embodiment of the present application.
- FIGS. 2A-2E are schematic diagrams of the electronic equipment provided by the embodiment of the present application.
- FIG. 2F is a schematic structural diagram of an electronic device provided in an embodiment of the present application.
- FIG. 3 is a schematic diagram of the software architecture of the electronic device provided by the embodiment of the present application.
- Fig. 4 is the interface diagram provided by the embodiment of the present application.
- FIG. 5 is a schematic diagram of a method for recognizing face angles provided by an embodiment of the present application.
- FIG. 6 is an interface diagram provided by the embodiment of the present application.
- Fig. 7 is a schematic diagram of the method flow provided by the embodiment of the present application.
- Figure 8 is a schematic diagram of the detection method flow and interface provided by the embodiment of the present application.
- Figure 9 is a schematic diagram of the detection method flow and interface provided by the embodiment of the present application.
- Figure 10 is an interface diagram provided by the embodiment of the present application.
- Figure 11 is a schematic diagram of the detection method flow and interface provided by the embodiment of the present application.
- Fig. 12A is a schematic diagram of the flow of the detection method provided by the embodiment of the present application.
- Fig. 12B is a schematic diagram of the detection method flow provided by the embodiment of the present application.
- Figure 13 is a schematic diagram of the detection method flow and interface provided by the embodiment of the present application.
- Figure 14 is a schematic diagram of the flow of the detection method provided by the embodiment of the present application.
- Fig. 15 is a schematic diagram of the training and use of the classifier provided by the embodiment of the present application.
- Fig. 16 is a schematic diagram of the flow of the detection method provided by the embodiment of the present application.
- Figure 17 is a schematic diagram of the detection method flow provided by the embodiment of the present application.
- Figure 18 is a schematic diagram of the detection method flow provided by the embodiment of the present application.
- FIG 19 and Figure 20 are schematic diagrams of interfaces provided by the embodiment of this application.
- Figure 21 is a schematic diagram of the device provided by the embodiment of the present application.
- FIG. 22 is a schematic diagram of a chip system provided by an embodiment of the present application.
- UV Imaging The wavelengths of ultraviolet light are not in the visible light range, therefore, to the human eye, ultraviolet light is invisible.
- ultraviolet light can be used as a light source, and an ultraviolet imaging system can be used for imaging. This technology of capturing ultraviolet light and imaging it with an ultraviolet imaging system can be called ultraviolet imaging technology. With ultraviolet imaging, users can observe the ultraviolet image formed by the object under ultraviolet irradiation.
- the ultraviolet imaging system includes an ultraviolet camera module.
- the ultraviolet camera module can be a camera capable of capturing ultraviolet rays, which is called an ultraviolet camera.
- the ultraviolet camera can include components such as a lens and an image sensor.
- the lens can be made of materials such as but not limited to quartz glass, which can transmit ultraviolet rays.
- Image sensors record information about the light that passes through the lens.
- the ultraviolet imaging system can be applied to skin detection (including but not limited to sunscreen application detection), so as to detect the skin's absorption or reflection of ultraviolet rays.
- ultraviolet rays when ultraviolet rays irradiate different parts of a character, they can be reflected or absorbed by human body parts.
- ultraviolet rays will be reflected on parts such as the neck where no sunscreen is applied, and the reflected ultraviolet rays can pass through the ultraviolet lens, and then record ultraviolet information on the image sensor.
- the pixel value of the imaging area of the neck and other parts is relatively high.
- the sunscreen will protect the skin and absorb part of the ultraviolet rays, so that no ultraviolet rays or less ultraviolet rays are reflected into the ultraviolet lens, and the image sensor will not record or record less. Apply sunscreen on the face The light information of the part.
- the pixel value of the imaging area of the sunscreen-applied part of the face is relatively low, for example, the imaging area 1 of the sunscreen-applied face shown in FIG. 1 is black.
- the embodiment of the present application provides a detection method, which can be applied in an electronic device, and the electronic device is equipped with an ultraviolet supplementary light and an ultraviolet camera.
- the electronic device in the embodiment of the present application can be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, a notebook computer, a smart cosmetic mirror (such as shown in Figure 2A), etc.
- the mobile phone in the embodiment of the present application can be a folding screen mobile phone or For mobile phones with non-folding screens, the embodiments of the present application do not impose any restrictions on the specific types of electronic devices.
- the layout of the cameras on the electronic device 100 can be referred to FIG. 2B , where the front of the electronic device 100 is the plane where the display screen 194 is located.
- the camera 1931 is located on the front of the electronic device 100 , so the camera is a front camera.
- the camera 1932 is located on the back of the electronic device 100 , and the camera is a rear camera.
- Cameras can include UV cameras and color cameras. Color cameras include but are not limited to red, green, blue (RGB) cameras.
- the solution of the embodiment of the present application may be applied to an electronic device 100 with a folding screen (that is, the display screen can be folded) having multiple display screens.
- the folding screen may be a flexible folding screen.
- the flexible folding screen includes a folding shaft made of flexible material. Part or all of the flexible folding screen is made of flexible materials. For example: only the foldable part (such as the folding shaft) of the flexible folding screen is made of flexible material, and the other parts are made of rigid material; or, all of the flexible folding screen is made of flexible material.
- the folding screen can be folded along the folding axis to form at least two sub-screens.
- FIG. 2B shows a folding screen electronic device 100 .
- the foldable screen is folded inwardly (or outwardly) along the folding edge, so that the foldable screen forms at least two sub-screens (for example, A sub-screen and B sub-screen).
- there is a display screen (such as a C screen) on the outer side of the fold. If the electronic device 100 is provided with a camera on the surface where the C-screen is located. Then, in the unfolded scene of the electronic device 100 as shown in (c) of FIG.
- the camera on the C-screen is on the back of the electronic device 100 and can be regarded as a rear camera.
- the camera on the C-screen becomes the front of the electronic device 100 and can be regarded as a front camera. That is to say, the front camera and the rear camera in this application do not limit the nature of the camera itself, but are only an illustration of a positional relationship.
- the ultraviolet camera can be set on the C screen (by digging a hole or setting up an under-screen camera, etc.). In this way, the user can collect the user's facial image through the camera, and check the application of sunscreen through the C screen.
- the ultraviolet camera can also be set on the A screen (or B screen), so that the user can collect facial images through the camera, and check the application of sunscreen through the A screen (or B screen).
- the folding screen of the folding screen electronic device may form multiple (such as 2, 3, etc.) sub-screens.
- the flexible folding screen shown in (1) in FIG. 2C may include a folding line 030 and a folding line 031 . After being folded longitudinally along the folding line 030, a sub-screen 032, a sub-screen 033, and a sub-screen 034 as shown in (2) of FIG. 2C can be formed.
- the screen arrangement of the folding screen electronic device may be a top-bottom screen arrangement such as that shown in (d) in FIG. 2B, or may also be a left-right screen arrangement such as that shown in (1) or (2) in FIG. 2D wait.
- the embodiment of the present application does not limit the screen arrangement of the folding screen electronic device.
- the flexible folding screen shown in (1) in FIG. 2D after being folded laterally along the folding line 040, can form a sub-screen 041 and a sub-screen 042 as shown in (2) in FIG. 2D.
- the folding screen device may be a mobile phone in a folded state (refer to the state shown in (1) of FIG. 2E ), and may be a tablet computer in an unfolded state.
- the folding screen may be a single-sided screen (that is, only one side can display a user interface) or a double-sided screen (that is, both opposite sides can display a user interface).
- a single-sided folding screen when the folding screen is turned toward the side that can display the user interface (that is, the front of the single-sided folding screen), it can be called forward folding; when the folding screen is turned toward the side that can display the user interface Folding is performed on the opposite side (ie, the back of the single-sided folding screen), which can be called reverse folding.
- FIG. 2D represents a schematic diagram of forward folding
- FIG. 2E represents a schematic diagram of reverse folding.
- a folding screen device can determine whether it is currently being folded forward or backward.
- FIG. 2F shows a schematic structural diagram of the electronic device 100 .
- the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (universal serial bus, USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, and an antenna 2 , mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, earphone jack 170D, sensor module 180, button 190, motor 191, indicator 192, camera 193, display screen 194, and A subscriber identification module (subscriber identification module, SIM) card interface 195 and the like.
- SIM subscriber identification module
- the processor 110 may include one or more processing units, for example: the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processing unit (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), controller, memory, video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural network processor (neural-network processing unit, NPU) wait. Wherein, different processing units may be independent devices, or may be integrated in one or more processors.
- application processor application processor, AP
- modem processor graphics processing unit
- GPU graphics processing unit
- image signal processor image signal processor
- ISP image signal processor
- controller memory
- video codec digital signal processor
- DSP digital signal processor
- baseband processor baseband processor
- neural network processor neural-network processing unit, NPU
- the controller may be the nerve center and command center of the electronic device 100 .
- the controller can generate an operation control signal according to the instruction opcode and timing signal, and complete the control of fetching and executing the instruction.
- a memory may also be provided in the processor 110 for storing instructions and data.
- the memory in processor 110 is a cache memory.
- the memory may hold instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instruction or data again, it can be directly called from the memory. Repeated access is avoided, and the waiting time of the processor 110 is reduced, thereby improving the efficiency of the system.
- the processor 110 is configured to call the ultraviolet camera to capture ultraviolet images when an operation triggering the detection is detected, and detect the application of sunscreen according to the ultraviolet images. Afterwards, the processor 110 may control to prompt the user about the application of sunscreen through the display screen.
- the application of sunscreen includes but is not limited to whether the user applies sunscreen, the location where the sunscreen is applied, the location where the sunscreen is not applied, whether the sunscreen is applied evenly at the location where the sunscreen is applied, the location where the sunscreen is not applied evenly, and the locations where the sunscreen is not applied evenly. Information on how thick to apply sunscreen and recommended information on sunscreen.
- the processor 110 controls the ultraviolet camera to capture ultraviolet images, which may be to control the ultraviolet camera to capture multiple ultraviolet images from different angles. Afterwards, the processor 110 can detect the application of sunscreen according to multiple ultraviolet images from different angles. In this way, the detection error caused by the reflective area in the part-angle ultraviolet image can be offset, and the accuracy of the detection result can be improved.
- the processor 110 controls to remind the user of the application of sunscreen through the display screen, which may be to mark different facial areas in the color image, such as marking the position where the sunscreen is not applied evenly in the color image, etc.
- the color image can record clearer skin details, help users clearly identify the application of sunscreen, and guide users to take more detailed sun protection measures. For example, users can finely reapply according to the application of sunscreen.
- the charging management module 140 is configured to receive a charging input from a charger.
- the power management module 141 is used for connecting the battery 142 , the charging management module 140 and the processor 110 .
- the power management module 141 receives the input of the battery 142 and/or the charging management module 140 to provide power for the processor 110 , the display screen 194 , the camera 193 and so on.
- the wireless communication function of the electronic device 100 can be realized by the antenna 1 , the antenna 2 , the mobile communication module 150 , the wireless communication module 160 , a modem processor, a baseband processor, and the like.
- the mobile communication module 150 can provide wireless communication solutions including 2G/3G/4G/5G applied on the electronic device 100 .
- the wireless communication module 160 can provide wireless local area networks (wireless local area networks, WLAN) (such as wireless fidelity (Wi-Fi) network), bluetooth (bluetooth, BT) and other wireless communication functions applied on the electronic device 100. solution.
- WLAN wireless local area networks
- Wi-Fi wireless fidelity
- BT bluetooth
- the electronic device 100 realizes the display function through the GPU, the display screen 194 , and the application processor.
- the GPU is a microprocessor for image processing, and is connected to the display screen 194 and the application processor. GPUs are used to perform mathematical and geometric calculations for graphics rendering.
- Processor 110 may include one or more GPUs that execute program instructions to generate or change display information.
- the display screen 194 is used to display images, videos and the like.
- the display screen 194 includes a display panel.
- the electronic device 100 may include 1 or N display screens 194 , where N is a positive integer greater than 1.
- the electronic device 100 can realize the shooting function through the ISP, the camera 193 , the video codec, the GPU, the display screen 194 and the application processor.
- the ISP is used for processing the data fed back by the camera 193 .
- the light is transmitted to the photosensitive element of the camera through the lens, and the optical signal is converted into an electrical signal, and the photosensitive element of the camera transmits the electrical signal to the ISP for processing, and converts it into an image visible to the naked eye.
- ISP can also perform algorithm optimization on image noise, brightness, and skin color.
- ISP can also optimize the exposure, color temperature and other parameters of the shooting scene.
- the ISP may be located in the camera 193 .
- the ISP can control the photosensitive element to expose and take pictures according to the shooting parameters.
- Camera 193 is used to capture still images or video.
- the object generates an optical image through the lens and projects it to the photosensitive element.
- the photosensitive element may be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor.
- CMOS complementary metal-oxide-semiconductor
- the photosensitive element converts the light signal into an electrical signal, and then transmits the electrical signal to the ISP to convert it into a digital image signal.
- the ISP outputs the digital image signal to the DSP for processing.
- DSP converts digital image signals into standard RGB, YUV and other image signals.
- the electronic device 100 may include 1 or N cameras 193 , where N is a positive integer greater than 1.
- the camera 193 may be located in the edge area of the electronic device, and may be an under-screen camera, a liftable camera, or a hole-digging camera.
- the camera 193 may include a rear camera, and may also include a front camera. The embodiment of the present application does not limit the specific position and shape of the camera 193 .
- the camera 193 includes a common camera and an ultraviolet camera.
- the ordinary camera includes a color camera.
- the color camera may be, for example but not limited to, an RGB camera.
- the color camera can be used to collect images, and the collected images can be used to determine whether there is a human face waiting to be detected. Captured images can also be used to flag sunscreen application. For example, a phone could mark in a color image where sunscreen was not applied evenly.
- the ultraviolet camera can be used to take ultraviolet images, and the electronic device 100 can detect the application of sunscreen according to the taken ultraviolet images.
- the ultraviolet camera can also be used to capture ultraviolet rays in the current environment.
- the image sensor such as CMOS
- the image sensor in the ultraviolet camera captures the ultraviolet rays in the current environment.
- the electronic device 100 can determine the intensity of ultraviolet rays in the current environment according to the captured ultraviolet rays.
- the electronic device 100 may further include an ultraviolet supplementary light.
- the ultraviolet supplementary light can emit ultraviolet rays and can be used as an imaging light source for ultraviolet cameras.
- the ultraviolet supplementary light in order to avoid excessive ultraviolet irradiation of the user by the ultraviolet supplementary light, only when the ambient ultraviolet intensity is lower than a certain threshold, that is, the ambient ultraviolet intensity cannot reach the ultraviolet intensity required by the ultraviolet camera for imaging, Turn on the UV light.
- the ambient ultraviolet intensity is higher than the threshold, the electronic device does not turn on the ultraviolet supplementary light.
- the electronic device 100 acquires the ambient ultraviolet intensity by detecting the ultraviolet intensity through its own sensor, or acquires the ambient ultraviolet intensity from the cloud side.
- the embodiment of the present application does not limit the manner in which the electronic device 100 acquires the intensity of ambient ultraviolet rays.
- Digital signal processors are used to process digital signals. In addition to digital image signals, they can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the energy of the frequency point.
- Video codecs are used to compress or decompress digital video.
- the electronic device 100 may support one or more video codecs.
- the electronic device 100 can play or record videos in various encoding formats, for example: moving picture experts group (moving picture experts group, MPEG) 1, MPEG2, MPEG3, MPEG4 and so on.
- MPEG moving picture experts group
- the NPU is a neural-network (NN) computing processor.
- NN neural-network
- Applications such as intelligent cognition of the electronic device 100 can be realized through the NPU, such as image recognition, face recognition, speech recognition, text understanding, and the like.
- the NPU uses image recognition technology to identify whether the image collected by the camera 193 contains a face image.
- the external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, so as to expand the storage capacity of the electronic device 100.
- the external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function. Such as saving music, video and other files in the external memory card.
- the internal memory 121 may be used to store computer-executable program codes including instructions.
- the processor 110 executes various functional applications and data processing of the electronic device 100 by executing instructions stored in the internal memory 121 and/or instructions stored in a memory provided in the processor.
- the electronic device 100 can implement audio functions through the audio module 170 , the speaker 170A, the receiver 170B, the microphone 170C, the earphone interface 170D, and the application processor. Such as music playback, recording, etc.
- the audio module 170 is used to convert digital audio data into an analog audio electrical signal output, and is also used to convert an analog audio electrical signal input into digital audio data.
- the audio module 170 may include an analog/digital converter and a digital/analog converter.
- the audio module 170 is used to convert the analog audio electrical signal output by the microphone 170C into digital audio data.
- the audio module 170 may also be used to encode and decode audio data.
- the audio module 170 may be set in the processor 110 , or some functional modules of the audio module 170 may be set in the processor 110 .
- Speaker 170A also called “horn” is used to convert analog audio electrical signals into sound signals. Electronic device 100 can listen to music through speaker 170A, or listen to hands-free calls.
- Receiver 170B also called “earpiece” is used to convert analog audio electrical signals into sound signals.
- the receiver 170B can be placed close to the human ear to receive the voice.
- the microphone 170C also called “microphone” or “microphone”, is used to convert sound signals into analog audio electrical signals.
- the user can put his mouth close to the microphone 170C to make a sound, and input the sound signal to the microphone 170C.
- the microphone 170C may be a built-in component of the electronic device 100 , or may be an external accessory of the electronic device 100 .
- the electronic device 100 may include one or more microphones 170C, where each microphone or multiple microphones cooperate to collect sound signals from various directions, and convert the collected sound signals into analog audio electrical signals. It can also achieve noise reduction, identify the source of sound, or directional recording functions.
- the sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, bone conduction sensor 180M, etc.
- the gyroscope sensor 180B can be used to determine the motion posture of the electronic device 100 .
- the angular velocity of the electronic device 100 around three axes may be determined by the gyro sensor 180B.
- the gyro sensor 180B can be used for image stabilization.
- the gyro sensor 180B detects the shaking angle of the electronic device 100, calculates the distance that the lens module needs to compensate according to the angle, and allows the lens to counteract the shaking of the electronic device 100 through reverse movement to achieve anti-shake.
- the gyro sensor 180B can also be used for navigation and somatosensory game scenes.
- the distance sensor 180F is used to measure the distance.
- the electronic device 100 may measure the distance by infrared or laser. In some embodiments, in a shooting scene, the electronic device 100 may use the distance sensor 180F for distance measurement to achieve fast focusing.
- Touch sensor 180K also known as "touch panel”.
- the touch sensor 180K can be disposed on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a “touch screen”.
- the touch sensor 180K is used to detect a touch operation on or near it.
- the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100 .
- the electronic device 100 may include more or fewer components than shown in the figure, or combine certain components, or separate certain components, or arrange different components.
- the illustrated components can be realized in hardware, software or a combination of software and hardware.
- the software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture.
- the embodiment of the present invention uses a layered architecture as an example to illustrate the software structure of the electronic device 100 .
- FIG. 3 is a block diagram of the software structure of the electronic device 100 according to the embodiment of the present invention.
- the layered architecture divides the software into several layers, and each layer has a clear role and division of labor. Layers communicate through software interfaces.
- the operating system of the electronic device (such as the system to be detected) is divided into four layers, which are respectively the kernel layer, the hardware abstract layer (hardware abstract layer, HAL), the application program framework layer, from bottom to top. and the application layer.
- the kernel layer is the layer between hardware and software.
- the kernel layer includes at least camera driver, audio driver, display driver, and sensor driver.
- Sensor drivers include but are not limited to image sensor drivers and acoustic sensor drivers.
- a kernel layer such as a camera driver is invoked to turn on the camera.
- the image sensor is called by the image sensor driver to complete the image acquisition.
- the hardware abstract layer (hardware abstract layer, HAL) is located between the kernel layer and the application framework layer, and is used to define the interface for driving the hardware implementation of the application program, and convert the value implemented by the driving hardware into the software implementation programming language. For example, identify the value driven by the camera, convert it into a software program language and upload it to the application framework layer, and then call the corresponding function.
- the HAL can upload the face image collected by the camera 193 to the application framework layer for further processing.
- the application framework layer provides an application programming interface (application programming interface, API) and a programming framework for applications in the application layer.
- the application framework layer obtains the original input event from the kernel layer via the HAL, and identifies the control corresponding to the input event.
- the application framework layer includes some predefined functions.
- the application framework layer can include view system, phone manager, resource manager, notification manager, window manager, etc.
- the application framework layer includes a first module.
- the first module is used to call the first camera to collect the first image through the camera driver, and call the second camera to collect the second image through the camera driver. Afterwards, a first area is determined on the second image, and a second area corresponding to the first area is determined on the first image according to the first area. Afterwards, the first module may invoke a display driver to display a user interface, where the user interface includes the first face and first information, and the first information is used to indicate the second area.
- the first module can also be arranged in other layers, and the first module can also be divided into more sub-modules. Each sub-module is used to perform a corresponding function.
- the view system includes visual controls, such as controls for displaying text, controls for displaying pictures, and so on.
- the view system can be used to build applications.
- a display interface can consist of one or more views.
- a display interface including a text message notification icon may include a view for displaying text and a view for displaying pictures.
- the phone manager is used to provide communication functions of the electronic device 100 . For example, the management of call status (including connected, hung up, etc.).
- the resource manager provides various resources for the application, such as localized strings, icons, pictures, layout files, video files, and so on.
- the notification manager enables the application to display notification information in the status bar, which can be used to convey notification-type messages, and can automatically disappear after a short stay without user interaction.
- the notification manager is used to notify the download completion, message reminder, etc.
- the notification manager can also be a notification that appears on the top status bar of the system in the form of a chart or scroll bar text, such as a notification of an application running in the background, or a notification that appears on the screen in the form of a dialog window.
- prompting text information in the status bar issuing a prompt sound, vibrating the electronic device, and flashing the indicator light, etc.
- a window manager is used to manage window programs.
- the window manager can get the size of the display screen, determine whether there is a status bar, lock the screen, capture the screen, etc.
- the application layer can consist of a series of application packages.
- the application package may include applications such as camera, video, call, WLAN, music, short message, Bluetooth, map, calendar, gallery, and navigation.
- the application layer and the application framework layer run in virtual machines.
- the virtual machine executes the java files of the application program layer and the application program framework layer as binary files.
- the virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.
- the sunscreen of the embodiment of the present application can be activated.
- the sunscreen application detection method according to the embodiment of the present application can be activated.
- the embodiment of the present application does not limit the manner of enabling the sunscreen application detection function.
- the technical solutions of the embodiments of the present application are introduced by taking the method of turning on the sunscreen application detection method by turning on the sunscreen detection switch in the mirror application as an example.
- the user can instruct the mobile phone to start the mirror application and display the shooting preview interface by means of touch operation, button operation, air gesture operation or voice operation.
- the mobile phone displays the main interface as shown in (a) in Figure 4.
- a common camera such as an RGB camera
- the mobile phone starts and starts the mirror application, and calls a common camera to collect images to display the shooting preview interface shown in (b) in FIG. 4 .
- the mobile phone starts the mirror application, and calls a common camera to capture images, and obtains the shooting preview interface shown in (b) in FIG. 4 .
- the shooting preview interface may include one or more controls for realizing different functions.
- the setting control 401 can be used to set the functions of the mirror application.
- the sun protection detection control 402 can be used to enable the sun protection detection function of the embodiment of the present application.
- the shooting preview interface may further include other controls.
- controls that can be used to turn on or off the filter function controls that can be used to turn on or off the flash function, controls that can be used to watch beauty tutorials, controls to control taking pictures, controls to adjust focus, and controls to adjust screen brightness.
- the mobile phone detects, for example, that the user clicks on the sunscreen detection control 402 on the shooting preview interface, and then starts the sunscreen application detection function.
- the sunscreen application detection function is turned on, the mobile phone can detect the application of sunscreen.
- the ultraviolet camera an example of the second camera
- the mobile phone can call the ultraviolet camera to collect one or more ultraviolet images, The application of sunscreen is detected based on the one or more ultraviolet images.
- the mobile phone can continue to call the color camera (an example of the first camera) Acquire color images.
- the mobile phone can automatically call the ultraviolet camera to collect the ultraviolet image. Acquire color images.
- the ultraviolet camera and the color camera can collect images synchronously or alternately.
- the mobile phone can be equipped with an ultraviolet supplementary light, and the mobile phone can detect and judge whether to turn on the ultraviolet supplementary light when collecting ultraviolet images based on the intensity of ambient ultraviolet rays.
- the mobile phone can call the ultraviolet camera to capture ambient ultraviolet rays, and determine the intensity of ambient ultraviolet rays according to the captured ultraviolet rays.
- the mobile phone turns on the ultraviolet fill light.
- the mobile phone calls the ultraviolet camera (the ultraviolet fill light may not be turned on) to collect a preset number of ultraviolet images.
- the ultraviolet camera is turned off, so as to reduce the power consumption caused by the ultraviolet camera.
- the mobile phone may not turn off the color camera.
- the image collected by the color camera can be displayed in the preview interface for the user to watch.
- the images captured by the UV camera can be used to determine the application of sunscreen.
- turning off the ultraviolet supplementary light may be: automatically turning off the ultraviolet supplementary light after detecting that the collection of a preset number of ultraviolet images is completed.
- the UV supplementary light is automatically turned off.
- the ultraviolet supplementary light can be set to be linked with the ultraviolet camera. Under this setting, if there is a human face in the image detected by the ordinary camera, the UV camera and the linked UV fill light can be turned on, and a preset number of UV images can be collected through the UV camera. In this linkage setting method, since the ultraviolet camera and the ultraviolet supplementary light can be turned on in linkage, the complexity of realizing the acquisition of ultraviolet images can be reduced.
- the mobile phone turns on the ultraviolet camera, which can also be triggered by the user's operation.
- a control for triggering the ultraviolet camera to collect ultraviolet images is set in the user interface, and when a user's preset operation on the control is detected, the mobile phone can turn on the ultraviolet camera to collect ultraviolet images.
- the color camera captures color images synchronously.
- the mobile phone can detect the application of sunscreen based on the collected ultraviolet images. After detecting the application of sunscreen, the mobile phone prompts the user to apply the information of the sunscreen.
- the sunscreen application information includes but is not limited to any one or more of the following information: whether the user applies sunscreen, the location where the sunscreen is applied, the location where the sunscreen is not applied, whether the sunscreen is applied evenly, and whether the sunscreen is not applied evenly location, information on the thickness of sunscreen application in each area, and sunscreen recommendation information.
- the mobile phone detects the user's click operation on the sunscreen detection control 402, then detects the application of sunscreen, and can present the prompt box shown in (c) of Figure 4 to the user 403.
- the user can intuitively determine the precise position where the sunscreen is not applied uniformly or thinly applied from the mark of the mobile phone, so that the sunscreen can be reapplied according to the precise position, and the user can be fine-tuned. sun protection guidelines.
- the mobile phone can collect the ultraviolet image of the front face, and determine the application of sunscreen according to the ultraviolet image of the front face.
- the mobile phone can prompt the user to "align the front face with the mobile phone" before collecting the ultraviolet image of the front face, so that the user can align the front face with the mobile phone under the prompt.
- the above-mentioned preset number of ultraviolet images collected by the ultraviolet camera can be multiple ultraviolet images, and the multiple ultraviolet images are images collected when the user's face turns to different angles, and the multiple ultraviolet images include people from different angles. face image information. Different UV images can correspond to different facial angles.
- the angle here may be the angle of the user's face relative to the mobile phone.
- the mobile phone may determine the angle of the human face in the image relative to the mobile phone based on the positional relationship between some key points of human faces detected in the image.
- the key points of the face used to detect the relationship such as angle and position can be the key point P1 of the left face, the key point P2 near the nose wing, and the key point P3 of the right face.
- the position relationship ratio between the key points (P2-P1)/( P3-P2).
- the mobile phone can obtain the angle of the collected image according to the relationship between the positional ratio of the key points and the actual left-right rotation angle.
- P2-P1 represents the distance between P2 and P1
- P3-P2 represents the distance between P3 and P2.
- the mobile phone calculates the ratio of the positional relationship of the facial key points, and determines that the ratio corresponds to the frontal face angle.
- the image collected by the mobile phone is the image 1402 shown in (b) of Figure 5
- the mobile phone calculates the positional relationship ratio of the key points of the face, and determines that the ratio corresponds to the user's face turning 30 degrees to the right.
- the images 1401 and 1402 may be color images or ultraviolet images.
- the RGB camera and the UV camera can work synchronously, which means that at the same moment, the mobile phone calls the RGB camera and the UV camera at the same time. Moreover, the mobile phone can recognize the key points of the face of the RGB image collected by the RGB camera, and judge the angle of the face in the RGB image to the mobile phone according to the key points of the face of the RGB image. The angle of the face relative to the mobile phone in the simultaneously acquired UV image.
- the RGB camera may not work synchronously with the UV camera.
- the mobile phone can determine the key points of the face in the RGB image, and identify the key points of the face in the ultraviolet image according to the image matching algorithm and the key points of the face in the RGB image. Afterwards, the mobile phone can determine the angle of the human face in the ultraviolet image relative to the mobile phone according to the key points of the human face in the ultraviolet image.
- the mobile phone collects multiple facial ultraviolet images from multiple angles, which may be during the process of the user actively turning the head (turning the head left or right), and the mobile phone collects multiple facial ultraviolet images from multiple angles .
- the user opens the mirror application, and instructs the mobile phone to enable the sun protection detection function by, for example, clicking on the sun protection detection control 402 shown in (b) of FIG.
- the RGB image is collected by the RGB camera.
- the mobile phone can calculate the angle of the human face relative to the mobile phone in each RGB image, and select the target RGB image whose angle meets the preset conditions from multiple RGB images according to the calculated angle, and the corresponding ultraviolet light of the target RGB image The image can be used as a UV image to calculate sunscreen application.
- the angle conforming to the preset condition may mean that the angle of the human face in the RGB image relative to the mobile phone is within the first preset angle range of left rotation, or within the second preset angle range of right rotation, or within the angle range of facing the mobile phone .
- the angle of the face relative to the mobile phone is between 5 degrees to the left and 5 degrees to the right as an example
- the mobile phone has 10 RGB images, and in one RGB image, the angle of the face relative to the mobile phone is 25 degrees to the left, in one RGB image, the angle of the face relative to the mobile phone is 20 degrees to the right, and in one RGB image
- the angle of the human face relative to the mobile phone is rotated 25 degrees to the right, and the angle of the human face relative to the mobile phone is rotated 2 degrees in one RGB image.
- the mobile phone regards these 4 RGB images as the target RGB images that meet the preset conditions , the UV images corresponding to these 4 RGB images can
- first preset angle the second preset angle
- facing angle can be flexibly set separately.
- preset conditions may also be set separately, which is not limited in this embodiment of the present application.
- the mobile phone collects 10 RGB images (RGB images 1-10) and 10 UV images (UV images 1-10), wherein the RGB image L (L is a positive integer, L is greater than or equal to 1, and less than or equal to 10) It is collected synchronously with the ultraviolet image L.
- the mobile phone can use the UV image 3-5 corresponding to the RGB image 3-RGB image 5 as the UV used to calculate the application of sunscreen. image.
- the mobile phone collects multiple facial ultraviolet images from multiple angles, or the mobile phone prompts the user to turn the head, and the mobile phone collects facial ultraviolet images from different angles during the process of turning the head under the prompt of the mobile phone. image.
- the mobile phone detects the user's operation such as clicking the sun protection detection control 402 shown in (b) of FIG. 4 , the mobile phone starts the sun protection detection function.
- the mobile phone may prompt the user to face the mobile phone in the preview interface, and collect RGB images through RGB images, and collect ultraviolet images through ultraviolet cameras. The mobile phone can judge the angle of the face in the RGB image relative to the mobile phone according to the RGB image.
- the mobile phone After determining that the face is facing the mobile phone, the RGB image of the front face (an example of the third image) and the ultraviolet image of the front face (face When the angle relative to the mobile phone satisfies the second condition, that is, meets the angle range of the frontal face), the mobile phone can continue to collect RGB images from other angles and ultraviolet images from other angles.
- the mobile phone may also present a prompt box 405 (an example of the second information) shown in (b) of FIG. 6 in the preview interface, to prompt the user to turn their head to the left.
- the mobile phone collects an RGB image (the fourth image) through the RGB image, and collects an ultraviolet image (the sixth image) through the ultraviolet camera.
- the mobile phone can continue to collect RGB images from other angles and UV images from other angles. Later, optionally, after determining that the RGB image of the left side face and the ultraviolet image of the left side face have been collected, the mobile phone can also present a prompt box 406 shown in (c) of FIG. Turn your head to the right.
- the image presented on the preview interface may be a color image.
- the mobile phone can call the ultraviolet camera to collect multiple ultraviolet images and call the color camera to collect multiple color images.
- the color image can be used to present the preview interface and assist in determining the angle of the face in the ultraviolet image relative to the mobile phone, and multiple ultraviolet images can be used to determine the application of sunscreen.
- the mobile phone can present a prompt box such as that shown in (d) of Figure 6, so as to remind the user of the application of sunscreen and provide effective and detailed guidance to the user on sun protection.
- the interface for the mobile phone to prompt the user to turn the head is not limited to the interface shown in FIG. 6 .
- directly prompt the user "Please turn your head left or right to collect multi-angle face images".
- the frontal face image mentioned in the embodiment of the present application may be an image within a certain angle deviation range, and is not limited to an image in which the face is strictly facing the mobile phone.
- the mobile phone presents the prompt box shown in (a) of FIG. 6 , which may be to present the prompt box 405 immediately after detecting the user's operation such as clicking the sun protection detection control 402 shown in FIG. 4 (b) . It may also be that after the user's operation such as clicking on the sun protection detection control 402 is detected, the prompt box 405 is presented when the user does not detect turning the head within a preset period of time (which can be set flexibly). The embodiment of the present application does not limit the timing at which the mobile phone prompts the user to turn the head.
- the mobile phone recognizes the face image in the RGB image.
- RGB images have richer imaging details and higher quality than ultraviolet images. Therefore, the mobile phone can recognize the face in the RGB image and match the corresponding area of the recognized face to the UV image.
- the RGB image may be collected by calling the RGB camera of the mobile phone after the user opens the mirror application, and the timing of collecting the RGB image is not limited.
- the timing of collecting the RGB image is not limited.
- the synchronous acquisition of the RGB camera and the ultraviolet camera after the ultraviolet camera is turned on is used as an example for illustration, but the technical solution of the embodiment of the present application is not limited thereto.
- the mobile phone recognizes the face image in the RGB image based on the face recognition algorithm.
- the mobile phone can extract facial feature points from the recognized face image, and mark eyes, eyebrows, mouth and other facial parts in the face image that usually do not apply sunscreen according to the extracted facial feature points. Later, when the mobile phone calculates the application of sunscreen, it does not need to calculate the eyes, eyebrows and other areas of the face image. In this way, the calculation amount of the mobile phone can be reduced.
- the mobile phone extracts the facial feature points in the face image, and as shown in (b) of Figure 8, marks the facial feature points in the face image, such as eyes and mouth The area surrounded by their respective feature points.
- the outline surrounding the imaging area such as the eye is taken as a box, and the box area is marked as an example.
- the outline surrounding the imaging area such as the eye can also be other
- the shape, such as a circle can also be, for example, a shape that fits the contours of the edges of imaging areas such as eyes.
- the eyes, lips and other areas that do not need to be applied with sunscreen can be removed from the face image to obtain the fourth area.
- the follow-up that is, the face area processed in the subsequent steps is to remove the eyes, lips, eyebrows, etc. and other areas (fourth area).
- the mobile phone may determine the first area in the fourth area of the second image according to the pixel values of the pixels included in the fourth area, without performing traversal processing on pixels outside the fourth area.
- this part of the area is not eliminated, but only this part of the area is marked, and subsequently, the mobile phone can choose not to process the pixels of this part of the area according to the mark made.
- the mobile phone acquires the human face (an example of the second human face) in the ultraviolet image (an example of the second image) according to the human face (the first human face) in the RGB image (an example of the first image).
- the first human face and the second human face belong to the same user.
- the mobile phone may acquire the human face image in the ultraviolet image through, for example, an image matching algorithm.
- the image matching algorithm includes but not limited to a scale-invariant feature transform (scale-invariant feature transform, SITF) algorithm.
- SITF scale-invariant feature transform
- the mobile phone can extract the key points of the face image in the RGB image. Key points can be, but are not limited to, points that will not disappear due to factors such as illumination, scale, and rotation, such as the lowest point of the chin, the highest point of the nose, and so on.
- the mobile phone can also obtain the feature vector of each key point of the face, and locate each key point in the ultraviolet image according to the feature vector of each key point of the face in the RGB image. In this way, the mobile phone can determine the face image according to the key points of the face in the ultraviolet image.
- the mobile phone obtains the eigenvector 1 of the lowest point of the chin and the eigenvector 2 of the highest point of the alar in the RGB image, and finds the eigenvectors in the ultraviolet image.
- the pixel close to eigenvector 1 is used as the lower edge point of the chin, and the pixel point whose eigenvector is closest to eigenvector 2 is found in the ultraviolet image as the highest point of the alar.
- the mobile phone may recognize the face image in the ultraviolet image according to the face image in the RGB image. Among them, the mobile phone can also identify the positions of the eyes and lips in the ultraviolet image according to the positions of the eyes and lips in the RGB image. Subsequently, the processing of the positions of the eyes and lips can be different from other areas of the face.
- the mobile phone may also execute S103.
- the above-mentioned embodiment is mainly described by taking the color camera as an RGB camera, and the collected color image is an RGB image as an example.
- the color camera in the embodiment of the present application can also be other types of cameras.
- the collected The color image can be of other types.
- the mobile phone preprocesses the face image in the ultraviolet image.
- the preprocessing includes but not limited to filtering processing, so as to eliminate the influence of noise in the image.
- Filtering may include, but is not limited to, mean filtering, median filtering, Gaussian filtering, and the like.
- the mobile phone may perform preprocessing on all regions of the face image in the ultraviolet image. Or, in order to reduce the amount of preprocessing calculations, the mobile phone can also preprocess the areas in the face image except the eyes, lips, etc. (that is, areas that usually do not need to apply sunscreen), that is, areas other than the marked area.
- the mobile phone recognizes the application of sunscreen based on the face image in the ultraviolet image.
- the mobile phone may determine the first area on the second face of the ultraviolet image (second image) according to the image features of the ultraviolet image, and determine the application of sunscreen in the first area.
- the first area is any one or more of the following areas: an area where sunscreen is applied, an area where sunscreen is not applied, an area where sunscreen is unevenly applied, an area where sunscreen is evenly applied, and the thickness of sunscreen applied is greater than In the area of the first threshold, the area where the thickness of the sunscreen application is smaller than the second threshold.
- Image features include but are not limited to one or more of the following features: the average pixel value of the image area, the maximum pixel value, the minimum pixel value, the difference between the pixel values of adjacent pixels in the image area, the color of each pixel in the image area, the image Regional contrast, histogram, histogram of oriented gradients (HOG), standard deviation, color scale, mean square deviation, variance.
- the color value of a pixel includes the pixel value of each channel of RGB.
- the principle of determining the sunscreen application area based on the pixel value in the UV image feature is introduced.
- sunscreen can absorb ultraviolet rays, so in the ultraviolet image, the brightness of the area where sunscreen is applied is lower, and the corresponding pixel value is usually lower, and the brightness of the area without sunscreen is higher, corresponding to Pixel values are usually higher.
- the mobile phone can detect the pixel value of the face image, and use the area with a pixel value higher than the threshold as the area without sun protection, and the area with the pixel value less than or equal to the threshold as the area with sun protection.
- an area whose pixel value is higher than the pixel threshold may be referred to as a highlighted area. Areas with pixel values less than or equal to the threshold can be called non-highlight areas.
- a pixel window with a size of W1*W2 may be set.
- the face image in the ultraviolet image can be traversed through the sliding window. Every time the window slides to a position, the average pixel value ⁇ F of the face image in the window area is counted once, and the window area of ⁇ F ⁇ T1 is divided into non-highlight areas , divide the window area of ⁇ F>T1 into the highlighted area.
- the mobile phone detects that the imaging region in the ultraviolet image whose pixel average value is higher than the threshold (lighter in color) is the region 408a, then the region 408a can be regarded by the mobile phone as a place where no sunscreen is applied The corresponding imaging area of the site.
- the imaging area 407a whose pixel mean value is smaller than the threshold (darker color) is regarded by the mobile phone as the imaging area corresponding to the part where the sunscreen is applied.
- the mobile phone may iterate over the pixel values of each pixel, classify pixels with pixel values greater than the threshold into highlighted areas, and classify pixels with pixel values less than or equal to the threshold into non-highlight areas.
- the mobile phone can also determine the uniformity and thickness of the sunscreen application based on the pixel values included in the ultraviolet image. Taking Figure 1 as an example, compared with Area 1, the application of sunscreen in Area 2 is thinner, so the imaging pixel value of Area 2 in the ultraviolet image is higher and the color is lighter.
- the mobile phone After the mobile phone determines the application of sunscreen (such as the area where the sunscreen is applied and the area where the sunscreen is not applied), the mobile phone can prompt the user for the application of the sunscreen.
- the application of sunscreen can be marked in a color image or a grayscale image.
- Figure 8 describes the process of labeling sunscreen application in color images
- Figure 9 describes the process of labeling sunscreen application in grayscale images. A detailed description will be given below in conjunction with FIG. 8 and FIG. 9 .
- the implementation process of (a) and (b) in FIG. 9 can refer to (a) and (b) in FIG. 8 , which will not be repeated here.
- the mobile phone may mark prompt information (first information) in the face (first face) of the color image.
- the color image may be collected by the mobile phone calling the color camera after the user opens the mirror application.
- the embodiment of the present application does not limit the timing of collecting color images. For example, it may be that when the ultraviolet camera collects the ultraviolet image, the color camera collects the color image synchronously, or the color camera and the ultraviolet camera collect alternately.
- the mobile phone can use an image matching algorithm to determine the second area corresponding to the first area on the face of the color image, that is, match the first area in the ultraviolet image to the second region of the color image. That is, sunscreen reminders can be marked in color images.
- the mobile phone determines that the area where sunscreen is applied is 407a (an example of the first area) according to the ultraviolet image shown in (c) of FIG. 407a' (an example of the second area) of the RGB image shown in (d), and mark the area 407a' where the sunscreen is applied in the RGB image.
- the mobile phone can also match the unapplied sunscreen area 408a (another example of the first area) of the UV image to the area 408a' of the RGB image, and mark the area 408a' (another example of the second area) in the RGB image. example).
- the second area in the color image is the area corresponding to the first area in the ultraviolet image.
- the second area is any one or more of the following areas: an area where sunscreen is applied, an area where sunscreen is not applied, an area where sunscreen is applied unevenly, an area where the thickness of sunscreen is greater than the first threshold, and where sunscreen is applied Regions with a thickness less than the second threshold.
- different colors may be used to mark different application conditions of the sunscreen so that the user can clearly identify the application conditions of the sunscreen.
- the difference between colors used to represent different application situations may be relatively large. For example, areas with sunscreen applied are marked with a green outline (an example of the first message) to indicate where sunscreen is applied, and areas without sunscreen are marked with a red outline to indicate where no sunscreen is applied . As another example, mark only the places where no sunscreen is applied, or, mark only the places where sunscreen is applied. Labeling sunscreen application information in color images makes it easier for users to understand these labeling information. Moreover, color images can often record more detailed skin conditions, which is more conducive to guiding users to reapply sunscreen.
- the mobile phone may present a prompt text on a corresponding area of the face in the color image. For example, near the area where sunscreen is applied, the user is prompted that this area is the area where sunscreen is applied.
- the mobile phone may hide the prompt text, and when it is detected that the user clicks on a corresponding area, the mobile phone presents the prompt text. For example, when it is detected that the user clicks on the area where sunscreen is applied, the mobile phone then prompts the user that this area is the area where sunscreen is applied.
- the first information is also used to indicate the uniformity of sunscreen application and/or the thickness of sunscreen application and/or whether to apply sunscreen in the second area.
- the mobile phone can also explicitly prompt the thickness and uniformity of the sunscreen in the form of text and the like. Or, when the user taps the area surrounded by the dotted frame, the mobile phone displays text prompting the uniformity and thickness of the sunscreen, etc.
- the mobile phone may also mark the application of sunscreen in the grayscale image.
- the grayscale image may be an image obtained by processing an ultraviolet image.
- the ultraviolet image can be processed to increase the difference in pixel difference between the sunscreen-applied area and the non-sunscreen-applied area.
- the color of the area with sunscreen in the ultraviolet image can be darkened, and the color of the area without sunscreen in the ultraviolet image can be lightened, so that the area with sunscreen in the ultraviolet image and the area without sunscreen The contrast of color depth deepens.
- the grayscale image used to mark the application of sunscreen may be a binary image. That is to say, the ultraviolet image can be processed to obtain the corresponding binary image, and the application of sunscreen can be marked on the binary image.
- the binary image that is, the pixel value of the pixel in the image
- the difference between the two pixel values of the binary image may be set to be greater than a certain threshold.
- the pixel value of any pixel in the binary image may be 255 (indicating white), or 0 (indicating black).
- the mobile phone calculates the binary image corresponding to the ultraviolet image, which can be implemented as follows: the mobile phone traverses the pixels included in the face image in the ultraviolet image, and for traversal For pixels whose pixel value is greater than the threshold value, the pixel value of the pixel at the corresponding position in the binary image is set to white by the mobile phone (the pixel value of the white pixel can be 255, for example), and for the pixel value traversed that is less than or equal to the threshold value Pixel, the pixel value of the corresponding position of the pixel in the binary image is set by the mobile phone as black (the pixel value of the black pixel may be 0, for example).
- the mobile phone traverses each pixel included in the ultraviolet face image as shown in (c) of FIG. is less than the binarization threshold, then, the shadow area 407a of the ultraviolet image is in the corresponding position in the binary image, that is, the pixel value of the pixel included in the area 407a "is set to black.
- the pixel value of the pixel included in the white area 408a is greater than the threshold ( Corresponding to the area where sunscreen may not be applied), then, the white area 408a of the ultraviolet image is at the corresponding position in the binary image, that is, the pixel value of the pixel included in the area 408a' is set to white.
- the embodiment of the present application does not limit the specific implementation of calculating the binary image by the mobile phone and the threshold used when calculating the binary image.
- the shaded area 407a in the ultraviolet image represents the area where sunscreen is applied, wherein the higher the density of the oblique lines in the shadow, the thicker the sunscreen is applied (the corresponding area's The lower the pixel value), the lower the density of the slash, which means the thinner the sunscreen is applied (the higher the pixel value of the corresponding area).
- the pixel values of the area where the sunscreen is thinly applied are relatively close to the pixel values of the area where no sunscreen is applied, and the user may apply sunscreen.
- the phone can mark whether sunscreen is applied or not through binary images.
- the mobile phone may calculate a binary image corresponding to the ultraviolet image shown in (c) of FIG. 9 , and the binary image is shown in (d) of FIG. 9 . It can be seen from (d) of FIG. 9 that the difference between the two pixel values in the binary image is relatively large. In this way, the color contrast between the area where sunscreen is applied and the area where sunscreen is not applied is deepened, and the user can more easily and intuitively distinguish which parts are applied with sunscreen and which parts are not.
- marking the application of sunscreen in the grayscale image may also be directly marking the application of sunscreen in the ultraviolet image. For example, for areas in the UV image where sunscreen is not applied, mark the area with a shade or a dashed box.
- the mobile phone may provide two option controls of a grayscale view and a color view.
- Mark sunscreen application on the color image when it detects an action from the user indicating use of the color view, for example, when the phone detects that the user taps a color view control.
- a grayscale image (such as a binary image) is presented, and the application of sunscreen is marked by the grayscale image.
- the imaging features of moles on the face are similar to those of the parts where sunscreen is applied. Therefore, if the user has moles on the face, the moles may be misjudged as the parts where sunscreen is applied , which means that moles are likely to interfere with the test results of sunscreen.
- the imaging characteristics of the part where sunscreen is applied but the reflection is similar to the imaging characteristics of the part where no sunscreen is applied, and the pixel values are higher. Based on the characteristics of the reflective part, during the detection process, the part with sunscreen applied but the reflective part may be misjudged as the part without sunscreen.
- the mobile phone can exclude the imaging area of interference factors such as moles, that is, the interference area is not determined as the area where sunscreen is applied. Considering that when users apply sunscreen, the area where the sunscreen is usually applied will be larger.
- the mobile phone determines a plurality of target sub-regions in the fourth region according to the pixel values of the pixels contained in the fourth region (the region of the human face excluding eyes, eyebrows, and lips).
- a target sub-region whose area is greater than or equal to a fourth threshold is determined as the first region.
- the target sub-area satisfies a third condition, and the third condition includes but is not limited to any one or more of the following: the pixel mean value of the target sub-area is less than the first threshold, and the pixels contained in the target sub-area value is less than the second threshold. That is to say, for a non-highlight area in the ultraviolet image whose area is smaller than the area threshold, the mobile phone does not determine it as an area for applying sunscreen. For the non-highlight area in the UV image whose area is greater than or equal to the area threshold, the mobile phone determines it as the area where sunscreen is applied.
- areas in the ultraviolet image where the pixel values are smaller than the pixel value threshold are 407a and 409a.
- the mobile phone determines the corresponding part as the part where sunscreen is applied.
- the part where the sunscreen is applied calculated by the mobile phone is the part corresponding to the area 407a, and the part where the sunscreen is not applied calculated by the mobile phone is the part corresponding to the area 408a.
- the mobile phone may display a prompt box as shown in (d) of FIG. 11 to prompt the user to reapply the parts where no sunscreen has been applied.
- the facial ultraviolet images from multiple angles can be processed, and according to the processing results of the multi-angle facial ultraviolet images Screen for reflective areas to identify areas where sunscreen is applied and areas where sunscreen is not.
- the above step S104 may include the following steps:
- the mobile phone acquires the highlighted area 408a and the non-highlighted area 407a of the human face image in the ultraviolet image 901 .
- the ultraviolet images 901-903 are three ultraviolet images collected by the mobile phone through the ultraviolet camera after the sun protection detection function is turned on.
- the ultraviolet images 901-903 meet the preset angle conditions, for example, the angle of the face in the image relative to the mobile phone is within the range of 30 degrees to the left and 30 degrees to the right.
- the ultraviolet image 901 may be a reference image selected from multiple ultraviolet images from multiple angles.
- the reference image is an image including more imaging details.
- the reference image may be an ultraviolet image from a frontal perspective.
- the reference image can be determined flexibly, and the manner of determining the reference image is not limited.
- the mobile phone can determine the human face image in the ultraviolet image 901 according to the scheme of the above-mentioned embodiment, and can determine the human face image in areas other than the eyes and lips of the human face image (the area where sunscreen is usually not applied). Find highlighted areas and non-highlighted areas in order to reduce computational complexity.
- threshold segmentation may be performed according to the pixel value, so as to locate the highlighted region and the non-highlighted region in the ultraviolet image 901 .
- the threshold is set to T1 (can be set flexibly)
- the mobile phone can traverse the pixels in the image 901.
- the mobile phone will use this pixel as the composition of the highlighted area
- the pixel value f(x,y) ⁇ T1 the mobile phone regards this pixel as the constituent pixel of the non-highlight area.
- the mobile phone can also calculate the relative position coordinates of the highlighted area relative to one or more specific parts of the eyes, nose, and lips, and the mobile phone can calculate the relative position coordinates of the non-highlight area relative to one or more of the eyes, nose, and lips.
- the relative location coordinates of a particular part can be calculated.
- the mobile phone calculates that the highlighted area in the ultraviolet image 901 is 408a and the non-highlighted area 407a.
- an image matching algorithm can be used to match corresponding regions in ultraviolet images from different angles.
- the mobile phone can acquire the target area 408b corresponding to the highlighted area 408a in the ultraviolet image 902 .
- the positional relationship between the highlighted area 408a and specific parts such as nose and eyes in the ultraviolet image 901 and the positional relationship between the target area 408b and specific parts such as nose and eyes in the ultraviolet image 902 are consistent.
- the mobile phone can acquire the target area 408c corresponding to the highlighted area 408a in the ultraviolet image 903 .
- step S203a Determine whether the target areas 408b, 408c include target non-highlight areas. If at least one target area includes a target non-highlight area, execute step S204a, and if none of the target areas includes a target non-highlight area, execute step S205a.
- the target area 408b of the ultraviolet image 902 includes a non-highlight area
- the highlighted area 408a in the ultraviolet image 901 probably includes a reflective area and an area without sunscreen.
- directly determining the highlighted area 408a as an area where no sunscreen is applied may lead to misjudgment.
- the following step S204a may be performed to remove a region that may cause a misjudgment from the highlighted region 408a, and the removed region may be replaced by a corresponding non-highlighted region.
- one or more target areas corresponding to the highlighted areas in the frontal ultraviolet image do not include non-highlighted area, that is, all the areas of the UV image at all angles are highlighted areas, which means that the highlighted area 408a in the UV image of the frontal face is indeed caused by not applying sunscreen. Then, the following areas can be directly executed without adjusting the area to be detected. Step S205a.
- the mobile phone detects whether it includes a non-highlighted area. After detection, the target area 408c includes a non-highlight area 411a.
- S204a according to the highlighted area 408a, the non-highlighted area 407a and the target non-highlighted area of the ultraviolet image 901, determine the area where sunscreen is applied and the area where sunscreen is not applied.
- the mobile phone can screen reflective parts according to ultraviolet images 901 , 902 , and 903 .
- the pixel value of the shaded area 407a is smaller than the pixel value threshold, and the pixel value of the area 408a is greater than the pixel value threshold.
- the mobile phone may preliminarily determine the part corresponding to the area 408a as the part where sunscreen may not be applied according to the above method, and preliminarily determine the area 407a as the area where the sunscreen is applied.
- the white pixels in the region 408a may be due to the fact that no sunscreen is applied to the part corresponding to the region 408a, or it may be because the part corresponding to the region 408a is not applied with sunscreen, and the other part is coated with sunscreen but reflects light.
- the mobile phone can be in the corresponding positions of the ultraviolet image 902 shown in Figure 13(b) and the ultraviolet image 903 shown in Figure 13(c), that is, in the areas 408b and 408c Find if there is a non-highlighted region of interest.
- the phone After searching, there is a target non-highlight area 410a in the area 408b, which indicates that the part corresponding to the target non-highlight area 410a is likely to be covered with sunscreen but reflects light at a frontal angle.
- the phone determines area 410a as the area to apply sunscreen.
- the mobile phone determines the area target non-highlight area 411a as the area where sunscreen is applied.
- the area where sunscreen is finally calculated by the mobile phone includes area 407a, area 410a and area 411a, and the area where no sunscreen is applied is the area in 408a except area 410a and area 411a.
- (d) of FIG. 13 shows areas where sunscreen is applied and areas that are not applied. It should be noted that (d) of FIG. 13 only exemplarily shows the area where the sunscreen is applied and the area that is not applied, and the mobile phone may not generate the image shown in (d) of FIG. 13 during actual processing.
- the mobile phone may display the prompt box 801 shown in FIG. 13 , wherein the area enclosed by the dotted frame is the area where sunscreen is applied, and the area outside the dotted frame is the area where no sunscreen is applied.
- S205a according to the non-highlight area 407a and the highlighted area 408a of the image 901, determine the area where the sunscreen is applied and the area where the sunscreen is not applied.
- the mobile phone preliminarily determines the candidate area (such as the area 408a shown in FIG. Areas not covered with sunscreen.
- the mobile phone after the mobile phone preliminarily determines the area 408a where no sunscreen is applied according to the ultraviolet image 901, it searches the ultraviolet image 902 and the ultraviolet image 903, if the corresponding position of the ultraviolet image 902 (area 408b) and the corresponding position of the ultraviolet image 903 There is no target non-highlight area in (408c), which means that there is no misjudgment problem caused by the reflection problem, and the area 408a is indeed an area where no sunscreen is applied. That is, the mobile phone finally determines that the area without sunscreen is 408a, and the area with sunscreen is 407a.
- the embodiment of the present application also provides a detection method, which can determine the area where sunscreen is applied and the area where sunscreen is not applied.
- the method includes:
- the mobile phone acquires the highlighted area and/or the non-highlighted area of the face image in the multiple ultraviolet images.
- the ultraviolet image may include highlighted regions and/or non-highlighted regions.
- the mobile phone can determine the non-highlight area in the ultraviolet image according to the highlighted area in the ultraviolet image, and similarly, the mobile phone can determine the non-highlight area according to the non-highlight area in the ultraviolet image.
- step S201b can be implemented as: the mobile phone acquires the highlighted area of the face image in each of the multiple ultraviolet images, and can determine the non-highlighted area according to the highlighted area. Or, the mobile phone acquires the non-highlight area of the face image in each ultraviolet image, and can determine the highlight area according to the non-highlight area. Or, the mobile phone obtains the non-highlight area and the high light area of the face image in each ultraviolet image.
- the ultraviolet images collected by the channel ultraviolet camera of the mobile phone include ultraviolet images 901-903.
- the mobile phone can respectively determine the face images 1-3 in the ultraviolet images 901-903 according to the method of the above-mentioned embodiment, and respectively determine the highlighted area and the non-highlighted area in the face images 1-3.
- FIG. 13 Exemplarily, in FIG.
- the highlighted area of face image 1 in ultraviolet image 901 is 408a
- the non-highlighted area is 407a
- the highlighted area of human face image 2 in ultraviolet image 902 is 408b
- the highlighted area of the face image in the ultraviolet image 903 is 408c
- the non-highlighted areas are 407c and 411a.
- the mobile phone determines the area where the sunscreen is applied and the area where the sunscreen is not applied according to the highlighted area and/or the non-highlighted area.
- the highlighted area of the face image 1 is 408a
- the non-highlighted area is 407a
- the highlighted area of the human face image 2 in the ultraviolet image 902 is 408b
- the non-highlighted areas are 410a and 407b
- the highlighted region of the face image in the ultraviolet image 903 is 408c
- the non-highlighted region is 407c and 411a
- the region 407a in the ultraviolet image corresponds to 407b in the ultraviolet image 902
- corresponds to 407c in the ultraviolet image 903 in the ultraviolet image
- the region 408a of corresponds to 408b in the ultraviolet image 902 and corresponds to 408c in the ultraviolet image 903 .
- the area 411a in the ultraviolet image 903 does not have a corresponding relationship with the area 410a in the ultraviolet image 902, and the non-highlight areas corresponding to the areas 410a and 411a do not exist in the ultraviolet image 901.
- the area where the mobile phone determines to apply sunscreen is shown in Figure 13 As shown in (d) 407a, 410a, 411a, the area without sunscreen is the area in 408a except 410a and 411a.
- the mobile phone can obtain the non-highlight region 407a in the ultraviolet image 901, the non-highlight regions 407b and 410a in the ultraviolet image 902, and the non-highlight regions 407c and 411a in the ultraviolet image 903, and according to these non-highlight regions Highlight areas, identify highlighted areas, and identify areas where sunscreen is applied and areas where sunscreen is not applied.
- the mobile phone can obtain the highlighted area 408a in the ultraviolet image 901, the highlighted area of the ultraviolet image 902, that is, the areas in the human face other than the eyes, eyebrows, mouth, 407b, and 410a, and the highlighted area in the ultraviolet image 903.
- Highlighted areas that is, areas other than the eyes, eyebrows, mouth, 407c, and 411a of the face, and based on these highlighted areas, determine the non-highlighted areas, and determine the area where sunscreen is applied and the area where sunscreen is not applied .
- the embodiment of the present application also provides a detection method, which can be used to detect the application uniformity of the sunscreen. As a possible implementation, see Figure 14.
- a detection method which can be used to detect the application uniformity of the sunscreen.
- Figure 14 see Figure 14.
- multiple image features can be extracted from the area to be detected, and the feature vector composed of multiple image features can be input into the classifier, and the classifier will output the sun protection of the corresponding parts Evenness of cream application.
- the area to be detected may be an area where sunscreen is applied.
- the manner of determining the application area of sunscreen may refer to corresponding embodiments such as FIG. 12A or FIG. 12B , which will not be repeated here.
- the feature vector includes a plurality of image features.
- N is a positive integer
- the samples are the feature vectors and their labels of image regions whose application uniformity is known ( Characterize whether the corresponding part of the image area is evenly coated with sunscreen), and train multiple samples to obtain a classifier.
- data such as training feature vectors may be processed, such as smoothing and normalization.
- the normalization process can reduce the complexity of the algorithm. Smoothing can further include operations such as noise reduction and fitting to reduce the impact of statistical errors.
- the classifier may be evaluated and tested.
- the recognition rate of the classifier reaches a certain threshold, it indicates that the classifier has been trained.
- the recognition rate of the classifier is low, the classifier can continue to be trained until the recognition accuracy of the classifier reaches a certain threshold.
- the training process of the classifier can be performed on the device side (such as a terminal such as a mobile phone) or on the cloud side (such as a server). Training can be offline training or online training. The embodiment of the present application does not limit the specific training method of the classifier. Subsequently, the trained classifier can output whether the sunscreen in the corresponding image area is evenly applied according to the input feature vector of the image area whose application uniformity is unknown.
- Fig. 16 shows the flow of the method for identifying the uniformity of sunscreen application based on the classifier. As shown in Figure 16, the method includes:
- the area to be detected is the area where sunscreen is applied.
- the specific implementation of determining the area to apply sunscreen can refer to the corresponding embodiments such as FIG. 12A or FIG. 12B , which will not be repeated here.
- the mobile phone divides the area to be detected (that is, the area where sunscreen is applied) into blocks, and the multiple image areas obtained are respectively Area 1-Area 3 shown in (b) of Figure 17, and respectively Extract feature vectors for regions 1-3.
- the eigenvector of region 1 is The eigenvectors of region 2 are The eigenvectors of region 3 are Among them, f ji is the ith (i is a positive integer) image feature of region j (j is 1, 2, 3).
- the three eigenvectors can be combined into the following eigenmatrix:
- the boxes not marked with oblique lines that is, the white boxes represent the pixels where no sunscreen is applied.
- Boxes marked with slashes represent pixels where sunscreen was applied.
- the density of the slanted lines can represent the high or low pixel value. In one example, the higher the slash density, the smaller the pixel value.
- the pixel value is used to represent the thickness of the smear.
- the mobile phone inputs the feature vectors of image region 1-region 3 into the classifier, that is, The three feature vectors are input into the classifier respectively, or the three feature vectors are combined to form a feature matrix, and the feature matrix is input into the classifier.
- the classifier detects whether the sunscreen application in the image area 1-area 3 is uniform according to the input feature vector.
- the classifier outputs the classification results of region 1-region 3: region 1 ⁇ ->1, region 2 ⁇ ->0, region 3 ⁇ ->0.
- 1 means that the sunscreen is applied unevenly
- 0 means that the sunscreen is applied evenly. That is, the application of the sunscreen in the image area 1 is uneven, and the application of the sunscreen in the image areas 2 and 3 is uniform.
- step S303 Determine whether the area of each area divided into the area to be detected is smaller than a threshold. If yes, execute step S304. If not, select the image area whose classification result is that the sunscreen is not evenly applied from the multiple areas divided into the area to be detected, update this part of the image area as the area to be detected, and jump to step S301.
- a threshold can be set.
- T3 the area of the image area divided into the area to be detected
- the detection granularity requirement of skin sunscreen application uniformity is met, and no further division of smaller image areas is required.
- step S304 it is necessary to divide the larger image area to form a smaller image area, and to perform smear uniformity detection on the smaller image area until The size and granularity of the image area detected by the smear uniformity meet the threshold, and step S304 is executed.
- the mobile phone can jump to step S301 to divide the area to be detected (area 1) into multiple areas. As shown in FIG. 18, area 1 is divided into area 4-area 6. Moreover, the mobile phone can extract feature vectors of multiple regions 4-6, for example, The eigenvectors of region 4-region 6 can also be combined into the following eigenmatrix:
- iter is a preset coefficient
- M is the number of n+1 subdivisions
- N is the number of nth subdivisions
- iter, M, and N are all positive integers
- n is an integer greater than or equal to 1.
- the mobile phone may divide the image area in other manners, which are not limited in this embodiment of the present application.
- the area of the image area included in the area to be detected may be different.
- the area to be detected includes area 1-area 3 shown in Figure 17, and area 1 may further include area 4-area 6 shown in Figure 18.
- area 4 is an area where sunscreen is applied unevenly, and areas where sunscreen is evenly applied are areas 2, 35, and 6.
- the mobile phone calculates the pixel mean value of each image area included in the area to be detected, and judges the thickness of the sunscreen application in each image area according to the pixel mean value. By detecting the average value of pixels in the uniform area and the non-uniform area ⁇ F.
- the sunscreen application thickness can be divided into several levels according to the pixel mean value. For example, it is divided into several pixel value intervals [0, T4), [T4, T5), [T5, T6), [T6, T7), [T7, 255], if the pixel mean value ⁇ F ⁇ threshold T4 , it is determined that the sunscreen in this area is very thick (too thick).
- T4 ⁇ F ⁇ threshold T5 it is determined that the thickness of the sunscreen is moderate. If T5 ⁇ F ⁇ threshold T6, it is determined that the sunscreen is thin. If T7 > ⁇ F ⁇ T6, it is determined that the sunscreen is very thin (too thin), and if ⁇ F>T7, it is determined that no sunscreen is applied.
- the mobile phone calculates the average pixel value of each image area in area 2, area 3, and area 4-area 6 shown in Figure 17, assuming that the pixel average values of areas 2, 3, 5, and 6 are T4 ⁇ F ⁇ Threshold T5, it is determined that the thickness of the sunscreen in areas 2, 3, 5, and 6 is moderate, and the average pixel value of area 4 is F ⁇ T4, then the mobile phone can determine that the thickness of the sunscreen in this area is relatively thick. It can be seen that from the sunscreen application uniformity detection process, the mobile phone can determine the uneven application of sunscreen, and can also determine the thickness of the uneven application of sunscreen. In some embodiments, the mobile phone can also determine other information about sunscreen application through the detection process of uniformity of sunscreen application, such as the location where no sunscreen is applied, the thickness of the application where the sunscreen is evenly applied, and the like.
- the mobile phone After the mobile phone determines the application of sunscreen, it can prompt the user for the application of sunscreen. Including, but not limited to, indicating the position where sunscreen is not applied, indicating the position where the sunscreen is applied unevenly, and indicating the thickness of the sunscreen in the designated area.
- the prompt information can be marked in the color image, and different application conditions can be marked with different colors and interface effects.
- Interface effects include, but are not limited to, flickering effects, glowing effects, and shadow effects. For example, if a certain area is an area where sunscreen is applied unevenly, and sub-area A of this area is an area where the application thickness is thin, then sub-area A is marked by the mobile phone with a red and flashing effect. In this way, the user's attention Concentrate on areas where sunscreen is thinner and reapply sunscreen to that area.
- some interface effects can also be used to distinguish the application of sunscreen in the grayscale image.
- the grayscale image there are all black pixel areas.
- the black pixel area 1 marks the flickering effect.
- the flickering effect can indicate that the sunscreen is applied thinly.
- the black pixel area 2 is not marked with the flickering effect.
- the unmarked flickering effect can indicate that the sunscreen is applied with a moderate thickness .
- pixel value intervals may be further divided according to actual application scenarios, and there is no limit to the thickness of the sunscreen represented by each pixel value interval.
- the pixel value is between 0 and 255 as an example. In other embodiments, when more or less bits are used to represent the color, the pixel value can also have other values The scope is unified here.
- the colors and interface effects of different regions can be preset by the system or set by the user.
- the mobile phone can explain the meaning of the application of sunscreen represented by each color and interface effect in the form of text on the prompt interface.
- the mobile phone explains the meaning of the application of sunscreen represented by each color and interface effect in text or other forms.
- the embodiment of the present application also provides a detection method, which determines the third area on the sixth image, and according to the first area and the third area in the above-mentioned embodiment, determines the difference between the first face and the first face in the first image. area and the second area corresponding to the third area.
- the first image and the sixth image are images from different angles.
- the third area is any one or more of the following areas: an area where sunscreen is applied, an area where sunscreen is not applied, an area where sunscreen is applied unevenly, an area where the thickness of sunscreen is greater than the first threshold, and where sunscreen is applied Regions with a thickness less than the second threshold.
- the first image is an ultraviolet image 901
- the sixth image is an image 902.
- the first area determined by the mobile phone on the ultraviolet image 901 is the non-highlight area 407a
- the mobile phone is on the ultraviolet image 902
- the determined third area is the non-highlight area 407b and 410a
- the mobile phone can determine the second area corresponding to 407b, 410a, and 407a on the ultraviolet image 901 according to the first area and the third area, that is, 407a and 410a. Therefore, the mobile phone can determine that the areas where the sunscreen is applied are 407a and 410a.
- the mobile phone can also collect multiple images from different angles, for example, collect the ultraviolet images 901-903 shown in FIG. 13 , and the mobile phone can determine the application of sunscreen according to the areas in the three ultraviolet images.
- the embodiment of the present application also provides a detection method, the mobile phone can obtain the current weather state, including but not limited to obtain the current ultraviolet intensity of the environment. In this way, the mobile phone can recommend information about sunscreen suitable for the current weather state to the user according to the current antenna state. Including but not limited to prompting the user with the sun protection factor (sun protection factor, SPF) of the sunscreen suitable for the current weather state, and the thickness of the sunscreen.
- SPF sun protection factor
- the intensity of ultraviolet rays can be divided into 5 levels (of course, it can be divided into more or less levels under different grading methods).
- the higher the level of ultraviolet intensity the stronger the ultraviolet rays, and users need to take stronger sun protection measures.
- the mobile phone obtains that the current ultraviolet intensity is strong, it can guide the user to protect himself from the sun by one or more of the following methods: recommending a sunscreen with a higher SPF to the user, and prompting the user to apply thicker sunscreen.
- a sunscreen with a higher SPF usually has a stronger ability to absorb ultraviolet rays and is more irritating to the skin. Therefore, when the mobile phone recommends sunscreen to users, it does not recommend the sunscreen with the highest SPF to users. , but choose the appropriate sunscreen according to the outdoor ultraviolet intensity, so as to avoid recommending sunscreens with too high SPF to irritate the user's skin.
- the mobile phone can also detect the user's skin information, such as skin type information, whether it is sensitive skin, oily skin, mixed skin, etc.
- the mobile phone can recommend sunscreens suitable for the user's skin type to the user based on the user's skin information and other information (including but not limited to weather status information), such as recommending a suitable SPF and application thickness.
- the mobile phone obtains the weather status information from the cloud side (such as a server), or the mobile phone detects the weather status information by itself.
- the cloud side such as a server
- the mobile phone can detect the current location information, and when it is currently indoors, the mobile phone can obtain the ultraviolet intensity from the server. When currently outdoors, the mobile phone can capture ultraviolet rays through the image sensor in the ultraviolet camera and judge the intensity of ultraviolet rays. For example, in the process of collecting ultraviolet images, the mobile phone judges the intensity of ultraviolet rays according to the ultraviolet rays captured by the ultraviolet camera.
- the mobile phone can call such as color camera Detect faces. After detecting a human face, the mobile phone can call the ultraviolet camera to collect ultraviolet images, and detect the application of sunscreen on the user's face according to the ultraviolet images. During this process, the mobile phone can determine the ambient ultraviolet intensity based on the ultraviolet rays captured by the ultraviolet camera.
- the mobile phone can prompt the user with information about sunscreen application through the user interface (UI) according to the intensity of the ambient ultraviolet light and the application of sunscreen on the user's face, including but not limited to one or more of the following information:
- UI user interface
- the user it is recommended that the user use the SPF of the sunscreen, whether the sunscreen is applied evenly on the face, where the sunscreen is applied unevenly on the face, and where the sunscreen is not applied on the face.
- a prompt box 2401 is displayed, which is used to prompt the user for the current ultraviolet intensity, the position where sunscreen needs to be reapplied, and the User prompts for desired SPF.
- a prompt box is displayed to remind the user of the current ultraviolet intensity and to remind the user that the sunscreen is applied evenly on the face.
- the embodiment of the present application also provides a detection method.
- the mobile phone can detect whether there is a previous detection result within a preset period of time. If the detection has not been performed within a preset period of time, the mobile phone can detect the application of sunscreen and prompt the user for the application of sunscreen. If the test has been performed within a preset period of time, the phone can prompt the user with the result of the most recent sunscreen application test. That is to say, not in all cases, the mobile phone will re-detect, so that the calculation load of the mobile phone can be reduced.
- the preset time period may be related to the sunscreen action time of the sunscreen and the like. For example, assuming that the statistics show that sunscreen usually works for 3-4 hours, then there is usually no need to reapply sunscreen within 3-4 hours.
- the mobile phone after the mobile phone detects the user's operation such as clicking on the sunscreen detection control 402, if the mobile phone detects that the sunscreen detection process has been performed within 3 hours, it prompts the user that there is no need to reapply.
- the mobile phone can also present the last detection result of the sunscreen to the user. Conversely, if the mobile phone detects that the sunscreen detection process has not been performed within 3 hours, it will prompt the user with the detection result.
- the mobile phone can determine whether the detection is performed in the morning, if the detection is not performed in the morning, the detection is performed, and the detection result is prompted to the user. If the mobile phone detects that the detection has been carried out in the morning, and the sunscreen has been applied or reapplied in place, then the current time information will be obtained, and it will be judged whether the current time belongs to the preset time period for sunscreen detection (for example, 12:00-14:00 ). If the current time is not between 12:00-14:00, it means that the sunscreen applied in the morning has not expired, and there is no need to re-test. If the current time is between 12:00-14:00, it means that the sunscreen applied in the morning may be invalid. Then, the mobile phone will re-test and prompt the user with the result of this test.
- the preset time period for sunscreen detection for example, 12:00-14:00
- a shortcut icon can also be set separately for the sun protection detection function in applications such as mirrors.
- the mobile phone After detecting the user's operation such as clicking the shortcut icon, the mobile phone can perform detection.
- the mobile phone may prompt the user that sunscreen application is being detected. After the mobile phone detection is completed, the user can be prompted for the detection result. It can be seen that in this scenario, the operation that triggers the mobile phone to start the sunscreen detection process opens the shortcut application of sunscreen detection for the user.
- the user can also perform an operation such as long pressing the mirror application to trigger the mobile phone to pop up the mirror function box 2801 .
- the user can enable the sunscreen detection function by, for example, clicking on the sunscreen detection option, then the mobile phone will detect the application of sunscreen, and can prompt the user for the application of sunscreen.
- detecting whether to apply sunscreen can also be implemented as: the mobile phone inputs the face image into a classifier for detecting whether to apply sunscreen, and then the mobile phone determines the area where sunscreen is applied and the area where sunscreen is not applied according to the classification result of the classifier area with sunscreen.
- the application of sunscreen in the embodiment of the present application can be determined according to the recognition result of the classifier. For the training and use process of the classifier used to detect whether to apply sunscreen, refer to the corresponding embodiment in FIG. 15 , which will not be repeated here.
- the mobile phone after the mobile phone prompts the user about the application of sunscreen, if the user has not exited the mirror application, the mobile phone detects that the user is still using the mirror application for reapplying through the RGB camera, and the mobile phone continues to detect the user's sunscreen application.
- Cream touch-up condition For example, the mobile phone can turn on the ultraviolet camera to collect face images at a certain time interval, so as to detect the user's reapplying of sunscreen until the user's reapplied sunscreen is sufficient to resist the ultraviolet rays of the current environment.
- the mobile phone may not actively detect the user's application of sunscreen, but trigger the detection process again after detecting the user's operation such as clicking the sunscreen detection control.
- the interfaces listed in the embodiments of the present application are exemplary, and the embodiments of the present application do not limit the implementation forms of the interfaces.
- both the color view and the grayscale view can be provided.
- the mobile phone may stop displaying the sun protection detection control 402, so that the screen has more display space to display other necessary controls.
- the above-mentioned embodiments mainly use face detection as an example, but it should be understood that the detection method in the embodiment of the present application may also be applicable to various other objects requiring sun protection. Examples include, but are not limited to, human faces and other parts of the body. Alternatively, it can also be applied to non-human objects, for example, parts of a vehicle that need sun protection.
- the above embodiment mainly uses the RGB image to identify the human face image, and then searches for the human face image in the ultraviolet image according to the human face image in the RGB image as an example to describe the human face image recognition process.
- the ultraviolet image can also be directly processed using the face recognition algorithm, so as to recognize the human face image in the ultraviolet image and extract feature points such as eyes and lips.
- the embodiment of the present application does not limit the specific implementation manner of acquiring the human face image in the ultraviolet image.
- the above-mentioned embodiment mainly uses three ultraviolet images from different angles to exclude the influence of reflective parts on sunscreen detection as an example. for other.
- the color camera and the ultraviolet camera collect synchronously, which may be collected at the same time, or collected within a preset time interval, may be collected first by the ultraviolet camera, and collected by the color camera within a preset time interval, or, The color camera collects first, and within a preset time interval, the ultraviolet camera collects.
- the embodiment of the present application does not limit the collection timing and collection conditions of the two.
- the embodiment of the present application mainly uses the determination of the application of sunscreen based on the pixel value of the ultraviolet image as an example for illustration, and the application of sunscreen may also be determined based on other image features of the ultraviolet image. For example, based on the variance, standard deviation, maximum pixel value, and minimum pixel value in the image features, the uniformity and thickness of sunscreen application can be determined.
- the apparatus may be the above-mentioned electronic device (such as a folding screen mobile phone).
- the apparatus may include: a display screen, memory and one or more processors.
- the display screen, memory and processor are coupled.
- the memory is used to store computer program code comprising computer instructions.
- the processor executes the computer instructions, the electronic device can execute various functions or steps performed by the mobile phone in the foregoing method embodiments.
- FIGS. 2A-2F For the structure of the electronic device, reference may be made to the electronic device shown in FIGS. 2A-2F .
- the core structure of the electronic device may be represented as the structure shown in FIG. 21 , and the core structure may include: a processing module 1301 , an input module 1302 , a storage module 1303 , and a display module 1304 .
- the processing module 1301 may include at least one of a central processing unit (CPU), an application processor (Application Processor, AP) or a communication processor (Communication Processor, CP).
- the processing module 1301 may perform operations or data processing related to control and/or communication of at least one of other elements of the user electronic device.
- the processing module 1301 can be configured to control the content displayed on the main screen according to a certain trigger condition. Or determine what is displayed on the screen according to preset rules.
- the processing module 1301 is also used to process the input instruction or data, and determine the display style according to the processed data.
- the input module 1302 is configured to obtain instructions or data input by the user, and transmit the obtained instructions or data to other modules of the electronic device.
- the input mode of the input module 1302 may include touch, gesture, screen approach, etc., and may also be voice input.
- the input module may be a screen of an electronic device, acquires user input operations, generates input signals according to the acquired input operations, and transmits the input signals to the processing module 1301 .
- the input module may be used to receive a sunscreen detection instruction input by a user, and/or perform other steps.
- the collection module 1306 is configured to collect data and transmit the collected data to other modules of the electronic device.
- the collection module 1306 may be a camera of the electronic device, and the camera may transmit the collected images to the processing module 1301, and/or perform other steps.
- Cameras include but are not limited to color cameras and ultraviolet cameras.
- the storage module 1303 may include a volatile memory and/or a nonvolatile memory.
- the storage module is used to store at least one instruction or data related to other modules of the user terminal device. Specifically, the storage module can record images collected by the camera.
- the display module 1304 may include, for example, a Liquid Crystal Display (LCD), a Light Emitting Diode (LED) display, an Organic Light Emitting Diode (OLED) display, a Micro Electro Mechanical System (MEMS) display or an electronic paper display. Used to display user-viewable content (eg, text, images, videos, icons, symbols, etc.). In the embodiment of the present application, the display module may be implemented as a display screen.
- LCD Liquid Crystal Display
- LED Light Emitting Diode
- OLED Organic Light Emitting Diode
- MEMS Micro Electro Mechanical System
- the structure shown in FIG. 21 can also have a communication module 1305, which is used to support the electronic device to communicate with other electronic devices.
- the communication module can be connected to a network via wireless communication or wired communication to communicate with other personal terminals or a network server.
- the wireless communication may employ at least one of cellular communication protocols, such as Long Term Evolution (LTE), Long Term Evolution-Advanced (LTE-A), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Universal Mobile Communications System (UMTS), Wireless Broadband (WiBro), or Global System for Mobile Communications (GSM).
- Wireless communications may include, for example, short-range communications.
- the short-range communication may include at least one of wireless fidelity (Wi-Fi), Bluetooth, near field communication (NFC), magnetic stripe transmission (MST), or GNSS.
- the embodiment of the present application also provides a chip system, as shown in FIG. 22 , the chip system includes at least one processor 1401 and at least one interface circuit 1402 .
- the processor 1401 and the interface circuit 1402 may be interconnected through wires.
- interface circuit 1402 may be used to receive signals from other devices, such as memory of an electronic device.
- the interface circuit 1402 may be used to send signals to other devices (such as the processor 1401).
- the interface circuit 1402 can read instructions stored in the memory, and send the instructions to the processor 1401 .
- the electronic device may be made to execute various steps in the foregoing embodiments.
- the chip system may also include other discrete devices, which is not specifically limited in this embodiment of the present application.
- the embodiment of the present application also provides a computer storage medium, the computer storage medium includes computer instructions, and when the computer instructions are run on the above-mentioned electronic device, the electronic device is made to perform the various functions or steps performed by the mobile phone in the above-mentioned method embodiment .
- the embodiment of the present application also provides a computer program product, which, when running on a computer, causes the computer to execute the various functions or steps performed by the mobile phone in the above method embodiments.
- the disclosed devices and methods can be implemented in other ways.
- the device embodiments described above are only illustrative.
- the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods.
- multiple units or components can be Incorporation or may be integrated into another device, or some features may be omitted, or not implemented.
- the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.
- the unit described as a separate component may or may not be physically separated, and the component displayed as a unit may be one physical unit or multiple physical units, that is, it may be located in one place, or may be distributed to multiple different places . Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.
- each functional unit in each embodiment of the present application may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit.
- the above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.
- the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a readable storage medium.
- the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the software product is stored in a storage medium Among them, several instructions are included to make a device (which may be a single-chip microcomputer, a chip, etc.) or a processor (processor) execute all or part of the steps of the methods described in the various embodiments of the present application.
- the aforementioned storage medium includes: various media that can store program codes such as U disk, mobile hard disk, read only memory (ROM), random access memory (random access memory, RAM), magnetic disk or optical disk.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Multimedia (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Databases & Information Systems (AREA)
- Evolutionary Computation (AREA)
- Computing Systems (AREA)
- Medical Informatics (AREA)
- Software Systems (AREA)
- Artificial Intelligence (AREA)
- Oral & Maxillofacial Surgery (AREA)
- Human Computer Interaction (AREA)
- Image Analysis (AREA)
Abstract
本申请提供检测方法及电子设备;涉及终端技术领域,电子设备具有紫外摄像头,通过紫外摄像头和RGB摄像头分别采集紫外图像和彩色图像,并将识别的紫外图像中的第一区域匹配到对应的彩色图像中,提高用户的使用体验;此外,通过采集不同角度的紫外图像进行防晒霜涂抹信息的联合分析,提高检测的精准度,能够对用户防晒进行精细化指导。
Description
本申请要求于2021年07月31日提交国家知识产权局、申请号为202110877247.1、发明名称为“检测方法及电子设备”的中国专利申请的优先权,其全部内容通过引用结合在本申请中。
本申请涉及电子技术领域,尤其涉及检测方法及电子设备。
目前,更多用户热衷于皮肤问题的预防。皮肤问题可包括黑头、色斑、皱纹等。其中,有许多皮肤问题可由紫外线照射产生。这种情况下,采取一些物理或化学方式降低紫外线的伤害就显得尤为重要。在化学防晒措施中,用户可以在需要防晒的部位涂抹防晒霜,如此可以吸收一部分紫外线,减轻紫外线对涂抹防晒霜部位的伤害。
但在一些场景中,用户经常忘记涂抹防晒霜,或者即便用户涂抹防晒霜,也可能涂抹的并不均匀,用户仍会暴露在强紫外线下的照射下,用户的一系列皮肤问题仍没有得到有效的解决方法。为此,本申请实施例提供一种检测方法。该检测方法可以但不限于检测防晒霜涂抹情况。
发明内容
本申请提供的检测方法及电子设备,电子设备具有紫外摄像头,通过紫外摄像头和RGB摄像头分别采集紫外图像和彩色图像,并将识别的紫外图像中的第一区域匹配到对应的彩色图像中,提高用户的使用体验。
为达到上述目的,本申请采用如下技术方案:
第一方面,本申请提供一种检测方法,该方法应用于电子设备或能够实现或匹配实现电子设备功能的装置(比如芯片),以电子设备为例,电子设备包括第一摄像头和第二摄像头;其中,第一摄像头为彩色摄像头,第二摄像头为紫外摄像头,方法包括:
通过第一摄像头采集第一图像,通过第二摄像头采集第二图像,其中,彩色图像包括第一内容。
其中,第一内容可以是彩色图像采集的待检测对象。待检测对象是待检测防晒霜涂抹情况的对象。待检测对象包括人体部位,比如脸部、手、胳膊等需要进行防晒的部位。待检测对象还可包括其他对象,比如,需要隔离紫外线进行防晒的车辆等物体。
基于紫外摄像头能够捕捉紫外线信息,且防晒霜能够吸收紫外线的特性,电子设备可根据紫外图像记录的紫外线信息能够确定防晒霜的涂抹情况,以便在紫外图像上确定防晒霜涂抹情况相关的区域,比如第一区域。之后,电子设备在彩色图像上确定与第一区域对应的第二区域,并显示用户界面,用户界面包括第一内容和用于指示第二区域的第一信息,即在彩色图像中标记防晒霜涂抹情况相关的第二区域。如此,用户可通过彩色图像观察防晒霜涂抹情况相关的第二区域。由于彩色图像成像细节更丰富,用户可根据彩色图像中标记的第二区域清晰准确的判断防晒霜涂抹情况,提高用 户的使用体验。并且,用户能够根据防晒霜涂抹情况采取防晒措施,比如补涂防晒霜。可见,通过本申请实施例的技术方案,能够对用户防晒进行精细化指导,降低用户被晒伤的概率。
在第一方面的一种可能的设计中,第一内容包含第一人脸;第二图像包含第二人脸,第一人脸和第二人脸属于同一用户;
在第二图像上确定第一区域,具体包括:
在第二图像的第二人脸上,确定第一区域;
在第一图像上确定与第一区域对应的第二区域,具体包括:
在第一图像的第一人脸上,确定与第一区域对应的第二区域。
如此一来,能够在紫外图像的人脸上确定防晒霜涂抹相关的区域,并将该区域匹配到彩色图像的人脸上的相应区域,使得用户能够根据彩色图像的人脸上的相应区域了解防晒霜涂抹情况,视觉效果更加直观,便于用户理解。
在第一方面的一种可能的设计中,第一区域为如下任一种或多种区域:涂抹防晒霜的区域,未涂抹防晒霜的区域,防晒霜涂抹不均匀的区域,防晒霜涂抹厚度大于第一阈值的区域,防晒霜涂抹厚度小于第二阈值的区域。
也就是说,电子设备能够精准检测到紫外图像的上述一种或多种区域,如此,能够提供给用户丰富的防晒提示,对用户防晒进行精细化指导。
在第一方面的一种可能的设计中,在通过第一摄像头采集第一图像,通过第二摄像头采集第二图像后,方法还包括:
通过第一摄像头采集第三图像,当第三图像满足第二条件时,通过第二摄像头采集第四图像;
确定第四图像上的第三区域;
根据第一区域,在第一图像上确定与第一区域对应的第二区域,包括:
根据第一区域和第三区域,在第一图像上确定与第一区域和第三区域对应的第二区域;
第二条件,包括:第三图像包括人脸,且第三图像包括的人脸相对于电子设备的角度在预设角度范围内。
该方案中,通过采集不同角度的紫外图像进行防晒霜涂抹信息的联合分析,能够提高检测的精准度,能够对用户防晒进行精细化指导。
在第一方面的一种可能的设计中,通过第一摄像头采集第一图像,通过第二摄像头采集第二图像,包括:在同一时刻,通过第一摄像头采集第一图像以及通过第二摄像头采集第二图像。
其中,紫外图像可用于检测防晒霜涂抹情况。彩色图像可用于检测人脸,还可用于识别人脸相对于电子设备的角度。
由于同一时刻,用户的脸部相对于电子设备的角度固定,因此,第一摄像头与第二摄像头同步采集图像,可以使得第一摄像头和第二摄像头所采集图像中人脸相对于电子设备的角度相同。这样一来,电子设备可以根据彩色图像中人脸相对于电子设备的角度推知紫外图像中人脸相对于电子设备的角度,以便确定紫外图像的可用性。
在第一方面的一种可能的设计中,通过第一摄像头采集第一图像,通过第二摄像头采集第二图像,包括:
通过第一摄像头采集第五图像;
当第五图像满足第一条件时,通过第一摄像头采集第一图像以及通过第二摄像头采集第二图像;
第一条件,包括:第五图像包括人脸。
在该检测方案中,电子设备先通过彩色摄像头采集彩色图像,在确定彩色图像中包括人脸,即存在待检测防晒霜涂抹情况的对象时,电子设备再通过紫外摄像头采集紫外图像(用于检测防晒霜涂抹情况),以及通过彩色摄像头采集彩色图像,以防止较早开启紫外摄像头导致的功耗问题。
在第一方面的一种可能的设计中,电子设备还包括紫外补光灯,通过第二摄像头采集第二图像,包括:打开紫外补光灯,通过第二摄像头采集第二图像。此种联动设置方式中,由于能够联动开启紫外摄像头和紫外补光灯,进而可使得采集紫外图像的实现复杂度较低。
在第一方面的一种可能的设计中,方法还包括:
确定环境紫外线强度;
通过第二摄像头采集第二图像,包括:
在环境紫外线强度小于第三阈值时,打开紫外补光灯,通过第二摄像头采集第二图像。
也就是说,只有在环境紫外线强度较小时,才开启紫外补光灯,因此,能够降低紫外补光灯对人体的伤害。
在第一方面的一种可能的设计中,在通过第一摄像头采集第一图像,通过第二摄像头采集第二图像后,以及通过第二摄像头采集第四图像之前,方法还包括:显示第二信息,第二信息用于提示用户调整人脸相对于电子设备的角度。通过向用户提示调整角度,使得电子设备能够采集多个角度的紫外图像,进而能够通过不同角度的紫外图像进行防晒霜涂抹信息的联合分析,提高检测的精准度。
在第一方面的一种可能的设计中,在第二图像的第二人脸上,确定第一区域,包括:
在第二图像的第二人脸上确定第四区域;
根据第四区域包含的像素的像素值,在第二图像的第四区域中确定第一区域。
在第一方面的一种可能的设计中,根据第四区域包含的像素的像素值,在第二图像的第四区域中确定第一区域,包括:
根据第四区域包含的像素的像素值,在第四区域中确定多个目标子区域,将多个目标子区域中面积大于或等于第四阈值的目标子区域确定为第一区域,目标子区域满足第三条件,第三条件包括:目标子区域的像素均值小于第五阈值,目标子区域包含的像素的像素值小于第六阈值。
采用该方案,手机可以在候选的防晒霜涂抹区域中排除黑痣等小面积区域,从而降低黑痣等部位被误判为涂抹防晒霜区域的概率,提高检测准确度。
在第一方面的一种可能的设计中,在第二图像上确定第一区域之前,方法还包括:
根据第一图像中的第一人脸,确定第二图像中的第二人脸。
该方案中,考虑到彩色图像比紫外图像的成像细节更丰富,画质更高。因此,手机可以在彩色图像中准确的识别人脸,并将识别出的人脸相应区域匹配到紫外图像中。
在第一方面的一种可能的设计中,第三区域为如下任一种或多种区域:涂抹防晒霜的区域,未涂抹防晒霜的区域,防晒霜涂抹不均匀的区域,防晒霜涂抹厚度大于第一阈值的区域,防晒霜涂抹厚度小于第二阈值的区域。
在第一方面的一种可能的设计中,第四区域不包括人脸中的眼睛、眉毛、嘴巴对应的区域。第四区域通常是无需涂抹防晒霜的区域。该方案中,从紫外图像的人脸中确定第四区域,并在第四区域中确定第一区域。由于能够不计算眼睛、眉毛、嘴巴等区域的像素值,因此能够降低计算量,提高电子设备的检测效率。
在第一方面的一种可能的设计中,第一信息还用于指示第二区域的防晒霜涂抹均匀程度和/或防晒霜涂抹薄厚程度和/或是否涂抹防晒霜。如此,用户能够通过电子设备显示的第一信息,获悉准确的防晒霜涂抹情况,便于防晒指导。
第二方面,提供一种电子设备,电子设备包括处理器、显示屏、第一摄像头和第二摄像头;其中,第一摄像头为彩色摄像头,第二摄像头为紫外摄像头;
第一摄像头,用于采集第一图像;
第二摄像头,用于采集第二图像,其中,第一图像包含第一内容;
处理器,用于在第二图像上确定第一区域;根据第一区域,在第一图像上确定与第一区域对应的第二区域;
显示屏,用于显示用户界面,用户界面包括第一内容和第一信息,第一信息用于指示第二区域。
在第二方面的一种可能的设计中,第一内容包含第一人脸;第二图像包含第二人脸,第一人脸和第二人脸属于同一用户;
在第二图像上确定第一区域,具体包括:
在第二图像的第二人脸上,确定第一区域;
在第一图像上确定与第一区域对应的第二区域,具体包括:
在第一图像的第一人脸上,确定与第一区域对应的第二区域。
在第二方面的一种可能的设计中,第一区域为如下任一种或多种区域:涂抹防晒霜的区域,未涂抹防晒霜的区域,防晒霜涂抹不均匀的区域,防晒霜涂抹厚度大于第一阈值的区域,防晒霜涂抹厚度小于第二阈值的区域。
在第二方面的一种可能的设计中,第一摄像头,还用于采集第三图像,第二摄像头,用于当第三图像满足第二条件时,采集第四图像;
处理器,用于确定第四图像上的第三区域;
根据第一区域,在第一图像上确定与第一区域对应的第二区域,包括:
根据第一区域和第三区域,在第一图像上确定与第一区域和第三区域对应的第二区域;
第二条件,包括:第三图像包括人脸,且第三图像包括的人脸相对于电子设备的 角度在预设角度范围内。
在第二方面的一种可能的设计中,第一摄像头,用于采集第一图像,第二摄像头,用于采集第二图像,包括:在同一时刻,第一摄像头采集第一图像以及第二摄像头采集第二图像。
在第二方面的一种可能的设计中,第一摄像头,用于采集第一图像,第二摄像头,用于采集第二图像,包括:
第一摄像头,用于采集第五图像;当第五图像满足第一条件时,采集第一图像,以及第二摄像头,用于采集第二图像;
第一条件,包括:第五图像包括人脸。
在第二方面的一种可能的设计中,电子设备还包括紫外补光灯,第二摄像头,用于采集第二图像,包括:打开紫外补光灯,第二摄像头采集第二图像。
在第二方面的一种可能的设计中,处理器,还用于确定环境紫外线强度;
第二摄像头,用于采集第二图像,包括:
在环境紫外线强度小于第三阈值时,打开紫外补光灯,第二摄像头采集第二图像。
在第二方面的一种可能的设计中,显示屏,还用于显示第二信息,第二信息用于提示用户调整人脸相对于电子设备的角度。
在第二方面的一种可能的设计中,在第二图像的第二人脸上,确定第一区域,包括:
在第二图像的第二人脸上确定第四区域;
根据第四区域包含的像素的像素值,在第二图像的第四区域中确定第一区域。
在第二方面的一种可能的设计中,根据第四区域包含的像素的像素值,在第二图像的第四区域中确定第一区域,包括:
根据第四区域包含的像素的像素值,在第四区域中确定多个目标子区域,将多个目标子区域中面积大于或等于第四阈值的目标子区域确定为第一区域,目标子区域满足第三条件,第三条件包括:目标子区域的像素均值小于第五阈值,目标子区域包含的像素的像素值小于第六阈值。
在第二方面的一种可能的设计中,处理器,还用于根据第一图像中的第一人脸,确定第二图像中的第二人脸。
在第二方面的一种可能的设计中,第三区域为如下任一种或多种区域:涂抹防晒霜的区域,未涂抹防晒霜的区域,防晒霜涂抹不均匀的区域,防晒霜涂抹厚度大于第一阈值的区域,防晒霜涂抹厚度小于第二阈值的区域。
在第二方面的一种可能的设计中,第四区域不包括人脸中的眼睛、眉毛、嘴巴对应的区域。第四区域通常是无需涂抹防晒霜的区域。
在第二方面的一种可能的设计中,第一信息还用于指示第二区域的防晒霜涂抹均匀程度和/或防晒霜涂抹薄厚程度和/或是否涂抹防晒霜。
第三方面,本申请提供一种电子设备,该电子设备具有实现如上述任一方面及其中任一种可能的实现方式中的检测方法的功能。该功能可以通过硬件实现,也可以通过硬件执行相应的软件实现。该硬件或软件包括一个或多个与上述功能 相对应的模块。
第四方面,本申请提供一种计算机可读存储介质,包括计算机指令,当计算机指令在电子设备上运行时,使得电子设备执行如任一方面及其中任一种可能的实现方式中任一项的检测方法。
第五方面,本申请提供一种计算机程序产品,当计算机程序产品在电子设备上运行时,使得电子设备执行如任一方面及其中任一种可能的实现方式中任一项的检测方法。
第六方面,提供一种电路***,电路***包括处理电路,处理电路被配置为执行如上述任一方面及其中任一种可能的实现方式中的检测方法。
第七方面,本申请实施例提供一种芯片***,包括至少一个处理器和至少一个接口电路,至少一个接口电路用于执行收发功能,并将指令发送给至少一个处理器,当至少一个处理器执行指令时,至少一个处理器执行如上述任一方面及其中任一种可能的实现方式中的检测方法。
图1为本申请实施例提供的紫外成像原理的示意图;
图2A-图2E为本申请实施例提供的电子设备的形态示意图;
图2F为本申请实施例提供的电子设备的结构示意图;
图3为本申请实施例提供的电子设备的软件架构示意图;
图4为本申请实施例提供的界面图;
图5为本申请实施例提供的识别脸部角度的方法示意图;
图6为本申请实施例提供的界面图;
图7为本申请实施例提供的方法流程的示意图;
图8为本申请实施例提供的检测方法流程以及界面示意图;
图9为本申请实施例提供的检测方法流程以及界面示意图;
图10为本申请实施例提供的界面图;
图11为本申请实施例提供的检测方法流程以及界面示意图;
图12A为本申请实施例提供的检测方法流程的示意图;
图12B为本申请实施例提供的检测方法流程的示意图;
图13为本申请实施例提供的检测方法流程以及界面示意图;
图14为本申请实施例提供的检测方法流程的示意图;
图15为本申请实施例提供的分类器的训练以及使用的示意图;
图16为本申请实施例提供的检测方法流程的示意图;
图17为本申请实施例提供的检测方法流程的示意图;
图18为本申请实施例提供的检测方法流程的示意图;
图19、图20为本申请实施例提供的界面示意图;
图21为本申请实施例提供的装置的示意图;
图22为本申请实施例提供的芯片***的示意图。
首先,对本申请实施例涉及的一些技术术语进行介绍。
紫外线成像:紫外线的波长不在可见光波段范围内,因此,对于人眼来说,紫外线不可见。在一些方案中,可以利用紫外线作为光源,使用紫外线成像***进行摄像,这种使用紫外线成像***捕捉紫外线并成像的技术可称为紫外线成像技术。通过紫外线成像,用户可以观察物体在紫外线照射下形成的紫外图像。
紫外线成像***包括紫外线摄像模组。紫外线摄像模组比如可以是可以捕获紫外线的摄像头,称为紫外线摄像头。如图1所示,紫外线摄像头可包括镜头、图像传感器等部件。该镜头可以是诸如但不限于石英玻璃一类的材质,可以透过紫外线。图像传感器可以记录穿过镜头的光线的信息。
本申请实施例中,考虑到紫外线成像***的特点,可以将紫外线成像***应用到皮肤检测(包括但不限于防晒霜涂抹检测)中,以便检测皮肤对紫外线的吸收或反射情况。
如图1所示,当紫外线照射到人物的不同部位,可产生反射或被人体部位吸收。比如,在没有涂抹防晒霜的脖颈等部位,会反射紫外线,反射的紫外线可以穿过紫外线镜头,进而在图像传感器上记录到紫外线的信息。相应的,脖颈等部位的成像区域的像素值较高。在涂抹有防晒霜的脸部区域,防晒霜会保护皮肤,吸收部分紫外线,进而没有紫外线或较少紫外线被反射进紫外线镜头,在图像传感器也就记录不到或较少记录脸部涂抹防晒霜部位的光线信息。相应的,人脸的防晒霜涂抹部位的成像区域的像素值较低,比如图1所示的涂抹防晒霜的人脸部位的成像区域1为黑色。
可以理解,在其他条件相同时,某个部位防晒霜涂抹越厚,则对紫外线的吸收能力通常越强,那么,该部位在紫外图像中的成像像素值通常越低(颜色越深)。反之,某个部位防晒霜涂抹越薄,则对紫外线的吸收能力通常越弱,那么,该部位在紫外图像中的成像像素值通常越高(颜色越浅)。仍以图1为例,与区域1相比,区域2的防晒霜涂抹更薄,那么,紫外图像中区域2的成像像素值更高,颜色越浅。
本申请实施例提供一种检测方法,可以应用在电子设备中,该电子设备上安装有紫外补光灯以及紫外摄像头。本申请实施例中的电子设备可以是手机、平板电脑、可穿戴设备、车载设备、笔记本电脑、智能化妆镜(比如图2A所示)等,本申请实施例中的手机可以是折叠屏手机或非折叠屏手机,本申请实施例对电子设备的具体类型不作任何限制。
示例性的,电子设备100上摄像头的布局可以参见图2B,其中,电子设备100正面为显示屏194所在的平面。如图2B中(a)所示,摄像头1931位于电子设备100正面,则摄像头为前置摄像头。如图2B中(b)所示,摄像头1932位于电子设备100背面,则摄像头为后置摄像头。摄像头可包括紫外摄像头和彩色摄像头。彩色摄像头包括但不限于红绿蓝三基色(red,green,blue,RGB)摄像头。
可选的,本申请实施例的方案可以应用于具有多个显示屏的折叠屏(即显示屏能够折叠)的电子设备100上。
在一些实施例中,折叠屏可以为柔性折叠屏。其中,柔性折叠屏包括采用柔性材质制作的折叠轴。该柔性折叠屏的部分或全部采用柔性材质制作。例如:该 柔性折叠屏中只有可折叠的部分(如折叠轴)采用柔性材质制作,其它部分采用刚性材质制作;或者,该柔性折叠屏全部采用柔性材质制作。该折叠屏可沿折叠轴折叠形成至少两个子屏。
示例性的,图2B中(c)示出了一种折叠屏电子设备100。响应于用户的操作,如图2B中(d)所示,沿折叠边向内折叠(或向外折叠)可折叠屏幕,使得可折叠屏幕形成至少两个子屏(例如A子屏和B子屏)。可选的,如图2B中(e)所示,在折叠的外侧有显示屏(例如C屏)。若电子设备100在C屏所在表面设置有摄像头。那么,在如图2B中(c)所示的电子设备100未折叠场景中,C屏上的摄像头在电子设备100的背面,可以视为后置摄像头。在如图2B中(e)所示的电子设备100已折叠场景中,C屏上的摄像头变为在电子设备100的正面,可以视为前置摄像头。也就是说,本申请中前置摄像头和后置摄像头并不对摄像头本身的性质进行限制,仅为一种位置关系的说明。
以图2B中的折叠屏手机为例,紫外摄像头可设置在诸如C屏(以挖孔或屏下摄像头等方式设置)上。这样一来,用户可通过该摄像头采集用户面部图像,并通过C屏查看防晒霜涂抹情况。紫外摄像头还可设置在A屏(或B屏)上,这样一来,用户可通过该摄像头采集面部图像,并通过A屏(或B屏)查看防晒霜涂抹情况。
可选的,折叠屏电子设备的折叠屏可形成多个(比如2个、3个等)子屏。示例性的,如图2C中(1)所示的柔性折叠屏,可以包括折叠线030和折叠线031。在沿折叠线030纵向折叠后,可以形成如图2C中(2)所示的子屏032、子屏033和子屏034。
可选的,折叠屏电子设备的屏幕排列方式可以是诸如图2B中(d)所示的上下屏幕排列,或者,也可以为诸如图2D中(1)或(2)所示的左右屏幕排列等。本申请实施例对折叠屏电子设备的屏幕排列方式不做限制。如图2D中(1)所示的柔性折叠屏,在沿折叠线040横向折叠后,可以形成如图2D中(2)所示的子屏041、和子屏042。例如,该折叠屏设备在折叠状态(可参见图2E的(1)所示状态)可以为手机,在展开状态可以为平板电脑。
在本申请的实施例中,折叠屏可以是单面屏(即只有一面可以显示用户界面)也可以是双面屏(即相对的两面均可显示用户界面)。
对于单面折叠屏来说,折叠屏朝可显示用户界面的一面(即单面折叠屏的正面)进行翻折的情况,可以称为正向翻折;折叠屏朝可显示用户界面的一面的相对面(即单面折叠屏的背面)进行翻折的情况,可以称为反向翻折。例如,图2D中(2)表示正向翻折的示意图,图2E中(2)表示反向翻折的示意图。折叠屏设备可以确定当前是正向翻折还是反向翻折。
示例性的,图2F示出了电子设备100的一种结构示意图。电子设备100可以包括处理器110,外部存储器接口120,内部存储器121,通用串行总线(universal serial bus,USB)接口130,充电管理模块140,电源管理模块141,电池142,天线1,天线2,移动通信模块150,无线通信模块160,音频模块170,扬声器170A,受话器170B,麦克风170C,耳机接口170D,传感器模块180,按键190,马达 191,指示器192,摄像头193,显示屏194,以及用户标识模块(subscriber identification module,SIM)卡接口195等。
处理器110可以包括一个或多个处理单元,例如:处理器110可以包括应用处理器(application processor,AP),调制解调处理器,图形处理器(graphics processing unit,GPU),图像信号处理器(image signal processor,ISP),控制器,存储器,视频编解码器,数字信号处理器(digital signal processor,DSP),基带处理器,和/或神经网络处理器(neural-network processing unit,NPU)等。其中,不同的处理单元可以是独立的器件,也可以集成在一个或多个处理器中。
其中,控制器可以是电子设备100的神经中枢和指挥中心。控制器可以根据指令操作码和时序信号,产生操作控制信号,完成取指令和执行指令的控制。
处理器110中还可以设置存储器,用于存储指令和数据。在一些实施例中,处理器110中的存储器为高速缓冲存储器。该存储器可以保存处理器110刚用过或循环使用的指令或数据。如果处理器110需要再次使用该指令或数据,可从存储器中直接调用。避免了重复存取,减少了处理器110的等待时间,因而提高了***的效率。
在本申请的一些实施例中,处理器110用于在检测到触发检测的操作时,调用紫外摄像头拍摄紫外图像,并根据紫外图像对防晒霜涂抹情况进行检测。之后,处理器110可以控制通过显示屏向用户提示防晒霜涂抹情况。其中,防晒霜涂抹情况包括但不限于用户是否涂抹防晒霜、涂抹防晒霜的位置、未涂抹防晒霜的位置、涂抹防晒霜的位置防晒霜是否涂抹均匀、防晒霜未涂抹均匀的位置、各区域防晒霜涂抹的薄厚信息、防晒霜推荐信息。
作为一种可能的实现方式,处理器110控制紫外摄像头拍摄紫外图像,可以是控制紫外摄像头拍摄多张不同角度的紫外图像。之后,处理器110可以根据多张不同角度的紫外图像检测防晒霜涂抹情况。如此一来,可以抵消掉部分角度紫外图像中诸如反光区域导致的检测误差,提升检测结果的准确性。
作为一种可能的实现方式,处理器110控制通过显示屏向用户提示防晒霜涂抹情况,可以是在彩色图像中标记不同的面部区域,比如在彩色图像中标记防晒霜未涂抹均匀的位置等,彩色图像能够记录更加清晰的皮肤细节,有助于用户清晰辨别防晒霜涂抹情况,指导用户进行更加精细的防晒措施,比如,用户可以根据防晒霜涂抹情况进行精细的补涂。
充电管理模块140用于从充电器接收充电输入。
电源管理模块141用于连接电池142,充电管理模块140与处理器110。电源管理模块141接收电池142和/或充电管理模块140的输入,为处理器110,显示屏194,摄像头193等供电。
电子设备100的无线通信功能可以通过天线1,天线2,移动通信模块150,无线通信模块160,调制解调处理器以及基带处理器等实现。
移动通信模块150可以提供应用在电子设备100上的包括2G/3G/4G/5G等无线通信的解决方案。无线通信模块160可以提供应用在电子设备100上的包括无线局域网(wireless local area networks,WLAN)(如无线保真(wireless fidelity,Wi-Fi) 网络),蓝牙(bluetooth,BT)等无线通信的解决方案。
电子设备100通过GPU,显示屏194,以及应用处理器等实现显示功能。GPU为图像处理的微处理器,连接显示屏194和应用处理器。GPU用于执行数学和几何计算,用于图形渲染。处理器110可包括一个或多个GPU,其执行程序指令以生成或改变显示信息。
显示屏194用于显示图像,视频等。显示屏194包括显示面板。在一些实施例中,电子设备100可以包括1个或N个显示屏194,N为大于1的正整数。
电子设备100可以通过ISP,摄像头193,视频编解码器,GPU,显示屏194以及应用处理器等实现拍摄功能。
ISP用于处理摄像头193反馈的数据。例如,拍照时,打开快门,光线通过镜头被传递到摄像头感光元件上,光信号转换为电信号,摄像头感光元件将电信号传递给ISP处理,转化为肉眼可见的图像。ISP还可以对图像的噪点,亮度,肤色进行算法优化。ISP还可以对拍摄场景的曝光,色温等参数优化。在一些实施例中,ISP可以设置在摄像头193中。例如,ISP可以根据拍摄参数控制感光元件进行曝光和拍照。
摄像头193用于捕获静态图像或视频。物体通过镜头生成光学图像投射到感光元件。感光元件可以是电荷耦合器件(charge coupled device,CCD)或互补金属氧化物半导体(complementary metal-oxide-semiconductor,CMOS)光电晶体管。感光元件把光信号转换成电信号,之后将电信号传递给ISP转换成数字图像信号。ISP将数字图像信号输出到DSP加工处理。DSP将数字图像信号转换成标准的RGB,YUV等格式的图像信号。
在一些实施例中,电子设备100可以包括1个或N个摄像头193,N为大于1的正整数。其中,摄像头193可以位于电子设备的边缘区域,可以为屏下摄像头,也可以是可升降的摄像头,或者挖孔摄像头。摄像头193可以包括后置摄像头,还可以包括前置摄像头。本申请实施例对摄像头193的具***置和形态不予限定。
在本申请实施例中,摄像头193包括普通摄像头以及紫外摄像头。
其中,普通摄像头包括彩色摄像头。彩色摄像头比如可以但不限于是RGB摄像头。
本申请实施例中,彩色摄像头可用于采集图像,采集的图像可被用来确定是否存在人脸等待检测部位。采集的图像还可用来标记防晒霜涂抹情况。比如,手机可以在彩色图像中标记防晒霜没有涂抹均匀的位置。
紫外摄像头,可用来拍摄紫外图像,电子设备100可根据拍摄的紫外图像检测防晒霜涂抹情况。紫外摄像头,还可用来捕获当前环境中的紫外线。具体的,紫外摄像头中的图像传感器(比如CMOS)捕获当前环境下的紫外线。电子设备100可根据捕获的紫外线确定当前环境下的紫外线强度。
本申请实施例中,电子设备100还可以包括紫外补光灯。该紫外补光灯可发射紫外线,可用于紫外摄像头的成像光源。作为一种可能的实现方式,为了避免紫外补光灯对用户的紫外线照射过度,仅在环境紫外线强度低于一定阈值,即环境紫外线强度达不到紫外摄像头成像所需的紫外线强度的情况下,才开启紫外补 光灯。在环境紫外线强度高于阈值的情况下,电子设备不开启紫外补光灯。
其中,电子设备100获取环境紫外线强度,可以是通过自身的传感器检测紫外线强度,或者,可以从云侧获取环境紫外线强度。本申请实施例对电子设备100获取环境紫外线强度的方式不做限制。
数字信号处理器用于处理数字信号,除了可以处理数字图像信号,还可以处理其他数字信号。例如,当电子设备100在频点选择时,数字信号处理器用于对频点能量进行傅里叶变换等。
视频编解码器用于对数字视频压缩或解压缩。电子设备100可以支持一种或多种视频编解码器。这样,电子设备100可以播放或录制多种编码格式的视频,例如:动态图像专家组(moving picture experts group,MPEG)1,MPEG2,MPEG3,MPEG4等。
NPU为神经网络(neural-network,NN)计算处理器,通过借鉴生物神经网络结构,例如借鉴人脑神经元之间传递模式,对输入信息快速处理,还可以不断的自学习。通过NPU可以实现电子设备100的智能认知等应用,例如:图像识别,人脸识别,语音识别,文本理解等。
在一些实施例中,NPU利用图像识别技术,识别摄像头193采集到的图像中是否包含人脸图像。
外部存储器接口120可以用于连接外部存储卡,例如Micro SD卡,实现扩展电子设备100的存储能力。外部存储卡通过外部存储器接口120与处理器110通信,实现数据存储功能。例如将音乐,视频等文件保存在外部存储卡中。
内部存储器121可以用于存储计算机可执行程序代码,所述可执行程序代码包括指令。处理器110通过运行存储在内部存储器121的指令,和/或存储在设置于处理器中的存储器的指令,执行电子设备100的各种功能应用以及数据处理。
电子设备100可以通过音频模块170,扬声器170A,受话器170B,麦克风170C,耳机接口170D,以及应用处理器等实现音频功能。例如音乐播放,录音等。
音频模块170用于将数字音频数据转换成模拟音频电信号输出,也用于将模拟音频电信号输入转换为数字音频数据,音频模块170可以包括模/数转换器和数/模转换器。例如,音频模块170用于将麦克风170C输出的模拟音频电信号转换为数字音频数据。音频模块170还可以用于对音频数据进行编码和解码。在一些实施例中,音频模块170可以设置于处理器110中,或将音频模块170的部分功能模块设置于处理器110中。
扬声器170A,也称“喇叭”,用于将模拟音频电信号转换为声音信号。电子设备100可以通过扬声器170A收听音乐,或收听免提通话。
受话器170B,也称“听筒”,用于将模拟音频电信号转换成声音信号。当电子设备100接听电话或语音信息时,可以通过将受话器170B靠近人耳接听语音。
麦克风170C,也称“话筒”,“传声器”,用于将声音信号转换为模拟音频电信号。当拨打电话或发送语音信息时,用户可以通过人嘴靠近麦克风170C发声,将声音信号输入到麦克风170C。其中,该麦克风170C可以是电子设备100的内置部件,也可以是电子设备100的外接配件。
在一些实施例中,电子设备100可以包括一个或多个麦克风170C,其中每一麦克风或多个麦克风合作可以实现采集各个方向的声音信号,并将采集到的声音信号转换为模拟音频电信号的功能,还可以实现降噪,识别声音来源,或定向录音功能等。
其中传感器模块180可以包括压力传感器180A,陀螺仪传感器180B,气压传感器180C,磁传感器180D,加速度传感器180E,距离传感器180F,接近光传感器180G,指纹传感器180H,温度传感器180J,触摸传感器180K,环境光传感器180L,骨传导传感器180M等。
其中,陀螺仪传感器180B,可以用于确定电子设备100的运动姿态。在一些实施例中,可以通过陀螺仪传感器180B确定电子设备100围绕三个轴(即,x,y和z轴)的角速度。陀螺仪传感器180B可以用于拍摄防抖。示例性的,当按下快门,陀螺仪传感器180B检测电子设备100抖动的角度,根据角度计算出镜头模组需要补偿的距离,让镜头通过反向运动抵消电子设备100的抖动,实现防抖。陀螺仪传感器180B还可以用于导航,体感游戏场景。
距离传感器180F,用于测量距离。电子设备100可以通过红外或激光测量距离。在一些实施例中,在拍摄场景下,电子设备100可以利用距离传感器180F测距以实现快速对焦。
触摸传感器180K,也称“触控面板”。触摸传感器180K可以设置于显示屏194,由触摸传感器180K与显示屏194组成触摸屏,也称“触控屏”。触摸传感器180K用于检测作用于其上或附近的触摸操作。
可以理解的是,本申请实施例示意的结构并不构成对电子设备100的具体限定。在本申请另一些实施例中,电子设备100可以包括比图示更多或更少的部件,或者组合某些部件,或者拆分某些部件,或者不同的部件布置。图示的部件可以以硬件,软件或软件和硬件的组合实现。
电子设备100的软件***可以采用分层架构,事件驱动架构,微核架构,微服务架构,或云架构。本发明实施例以分层架构为例,示例性说明电子设备100的软件结构。
图3是本发明实施例的电子设备100的软件结构框图。
分层架构将软件分成若干个层,每一层都有清晰的角色和分工。层与层之间通过软件接口通信。在一些实施例中,将电子设备的操作***(例如And待检测区域d***)分为四层,从下至上分别为内核层,硬件抽象层(hardware abstract layer,HAL),应用程序框架层,以及应用程序层。
内核层是硬件和软件之间的层。内核层至少包含摄像头驱动,音频驱动,显示驱动,传感器驱动。传感器驱动包括但不限于图像传感器驱动、声敏传感器驱动。
在一些实施例中,比如在人脸识别场景中,内核层的诸如摄像头驱动被调用以开启摄像头。再比如,通过图像传感器驱动调用图像传感器完成图像采集。
硬件抽象层(hardware abstract layer,HAL)位于内核层和应用程序框架层之间,用于定义驱动应用程序硬件实现的接口,将驱动硬件实现的值转化为软件实 现程序语言。例如识别摄像头驱动的值,将其转化为软件程序语言上传至应用程序框架层,进而实现调用相应功能。
在一些实施例中,HAL可以将摄像头193采集到的人脸图像上传至应用程序框架层进行进一步的处理。
应用程序框架层为应用程序层的应用程序提供应用编程接口(application programming interface,API)和编程框架。应用程序框架层从内核层经由HAL获取原始输入事件,识别该输入事件所对应的控件。应用程序框架层包括一些预先定义的函数。
如图3所示,应用程序框架层可以包括视图***,电话管理器,资源管理器,通知管理器,窗口管理器等。
在一些实施例中,应用框架层包括第一模块。第一模块用于通过摄像头驱动调用第一摄像头采集第一图像,通过摄像头驱动调用第二摄像头采集第二图像。之后,在所述第二图像上确定第一区域,并根据所述第一区域,在所述第一图像上确定与所述第一区域对应的第二区域。之后,第一模块可调用显示驱动显示用户界面,所述用户界面包括所述第一人脸和第一信息,所述第一信息用于指示所述第二区域。
可选的,第一模块还可以是设置在其他层中,并且,第一模块还可以划分为更多子模块。每个子模块用于执行相应功能。
视图***包括可视控件,例如显示文字的控件,显示图片的控件等。视图***可用于构建应用程序。显示界面可以由一个或多个视图组成的。例如,包括短信通知图标的显示界面,可以包括显示文字的视图以及显示图片的视图。
电话管理器用于提供电子设备100的通信功能。例如通话状态的管理(包括接通,挂断等)。
资源管理器为应用程序提供各种资源,比如本地化字符串,图标,图片,布局文件,视频文件等等。
通知管理器使应用程序可以在状态栏中显示通知信息,可以用于传达告知类型的消息,可以短暂停留后自动消失,无需用户交互。比如通知管理器被用于告知下载完成,消息提醒等。通知管理器还可以是以图表或者滚动条文本形式出现在***顶部状态栏的通知,例如后台运行的应用程序的通知,还可以是以对话窗口形式出现在屏幕上的通知。例如在状态栏提示文本信息,发出提示音,电子设备振动,指示灯闪烁等。
窗口管理器用于管理窗口程序。窗口管理器可以获取显示屏大小,判断是否有状态栏,锁定屏幕,截取屏幕等。
应用程序层可以包括一系列应用程序包。
如图3所示,应用程序包可以包括相机,视频,通话,WLAN,音乐,短信息,蓝牙,地图,日历,图库,导航等应用程序。
应用程序层和应用程序框架层运行在虚拟机中。虚拟机将应用程序层和应用程序框架层的java文件执行为二进制文件。虚拟机用于执行对象生命周期的管理,堆栈管理,线程管理,安全和异常的管理,以及垃圾回收等功能。
以下将以电子设备为具有诸如图2F和图3所示结构的手机为例,在一些实施 例中,手机可以在检测到用户开启了诸如相机、镜子等应用时,开启本申请实施例的防晒霜涂抹检测方法。或者,手机也可以是在检测到用户向手机输入了预设指令之后,开启本申请实施例的防晒霜涂抹检测方法。本申请实施例并不限制开启防晒霜涂抹检测功能的方式。如下,以通过镜子应用中的开启防晒检测的开关开启防晒霜涂抹检测方法为例,对本申请实施例的技术方案进行介绍。
可选的,用户可以通过触摸操作、按键操作、隔空手势操作或语音操作等方式,指示手机启动镜子应用,并显示拍摄预览界面。示例性的,手机显示如图4中(a)所示的主界面,在检测到用户的诸如点击镜子应用图标的操作时,手机启动启动镜子应用,并调用普通摄像头(比如RGB摄像头)采集图像,显示图4中(b)所示的拍摄预览界面。或者,在检测到用户打开镜子应用的语音指示操作时,手机启动镜子应用,并调用普通摄像头捕捉图像,得到图4中(b)所示的拍摄预览界面。拍摄预览界面可包括一个或多个控件,用于实现不同功能。比如,包括设置控件401、防晒检测控件402。其中,设置控件401可用于对镜子应用的功能进行设置。防晒检测控件402可用于开启本申请实施例的防晒检测功能。
可选的,拍摄预览界面还可以包括其他控件。比如,可用于开启或关闭滤镜功能的控件、可用于开启或关闭闪光灯功能的控件、可用于观看美妆教程的控件、控制拍照的控件、用于调整焦距的控件、调整屏幕亮度的控件。
在一些实施例中,手机检测到诸如用户在拍摄预览界面的对防晒检测控件402的点击操作,则开启防晒霜涂抹检测功能。在开启防晒霜涂抹检测功能的情况下,手机可检测防晒霜的涂抹情况。作为一种可能的实现方式,考虑到紫外摄像头(第二摄像头的一种示例)可以捕捉紫外线的成像特点以及防晒霜有吸收紫外线的特点,手机可以调用紫外摄像头采集一张或多张紫外图像,并根据该一张或多张紫外图像检测防晒霜的涂抹情况。
作为一种可能的实现方式,在触发手机开启防晒检测功能,比如用户点击如图4的(b)所示的防晒检测控件402后,手机可继续调用彩色摄像头(第一摄像头的一种示例)采集彩色图像。
在一些实施例中,若通过彩色摄像头检测到采集的图像(第五图像的一个示例)满足第一条件,比如该图像中存在人脸,则手机可以自动调用紫外摄像头采集紫外图像,通过彩色摄像头采集彩色图像。其中,紫外摄像头与彩色摄像头可以是同步采集,或者交替采集图像。
在一些实施例中,考虑到紫外线对人体的副作用以及紫外摄像头成像的效果,手机可以设置紫外补光灯,手机可以检测并基于环境紫外线强度判断是否在采集紫外图像时开启紫外补光灯。作为一种可能的实现方式,若通过彩色摄像头检测到采集的图像(第五图像的一个示例)存在人脸,则手机可以调用紫外摄像头捕获环境紫外线,并根据捕获的紫外线确定环境紫外线强度。在一些情况下,若环境的当前紫外线强度低于第三阈值,说明环境的紫外线强度达不到紫外摄像头成像的所需强度,则手机开启紫外补光灯,在开启紫外补光灯的情况下调用紫外摄像头采集预设数目的紫外图像,以及调用彩色摄像头采集彩色图像。在另一些情况下,若环境的当前紫外线强度高于第三阈值,则手机调用紫外摄像头(可以不 开启紫外补光灯)采集预设数目的紫外图像。可选的,在采集完预设数目的紫外图像后,关闭紫外摄像头,以降低紫外摄像头工作带来的功耗。
需要说明的是,在开启紫外摄像头后,手机可以不关闭彩色摄像头。其中,彩色摄像头采集的图像可以显示在预览界面中,提供给用户观看。紫外摄像头采集的图像可用来确定防晒霜的涂抹情况。
其中,在开启紫外补光灯用以补充紫外摄像头的光源强度的情况下,为降低紫外补光灯对用户的紫外照射,还需及时关闭紫外补光灯。本申请实施例中,关闭紫外补光灯,可以是:在检测到预设数目的紫外图像采集完成后,自动关闭紫外补光灯。或者,在检测到紫外补光灯的开启时长达到t1时,自动关闭紫外补光灯。或者,还可以结合开启时长和预设数目的紫外图像是否采集完成来判断是否关闭紫外补光灯。比如,在检测到紫外补光灯的开启时长达到t1,且预设数目的紫外图像采集完成后,自动关闭紫外补光灯。
在另一些实施例中,可以设置紫外补光灯与紫外摄像头联动。此种设置下,若通过普通摄像头检测到图像中存在人脸,则可以开启紫外摄像头以及联动的紫外补光灯,并可以通过紫外摄像头采集预设数目的紫外图像。此种联动设置方式中,由于能够联动开启紫外摄像头和紫外补光灯,进而可使得采集紫外图像的实现复杂度较低。
在另一些实施例中,手机开启紫外摄像头,还可以由用户的操作触发。比如,用户界面中设置用于触发紫外摄像头采集紫外图像的控件,当检测到用户对该控件的预设操作后,手机可开启紫外摄像头采集紫外图像。可选的,彩色摄像头同步采集彩色图像。
之后,手机可根据采集的紫外图像,检测防晒霜的涂抹情况。在检测到防晒霜的涂抹情况后,手机向用户提示防晒霜的涂抹信息。其中,防晒霜的涂抹信息包括但不限于如下任一项或多项信息:用户是否涂抹防晒霜、涂抹防晒霜的位置、未涂抹防晒霜的位置、防晒霜是否涂抹均匀、防晒霜未涂抹均匀的位置、各区域防晒霜涂抹的薄厚信息、防晒霜推荐信息。
示例性的,如图4的(b)所示,手机检测到用户对防晒检测控件402的点击操作,则检测防晒霜的涂抹情况,并可以向用户呈现图4的(c)所示提示框403。
通过本申请实施例的防晒霜涂抹检测方案,用户可以直观的从手机的标记确定防晒霜没有涂抹均匀或者涂抹较薄的精确位置,以便根据该精确位置进行防晒霜补涂,能够对用户进行精细的防晒指导。
可选的,为了提升检测结果的精确度,还可以采集特定角度的紫外图像,并根据特定角度的紫外图像确定防晒霜涂抹情况。比如,手机可以采集正脸的紫外图像,并根据正脸的紫外图像确定防晒霜涂抹情况。可选的,手机可以在采集正脸的紫外图像前,向用户提示“将正脸对准手机”,以便用户在提示下将正脸对准手机。
可选的,考虑到在某些场景中,人脸的某些部位在所采集的某个角度紫外图像中出现诸如反光等问题,导致由该角度紫外图像判断防晒霜涂抹情况时判断不准确,本申请实施例中,可以采集不同角度的脸部紫外图像,并根据不同角度的 脸部紫外图像判断防晒霜涂抹情况。也就是说,紫外线摄像头采集的上述预设数目的紫外图像可以为多张紫外图像,且该多张紫外图像为用户脸部转到不同角度时采集的图像,多张紫外图像包括不同角度的人脸图像信息。不同紫外图像可以对应不同的面部角度。这里的角度可以是用户脸部相对于手机的角度。
如此一来,对人脸的某个部位来说,可以有不同角度的图像信息,根据该部位的不同角度的图像信息,可以更准确地判断该部位的防晒霜涂抹情况。
作为一种可能的实现方式,手机可以基于图像中检测到的部分人脸关键点之间的位置关系确定该图像中人脸相对于手机的角度。其中,用于检测角度、位置等关系的面部关键点可以是左脸关键点P1,鼻翼附近关键点P2,右脸关键点P3,关键点之间的位置关系比例=(P2-P1)/(P3-P2)。手机可根据关键点的位置关系比例与实际左右转动角度之间的关系,得到所采集图像的角度。其中,P2-P1表示P2与P1之间的距离,P3-P2表示P3与P2之间的距离。示例性的,当手机采集到的图像为如图5的(a)所示的图像1401,则手机根据计算面部关键点的位置关系比例,并确定该比例对应正脸角度。当手机采集到的图像为如图5的(b)所示图像1402,则手机根据计算面部关键点的位置关系比例,并确定该比例对应用户的脸部向右转动了30度。其中,图像1401、1402可以是彩色图像或紫外图像。
作为一种可能的实现方式,RGB摄像头与紫外摄像头可同步工作,意味着,在同一时刻,手机同时调用RGB摄像头以及紫外摄像头。并且,手机可识别RGB摄像头采集的RGB图像的人脸关键点,并根据RGB图像的人脸关键点判断RGB图像中人脸对于手机的角度,这样一来,手机可据此推知与该RGB图像同步采集的紫外图像中人脸相对于手机的角度。
作为另一种可能的实现方式,RGB摄像头可以不与紫外摄像头同步工作。此种情况下,手机可以确定RGB图像中的人脸关键点,并根据图像匹配算法以及RGB图像中的人脸关键点识别紫外图像中的人脸关键点。之后,手机可根据紫外图像中的人脸关键点确定紫外图像中人脸相对于手机的角度。
可选的,手机采集多个角度的多个脸部紫外图像,可以是用户主动转动头部(向左转动或向右转动头部)过程中,手机采集多个角度的脸部紫外图像。示例性的,用户打开镜子应用,并通过诸如点击图4的(b)所示防晒检测控件402指示手机开启防晒检测功能,手机可以通过紫外摄像头实时或按照一定周期采集紫外图像,并在采集紫外图像的同时通过RGB摄像头采集RGB图像。可选的,手机可以计算每张RGB图像中人脸相对于手机的角度,并根据计算得到的角度,从多张RGB图像中选择角度符合预设条件的目标RGB图像,目标RGB图像对应的紫外图像可作为用来计算防晒霜涂抹情况的紫外图像。角度符合预设条件可以指,RGB图像中人脸相对于手机的角度在左转第一预设角度范围内,或者在右转第二预设角度范围内,或者在正对手机的角度范围内。示例性的,以第一预设角度、第二预设角度均为10-30度,人脸正对手机是人脸相对于手机的角度在左转5度到右转5度之间为例,手机了10张RGB图像,且其中1张RGB图像中人脸相对于手机的角度为左转25度,1张RGB图像中人脸相对于手机的角度为右转20度,1张RGB图像中人脸相对于手机的角度为右转25度,1张RGB图像中人脸相对于 手机的角度为左转2度,那么,手机将这4张RGB图像作为符合预设条件的目标RGB图像,这4张RGB图像对应的紫外图像可作为用来计算防晒霜涂抹情况的紫外图像。
当然,第一预设角度,第二预设角度,正对角度可以另行灵活设置。预设条件也可以另行设置,本申请实施例对此不做限制。
示例性的,手机采集10张RGB图像(RGB图像1-10)和10张紫外图像(紫外图像1-10),其中,RGB图像L(L是正整数,L大于或等于1,且小于或等于10)与紫外图像L是同步采集的。其中,RGB图像3-RGB图像5在左转30度至右转30度范围内,则手机可以将RGB图像3-RGB图像5对应的紫外图像3-5作为用来计算防晒霜涂抹情况的紫外图像。
或者,可选的,手机采集多个角度的多个脸部紫外图像,也可以是手机提示用户转动头部,用户在手机的提示下转动头部的过程中,手机采集不同角度的脸部紫外图像。手机检测到用户的诸如点击图4的(b)所示防晒检测控件402的操作,则手机开启防晒检测功能。可选的,如图6的(a)所示,手机可在预览界面中提示用户正脸对准手机,并通过RGB图像采集RGB图像,并通过紫外摄像头采集紫外图像。手机可根据RGB图像判断RGB图像中人脸相对于手机的角度,在确定人脸正对手机,即采集到正脸的RGB图像(第三图像的一个示例)以及正脸的紫外图像(人脸相对于手机的角度满足第二条件,即符合正脸角度范围)的情况下,手机可继续采集其他角度的RGB图像以及采集其他角度的紫外图像。可选的,在确定已采集到正脸的RGB图像以及正脸的紫外图像后,手机还可以在预览界面中呈现图6的(b)所示提示框405(第二信息的一个示例),以提示用户向左转动头部。手机通过RGB图像采集RGB图像(第四图像),并通过紫外摄像头采集紫外图像(第六图像)。在手机根据采集的RGB图像确定人脸角度满足左转一定角度的情况下,手机可继续采集其他角度的RGB图像以及采集其他角度的紫外图像。之后,可选的,在确定已采集到左侧脸的RGB图像以及左侧脸的紫外图像后,手机还可以再预览界面中呈现图6的(c)所示提示框406,以提示用户向右转动头部。可选的,预览界面呈现的图像可以是彩色图像。在用户转动头部的过程中,手机可调用紫外摄像头相应采集多张紫外图像以及调用彩色摄像头采集多张彩色图像。其中,彩色图像可用来呈现预览界面以及辅助确定紫外图像中人脸相对于手机的角度,多张紫外图像可用来确定防晒霜涂抹情况。之后,手机可呈现诸如图6的(d)所示提示框,以便向用户提示防晒霜的涂抹情况,对用户防晒进行有效精细的指导。
当然,手机提示用户转动头部的界面不限于图6所示界面。比如,直接提示用户“请向左或向右转动头部,以便采集多角度的人脸图像”。
需要说明的是,本申请实施例中提及的正脸图像,可以是在一定角度偏差范围内的图像,并非限制为人脸严格正对手机的图像。
可选的,手机呈现图6的(a)所示提示框,可以是在检测到用户的诸如点击图4的(b)所示防晒检测控件402的操作之后即刻呈现提示框405。也可以是在检测到用户的诸如点击防晒检测控件402的操作之后,在预设时段(可以灵活设 置)内没有检测到用户转动头部的情况下呈现提示框405。本申请实施例并不限制手机提示用户转动头部的时机。
如下,仍以在镜子应用开启防晒检测功能为例,对基于紫外图像检测防晒霜涂抹情况的具体实现方法进行详细阐述。
首先,介绍检测是否涂抹防晒霜的流程。如图7所示,该流程包括如下步骤:
S101、手机识别RGB图像中的人脸图像。
作为一种可能的实现方式,考虑到RGB图像比紫外图像的成像细节更丰富,画质更高。因此,手机可以在RGB图像中识别人脸,并将识别出的人脸相应区域匹配到紫外图像中。
其中,RGB图像可以是在用户打开镜子应用后,手机调用RGB摄像头采集的,并不限制采集RGB图像的时机。比如,可以是在紫外摄像头采集紫外图像同时,RGB摄像头同步采集RGB图像,或者RGB摄像头与紫外摄像头交替采集。本申请实施例中主要以开启紫外摄像头后,RGB摄像头与紫外摄像头同步采集为例进行说明,但本申请实施例的技术方案不限于此。
作为一种可能的实现方式,手机基于人脸识别算法识别RGB图像中的人脸图像。可选的,手机可从识别出的人脸图像中提取面部特征点,并根据提取的面部特征点标记人脸图像中的眼睛、眉毛、嘴巴等通常不会涂抹防晒霜的面部部位。后续,手机在计算防晒霜的涂抹情况时,可以不对人脸图像的眼睛、眉毛等区域进行计算处理。如此,能够降低手机的计算量。示例性的,如图8的(a)所示,手机提取人脸图像中的面部特征点,并可以图8的(b)所示,在人脸图像中标记面部特征点中眼部、嘴巴等各自特征点包围的区域。
需要说明的是,图8中以包围眼部等成像区域的轮廓为方框,并标记方框区域为例进行举例,在另一些实施例中,包围眼部等成像区域的轮廓还可以为其他形状,比如圆形,再比如还可以为贴合眼部等成像区域边缘轮廓的形状。
本申请实施例中,可以在人脸图像中剔除眼部、嘴唇等无需涂抹防晒霜的区域,得到第四区域,后续,也就是后续步骤中所处理的人脸区域是剔除眼睛、嘴唇、眉毛等区域的区域(第四区域)。比如,手机可根据第四区域包含的像素的像素值,在所述第二图像的第四区域中确定第一区域,而不对第四区域外的像素进行遍历处理。
或者,并不剔除这部分区域,而是仅对这部分区域做出标记,后续,手机可以根据做出的标记选择不处理这部分区域的像素。
S102、手机根据RGB图像(第一图像的一个示例)中的人脸(第一人脸),获取紫外图像(第二图像的一个示例)中的人脸(第二人脸的一个示例)。
其中,第一人脸和所述第二人脸属于同一用户。
作为一种可能的实现方式,在手机获取RGB图像中的人脸图像后,手机可通过诸如图像匹配算法,获取紫外图像中的人脸图像。示例性的,图像匹配算法包括但不限于尺度不变特征变换(scale-invariant feature transform,SITF)算法。在SITF算法中,手机可以在RGB图像中提取人脸图像的关键点。关键点可以但不限于是不会因光照、尺度、旋转等因素而消失的点,比如下巴最下缘点、鼻翼最高 点等。手机还可以获取人脸各关键点的特征向量,并根据RGB图像中人脸各关键点的特征向量在紫外图像中定位各关键点。如此,手机能够根据紫外图像的人脸关键点确定人脸图像。
以在紫外图像中定位下巴最下缘点、鼻翼最高点为例,手机获取RGB图像中下巴最下缘点的特征向量1,鼻翼最高点的特征向量2,并在紫外图像中查找特征向量最接近特征向量1的像素点作为下巴最下缘点,在紫外图像中查找特征向量最接近特征向量2的像素点作为鼻翼最高点。
示例性的,如图8的(b)以及(c)所示,手机可以根据RGB图像中的人脸图像识别紫外图像中的人脸图像。其中,手机还可以根据RGB图像中的眼部、嘴唇等位置识别紫外图像中的眼部、嘴唇等位置,后续,对眼部、嘴唇等位置的处理过程可以不同于脸部的其他区域。
本申请实施例中,可选的,获取紫外图像中的人脸图像后,手机还可以执行S103。
需要说明的是,上述实施例主要以彩色摄像头为RGB摄像头,且采集的采集彩色图像为RGB图像为例进行说明,本申请实施例中的彩色摄像头还可以是其他类型的摄像头,相应的,采集的彩色图像可以为其他类型。
S103、手机对紫外图像中的人脸图像进行预处理。
其中,预处理包括但不限于滤波处理,以便消除图像中噪声的影响。滤波可以包括但不限于均值滤波、中值滤波、高斯滤波等。
在一些实施例中,手机可以对紫外图像中人脸图像的全部区域进行预处理。或者,为了降低预处理的计算量,手机还可以对人脸图像中除眼部、嘴唇等(即通常无需涂抹防晒霜的区域)区域,即标记区域之外的区域进行预处理。
S104、手机基于紫外图像中的人脸图像,识别防晒霜涂抹情况。
本申请实施例中,手机可根据紫外图像的图像特征,在紫外图像(第二图像)的第二人脸上,确定第一区域,并确定第一区域的防晒霜涂抹情况。
其中,所述第一区域为如下任一种或多种区域:涂抹防晒霜的区域,未涂抹防晒霜的区域,防晒霜涂抹不均匀的区域,防晒霜涂抹均匀的区域,防晒霜涂抹厚度大于第一阈值的区域,防晒霜涂抹厚度小于第二阈值的区域。图像特征包括但不限于如下一个或多个特征:图像区域的平均像素值、最大像素值、最小像素值、图像区域中相邻像素的像素值的差值、图像区域中各像素的颜色、图像区域的对比度、直方图、方向梯度直方图(histogram of oriented gradients,HOG)、标准差、色阶、均方差、方差。像素的颜色值包括RGB各通道的像素值。
如下介绍根据紫外图像的图像特征确定第一区域的原理。
首先,介绍基于紫外图像特征中的像素值确定涂抹防晒霜区域的原理。上文已提及,防晒霜能够吸收紫外线,因此,在紫外图像中,涂抹防晒霜的区域的亮度较低,对应的像素值通常较低,未涂抹防晒霜的区域的亮度较高,对应的像素值通常较高。基于该原理,手机可以检测人脸图像的像素值,并将像素值高于阈值的区域作为未涂抹防晒的区域,将将像素值小于或等于阈值的区域作为涂抹防晒的区域。
本申请实施例中,像素值高于像素阈值的区域,可称为高亮区域。像素值小于或等于阈值的区域,可称为非高亮区域。
作为一种可能的实现方式,可以设置大小为W1*W2的像素窗口。可通过滑动窗口遍历紫外图像中的人脸图像,窗口每滑动到一个位置时,就统计一次人脸图像在窗口区域内的像素均值ˉF,并将ˉF﹤T1的窗口区域划分为非高亮区域,将ˉF>T1的窗口区域划分为高亮区域。
示例性的,仍参见图8的(c),手机检测到紫外图像中像素均值高于阈值(颜色较浅)的成像区域为区域408a,那么,区域408a可被手机视为没涂抹防晒霜的部位对应的成像区域。反之,像素均值小于阈值(颜色较深)的成像区域407a被手机视为涂抹防晒霜部位对应的成像区域。
或者,手机可以遍历每个像素的像素值,将像素值大于阈值的像素划分进高亮区域,将像素值小于或等于该阈值的像素划分进非高亮区域。
可以理解,在其他条件相同时,某个部位防晒霜涂抹越厚,则对紫外线的吸收能力通常越强,那么,该部位在紫外图像中的成像像素值通常越低(颜色越深)。反之,某个部位防晒霜涂抹越薄,则对紫外线的吸收能力通常越弱,那么,该部位在紫外图像中的成像像素值通常越高(颜色越浅)。基于该原理,手机还可以根据紫外图像中包括的像素值确定防晒霜的涂抹均匀程度,以及涂抹薄厚程度。以图1为例,与区域1相比,区域2的防晒霜涂抹更薄,那么,紫外图像中区域2的成像像素值更高,颜色越浅。
手机在确定防晒霜涂抹情况(比如涂抹防晒霜的区域,没有涂抹防晒霜的区域)后,可以向用户提示防晒霜涂抹情况。本申请实施例中,可以在彩色图像或灰度图像中标记防晒霜涂抹情况。图8介绍了在彩色图像中标记防晒霜涂抹情况的过程,图9介绍了在灰度图像中标记防晒霜涂抹情况的过程。如下结合图8、图9进行具体阐述。其中,图9的(a)、(b)的实现过程可参见图8的(a)、(b),本文不再赘述。
在一些实施例中,手机可以在彩色图像的人脸(第一人脸)中标记提示信息(第一信息)。其中,该彩色图像可以是在用户打开镜子应用后,手机调用彩色摄像头采集的。本申请实施例并不限制采集彩色图像的时机。比如,可以是在紫外摄像头采集紫外图像同时,彩色摄像头同步采集彩色图像,或者彩色摄像头与紫外摄像头交替采集。
作为一种可能的实现方式,手机可通过图像匹配算法,在所述彩色图像的所述人脸上,确定与上述第一区域对应的第二区域,即,将紫外图像中的第一区域匹配到彩色图像的第二区域。也就是说,可以在彩色图像中标记防晒霜提示信息。示例性的,手机根据图8的(c)所示紫外图像确定出防晒霜涂抹的区域为407a(第一区域的一个示例),并通过图像匹配算法将涂抹防晒霜的区域407a匹配到图8的(d)所示RGB图像的407a’(第二区域的一个示例),并在该RGB图像中标记出涂抹防晒霜的区域407a’。可以理解,手机还可以将紫外图像的未涂抹防晒霜区域408a(第一区域的又一示例)匹配到RGB图像的区域408a’,并在RGB图像中标记区域408a’(第二区域的又一示例)。
其中,彩色图像中的第二区域是紫外图像中第一区域对应的区域。所述第二区域为如下任一种或多种区域:涂抹防晒霜的区域,未涂抹防晒霜的区域,防晒霜涂抹不均匀的区域,防晒霜涂抹厚度大于第一阈值的区域,防晒霜涂抹厚度小于第二阈值的区域。
可选的,为了用户能够清晰辨别防晒霜的涂抹情况,可以用不同颜色标记防晒霜的不同涂抹情况。可选的,为了更加容易区分不同区域,用于表征不同涂抹情况的颜色之间的差异可以比较大。比如,用绿色轮廓线(第一信息的一个示例)标记涂抹防晒霜的区域,以便提示有防晒霜涂抹的位置,用红色轮廓线标记未涂抹防晒霜的区域,以便提示没有防晒霜涂抹的位置。再比如,仅标记没有涂抹防晒霜的位置,或者,仅标记涂抹防晒霜的位置。在彩色图像中标记防晒霜涂抹信息,可以更便于用户理解这些标记信息。并且,彩色图像往往能记录更加精细的皮肤状态,更利于指导用户进行防晒霜补涂。
可选的,手机可以在彩色图像的脸部相应区域呈现提示文本。比如,在涂抹防晒霜的区域附近,向用户提示该区域为涂抹防晒霜的区域。
可选的,手机可以隐藏提示文本,待检测到用户点击相应区域的情况下,手机呈现提示文本。比如,当检测到用户点击涂抹防晒霜的区域后,手机再向用户提示该区域是涂抹防晒霜的区域。
本实施例中,通过在彩色图像中标记诸如不同颜色的轮廓线,自动标记提示文本,基于用户操作标记提示文本等方式,能够显式或隐式向用户提示防晒霜涂抹情况。
可选的,所述第一信息还用于指示所述第二区域的防晒霜涂抹均匀程度和/或防晒霜涂抹薄厚程度和/或是否涂抹防晒霜。示例性的,如图4所示,手机还可以显式的以文本等形式提示防晒霜的涂抹薄厚程度,均匀程度等。或者,在用户点击虚线框包围的区域时,手机显示文本提示防晒霜的涂抹均匀度、薄厚程度等。
在另一些实施例中,手机在根据紫外图像确定防晒霜涂抹情况后,还可以在灰度图像中标记防晒霜涂抹情况。灰度图像可以是对紫外图像进行处理得到的图像。
作为一种可能的实现方式,可以对紫外图像进行处理,以加大涂抹防晒霜区域与未涂抹防晒霜区域之间的像素差值差异。比如,可以将紫外图像中涂抹有防晒霜的区域的颜色加深,将紫外图像中没有涂抹防晒霜的区域的颜色变浅,使得紫外图像中涂抹有防晒霜的区域和没有涂抹防晒霜的区域的颜色深浅对比加深。
可选的,用于标记防晒霜涂抹情况的灰度图像可以是二值图像。也就是说,可以对紫外图像进行处理,得到相应的二值图像,并在该二值图像上标记防晒霜涂抹情况。其中,二值图像,即图像中像素的像素值有两种情况。可选的,为了加大涂抹区域与未涂抹区域之间的像素值差异,可以设置二值图像的两种像素值之间的差异大于一定阈值。示例性的,二值图像中任意像素的像素值可以是255(表示白色),或者0(表示黑色)。
以二值图像包括黑色像素和白色像素为例,作为一种可能的实现方式,手机计算紫外图像对应的二值图像,可以实现为:手机遍历紫外图像中人脸图像所包 括的像素,对于遍历到的像素值大于阈值的像素,该像素在二值图像中相应位置的像素值被手机设置为诸如白色(白色像素的像素值比如可以是255),对于遍历到的像素值小于或等于阈值的像素,该像素在二值图像中相应位置的像素值被手机设置为诸如黑色(黑色像素的像素值比如可以是0)。示例性的,手机遍历如图9的(c)所示紫外人脸图像中所包括的各个像素,其中,紫外人脸图像的阴影区域407a所包括像素(对应涂抹防晒霜的区域)的像素值小于二值化阈值,那么,紫外图像的该阴影区域407a在二值图像中相应位置,即区域407a”所包括像素的像素值被设置为黑色。白色区域408a所包括像素的像素值大于阈值(对应可能没有涂抹防晒霜的区域),那么,紫外图像的白色区域408a在二值图像中相应位置,即区域408a’所包括像素的像素值被设置为白色。
需要说明的是,本申请实施例对手机计算二值图像的具体实现,以及计算二值图像时使用的阈值不做限制。
示例性的,如图9的(c)所示,假设紫外图像中阴影区域407a表示涂抹防晒霜的区域,其中,阴影中斜线的密度越高,表示防晒霜涂抹的越厚(对应区域的像素值越低),斜线的密度越低,表示防晒霜涂抹的越薄(对应区域的像素值越高)。用户肉眼观察的情况下,很可能难以辨别一些像素值比较接近的区域,比如,防晒霜涂抹薄的区域的像素值与没有涂抹防晒霜的区域的像素值比较接近,则用户可能将涂抹防晒霜的区域误判断为未涂抹防晒霜的区域,无法清晰辨别防晒霜的涂抹情况。为了提高涂抹和未涂抹区域之间的辨识度,手机可以通过二值图像标记是否涂抹防晒霜。示例性的,手机可以计算如图9的(c)所示紫外图像对应的二值图像,该二值图像如图9的(d)所示。由图9的(d)可以看出,该二值图像中两种像素值之间的差异较大。如此一来,涂抹防晒霜的区域和未涂抹防晒霜的区域之间的色彩对比加深,用户可更加方便直观的辨别出哪些部位涂抹了防晒霜,哪些部位没有涂抹防晒霜。
在另一些实施例中,在灰度图像中标记防晒霜涂抹情况,也可以是直接在紫外图像中标记防晒霜涂抹情况。比如,对于紫外图像中没有涂抹防晒霜的区域,用阴影或虚线框标记出该区域。
在另一些实施例中,如图10所示,手机可提供灰度视图和彩色视图两种选项控件。在检测到用户指示使用彩色视图的操作时,比如,手机检测到用户点击彩色视图控件时,在彩色图像上标记防晒霜涂抹情况。在检测到用户指示使用灰度视图的操作时,比如,手机检测到用户点击灰度视图控件时,则呈现灰度图像(比如二值图像),并通过灰度图像标记防晒霜涂抹情况。
考虑到某些情况下,可能有干扰因素影响检测。比如,在紫外图像中,面部黑痣的成像特征与涂抹有防晒霜的部位的成像特征类似,因此,如果用户脸部有黑色痣,那么,黑痣有可能被误判为涂抹防晒霜的部位,意味着,黑痣很可能干扰防晒霜的检测结果。再比如,在紫外图像中,涂抹防晒霜但反光部位的成像特征与未涂抹防晒霜部位的成像特征类似,像素值均较高。基于该反光部位的该特点,在检测过程中,涂抹防晒霜但反光部位有可能误判为未涂抹防晒霜的部位。
可选的,为了降低检测结果的误判,进一步提升检测的识别准确度。比如, 为了降低黑痣等部位被误判为涂抹防晒霜部位的概率,手机可以排除诸如黑痣等干扰因素的成像区域,即不将干扰区域确定为涂抹防晒霜的区域。考虑到用户涂抹防晒霜时,通常涂抹防晒霜的面积会较大。作为一种可能的实现方式,手机根据所述第四区域(人脸不包括眼睛、眉毛、嘴唇的区域)包含的像素的像素值,在所述第四区域中确定多个目标子区域,将所述多个目标子区域中面积大于或等于第四阈值的目标子区域确定为所述第一区域。所述目标子区域满足第三条件,所述第三条件包括但不限于如下任一项或多项:所述目标子区域的像素均值小于第一阈值,所述目标子区域包含的像素的像素值小于第二阈值。也就是说,对于紫外图像中面积小于面积阈值的非高亮区域,手机不将其确定为涂抹防晒霜的区域。对于紫外图像中面积大于或等于面积阈值的非高亮区域,手机将其确定为涂抹防晒霜的区域。
示例性的,如图11的(a)、(b)、(c)所示,紫外图像中像素值小于像素值阈值(颜色较深)的区域为407a、409a。其中,对于面积大于或等于阈值的区域407a,手机将其对应的部位确定为涂抹防晒霜的部位。对于面积小于阈值的区域409a,由于其可能是干扰因素的成像,手机不将其对应的部位确定为涂抹防晒霜的部位。最终,手机计算得到的涂抹防晒霜的部位为区域407a对应的部位,手机计算出的未涂抹防晒霜的部位为区域408a对应的部位。可选的,手机可显示图11的(d)所示的提示框,以提示用户对没有涂抹防晒霜的部位进行补涂。
可选的,为了降低反光部位被误判为未涂抹防晒霜部位的概率,本申请实施例中可以对多个角度的脸部紫外图像进行处理,并根据多角度的脸部紫外图像的处理结果筛查反光部位,以便确定涂抹防晒霜的区域以及未涂抹防晒霜的区域。具体的,作为一种可能的实现方式,参见图12A,上述步骤S104可包括如下步骤:
S201a、手机获取紫外图像901中人脸图像的高亮区域408a以及非高亮区域407a。
如图13所示,紫外图像901-903是开启防晒检测功能后,手机通过紫外摄像头采集的三张紫外图像。紫外图像901-903满足预设的角度条件,比如满足图像中人脸相对于手机的角度在左转30度至右转30度范围内。其中,紫外图像901可以是从多角度的多张紫外图像中选择的基准图像。可选的,基准图像是包括更多成像细节的图像。在一个示例中,基准图像可以是正脸角度的紫外图像。当然,基准图像可以是灵活确定的,并不限制基准图像的确定方式。
作为一种可能的实现方式,手机可按照上述实施例的方案确定紫外图像901中的人脸图像,并可以在人脸图像的眼部、嘴唇(通常不涂抹防晒霜的区域)之外的区域查找高亮区域以及非高亮区域,以便降低计算复杂度。
作为一种可能的实现方式,对紫外图像901,可根据像素值进行阈值分割,以定位紫外图像901中的高亮区域以及非高亮区域。比如,设置阈值为T1(可灵活设置),手机可遍历图像901的像素,对于图像901中的像素,当像素值f(x,y)>T1,则手机将该像素作为高亮区域的构成像素,反之,当像素值f(x,y)<T1,则手机将该像素作为非高亮区域的构成像素。可选的,手机还可以计算高亮区域相对于眼睛、鼻子、嘴唇中一个或多个特定部位的相对位置坐标,以及手机可以计 算非高亮区域相对于眼睛、鼻子、嘴唇中一个或多个特定部位的相对位置坐标。
示例性的,如图13的(a)所示,手机计算出紫外图像901中的高亮区域是408a以及非高亮区域407a。
S202a、根据紫外图像901的高亮区域408a,获取紫外图像902中与高亮区域408a对应的目标区域408b,以及获取紫外图像903中与高亮区域408a对应的目标区域408c。
作为一种可能的实现方式,可以采用图像匹配算法匹配不同角度紫外图像中的相应区域。
示例性的,仍参见图13,根据高亮区域408a在紫外图像901中的位置以及图像匹配算法,手机可获取高亮区域408a在紫外图像902中对应的目标区域408b。其中,高亮区域408a与紫外图像901中鼻子、眼部等特定部位的位置关系,目标区域408b与紫外图像902中鼻子、眼部等特定部位的位置关系,这两个位置关系一致。
再示例性的,仍参见图13,根据高亮区域408a在紫外图像901中的位置坐标以及图像匹配算法,手机可获取高亮区域408a在紫外图像903中对应的目标区域408c。
S203a、判断目标区域408b、408c是否包括目标非高亮区域。若至少一个目标区域包括目标非高亮区域,则执行步骤S204a,若目标区域均不包括目标非高亮区域,则执行步骤S205a。
以紫外图像902为例,如图13所示,若紫外图像902的目标区域408b包括非高亮区域,则说明紫外图像901中的高亮区域408a很可能包括反光区域和未涂抹防晒霜区域。此种情况下,将高亮区域408a直接确定为没有涂抹防晒霜的区域,可能导致误判。为了避免此种误判,可执行下述步骤S204a,从高亮区域408a中剔除可能导致误判的区域,被剔除的该区域可以由相应的非高亮区域替代。反之,若正脸紫外图像中高亮区域对应的一个或多个目标区域(比如图13的(b)所示目标区域408b、图13的(c)所示目标区域408c)均不包括非高亮区域,即全部角度的紫外图像的该区域全部为高亮区域,则说明正脸紫外图像中的高亮区域408a确实由未涂抹防晒霜导致,那么,可以不调整待检测区域,直接执行下述步骤S205a。
再示例性的,仍参见图13,对于高亮区域408a对应的目标区域408c(包括在紫外图像903中),手机检测其中是否包括非高亮区域。经检测,目标区域408c包括非高亮区域411a。
S204a、根据紫外图像901的高亮区域408a,非高亮区域407a以及目标非高亮区域,确定涂抹防晒霜的区域以及没有涂抹防晒霜的区域。
示例性的,如图13所示,手机可以根据紫外图像901、紫外图像902、紫外图像903筛查反光部位。具体的,如图13的(a)所示,阴影区域407a的像素值小于像素值阈值,区域408a的像素值大于像素值阈值。手机可以根据上述方法将区域408a对应的部位初步确定为可能没有涂抹防晒霜的部位,将区域407a初步确定为涂抹防晒霜的区域。其中,区域408a的白色像素可能是由于该区域408a对应 的部位未涂抹防晒霜,或者,也可能由于区域408a对应的部分部位未涂抹防晒霜,另外部分虽涂抹防晒霜但反光。为了进一步检测区域408a是否确实对应未涂防晒霜部位,手机可以在图13的(b)所示紫外图像902以及图13的(c)所示紫外图像903的相应位置,即区域408b、408c中查找是否存在目标非高亮区域。经查找,区域408b中存在目标非高亮区域410a,这说明目标非高亮区域410a对应的部位很可能涂抹有防晒霜但在正脸角度下产生了反光。那么,手机将区域410a确定为涂抹防晒霜的区域。类似的,对于紫外图像903的目标区域408c中存在目标非高亮区域411a,那么,手机将区域目标非高亮区域411a确定为涂抹防晒霜的区域。如此,手机最终计算的涂抹防晒霜的区域包括区域407a,区域410a以及区域411a,没有涂抹防晒霜的区域为408a中除区域410a以及区域411a外的区域。可选的,图13的(d)示出了防晒霜涂抹区域以及未涂抹区域。需要说明的是,图13的(d)仅示例性示出了防晒霜涂抹区域以及未涂抹区域,手机在实际处理过程中,也可以不生成图13的(d)所示图像。
可选的,手机可以可显示图13所示的提示框801,其中,虚线框包围区域是涂抹防晒霜的部位,虚线框之外的区域是没有涂抹防晒霜的区域。
S205a、根据图像901的非高亮区域407a以及高亮区域408a,确定涂抹防晒霜的区域以及没有涂抹防晒霜的区域。
在另一些情况下,手机初步确定可能未涂抹防晒霜的候选区域(比如图13所示的区域408a)后,结合其他角度的紫外图像,没有发现诸如反光区域,这意味着,候选区域确实为未涂抹防晒霜的区域。仍以图13为例,手机根据紫外图像901初步确定未涂抹防晒霜的区域408a之后,查找紫外图像902以及紫外图像903,若紫外图像902的相应位置(区域408b)、紫外图像903的相应位置(408c)中均不存在目标非高亮区域,则说明不存在诸如反光问题导致的误判问题,区域408a确实是未涂抹防晒霜的区域。即,手机最终确定没有涂抹防晒霜的区域为408a,涂抹防晒霜的区域为407a。
本申请实施例还提供一种检测方法,使用该方法可以确定涂抹防晒霜的区域以及没有涂抹防晒霜的区域。参见图12B,该方法包括:
S201b、手机获取多张紫外图像中人脸图像的高亮区域和/或非高亮区域。
对于一张紫外图像来说,该紫外图像可包括高亮区域和/或非高亮区域。手机可根据该紫外图像中的高亮区域确定该紫外图像中的非高亮区域,类似的,手机可根据该紫外图像中的非高亮区域确定非高亮区域。基于该原理,步骤S201b可实现为:手机获取多张紫外图像的每张紫外图像中人脸图像的高亮区域,并可以根据高亮区域确定非高亮区域。或者,手机获取每张紫外图像中人脸图像的非高亮区域,并可以根据非高亮区域确定高亮区域。或者,手机获取每张紫外图像中人脸图像的非高亮区域以及高亮区域。
示例性的,仍参见图13,在开启防晒检测功能后,手机通道紫外摄像头采集的紫外图像包括紫外图像901-903。手机可按照上述实施例的方法分别确定紫外图像901-903中的人脸图像1-3,并分别确定人脸图像1-3中的高亮区域以及非高亮区域。示例性的,在图13中,紫外图像901中人脸图像1的高亮区域为408a,非 高亮区域为407a,紫外图像902中人脸图像2的高亮区域为408b,非高亮区域为410a以及407b,紫外图像903中人脸图像的高亮区域为408c,非高亮区域为407c以及411a。
S202b、手机根据高亮区域和/或非高亮区域,确定涂抹防晒霜的区域以及未涂抹防晒霜的区域。
示例性的,在紫外图像901中人脸图像1的高亮区域为408a,非高亮区域为407a,紫外图像902中人脸图像2的高亮区域为408b,非高亮区域为410a以及407b,紫外图像903中人脸图像的高亮区域为408c,非高亮区域为407c以及411a中,紫外图像中的区域407a对应紫外图像902中的407b,对应紫外图像903中的407c,紫外图像中的区域408a对应紫外图像902中的408b,对应紫外图像903中的408c。紫外图像903中的区域411a与紫外图像902中的区域410a并不具有对应关系,且紫外图像901中不存在于区域410a、411a对应的非高亮区域,手机确定涂抹防晒霜的区域为图13的(d)所示407a、410a、411a,没有涂抹防晒霜的区域为408a中除410a以及411a外的区域。
再示例性的,手机可获取紫外图像901中的非高亮区域407a,紫外图像902的非高亮区域407b,410a,紫外图像903中的非高亮区域407c,以及411a,并根据这些非高亮区域,确定高亮区域,并确定涂抹防晒霜的区域以及未涂抹防晒霜的区域。
再示例性的,手机可获取紫外图像901中的高亮区域408a,紫外图像902的高亮区域,即人脸中除眼睛、眉毛、嘴巴、407b、410a之外的区域,紫外图像903中的高亮区域,即人脸中除眼睛、眉毛、嘴巴、407c、411a之外的区域,并根据这些高亮区域,确定非高亮区域,并确定涂抹防晒霜的区域以及未涂抹防晒霜的区域。
本申请实施例还提供一种检测方法,使用该方法可以对涂抹防晒霜的涂抹均匀度进行检测。作为一种可能的实现方式,参见图14,本申请实施例中,可以从待检测区域提取多个图像特征,并将多个图像特征构成的特征向量输入分类器,由分类器输出相应部位防晒霜涂抹的均匀度。
其中,待检测区域可以是涂抹防晒霜的区域。可选的,确定防晒霜涂抹区域的方式可参见图12A或图12B等对应的实施例,这里不再赘述。
其中,特征向量包括多个图像特征。
首先,对本申请实施例涉及的分类器的训练过程进行介绍。训练过程如图15所示,训练用于识别防晒霜是否涂抹均匀的分类器,需要提供N(N为正整数)个样本,样本是涂抹均匀度已知的图像区域的特征向量及其标签(表征图像区域对应的部位是否均匀涂抹防晒霜),对多个样本进行训练即可得到分类器。
可选的,在训练分类器之前,可以对训练特征向量等数据进行处理,比如进行平滑处理,归一化处理。其中,归一化处理可降低算法的复杂度。平滑处理可进一步包括降噪、拟合等操作,以减少统计误差带来的影响。
可选的,为了提升分类器的识别准确率,可以对分类器进行评估、测试。当分类器的识别率达到一定阈值,说明该分类器已训练好。当分类器的识别率较低, 可以继续训练分类器,直至分类器的识别准确率达到一定阈值。
可选的,分类器的训练过程可以在端侧(比如手机等终端)或云侧(比如服务器)。训练可以是离线训练或在线训练。本申请实施例对分类器的具体训练方式不做限制。后续,训练好的分类器可根据输入的涂抹均匀度未知的图像区域的特征向量,输出相应图像区域的防晒霜是否涂抹均匀。
图16示出了基于分类器识别防晒霜涂抹均匀度的方法流程。如图16所示,该方法包括:
S301、将待检测区域划分为多块子区域,分别提取多块子区域的特征向量。
其中,待检测区域即涂抹防晒霜的区域。确定涂抹防晒霜区域的具体实现可参见图12A或图12B等对应的实施例,这里不再赘述。
示例性的,如图17所示,手机将该待检测区域(即涂抹防晒霜的区域)分块,得到多块图像区域分别是图17的(b)所示区域1-区域3,并分别提取区域1-3的特征向量。假设区域1的特征向量为
区域2的特征向量为
区域3的特征向量为
其中,f
ji为区域j(j为1,2,3)的第i(i为正整数)个图像特征。三个特征向量可组合为如下特征矩阵:
其中,图17的(a)所示待检测区域中,未标注斜线的方框,即白色方框表示没有涂抹防晒霜部位的像素。标注斜线的方框表示涂抹防晒霜部位的像素。斜线的密度可表征像素值高低。在一个示例中,斜线密度越高,像素值越小。可选的,像素值高低用用来表征涂抹薄厚。
302、将多块区域的特征向量输入分类器,得到分类器对该多块区域的识别结果。
示例性的,仍参见图17,手机将图像区域1-区域3的特征向量输入分类器,即将
这三个特征向量分别输入分类器,或者,将三个特征向量进行组合,形成特征矩阵,将特征矩阵输入分类器。之后,分类器根据输入的特征向量检测图像区域1-区域3的防晒霜涂抹是否均匀。经运算,分类器输出区域1-区域3的分类结果:区域1<->1,区域2<->0,区域3<->0。其中,1表示防晒霜涂抹不均匀,0表示防晒霜涂抹均匀。即,图像区域1的防晒霜涂抹不均匀,图像区域2、3的防晒霜涂抹均匀。
S303、判断待检测区域划分成的每块区域的面积是否小于阈值。若是,则执行步骤S304。若否,则从待检测区域划分成的多块区域中选择分类结果为防晒霜未涂抹均匀的图像区域,将这部分图像区域更新为待检测区域,并跳转到步骤S301。
可以理解,为了对皮肤防晒霜涂抹情况进行更精细的检测,可以对更小皮肤尺寸内的防晒霜涂抹均匀度进行检测。因此,可以设置阈值。当待检测区域划分成的图像区域的面积小于阈值T3时,认为满足皮肤防晒霜涂抹均匀度的检测粒度要求,可以不再进一步划分更小的图像区域。反之,当待检测区域划分成的图像区域的面积大于阈值时,需要再将较大的图像区域进行划分,形成较小的图像区域,并对较小的图像区域,进行涂抹均匀度检测,直至涂抹均匀度检测的图像区域尺寸粒度满足阈值,执行步骤S304。
示例性的,经判断,假设如图17所示,区域1-区域3中每块图像区域的面积大于阈值,则手机根据区域1-区域3的分类结果:区域1<->1(涂抹不均匀),区域2<->0(涂抹均匀),区域3<->0,从区域1-区域3中选择分类结果为防晒霜未涂抹均匀的图像区域,即选择区域1,并将区域1更新为后续步骤中的待检测区域。
之后,手机可跳转到步骤S301,将待检测区域(区域1)划分为多块区域,如图18所示,区域1被划分为区域4-区域6。并且,手机可分别提取多块区域4-区域6的特征向量,比如,
区域4-区域6的特征向量还可组合为如下特征矩阵:
需要说明的是,每次划分更小的图像区域时,将一块大区域划分成的小区域的数目可以不同或相同。
作为一种可能的实现方式,手机可根据M=N/iter划分更小图像区域。其中,iter为预设系数,M为n+1次分块的数目,N为第n次分块的数目,iter,M,N均为正整数,n为大于或等于1的整数。
示例性的,在首次划分图像区域时,待检测区域可以划分为M=16块图像区域J。以iter=2为例,第二次划分图像区域时,N=16,iter=2,那么,对于首次划分形成的图像区域J,将图像区域J划分为M=16/2=8块图像区域K。第三次划分图像区域时,N=8,iter=2,那么,对于第二次划分形成的图像区域K,将图像区域K划分为M=8/2块图像区域,以此类推。当然,手机划分图像区域的方式还可以为其他,本申请实施例对此不进行限制。
S304、根据分类器对待检测区域包括的多个图像区域的分类结果,获取防晒霜的涂抹均匀度。
需要说明的是,待检测区域包括的图像区域的面积可以不同。比如,待检测区域包括图17所示区域1-区域3,区域1可进一步包括图18所示的区域4-区域6其中,区域1-3的分类结果如图17所示,区域4-区域6的分类结果如图18所示。可以看出,区域4为防晒霜涂抹不均匀的区域,防晒霜涂抹均匀的区域为区域2、35、6。
作为一种可能的实现方式,手机计算待检测区域包括的每块图像区域的像素均值,并根据像素均值判断每块图像区域的防晒霜涂抹薄厚程度。通过检测均匀区域以及非均匀区域像素平均值ˉF。作为一种可能的设计,可以根据像素均值将防晒霜涂抹薄厚程度划分为几个层次。比如,划分为[0,T4)、[T4,T5)、[T5,T6)、[T6,T7)、[T7,255]这几个像素值区间,若某区域的像素均值ˉF<阈值T4,则确定该区域的防晒霜涂抹非常厚(过厚),若T4≤ˉF<阈值T5,则确定防晒霜涂抹厚度适中,若T5≤ˉF<阈值T6,则确定防晒霜涂抹较薄,若T7>ˉF≥T6,则确定防晒霜非常薄(过薄),若ˉF>T7,则确定没有涂抹防晒霜。示例性的,手机计算图17所示区域2、区域3、图18所示区域4-区域6中每块图像区域的像素均值,假设区域2,3,5,6的像素均值T4≤ˉF<阈值T5,则确定区域2,3,5,6的防晒霜涂抹厚度适中,区域4的像素均值F<T4,则手机可确定该区域的防晒霜涂抹厚度较厚。可 以看出,由防晒霜涂抹均匀度检测流程,手机可以确定防晒霜涂抹不均匀的位置,还可以确定防晒霜涂抹不均匀位置的涂抹薄厚程度。在一些实施例中,手机还可以通过防晒霜涂抹均匀度检测流程确定防晒霜涂抹的其他信息,比如没有涂抹防晒霜的位置,防晒霜涂抹均匀位置的涂抹薄厚程度等。
手机确定防晒霜涂抹情况后,可向用户提示防晒霜涂抹情况。包括但不限于提示未涂抹防晒霜的位置,提示涂抹不均匀的位置,提示指定区域的防晒霜涂抹薄厚程度。
作为一种可能的实现方式,手机在向提示防晒霜涂抹情况时,可以在彩色图像中标记提示信息,用不同颜色、界面效果标记不同的涂抹情况。界面效果包括但不限于闪烁效果、发光效果、阴影效果。比如,某个区域为防晒霜涂抹不均匀的区域,其中,该区域的子区域A为涂抹厚度薄的区域,则子区域A被手机以红色且闪烁效果标记,这样一来,用户的注意力会集中在防晒霜涂抹较薄的区域,并对该区域进行防晒霜补涂。
在另一些实施例中,灰度图像中也可以通过一些界面效果来区分防晒霜涂抹情况。比如,灰度图像中均为黑色像素区域,黑色像素区域1标记闪烁效果,闪烁效果可表示防晒霜涂抹较薄,黑色像素区域2未标记闪烁效果,未标记闪烁效果可表示防晒霜涂抹厚度适中。
需要说明的是,本申请实施例还可以根据实际应用场景另行划分像素值区间,并且,对各像素值区间代表的防晒霜涂抹薄厚程度不做限制。
并且,本申请实施例中以像素值在0到255之间为例进行举例,在另一些实施例中,当采用更多或更少位(bit)表示颜色时,像素值还可以有其他数值范围,在此统一说明。
可选的,不同区域的颜色、界面效果可以由***预设,也可以由用户设置。
可选的,手机可在提示界面上以文字等形式解释每种颜色、界面效果代表的防晒霜涂抹情况的含义。或者,也可以是用户点击相应区域后,手机以文字等形式说明每种颜色、界面效果代表的防晒霜涂抹情况的含义。
本申请实施例还提供一种检测方法,在第六图像上确定第三区域,并根据上述实施例的第一区域和该第三区域,在第一图像的第一人脸上确定与第一区域和所述第三区域对应的所述第二区域。其中,第一图像、第六图像为不同角度的图像。所述第三区域为如下任一种或多种区域:涂抹防晒霜的区域,未涂抹防晒霜的区域,防晒霜涂抹不均匀的区域,防晒霜涂抹厚度大于第一阈值的区域,防晒霜涂抹厚度小于第二阈值的区域。
示例性的,以图13为例,第一图像是紫外图像901,第六图像是图像902,假设手机在紫外图像901上确定的第一区域为非高亮区域407a,手机在紫外图像902上确定的第三区域是非高亮区域407b以及410a,那么,手机可根据第一区域和第三区域,在紫外图像901上确定与407b、410a、407a对应的第二区域,即407a以及410a。因此,手机可确定涂抹防晒霜的区域为407a以及410a。
再示例性的,手机还可以采集多张不同角度的图像,比如,采集图13所示的紫外图像901-903,手机可根据三张紫外图像中的区域确定防晒霜涂抹情况。
本申请实施例还提供一种检测方法,手机可以获取当前的天气状态,包括但不限于获取环境当前的紫外线强度。如此,手机可以根据当前的天线状态向用户推荐适合当前天气状态的防晒霜的信息。包括但不限于向用户提示适合当前天气状态的防晒霜的防晒指数(sun protection factor,SPF),防晒霜涂抹薄厚程度。
通常,紫外线强度可分为5级(当然不同分级方式下还可以划分为更多或更少级别),通常紫外线强度的级数越高则紫外线越强,需要用户进行更强的防晒措施。当手机获取到当前紫外线强度较强,则可以通过如下一种或多种方式指导用户防晒:向用户推荐更高防晒指数的防晒霜,向用户提示需要加厚涂抹的防晒霜。
需要说明的是,防晒指数越高的防晒霜对紫外线的吸收能力通常越强,对皮肤的刺激通常越大,因此,手机在向用户推荐防晒霜时,并非向用户推荐防晒指数最高的防晒霜,而是可根据室外紫外线强度选择合适的防晒霜,以避免推荐过高防晒指数的防晒霜导致刺激用户肌肤。
可选的,手机还可以检测用户的皮肤信息,比如肤质信息,是敏感型肌肤,油性肌肤,混合型肌肤等。手机可以根据用户的皮肤信息和其他信息(包括但不限于天气状态信息)向用户推荐适合用户肤质的防晒霜,比如,推荐合适的防晒指数、涂抹厚度等。
本申请实施例中,手机获取天气状态信息,可以是从云侧(比如服务器)获取,还可以是手机自己检测天气状态信息。
以手机获取紫外线强度为例,手机可以检测当前的位置信息,当当前在室内时,手机可以从服务器获取紫外线强度。当当前在室外时,手机可以通过紫外摄像头中的图像传感器捕获紫外线,并判断紫外线强度。比如,手机在采集紫外图像的过程中根据紫外摄像头捕获的紫外线判断紫外线强度。
以用户在室外使用手机进行检测为例,用户在室外使用手机的相机或者镜子等应用查看自己妆容时,当检测到用户触发防晒检测功能(比如点击防晒检测控件)时,手机可以调用诸如彩色摄像头检测人脸。在检测到人脸后,手机可以调用紫外摄像头采集紫外图像,并根据紫外图像对用户面部的防晒霜涂抹情况进行检测。该过程中,手机可以根据紫外摄像头捕获的紫外线确定环境紫外线强度。之后,手机可以根据环境紫外光强度以及用户面部的防晒霜涂抹情况,通过用户界面(user interface,UI)提示用户防晒霜涂抹的相关信息,包括但不限于如下一种或多种信息:当前紫外线的强度情况,推荐用户使用防晒霜的防晒指数,面部防晒霜涂抹是否均匀,面部不均匀涂抹防晒霜的位置,面部未涂抹防晒霜的位置。示例性的,如图19所示,手机在检测到当前紫外线强度以及用户的防晒霜涂抹情况后,显示提示框2401,用于向用户提示当前紫外线强度,需要补涂防晒霜的位置,以及向用户提示所需的防晒霜指数。再比如,如图20所示,手机在检测到当前紫外线强度以及用户的防晒霜涂抹情况后,显示提示框,用于向用户提示当前紫外线强度,以及向用户提示面部防晒霜涂抹均匀。
本申请实施例还提供一种检测方法,在检测到用户触发防晒检测功能的情况下,手机可以检测预设时段内是否存在上一次检测结果。如果在预设时段内没有 执行过检测,则手机可以对防晒霜涂抹情况进行检测,并向用户提示防晒霜涂抹情况。如果在预设时段内已执行过检测,则手机可以向用户提示最近一次防晒霜涂抹情况的检测结果。也就是说,并非在所有情况下,手机都会重新进行检测,如此,能够降低手机的运算量。
其中,预设时段可以与防晒霜的防晒作用时间等相关。比如,假设经统计发现防晒霜通常能够作用3-4小时,那么3-4小时内通常无需补涂防晒霜。
示例性的,手机检测到用户的诸如点击防晒检测控件402的操作后,若手机检测到3小时内已执行过防晒检测流程,则提示用户无需补涂。可选的,手机还可以将上一次防晒霜的检测结果呈现给用户。反之,若手机检测到3小时内没有执行过防晒检测流程,则向用户提示本次检测结果。
再示例性的,当检测到用户触发防晒检测功能时,手机可以判断早上是否有进行检测,若早上没有进行检测,则进行检测,并向用户提示检测结果。若手机检测到早上已进行检测,并且防晒霜已涂抹到位或已补涂到位,则获取当前时间信息,并判断当前时间是否属于预设的防晒检测的时间段(比如12:00-14:00)。若当前时间没有在12:00-14:00,则说明早上涂抹的防晒霜还没有失效,无需重新进行检测。若当前时间在12:00-14:00,则说明早上涂抹的防晒霜可能已失效,那么,手机重新进行检测,并向用户提示本次检测的结果。
需要说明的是,上述方法流程中的某些步骤还可以替换为其他步骤,或者增加或减少部分步骤。本申请实施例中并不限制方法流程包括的具体步骤。
示例性的,在一些实施例中,还可以将镜子等应用中的防晒检测功能单独设置一个快捷图标。当检测到用户的诸如点击该快捷图标的操作后,手机可进行检测。可选的,手机可提示用户正在检测防晒霜涂抹情况。手机检测完成后,可提示用户检测结果。可以看出,该场景中,触发手机开启防晒检测流程的操作为用户打开防晒检测这一快捷应用。
在一些实施例中,用户还可以进行诸如长按镜子应用等操作,触发手机弹出镜子功能框2801。用户可以通过诸如点击防晒检测选项开启防晒检测功能,那么,手机会检测防晒霜涂抹情况,并可以向用户提示防晒霜涂抹情况。
再示例性的,检测是否涂抹防晒霜,还可以实现为:手机将人脸图像输入用于检测是否涂抹防晒霜的分类器,之后手机根据分类器的分类结果确定涂抹防晒霜的区域以及没有涂抹防晒霜的区域。再比如,本申请实施例中的防晒霜涂抹情况均可以根据分类器的识别结果确定。用于检测是否涂抹防晒霜的分类器的训练以及使用过程可参见图15对应的实施例,这里不再赘述。
在一些实施例中,当手机向用户提示了防晒霜涂抹情况后,若用户仍未退出镜子应用,手机通过诸如RGB摄像头检测到用户仍使用镜子应用进行补涂,则手机可持续检测用户的防晒霜补涂情况。比如,手机可按一定时间间隔开启紫外摄像头采集人脸图像,以便检测用户的防晒霜补涂情况,直至用户补涂的防晒霜足以抵抗当前环境的紫外线。
当然,当手机向用户提示了防晒霜涂抹情况后,手机可以并不主动检测用户的防晒霜补涂情况,而是在检测到用户的诸如点击防晒检测控件的操作之后,才 再次触发检测流程。
本申请实施例列举的界面均是示例性的,本申请实施例并不限制各界面的实现形式。比如,在一些界面中,还可以提示没有涂抹防晒霜的位置以及涂抹防晒霜但未涂抹均匀的位置。再比如,在向用户提示防晒霜涂抹情况的界面,均可提供彩色视图和灰度视图两种选项。再比如,在检测到用户点击防晒检测控件402后的预设时间段内,手机可以停止显示防晒检测控件402,以便屏幕有更多的显示空间来显示其他必要控件。
需要说明的是,上述实施例中主要以面部检测作为实例,但应理解,本申请实施例的检测方法还可适用于其他各种需要防晒的对象。比如包括但不限于人脸、身体的其他部位。或者,也可以适用于非人体对象,比如,需要进行防晒的车辆的部位。
且,上述实施例主要以通过RGB图像识别人脸图像,再根据RGB图像中的人脸图像查找紫外图像的人脸图像为例对人脸图像识别过程进行说明。在另一些实施例中,还可以直接使用人脸识别算法处理紫外图像,以便识别紫外图像中的人脸图像,以及提取眼部、嘴唇等特征点。本申请实施例并不限制获取紫外图像中人脸图像的具体实现方式。
此外,上述实施例主要以根据不同角度的三张紫外图像排除反光部位对防晒检测的影响为例,需要说明的是,为了排除反光等对防晒检测的影响,不同角度的紫外图像的数目还可以为其他。
本申请实施例中,彩色摄像头与紫外摄像头同步采集,可以是同一时刻采集,也可以在预设时间间隔内采集,可以是紫外摄像头先采集,在预设时间间隔内,彩色摄像头采集,或者,彩色摄像头先采集,在预设时间间隔内,紫外摄像头采集。本申请实施例不限制两者的采集时机以及采集条件。
本申请实施例主要以基于紫外图像的像素值确定防晒霜涂抹情况为例进行说明,还可以基于紫外图像的其他图像特征确定防晒霜涂抹情况。比如,基于图像特征中的方差、标准差、最大像素值,最小像素值等确定防晒霜涂抹均匀程度、涂抹薄厚程度。
本申请另一些实施例提供了一种装置,该装置可以是上述电子设备(比如折叠屏手机)。该装置可以包括:显示屏、存储器和一个或多个处理器。该显示屏、存储器和处理器耦合。该存储器用于存储计算机程序代码,该计算机程序代码包括计算机指令。当处理器执行计算机指令时,电子设备可执行上述方法实施例中手机执行的各个功能或者步骤。该电子设备的结构可以参考图2A-图2F所示的电子设备。
其中,该电子设备的核心结构可以表示为图21所示的结构,该核心结构可包括:处理模块1301、输入模块1302、存储模块1303、显示模块1304。
处理模块1301,可包括中央处理器(CPU)、应用处理器(Application Processor,AP)或通信处理器(Communication Processor,CP)中的至少一个。处理模块1301可执行与用户电子设备的其他元件中的至少一个的控制和/或通信相关的操作或数据处理。具体地,处理模块1301可用于根据一定的触发条件,控制主屏上显示的内容。或者根据预设规则确定屏幕上显示的内容。处理模块1301还用于将输入的指令或数据进行处理,并根据处理 后的数据确定显示样式。
输入模块1302,用于获取用户输入的指令或数据,并将获取到的指令或数据传输到电子设备的其他模块。可选的,输入模块1302的输入方式可以包括触摸、手势、接近屏幕等,也可以是语音输入。例如,输入模块可以是电子设备的屏幕,获取用户的输入操作并根据获取到的输入操作生成输入信号,将输入信号传输至处理模块1301。在本申请实施例中,输入模块可用于接收用户输入的防晒检测指令,和/或执行其他步骤。
采集模块1306,用于采集数据,并将采集到的数据传输到电子设备的其他模块。可选的,采集模块1306可以是电子设备的摄像头,摄像头可以将采集的图像传输至处理模块1301,和/或执行其他步骤。摄像头包括但不限于彩色摄像头、紫外摄像头。
存储模块1303,可包括易失性存储器和/或非易失性存储器。存储模块用于存储用户终端设备的其他模块中的至少一个相关的指令或数据,具体地说,存储模块可记录摄像头采集的图像。
显示模块1304,可包括例如液晶显示器(LCD)、发光二极管(LED)显示器、有机发光二极管(OLED)显示器、微机电***(MEMS)显示器或电子纸显示器。用于显示用户可观看的内容(例如,文本、图像、视频、图标、符号等)。在本申请实施例中,显示模块可实现为显示屏幕。
可选的,图21所示结构还可通信模块1305,用于支持电子设备与其他电子设备通信。例如,通信模块可经由无线通信或有线通信连接到网络,以与其他个人终端或网络服务器进行通信。无线通信可采用蜂窝通信协议中的至少一个,诸如,长期演进(LTE)、高级长期演进(LTE-A)、码分多址(CDMA)、宽带码分多址(WCDMA)、通用移动通信***(UMTS)、无线宽带(WiBro)或全球移动通信***(GSM)。无线通信可包括例如短距通信。短距通信可包括无线保真(Wi-Fi)、蓝牙、近场通信(NFC)、磁条传输(MST)或GNSS中的至少一个。
本申请实施例还提供一种芯片***,如图22所示,该芯片***包括至少一个处理器1401和至少一个接口电路1402。处理器1401和接口电路1402可通过线路互联。例如,接口电路1402可用于从其它装置(例如电子设备的存储器)接收信号。又例如,接口电路1402可用于向其它装置(例如处理器1401)发送信号。示例性的,接口电路1402可读取存储器中存储的指令,并将该指令发送给处理器1401。当所述指令被处理器1401执行时,可使得电子设备执行上述实施例中的各个步骤。当然,该芯片***还可以包含其他分立器件,本申请实施例对此不作具体限定。
本申请实施例还提供一种计算机存储介质,该计算机存储介质包括计算机指令,当所述计算机指令在上述电子设备上运行时,使得该电子设备执行上述方法实施例中手机执行的各个功能或者步骤。
本申请实施例还提供一种计算机程序产品,当所述计算机程序产品在计算机上运行时,使得所述计算机执行上述方法实施例中手机执行的各个功能或者步骤。
通过以上实施方式的描述,所属领域的技术人员可以清楚地了解到,为描述的方便和简洁,仅以上述各功能模块的划分进行举例说明,实际应用中,可以根据需要而将上述功能分配由不同的功能模块完成,即将装置的内部结构划分成不同的功能模块,以完成以上描述的全部或者部分功能。
在本申请所提供的几个实施例中,应该理解到,所揭露的装置和方法,可以通过其它 的方式实现。例如,以上所描述的装置实施例仅是示意性的,例如,所述模块或单元的划分,仅仅为一种逻辑功能划分,实际实现时可以有另外的划分方式,例如多个单元或组件可以结合或者可以集成到另一个装置,或一些特征可以忽略,或不执行。另一点,所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口,装置或单元的间接耦合或通信连接,可以是电性,机械或其它的形式。
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的,作为单元显示的部件可以是一个物理单元或多个物理单元,即可以位于一个地方,或者也可以分布到多个不同地方。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
另外,在本申请各个实施例中的各功能单元可以集成在一个处理单元中,也可以是各个单元单独物理存在,也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现,也可以采用软件功能单元的形式实现。
所述集成的单元如果以软件功能单元的形式实现并作为独立的产品销售或使用时,可以存储在一个可读取存储介质中。基于这样的理解,本申请实施例的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的全部或部分可以以软件产品的形式体现出来,该软件产品存储在一个存储介质中,包括若干指令用以使得一个设备(可以是单片机,芯片等)或处理器(processor)执行本申请各个实施例所述方法的全部或部分步骤。而前述的存储介质包括:U盘、移动硬盘、只读存储器(read only memory,ROM)、随机存取存储器(random access memory,RAM)、磁碟或者光盘等各种可以存储程序代码的介质。
以上内容,仅为本申请的具体实施方式,但本申请的保护范围并不局限于此,任何在本申请揭露的技术范围内的变化或替换,都应涵盖在本申请的保护范围之内。因此,本申请的保护范围应以所述权利要求的保护范围为准。
Claims (18)
- 一种检测方法,其特征在于,所述方法应用于电子设备,所述电子设备包括第一摄像头和第二摄像头;其中,所述第一摄像头为彩色摄像头,所述第二摄像头为紫外摄像头,所述方法包括:通过第一摄像头采集第一图像,通过第二摄像头采集第二图像,其中,所述第一图像包含第一内容;在所述第二图像上确定第一区域;根据所述第一区域,在所述第一图像上确定与所述第一区域对应的第二区域;显示用户界面,所述用户界面包括所述第一内容和第一信息,所述第一信息用于指示所述第二区域。
- 根据权利要求1所述的方法,其特征在于,所述第一内容包含第一人脸;所述第二图像包含第二人脸,所述第一人脸和所述第二人脸属于同一用户;所述在所述第二图像上确定第一区域,具体包括:在所述第二图像的所述第二人脸上,确定所述第一区域;所述在所述第一图像上确定与所述第一区域对应的第二区域,具体包括:在所述第一图像的所述第一人脸上,确定与所述第一区域对应的所述第二区域。
- 根据权利要求1或2所述的方法,其特征在于,所述第一区域为如下任一种或多种区域:涂抹防晒霜的区域,未涂抹防晒霜的区域,防晒霜涂抹不均匀的区域,防晒霜涂抹厚度大于第一阈值的区域,防晒霜涂抹厚度小于第二阈值的区域。
- 根据权利要求1-3中任一项所述的方法,其特征在于,在所述通过第一摄像头采集第一图像,通过第二摄像头采集第二图像后,所述方法还包括:通过所述第一摄像头采集第三图像,当所述第三图像满足第二条件时,通过所述第二摄像头采集第四图像;确定所述第四图像上的第三区域;所述根据所述第一区域,在所述第一图像上确定与所述第一区域对应的第二区域,包括:根据所述第一区域和所述第三区域,在所述第一图像上确定与所述第一区域和所述第三区域对应的所述第二区域;所述第二条件,包括:所述第三图像包括人脸,且所述第三图像包括的人脸相对于所述电子设备的角度在预设角度范围内。
- 根据权利要求1-4中任一项所述的方法,其特征在于,通过第一摄像头采集第一图像,通过第二摄像头采集第二图像,包括:在同一时刻,通过所述第一摄像头采集所述第一图像以及通过所述第二摄像头采集所述第二图像。
- 根据权利要求1-5中任一项所述的方法,其特征在于,所述通过第一摄像头采集第一图像,通过第二摄像头采集第二图像,包括:通过所述第一摄像头采集第五图像;当所述第五图像满足第一条件时,通过所述第一摄像头采集所述第一图像以及通 过所述第二摄像头采集所述第二图像;所述第一条件,包括:所述第五图像包括人脸。
- 根据权利要求1-6中任一项所述的方法,其特征在于,所述电子设备还包括紫外补光灯,通过所述第二摄像头采集所述第二图像,包括:打开所述紫外补光灯,通过所述第二摄像头采集所述第二图像。
- 根据权利要求7所述的方法,其特征在于,所述方法还包括:确定环境紫外线强度;所述通过所述第二摄像头采集所述第二图像,包括:在所述环境紫外线强度小于第三阈值时,打开所述紫外补光灯,通过所述第二摄像头采集所述第二图像。
- 根据权利要求4-8中任一项所述的方法,其特征在于,在所述通过第一摄像头采集第一图像,通过第二摄像头采集第二图像后,以及通过所述第二摄像头采集第四图像之前,所述方法还包括:显示第二信息,所述第二信息用于提示用户调整人脸相对于所述电子设备的角度。
- 根据权利要求2-9中任一项所述的方法,其特征在于,在所述第二图像的第二人脸上,确定所述第一区域,包括:在所述第二图像的所述第二人脸上确定第四区域;根据所述第四区域包含的像素的像素值,在所述第二图像的第四区域中确定所述第一区域。
- 根据权利要求10所述的方法,其特征在于,根据所述第四区域包含的像素的像素值,在所述第二图像的第四区域中确定所述第一区域,包括:根据所述第四区域包含的像素的像素值,在所述第四区域中确定多个目标子区域,将所述多个目标子区域中面积大于或等于第四阈值的目标子区域确定为所述第一区域,所述目标子区域满足第三条件,所述第三条件包括:所述目标子区域的像素均值小于第五阈值,所述目标子区域包含的像素的像素值小于第六阈值。
- 根据权利要求2-10中任一项所述的方法,其特征在于,在所述第二图像上确定第一区域之前,所述方法还包括:根据所述第一图像中的第一人脸,确定所述第二图像中的第二人脸。
- 根据权利要求4所述的方法,其特征在于,所述第三区域为如下任一种或多种区域:涂抹防晒霜的区域,未涂抹防晒霜的区域,防晒霜涂抹不均匀的区域,防晒霜涂抹厚度大于第一阈值的区域,防晒霜涂抹厚度小于第二阈值的区域。
- 根据权利要求10所述的方法,其特征在于,所述第四区域不包括人脸中的眼睛、眉毛、嘴巴对应的区域。
- 根据权利要求1-14中任一项所述的方法,其特征在于,所述第一信息还用于指示所述第二区域的防晒霜涂抹均匀程度和/或防晒霜涂抹薄厚程度和/或是否涂抹防晒霜。
- 一种电子设备,其特征在于,包括:处理器,存储器,所述存储器与所述处理器耦合,所述存储器用于存储计算机程序代码,所述计算机程序代码包括计算机指令, 当所述处理器从所述存储器中读取所述计算机指令,使得所述电子设备执行权利要求1-15任一项所述的方法。
- 一种计算机可读存储介质,所述计算机可读存储介质中存储有指令,其特征在于,当所述指令在电子设备上运行时,使得所述电子设备执行权利要求1-15中任一项所述的方法。
- 一种包含指令的计算机程序产品,其特征在于,当所述计算机程序产品在电子设备上运行时,使得所述电子设备执行权利要求1-15中任一项所述的方法。
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202110877247.1A CN115700841A (zh) | 2021-07-31 | 2021-07-31 | 检测方法及电子设备 |
CN202110877247.1 | 2021-07-31 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2023011348A1 true WO2023011348A1 (zh) | 2023-02-09 |
Family
ID=85120768
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2022/108950 WO2023011348A1 (zh) | 2021-07-31 | 2022-07-29 | 检测方法及电子设备 |
Country Status (2)
Country | Link |
---|---|
CN (1) | CN115700841A (zh) |
WO (1) | WO2023011348A1 (zh) |
Families Citing this family (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN116959052A (zh) * | 2022-04-15 | 2023-10-27 | 华为技术有限公司 | 面部特征检测方法、可读介质和电子设备 |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN104715393A (zh) * | 2013-12-16 | 2015-06-17 | 国际商业机器公司 | 用于推荐正确防晒霜使用的方法和*** |
US20200107772A1 (en) * | 2018-10-08 | 2020-04-09 | HallStar Beauty and Personal Care Innovations Company | Protection and skin damage detection devices and system |
CN112016491A (zh) * | 2020-09-03 | 2020-12-01 | 北京优彩科技有限公司 | 一种智能卸妆镜及其控制方法和ai算法 |
CN113034354A (zh) * | 2021-04-20 | 2021-06-25 | 北京优彩科技有限公司 | 一种图像处理方法、装置、电子设备和刻度存储介质 |
-
2021
- 2021-07-31 CN CN202110877247.1A patent/CN115700841A/zh active Pending
-
2022
- 2022-07-29 WO PCT/CN2022/108950 patent/WO2023011348A1/zh unknown
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN104715393A (zh) * | 2013-12-16 | 2015-06-17 | 国际商业机器公司 | 用于推荐正确防晒霜使用的方法和*** |
US20200107772A1 (en) * | 2018-10-08 | 2020-04-09 | HallStar Beauty and Personal Care Innovations Company | Protection and skin damage detection devices and system |
CN112016491A (zh) * | 2020-09-03 | 2020-12-01 | 北京优彩科技有限公司 | 一种智能卸妆镜及其控制方法和ai算法 |
CN113034354A (zh) * | 2021-04-20 | 2021-06-25 | 北京优彩科技有限公司 | 一种图像处理方法、装置、电子设备和刻度存储介质 |
Also Published As
Publication number | Publication date |
---|---|
CN115700841A (zh) | 2023-02-07 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN112333380B (zh) | 一种拍摄方法及设备 | |
WO2021136050A1 (zh) | 一种图像拍摄方法及相关装置 | |
WO2021078001A1 (zh) | 一种图像增强方法及装置 | |
CN113129312B (zh) | 一种图像处理方法、装置与设备 | |
WO2020077511A1 (zh) | 一种拍摄场景下的图像显示方法及电子设备 | |
WO2020125410A1 (zh) | 一种图像处理的方法及电子设备 | |
US11759143B2 (en) | Skin detection method and electronic device | |
CN109961453B (zh) | 一种图像处理方法、装置与设备 | |
CN111651040B (zh) | 用于肌肤检测的电子设备的交互方法及电子设备 | |
EP3816932B1 (en) | Skin detection method and electronic device | |
CN112037162B (zh) | 一种面部痤疮的检测方法及设备 | |
WO2021258321A1 (zh) | 一种图像获取方法以及装置 | |
US20220180485A1 (en) | Image Processing Method and Electronic Device | |
WO2020029306A1 (zh) | 一种图像拍摄方法及电子设备 | |
CN113170037B (zh) | 一种拍摄长曝光图像的方法和电子设备 | |
EP3809361B1 (en) | Wrinkle detection method and electronic device | |
WO2022179604A1 (zh) | 一种分割图置信度确定方法及装置 | |
WO2023011348A1 (zh) | 检测方法及电子设备 | |
WO2021170129A1 (zh) | 一种位姿确定方法以及相关设备 | |
WO2021208677A1 (zh) | 眼袋检测方法以及装置 | |
CN115760931A (zh) | 图像处理方法及电子设备 | |
EP3809310B1 (en) | Method and electronic device for detecting open and closed states of eyes | |
CN111417982B (zh) | 一种色斑检测方法及电子设备 | |
WO2023065994A1 (zh) | 紫外线检测方法及电子设备 | |
WO2024082976A1 (zh) | 文本图像的ocr识别方法、电子设备及介质 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
NENP | Non-entry into the national phase |
Ref country code: DE |