WO2015186519A1 - 画像処理装置及び画像表示装置 - Google Patents
画像処理装置及び画像表示装置 Download PDFInfo
- Publication number
- WO2015186519A1 WO2015186519A1 PCT/JP2015/064466 JP2015064466W WO2015186519A1 WO 2015186519 A1 WO2015186519 A1 WO 2015186519A1 JP 2015064466 W JP2015064466 W JP 2015064466W WO 2015186519 A1 WO2015186519 A1 WO 2015186519A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- face
- image
- information
- subject
- facial
- Prior art date
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/161—Detection; Localisation; Normalisation
- G06V40/165—Detection; Localisation; Normalisation using facial parts and geometric relationships
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T11/00—2D [Two Dimensional] image generation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30196—Human being; Person
- G06T2207/30201—Face
Definitions
- the present invention relates to an image processing device that generates a suitable image when the imaging direction of an imaging unit and the display direction of a display unit are aligned, and an image display device including the image processing device.
- One is a mirror function that enables a work such as makeup that requires confirmation of one's face by reversing the captured image and displaying it as a mirror image.
- the other is a video chat or TV conference function that enables conversation with a remote partner by displaying a captured image on a display held by the remote partner.
- Patent Document 1 discloses a method for accurately creating a three-dimensional portrait of a person based on a two-dimensional face photograph of the person who is the subject. .
- the head three-dimensional model is generated after adjusting the orientation of the two-dimensional facial photograph, and the orientation of the two-dimensional facial photograph and the adjusted orientation are different values. It is difficult to generate a suitable image. For example, if the adjusted orientation is smaller than the orientation of the 2D face photo, the length between the subject's eyes and nose in the generated image will shrink, or the orientation adjusted relative to the orientation of the 2D face photo When is large, it is difficult to generate a suitable arbitrary face-facing image close to the actual face, for example, the length between the subject's eyes and nose in the generated image increases.
- the present invention has been invented in view of the above problems, and an object of the present invention is to provide an image processing apparatus that generates a suitable image when the imaging direction of the imaging unit and the display direction of the display unit are aligned. .
- a facial organ information detection unit that detects facial organ information that is a position of a facial organ of a subject from an input image, and a face that calculates facial orientation information of the subject from the facial organ information.
- a direction information calculation unit; and an arbitrary face direction image generation unit that generates an image in which the face direction of the subject is changed from the facial organ information and the face direction information, and the arbitrary face direction image generation unit includes the face
- the face orientation information is corrected based on the front face organ information that is the face organ arrangement in the front face of the subject, and then an image of an arbitrary face orientation is obtained.
- the preset positions of the facial organs such as eyes and nose in the front face of the subject and the face organ positions in the front face calculated from the face orientation information are used. Then, after correcting the deviation of the face orientation information, an arbitrary face orientation image is generated.
- an arbitrary face orientation image is generated in consideration of the face orientation of the subject and the arrangement of facial organs in the front face. And a suitable image can be generated.
- FIG. 1A is a functional block diagram showing an example of the configuration of the image processing apparatus 101 according to the first embodiment of the present invention and the image display apparatus with an imaging unit 102 including the image processing apparatus 101.
- 2 is a diagram illustrating an example of a device that generates a suitable image from a captured image and displays a generated image on a display unit 104.
- the image display device 102 includes an imaging unit 103, a display unit 104, a storage unit 105, an image processing device 101, and an input / output unit 106.
- the imaging unit 103 includes an imaging lens and an imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor), and can capture a still image or a moving image of the subject.
- an imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor), and can capture a still image or a moving image of the subject.
- the display unit 104 is a display screen such as an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) display, and displays information such as images and characters, an image of a subject, and the like.
- LCD Liquid Crystal Display
- organic EL Electro Luminescence
- the image processing apparatus 101 can be configured by, for example, a CPU (Central Processing Unit), a GPU (Graphic Processing Unit: image processing apparatus), and the like, and includes an imaging unit 103, a storage unit 105, an input / output unit 106, and the like.
- the image is acquired from the image and processed, and the processed image is output to the display unit 104, the storage unit 105, and the like.
- the image processing apparatus 101 includes, for example, a facial organ information detection unit 107, a face orientation information calculation unit 108, an arbitrary face orientation image generation unit 109, and a display control unit 110.
- the facial organ information detection unit 107 detects facial organ information, that is, the position of the subject's facial organ from the image input to the image processing apparatus 101.
- the face orientation information calculation unit 108 calculates the face orientation information of the subject based on the facial organ information detected by the facial organ information detection unit 107.
- the facial organ information detection unit 107 uses the facial organ information detected by the facial orientation information, the facial orientation information calculated by the facial orientation information calculation unit 108, and the frontal facial organ information representing the facial organ arrangement of the subject on the frontal face. An image changed to face orientation is generated.
- the frontal facial organ information is information generated in advance, and details will be described later.
- the display control unit 110 performs control to display the image generated by the arbitrary face orientation image generation unit 109 on the display unit 104.
- the storage unit 105 is, for example, a flash memory or a hard disk, and stores images, frontal facial organ information, and the like, and stores device-specific data.
- the input / output unit 106 inputs user commands and voices, a specified target face orientation, etc. to the image processing apparatus 101 and outputs voices, such as voice input / output devices such as key buttons, microphones and speakers.
- the above is a diagram showing a system configuration example of the first embodiment.
- FIG. 2 is a diagram for explaining facial organ information detected by the facial organ information detection unit 107.
- FIG. 3 is a diagram for explaining axes representing the face orientation of the subject
- FIG. 4 is a diagram for explaining the face orientation of the subject with respect to each axis of three-dimensional coordinates
- FIG. It is a figure explaining the calculation method of the inclination of the face to the z axis.
- FIG. 6 is a flowchart showing the flow of image processing in the present embodiment.
- step S1 processing is started (START), and in step S1, the image processing apparatus 101 acquires an image captured by the imaging unit 103.
- step S2 the facial organ information detection unit 107 detects the facial organ of the subject from the image. Extract information.
- the facial organ information is the position of the constituent elements of the face such as the eyes, nose and mouth of the subject.
- FIG. 2 the position of the center of the left eye of the subject H, which is one of facial organ information, is represented by coordinates 201.
- the position of the left eye of the subject H is coordinate 207
- the position of the left eye corner is coordinate 208
- the position of the right eye center is coordinate 209
- the position of the right eye head is coordinate 210
- the position of the right eye corner is coordinate 211
- the position of the mouth center is represented by coordinates 213.
- the facial organ information is described as 6 eyes, 1 nose, and 1 mouth, but further, elements such as eyes, nose, and mouth constituting the face, such as the left nose, right nose, left mouth, and right mouth. It is preferable to assign a plurality of points to one because the degree of deformation of the facial organ can be set finely and an image without a sense of incongruity can be generated.
- the method of detecting facial organ information from images includes, for example, a pattern matching method using a template, and statistically obtaining an identification function from a large number of facial organ images and learning samples of images other than facial organs (non-facial organs).
- Methods for detecting facial organ information P. Viola and M. Jones, “Rapid object detection using a boosting cascade of simple features”, Proc. IEEE Conf. CVPR, pp. 511-518 are known. This can be realized by using the method described above. As described above, facial organ information is detected.
- step S3 the face orientation information calculation unit 108 calculates the face orientation information of the subject using the facial organ information.
- the operation of the face orientation information calculation unit 108 will be described with reference to FIGS.
- the face orientation information calculation unit 108 calculates the face orientation of the subject using the facial organ information.
- the face direction of the subject is represented by the inclination with respect to the x-axis 301, the y-axis 302, and the z-axis 303 shown in FIG.
- the x-axis 301 is a horizontal axis that passes from the left end to the right end of the subject's face
- the y-axis 302 is the vertical axis from the lower end to the upper end of the subject's face
- the z-axis 303 is the front face of the subject's face. It is an axis that passes from the back to the back.
- FIGS. 4A and 4B when the subject H is viewed from above, when the face inclination 401 ( ⁇ 1) with respect to the x axis is a positive value (a), the face direction is the right direction, That is, it is leftward for the subject, and when the face inclination 402 ( ⁇ 1) is a negative value (b), it indicates that the direction of the face is leftward, that is, rightward for the subject.
- the inclination of the face with respect to the x axis is calculated from, for example, the ratio of the lengths of the left and right eyes.
- the lengths of the left and right eyes are respectively the lengths of the positions of the eyes and the corners of the eyes.
- the length L between the points A and B is the Euclidean distance between the points A and B. Can be calculated by the equation (1).
- the length 501 (L1) of the left eye is calculated from the coordinates of the position of the left eye's eye head and the coordinates of the position of the left eye's eye corner.
- the right eye length 502 (L2) is calculated from the coordinates of the position of the right eye corner and the coordinates of the right eye corner (FIG. 5A).
- the ratio C1 between the lengths of the left and right eyes is calculated using Equation (2).
- the length of the left eye is L1
- the length of the right eye is L2.
- An inclination ⁇ 1 of the subject's face with respect to the x-axis is calculated from the ratio C1 of the lengths of the left and right eyes using Expression (3).
- the face inclination with respect to the y-axis is calculated from the ratio between the length 503 of the subject's eyes and nose and the length 504 between the pupils and the face inclination ⁇ 1 with respect to the x-axis. .
- the length between the position coordinate 505 of the left eye center and the position coordinate 506 of the right eye center of the subject is calculated using Equation (1).
- the length between the pupils is calculated to be shorter when the inclination ⁇ 1 of the face with respect to the x-axis of the subject becomes larger. Therefore, in order to calculate the length between the pupils on the front face of the subject, Use to correct the length between pupils.
- the length between pupils before correction is L3
- the length between pupils after correction is L3 ′.
- the length 503 of the subject's eyes and nose is calculated using Expression 508 from the position coordinates 507 of the intermediate point between the center of the left eye and the center of the right eye of the subject and the position coordinates of the nose.
- a ratio C2 between the length L4 of the subject's eyes and the nose and the length L3 ′ between the corrected pupils is calculated using Equation (5).
- Equation (6) From the ratio C2 between the length L4 of the subject's eyes and nose and the corrected length L3 ′ between the pupils, the inclination ⁇ 2 of the subject's face with respect to the y-axis is calculated using Equation (6).
- K1 be the ratio of the length between the eyes and nose of the front face to the length between the pupils.
- This K1 may be obtained by photographing the front faces of a plurality of people in advance, and using an average value or an intermediate value of the ratio between the length of the eyes and nose and the length between the pupils calculated from them. A front face of the subject may be photographed, and a ratio calculated therefrom may be set. Through the above processing, the inclination ⁇ 2 of the subject's face with respect to the y-axis is calculated.
- the inclination of the subject's face with respect to the z-axis is calculated from the length 504 between the pupils and the vertical shift 509 between the left eye and the right eye, as shown in FIG.
- a vertical shift 509 between the left eye and the right eye is represented by an absolute value of a difference in y coordinate between the position coordinate 510 of the left eye and the position coordinate 511 of the right eye.
- the inclination ⁇ 3 of the subject's face with respect to the Z axis is calculated by Expression (7).
- G is the absolute value 509 of the difference between the y-coordinate between the left-eye position coordinate 510 and the right-eye position coordinate 511.
- face orientations ⁇ 1, ⁇ 2, and ⁇ 3 [degrees] representing the inclination of the face with respect to the x, y, and z axes of the subject are calculated as face orientation information, and output to the arbitrary face orientation image generation unit 109.
- FIG. 1B is a functional block diagram illustrating a configuration example of the arbitrary face orientation image generation unit 109.
- FIG. 7 is a diagram for explaining the relationship between feature points and moving points.
- FIG. 8 is a diagram for explaining an example in which the positions of feature points and target points are adjusted for each facial organ.
- FIG. 9 is a diagram for explaining frontal facial organ information.
- FIG. 10 is a diagram for explaining a method of changing the position of the facial organ.
- the arbitrary face orientation image generation unit 109 includes a target face orientation information acquisition unit 109-1, a front face determination unit 109-2, a front face organ information acquisition unit 109-3, and a corrected face organ.
- An information calculation unit 109-4, a face orientation information correction unit 109-5, and an arbitrary face orientation image generation processing unit 109-6 are included.
- the arbitrary face orientation image generation unit 109 performs a process of generating an arbitrary face orientation image by changing the position of the facial organ and deforming the facial organ.
- step S4 target face direction information designated by the user is acquired from the target face direction information acquisition unit 109-1 and the input / output unit 106.
- step S5 the front face determination unit 109-2 determines whether or not the subject's face is a front face based on the face orientation information calculated in step S3. If the subject's face is a front face, the processes of steps S9 to S10 are executed. If the subject's face is not a front face, the processes of steps S6 to S10 are executed.
- the frontal facial organ information acquisition unit 109-3 acquires the frontal facial organ information from the storage unit 105 in step S6.
- the corrected facial organ information calculation unit 109-4 calculates facial organ information in the front face from the facial organ information extracted in step S2 and the face orientation information calculated in step S3.
- step S8 the corrected face organ information calculation unit 109-4 corrects the face orientation information from the front face organ information acquired in step S6 and the face organ information in the front face calculated in step S7. That is, the position of the facial organ in the front face is calculated, and the calculated facial organ position is compared with the preset frontal facial organ information to correct the face orientation information.
- step S9 the arbitrary face orientation image generation processing unit 109-6 generates an arbitrary face orientation image from the facial organ information and the face orientation information.
- step S10 the arbitrary face orientation image generation unit 109 outputs the generated image to the display unit 104, whereby an appropriate image can be displayed on the display unit 104.
- the image display apparatus 102 operates (END). Thereby, the position of the face organ does not move to an unnatural position, and a suitable image with less discomfort is generated.
- step S8 as a method of changing the position of the facial organ in the image and deforming the shape, affine transformation, B-spline transformation, rigidMLS (S. Schaefer, T. McPail and J. Warren, “Image” Deformation using moving squares ", ACM Trans. Graph, vol. 25, no. 3, pp. 533-540, 2006).
- affine transformation, B-spline transformation, rigidMLS S. Schaefer, T. McPail and J. Warren, “Image” Deformation using moving squares ", ACM Trans. Graph, vol. 25, no. 3, pp. 533-540, 2006.
- a feature point set in an image and a destination point of the feature point that is, a target point are set, and coordinate conversion is performed so that the feature point and the target point are brought close to each other.
- the coordinate conversion using the feature points and the target points will be described with reference to FIGS.
- the position of the facial organ detected in the input face image 601 is set as a feature point 602, and a position shifted from the feature point by the number of pixels 603 in the y-axis direction is set as the target point 604 with respect to all the feature points.
- an image 605 in which the facial organ is moved by the number of pixels 603 in the y-axis direction can be generated.
- FIGS. 8 (a) and 8 (b) an example in which the positions of feature points and target points are adjusted for each facial organ will be described with reference to FIGS. 8 (a) and 8 (b).
- the difference between the feature point and the target point is changed between the facial organ eyes 702, the nose 703, and the mouth 704.
- the position of the face organ eye, that is, the feature point 702 which is shifted from the feature point 702 by the number of pixels 705 is set as the target point 706.
- the nose of the facial organ that is, the feature point 703, a position shifted by 707 pixels from the feature point 703 is set as the target point 708.
- the position of the facial organ mouth that is, the feature point 704 that is shifted from the feature point 704 by the number of pixels 709 is set as the target point 710.
- the length 713 between the eyes and the nose of the generated image 712 is longer than the length 711 between the eyes and the nose of the input image 701. It is possible to generate an image in which the positional relationship of the facial organs, that is, the arrangement of the facial organs is changed.
- the position of the facial organ in the front face of the subject is calculated (step S7), and the calculated facial organ position is compared with preset frontal facial organ information (step S7).
- S8) A method will be described.
- the front face of the subject is a face orientation in which the inclination ⁇ 1 with respect to the x-axis and the inclination ⁇ 2 with respect to the y-axis are 0 in the face orientation information. From the input face length L3 between the eyes, length L4 between the eyes and nose, and face orientation information ⁇ 1 and ⁇ 2, the length L3 ′ between the face pupils in the front face and the length of the eyes and nose L4 ′ is calculated.
- the length L3 ′ between the pupils in the front face is calculated by Expression (4) using the face direction information ⁇ 1 indicating the inclination of the face direction with respect to the x axis.
- the corrected eye-to-nose length L4 ′ is calculated by Expression (8).
- the frontal face organ information is generated from a front face image input in advance.
- the front face image input at this time may be a still image or a moving image.
- a moving image shot in real time from the imaging unit may be used. In this case, it is unlikely that all the frames of the captured video are front faces.
- the front face organ information is used by using the position of the face organ of the subject determined to be the front face in the past frame. Is generated. When there are a plurality of face organ positions of the subject determined as the front face in the past frame, an average value or an intermediate value of the face organ positions is used.
- the position of the front face organ is calculated using the golden ratio (standard value) of the human face.
- the golden ratio is an arrangement of an average human 801 facial organ.
- the ratio between the length 802 between the pupils and the length 803 between the eyes and the nose is set to be 32:25
- the ratio of the length 808 to the submandibular 807 is set to be 1: 1
- the ratio of the length 811 between the left eye 809 and the right eye 810 to the eye length 812 is
- the position of the facial organs in the generated image is set so as not to deviate from the average human facial organ layout, such as setting to be 1: 1.
- the arrangement of the facial organs on the frontal face of the subject can be used, and the frontal face of the subject is accurately generated. This is preferable. Also, if the input front face image is not determined to be a front face in a past frame even in a still image or video, the face arrangement on the front face of the subject is set using the golden ratio of the face. This is preferable because the position of the facial organ in the generated image can be brought close to the average human facial organ arrangement.
- step 9 in order to generate an arbitrary face-oriented image, two methods are used: a method of changing the position of the facial organ of the subject and a method of deforming the facial organ.
- a method for changing the position of the facial organ will be described with reference to FIG.
- the face organ position change is generated as face orientation information ⁇ 1, ⁇ 2, and ⁇ 3 calculated by the face orientation information calculation unit 108, and correction amounts ⁇ 1 ′ and ⁇ 2 ′ of face orientation information calculated using the front face position information. This is performed using target face orientation information ⁇ 1 and ⁇ 2 representing the desired face orientation.
- the difference between the face orientation information ⁇ 1 + ⁇ 1 ′, ⁇ 2 + ⁇ 2 ′ and the target face orientation information ⁇ 1, ⁇ 2 is the face orientation information to be changed.
- the movement amount 901 in the x-axis direction and the movement amount 902 in the y-axis direction of the position of the facial organ are calculated, the movement amount in the z-axis direction is added using the face orientation information ⁇ 3 with respect to the z-axis, and the face orientation is changed.
- the position of the facial organ is calculated.
- the length 903 between the pupils after the face direction is changed is calculated by Expression (11).
- the length between pupils after changing the face orientation is M3.
- the difference between the length 904 between the pupils in the input image and the length 903 between the pupils after changing the face direction is a movement amount 901 in the lateral direction of the facial organ, that is, the x-axis direction.
- a method for calculating the movement amount 902 in the y-axis direction will be described.
- the length 905 between the eyes and the nose after the change of the face direction is calculated by Expression (12).
- the length of the eyes and nose in the input image is L4, and the length of the eyes and nose after changing the face orientation is M4.
- the difference between the length 906 between the eyes and the nose in the input image and the length 905 between the eyes and the nose after changing the face orientation is the amount of movement 902 in the vertical direction of the facial organ, that is, the y-axis direction. Since the calculated movement amount 901 in the x-axis direction and movement amount 902 in the y-axis direction do not take into account the face inclination ⁇ 3 with respect to the z-axis, the target point (Qx, Qy) is calculated.
- the movement amount in the x-axis direction is E1
- the movement amount in the y-axis direction is E2
- the feature point in the input image is (Px, Py).
- the target point is set so that the eye length is aligned with the value calculated from the frontal facial organ information.
- the ratio L1: L2 between the left eye length L1 and the right eye length L2 is equal to the ratio D1: D2 between the left eye length D1 and the right eye length D2 calculated from the frontal facial organ information.
- Set the target point to correct the eye length.
- the ratio D1: D2 between the length of the left eye and the length of the right eye is the length D1 of the left eye and the length D2 of the right eye of the front face arrangement in the past frame.
- the target point is set so that the length L1 of the left eye is enlarged or reduced so that the length L1 of the left eye is equal to the length D1 of the left eye calculated from the frontal facial organ information.
- the target point is set so that the length of the right eye, the length of the nose, and the length of the mouth are also equal to the lengths of the corresponding facial organs calculated from the frontal facial organ information.
- the target point so as to change not only the length of the facial organ but also the size of the eyes, the direction of the nose, the position of the mouth corner, etc., because the effect of changing the face direction can be emphasized.
- the arbitrary face orientation image generation unit 109 performs image transformation on the input image using one or more feature points in the set input image and the target points corresponding to the feature points, and the facial organs An image whose position has been changed is generated.
- the arbitrary face orientation image generation unit 109 further deforms the facial organs using the frontal facial organ information with respect to the image generated as described above.
- the image display apparatus 102 As described above, the image display apparatus 102 according to the first embodiment operates.
- the image display apparatus 102 including the image processing apparatus 101 according to the present invention described above it is possible to appropriately perform image processing in consideration of the face direction of the subject and the front face arrangement, and to display a suitable image.
- ⁇ Second Embodiment> A second embodiment of the present invention will be described.
- the face orientation is large Even when all the facial organs are not detected, an effect as if the face orientation has changed can be obtained, which is preferable.
- the face orientation in the input image calculated from the facial organ information is 90 degrees with respect to the x-axis, that is, a side face and the target face orientation is 0 degrees, that is, when it is designated to generate a front face
- the processing of changing the position of the target face is performed at a stage where the target face direction is up to 70 degrees with respect to the x-axis, and in the process of deforming the facial organ, a process for correcting the shape of the eyes to face the front is performed. Emphasize the effect of changing.
- the upper left coordinate, upper right coordinate, lower left coordinate, lower right coordinate, and center coordinate of the face area of the subject are determined from the positional relationship between the left eye and right eye. calculate. Five points indicating the face area are set as feature points, and target points are set according to the face direction.
- the face orientation information of the subject in the input image is calculated only from the detected face organs, calculated from the positional relationship between the camera and the subject, Acquired using frame face orientation information.
- the influence of occlusion and facial distortion can be reduced even when the orientation of the face is greatly changed. Furthermore, even when all the facial organs are not detected, it is possible to obtain an effect as if the face orientation has changed.
- the difference between the present embodiment and the first embodiment is that the present embodiment is configured to include a transmission / reception unit 1103.
- the image display apparatus 1102 is connected to an external network via a transmission / reception unit 1103 and is connected to other communication devices.
- the transmission / reception unit 1103 is a communication unit or a cable of a mobile phone, and transmits / receives an image, face position information, face size information, and the like to / from the outside.
- an image captured by the image display device 1102 and facial organ information are transmitted to another image display device 1102 through the transmission / reception unit 1103, and an image and face captured by the other image display device 1102 are transmitted.
- Receive organ information That is, the captured image of the user 2 at a remote location is displayed on the image display device 1102 of the user 1, and the captured image of the user 1 is displayed on the image display device 1102 of the user 2.
- This is a configuration for realizing a video chat or video conference system.
- the image display device 1102 that captured the user 1 is identified as the first image display device
- the image display device 1102 that captured the user 2 is identified as the second image display device.
- step S ⁇ b> 21 the image processing apparatus 1101 captures the first image from the imaging unit 103.
- step S22 the facial organ information detection unit 107 detects first facial organ information from the first image.
- step S23 the face orientation information calculation unit 108 calculates the first face orientation information of the subject using the facial organ information.
- step S24 the transmission / reception unit 1103 transmits the first image, the first facial organ information, and the first face orientation information to the second image display device.
- step S25 the transmission / reception unit 1103 receives the second image, the second facial organ information, and the second face orientation information.
- step S ⁇ b> 26 the image processing apparatus 1101 acquires second target face orientation information from the input / output unit 106.
- step S27 the arbitrary face orientation image generation unit 109 determines whether the subject's face is a front face based on the second face orientation information. If the subject's face is a front face, steps S31 and S32 are executed. If the face of the subject is not a front face, steps S28 to S32 are executed.
- step S ⁇ b> 28 the image processing apparatus 1101 acquires frontal facial organ information from the storage unit 105.
- step S29 the arbitrary face orientation image generation unit 109 calculates facial organ information on the front face from the second facial organ information and the second facial orientation information.
- step S30 the arbitrary face orientation image generation unit 109 corrects the second face orientation information from the acquired front face organ information and the calculated face organ information of the front face.
- step S31 the arbitrary face orientation image generation unit 109 generates an arbitrary face orientation image from the second facial organ information and the second face orientation information.
- step S ⁇ b> 32 the arbitrary face orientation image generation unit 109 outputs the generated image to the display unit 104.
- the image display apparatus As described above, the image display apparatus according to the third embodiment operates.
- the image display device 1102 including the image processing device 1101 according to the present embodiment described above, when the user 1 and the user 2 are interacting in a video chat or a video conference, the facial organ information and the face transmitted together with the image.
- a suitable image can be displayed by generating an arbitrary face orientation image using the orientation information.
- the program that operates in the image processing apparatus according to the present invention may be a program that controls a CPU or the like (a program that causes a computer to function) so as to realize the functions of the above-described embodiments relating to the present invention.
- Information handled by these devices is temporarily stored in a RAM (Random Access Memory) during processing, and then stored in various ROMs and HDDs such as a ROM (Read Only Memory). Reading, correction and writing are performed.
- a program for realizing the functions of the components shown in FIGS. 1A and 1B is recorded on a computer-readable recording medium, and the program recorded on the recording medium is read into a computer system and executed by a CPU or the like. You may process each part by.
- the “computer system” includes hardware such as an OS (Operating System) and peripheral devices.
- the “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM or a CD-ROM, and a hard disk incorporated in a computer system.
- the “computer-readable recording medium” dynamically holds a program for a short time like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line.
- a volatile memory in a computer system serving as a server or a client in that case, and a program that holds a program for a certain period of time are also included.
- a part or all of the image processing apparatus in the above-described embodiment may be realized as an LSI that is typically an integrated circuit.
- Each functional block of the image processing apparatus may be individually formed into chips, or a part or all of them may be integrated into a chip.
- the method of circuit integration is not limited to LSI, and may be realized by a dedicated circuit or a general-purpose processor.
- an integrated circuit based on the technology can also be used.
- control lines and information lines indicate what is considered necessary for the explanation, and not all the control lines and information lines on the product are necessarily shown. All the components may be connected to each other.
- the present invention can be used for an image display device.
- the present invention includes the following disclosure.
- a facial organ information detector that detects facial organ information that is the position of the facial organ of the subject from the input image;
- a face orientation information calculating unit that calculates face orientation information of the subject from the facial organ information;
- An arbitrary face orientation image generator that generates an image in which the face orientation of the subject is changed from the facial organ information and the face orientation information, If the arbitrary face orientation image generation unit determines that the face is tilted according to the face orientation information, the face orientation information is corrected based on front facial organ information that is a facial organ arrangement on the front face of the subject.
- an image processing apparatus that generates an image with an arbitrary face orientation.
- the face orientation is determined using the preset positions of the facial organs such as eyes and nose in the front face of the subject and the face organ positions in the front face calculated from the face orientation information.
- an arbitrary face orientation image is generated.
- the arbitrary face orientation image generator generates an image in which the face orientation of the subject is changed by the process of changing the position of the face organ and the process of deforming the face organ from the face organ information and the face orientation information.
- the image processing apparatus according to (1), wherein the image processing apparatus is generated.
- the arbitrary face orientation image generation unit adjusts a ratio between a process of changing the position of the facial organ and a process of deforming the facial organ according to the facial organ information. Processing equipment.
- the arbitrary face orientation image generation unit uses the face arrangement of the subject determined to be a front face in the previous frame as front face organ information, and the position of the face organ calculated based on the golden ratio of the face.
- the image processing apparatus according to any one of (1) to (3), wherein the face orientation information is corrected using at least one of processing for converting to facial organ information.
- An image display device comprising: an imaging unit that images a subject; and the image processing device according to (1) that processes an image of the subject captured by the imaging unit.
- the face orientation information calculation unit Any one of (1) to (4), wherein the inclination of the face with respect to the x-axis, which is a horizontal axis passing from the left end to the right end of the face of the subject, is calculated from the ratio of the length of the organ in the horizontal direction.
- the lengths of the left and right eyes are respectively the lengths of the positions of the eyes and the corners of the eyes.
- the face orientation information calculation unit The inclination of the face with respect to the y axis, which is the vertical axis from the lower end to the upper end of the subject's face, is calculated from the ratio of the length of the subject's eyes and nose to the length between the pupils and the inclination of the face ⁇ 1 with respect to the x axis.
- the image processing apparatus according to (6).
- the face orientation information calculation unit The inclination of the subject's face with respect to the z-axis, which is an axis passing through the front to the back of the subject's face, is calculated from the length between the pupils and the vertical shift between the left and right eyes (6) or ( The image processing apparatus according to 7).
- the arbitrary face orientation image generation unit By setting a feature point set in the image and a target point that is the destination point of the feature point, and performing coordinate transformation to bring the feature point close to the target point, the facial features in the image The image processing apparatus according to (1) or (2), wherein the position is changed and the shape is changed.
- the arbitrary face orientation image generation unit performs image deformation on the input image using one or more feature points in the set input image and target points corresponding to the feature points, and the position of the facial organ is changed.
- the arbitrary face orientation image generation step corrects the face orientation information based on front face organ information that is a face organ arrangement in the front face of the subject when it is determined that the face is tilted according to the face orientation information.
- An image processing method comprising: generating an image with an arbitrary face orientation after application.
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Oral & Maxillofacial Surgery (AREA)
- Health & Medical Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Geometry (AREA)
- General Health & Medical Sciences (AREA)
- Human Computer Interaction (AREA)
- Multimedia (AREA)
- Image Processing (AREA)
- Image Analysis (AREA)
Abstract
Description
<第1実施の形態>
図1Aは、本発明の第1の実施の形態による画像処理装置101と、画像処理装置101を備える撮像部付き画像表示装置102の一構成例を示す機能ブロック図であり、撮像部103で被写体を撮像して、撮像した画像から好適な画像を生成し、生成画像を表示部104に表示する装置の例を示す図である。
<第2実施の形態>
本発明の第2の実施の形態について説明する。本実施の形態では、さらに、顔器官の位置を変更させる処理と、顔器官を変形させる処理を、被写体の顔器官情報に応じて、入力画像に施す割合を変化させると、顔向きが大きい場合や顔器官が全て検出されなかった場合でも、顔向きが変わったかのような効果を得ることができるため、好適である。例えば、顔器官情報から算出した入力画像中の顔向きがx軸に対して90度、すなわち横顔で、目標顔向きが0度、すなわち正面顔を生成するように指定されている場合、顔器官の位置を変更させる処理は目標顔向きがx軸に対して70度の段階までとし、顔器官を変形させる処理において、目の形が正面を向くように補正する処理を施すことで、顔向きを変更した効果を強調させる。
<第3の実施形態>
次に、本発明の第3の実施の形態に係る画像処理装置1101を備える画像表示装置1102の構成例について、図11を用いて説明する。図11において、図1と同じ構成要素には同じ番号を付しており、これらの要素は図1の実施の形態と同じ処理を行うため説明を省略する。
(付記)
本発明は以下の開示を含む。
(1)
入力された画像から被写体の顔器官の位置である顔器官情報を検出する顔器官情報検出部と、
前記顔器官情報から前記被写体の顔向き情報を算出する顔向き情報算出部と、
前記顔器官情報と前記顔向き情報とから前記被写体の顔向きを変更した画像を生成する任意顔向き画像生成部と、を備え、
前記任意顔向き画像生成部は前記顔向き情報により顔が傾いていると判定した場合に、前記被写体の正面顔における顔器官配置である正面顔器官情報に基づいて前記顔向き情報に補正を施した後、任意の顔向きの画像を生成することを特徴とする画像処理装置。
(2)
前記任意顔向き画像生成部は、前記顔器官情報と前記顔向き情報とから、前記顔器官の位置を変更する処理と前記顔器官を変形する処理とにより前記被写体の顔向きを変更した画像を生成することを特徴とする(1)に記載の画像処理装置。
(3)
前記任意顔向き画像生成部は、前記顔器官情報に応じて前記顔器官の位置を変更する処理と前記顔器官を変形する処理の割合を調整することを特徴とする(2)に記載の画像処理装置。
(4)
前記任意顔向き画像生成部は、前フレームにおいて正面顔であると判定された被写体の顔配置を正面顔器官情報とする処理と、顔の黄金比に基づいて算出された顔器官の位置を正面顔器官情報とする処理との少なくとも一つを用いて前記顔向き情報を補正することを特徴とする(1)から(3)までのいずれか1に記載の画像処理装置。
(5)
被写体を撮像する撮像部と、前記撮像部で撮像された前記被写体の画像を処理する(1)に記載の画像処理装置と、を備えることを特徴とする画像表示装置。
(6)
前記顔向き情報算出部は、
被写体の顔の左端から右端を通る水平の軸であるx軸に対する顔の傾きを、器官の水平方向の長さの比から算出することを特徴とする(1)から(4)までのいずれか1に記載の画像処理装置。
(7)
前記顔向き情報算出部は、
被写体の顔の下端から上端への垂直方向の軸であるy軸に対する顔の傾きを、被写体の目と鼻との長さと瞳孔間の長さとの比と、x軸に対する顔の傾きθ1から算出することを特徴とする(6)に記載の画像処理装置。
(8)
前記顔向き情報算出部は、
被写体の顔の前面から背面を通る軸であるz軸に対する被写体の顔の傾きを、瞳孔間の長さと左目と右目との縦方向のずれとから算出することを特徴とする(6)又(7)に記載の画像処理装置。
(9)
前記任意顔向き画像生成部は、
画像中に設定した特徴点と、その特徴点の移動先の点である目標点を設定し、前記特徴点と前記目標点とを近づけるような座標変換を行うことで、画像中の顔器官の位置を変更、及び、形を変形させることを特徴とする(1)又は(2)に記載の画像処理装置。
(10)
前記任意顔向き画像生成部は、設定した入力画像中の1つ以上の特徴点と、特徴点に対応する目標点を用いて、入力画像に画像変形を施し、顔器官の位置が変更された画像を生成することを特徴とする(1)又は(2)に記載の画像処理装置。
(11)
入力された画像から被写体の顔器官の位置である顔器官情報を検出する顔器官情報検出ステップと、
前記顔器官情報から前記被写体の顔向き情報を算出する顔向き情報算出ステップと、
前記顔器官情報と前記顔向き情報とから前記被写体の顔向きを変更した画像を生成する任意顔向き画像生成ステップと、を有し、
前記任意顔向き画像生成ステップは、前記顔向き情報により顔が傾いていると判定した場合に、前記被写体の正面顔における顔器官配置である正面顔器官情報に基づいて前記顔向き情報に補正を施した後、任意の顔向きの画像を生成することを特徴とする画像処理方法。
(12)
コンピュータに、(11)に記載の画像処理方法を実行させるためのプログラム。
Claims (5)
- 入力された画像から被写体の顔器官の位置である顔器官情報を検出する顔器官情報検出部と、
前記顔器官情報から前記被写体の顔向き情報を算出する顔向き情報算出部と、
前記顔器官情報と前記顔向き情報とから前記被写体の顔向きを変更した画像を生成する任意顔向き画像生成部と、を備え、
前記任意顔向き画像生成部は前記顔向き情報により顔が傾いていると判定した場合に、前記被写体の正面顔における顔器官配置である正面顔器官情報に基づいて前記顔向き情報に補正を施した後、任意の顔向きの画像を生成することを特徴とする画像処理装置。 - 前記任意顔向き画像生成部は、前記顔器官情報と前記顔向き情報とから、前記顔器官の位置を変更する処理と前記顔器官を変形する処理とにより前記被写体の顔向きを変更した画像を生成することを特徴とする請求項1に記載の画像処理装置。
- 前記任意顔向き画像生成部は、前記顔器官情報に応じて前記顔器官の位置を変更する処理と前記顔器官を変形する処理との割合を調整することを特徴とする請求項2に記載の画像処理装置。
- 前記任意顔向き画像生成部は、前フレームにおいて正面顔であると判定された被写体の顔配置を正面顔器官情報とする処理と、顔の黄金比に基づいて算出された顔器官の位置を正面顔器官情報とする処理との少なくとも一つを用いて前記顔向き情報を補正することを特徴とする請求項1から3までのいずれか1項に記載の画像処理装置。
- 被写体を撮像する撮像部と、前記撮像部で撮像された前記被写体の画像を処理する請求項1に記載の画像処理装置と、を備えることを特徴とする画像表示装置。
Priority Applications (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2016525760A JP6330036B2 (ja) | 2014-06-06 | 2015-05-20 | 画像処理装置及び画像表示装置 |
US15/316,254 US10296783B2 (en) | 2014-06-06 | 2015-05-20 | Image processing device and image display device |
CN201580029934.0A CN106462937B (zh) | 2014-06-06 | 2015-05-20 | 图像处理装置以及图像显示装置 |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2014118024 | 2014-06-06 | ||
JP2014-118024 | 2014-06-06 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2015186519A1 true WO2015186519A1 (ja) | 2015-12-10 |
Family
ID=54766594
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/JP2015/064466 WO2015186519A1 (ja) | 2014-06-06 | 2015-05-20 | 画像処理装置及び画像表示装置 |
Country Status (4)
Country | Link |
---|---|
US (1) | US10296783B2 (ja) |
JP (1) | JP6330036B2 (ja) |
CN (1) | CN106462937B (ja) |
WO (1) | WO2015186519A1 (ja) |
Cited By (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN107865473A (zh) * | 2016-09-26 | 2018-04-03 | 华硕电脑股份有限公司 | 人体特征测距装置及其测距方法 |
WO2019163124A1 (ja) * | 2018-02-26 | 2019-08-29 | 三菱電機株式会社 | 三次元位置推定装置及び三次元位置推定方法 |
JP2020518051A (ja) * | 2017-08-17 | 2020-06-18 | 平安科技(深▲せん▼)有限公司Ping An Technology (Shenzhen) Co.,Ltd. | 顔姿勢検出方法、装置及び記憶媒体 |
US10796166B2 (en) | 2016-03-23 | 2020-10-06 | Nec Corporation | Information processing for aggregating sensor information about persons entering and exiting a venue |
WO2023112606A1 (ja) * | 2021-12-14 | 2023-06-22 | 矢崎総業株式会社 | 顔向き判定システム |
JP7481398B2 (ja) | 2022-07-04 | 2024-05-10 | ソフトバンク株式会社 | 判定装置、プログラム、及び判定方法 |
Families Citing this family (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP6582557B2 (ja) * | 2015-05-28 | 2019-10-02 | 株式会社ソシオネクスト | 描画装置、描画方法および描画プログラム |
US11156473B2 (en) * | 2016-08-18 | 2021-10-26 | Sony Corporation | Information processing apparatus, information processing system, and information processing method |
JP6558388B2 (ja) * | 2017-03-14 | 2019-08-14 | オムロン株式会社 | 画像処理装置 |
CN111353354B (zh) * | 2018-12-24 | 2024-01-23 | 杭州海康威视数字技术股份有限公司 | 一种人体应激性信息识别方法、装置及电子设备 |
US20220124256A1 (en) * | 2019-03-11 | 2022-04-21 | Nokia Technologies Oy | Conditional display of object characteristics |
US11621979B1 (en) | 2020-12-31 | 2023-04-04 | Benjamin Slotznick | Method and apparatus for repositioning meeting participants within a virtual space view in an online meeting user interface based on gestures made by the meeting participants |
US11330021B1 (en) | 2020-12-31 | 2022-05-10 | Benjamin Slotznick | System and method of mirroring a display of multiple video feeds in videoconferencing systems |
US11546385B1 (en) * | 2020-12-31 | 2023-01-03 | Benjamin Slotznick | Method and apparatus for self-selection by participant to display a mirrored or unmirrored video feed of the participant in a videoconferencing platform |
Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2012226665A (ja) * | 2011-04-21 | 2012-11-15 | Canon Inc | 情報処理装置、その処理方法、プログラム及び撮像装置 |
JP2014049866A (ja) * | 2012-08-30 | 2014-03-17 | Sharp Corp | 画像処理装置及び画像表示装置 |
Family Cites Families (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2007193824A (ja) * | 2000-04-13 | 2007-08-02 | Fujifilm Corp | 画像処理方法 |
US7894666B2 (en) * | 2004-07-30 | 2011-02-22 | Canon Kabushiki Kaisha | Image processing method and apparatus, image sensing apparatus, and program |
US7783085B2 (en) * | 2006-05-10 | 2010-08-24 | Aol Inc. | Using relevance feedback in face recognition |
KR101460130B1 (ko) * | 2007-12-11 | 2014-11-10 | 삼성전자주식회사 | 휴대 단말기의 화상 통화 방법 및 장치 |
JP4569670B2 (ja) * | 2008-06-11 | 2010-10-27 | ソニー株式会社 | 画像処理装置、画像処理方法およびプログラム |
JP5239625B2 (ja) * | 2008-08-22 | 2013-07-17 | セイコーエプソン株式会社 | 画像処理装置、画像処理方法および画像処理プログラム |
JP5842541B2 (ja) | 2011-11-01 | 2016-01-13 | 大日本印刷株式会社 | 三次元ポートレートの作成装置 |
CN102547209B (zh) * | 2012-02-06 | 2015-07-22 | 华为技术有限公司 | 视讯设备控制方法、装置及视讯*** |
-
2015
- 2015-05-20 WO PCT/JP2015/064466 patent/WO2015186519A1/ja active Application Filing
- 2015-05-20 CN CN201580029934.0A patent/CN106462937B/zh not_active Expired - Fee Related
- 2015-05-20 US US15/316,254 patent/US10296783B2/en not_active Expired - Fee Related
- 2015-05-20 JP JP2016525760A patent/JP6330036B2/ja not_active Expired - Fee Related
Patent Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2012226665A (ja) * | 2011-04-21 | 2012-11-15 | Canon Inc | 情報処理装置、その処理方法、プログラム及び撮像装置 |
JP2014049866A (ja) * | 2012-08-30 | 2014-03-17 | Sharp Corp | 画像処理装置及び画像表示装置 |
Cited By (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10796166B2 (en) | 2016-03-23 | 2020-10-06 | Nec Corporation | Information processing for aggregating sensor information about persons entering and exiting a venue |
CN107865473A (zh) * | 2016-09-26 | 2018-04-03 | 华硕电脑股份有限公司 | 人体特征测距装置及其测距方法 |
JP2020518051A (ja) * | 2017-08-17 | 2020-06-18 | 平安科技(深▲せん▼)有限公司Ping An Technology (Shenzhen) Co.,Ltd. | 顔姿勢検出方法、装置及び記憶媒体 |
WO2019163124A1 (ja) * | 2018-02-26 | 2019-08-29 | 三菱電機株式会社 | 三次元位置推定装置及び三次元位置推定方法 |
WO2023112606A1 (ja) * | 2021-12-14 | 2023-06-22 | 矢崎総業株式会社 | 顔向き判定システム |
JP7418386B2 (ja) | 2021-12-14 | 2024-01-19 | 矢崎総業株式会社 | 顔向き判定システム |
JP7481398B2 (ja) | 2022-07-04 | 2024-05-10 | ソフトバンク株式会社 | 判定装置、プログラム、及び判定方法 |
Also Published As
Publication number | Publication date |
---|---|
JP6330036B2 (ja) | 2018-05-23 |
JPWO2015186519A1 (ja) | 2017-04-20 |
US20170147866A1 (en) | 2017-05-25 |
CN106462937B (zh) | 2019-12-31 |
US10296783B2 (en) | 2019-05-21 |
CN106462937A (zh) | 2017-02-22 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
JP6330036B2 (ja) | 画像処理装置及び画像表示装置 | |
JP6499583B2 (ja) | 画像処理装置及び画像表示装置 | |
JP5450739B2 (ja) | 画像処理装置及び画像表示装置 | |
US9613286B2 (en) | Method for correcting user's gaze direction in image, machine-readable storage medium and communication terminal | |
JP2019030007A (ja) | 複数のカメラを用いて映像を取得するための電子装置及びこれを用いた映像処理方法 | |
WO2022110591A1 (zh) | 基于连麦直播的直播画面处理方法、装置及电子设备 | |
JP4144492B2 (ja) | 画像表示装置 | |
JPWO2008012905A1 (ja) | 認証装置および認証用画像表示方法 | |
CN109325908B (zh) | 图像处理方法及装置、电子设备和存储介质 | |
WO2016165614A1 (zh) | 一种即时视频中的表情识别方法和电子设备 | |
JP2015012304A (ja) | 画像処理装置、画像処理方法、及び、プログラム | |
TWI615808B (zh) | 全景即時影像處理方法 | |
JP2009246408A (ja) | 対話装置、画像処理モジュール、画像処理方法およびプログラム | |
US20160350622A1 (en) | Augmented reality and object recognition device | |
US9536133B2 (en) | Display apparatus and control method for adjusting the eyes of a photographed user | |
JP2005142765A (ja) | 撮像装置及び方法 | |
CN107277362A (zh) | 用于生成高清图像的方法和装置 | |
JP2018136666A (ja) | 視線変換装置及び視線変換方法 | |
JP6103942B2 (ja) | 画像データ処理装置及び画像データ処理プログラム | |
JP2016139975A (ja) | 画像取得装置、画像取得方法及び画像取得のためのプログラム | |
WO2023005200A1 (zh) | 图像生成方法、装置及***、计算机可读存储介质 | |
CN115278071B (zh) | 图像处理方法、装置、电子设备和可读存储介质 | |
WO2023206475A1 (zh) | 图像处理方法、装置、电子设备及存储介质 | |
WO2022178724A1 (zh) | 图像拍摄方法、终端设备、拍摄装置和存储介质 | |
CN107516339B (zh) | 一种信息处理方法及信息处理装置 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 15803110 Country of ref document: EP Kind code of ref document: A1 |
|
ENP | Entry into the national phase |
Ref document number: 2016525760 Country of ref document: JP Kind code of ref document: A |
|
WWE | Wipo information: entry into national phase |
Ref document number: 15316254 Country of ref document: US |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 15803110 Country of ref document: EP Kind code of ref document: A1 |