WO2021240668A1 - Dispositif de détection de geste et procédé de détection de geste - Google Patents

Dispositif de détection de geste et procédé de détection de geste Download PDF

Info

Publication number
WO2021240668A1
WO2021240668A1 PCT/JP2020/020828 JP2020020828W WO2021240668A1 WO 2021240668 A1 WO2021240668 A1 WO 2021240668A1 JP 2020020828 W JP2020020828 W JP 2020020828W WO 2021240668 A1 WO2021240668 A1 WO 2021240668A1
Authority
WO
WIPO (PCT)
Prior art keywords
frame
hand
occupant
face
detected
Prior art date
Application number
PCT/JP2020/020828
Other languages
English (en)
Japanese (ja)
Inventor
太郎 熊谷
拓也 村上
Original Assignee
三菱電機株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 三菱電機株式会社 filed Critical 三菱電機株式会社
Priority to PCT/JP2020/020828 priority Critical patent/WO2021240668A1/fr
Publication of WO2021240668A1 publication Critical patent/WO2021240668A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
    • G01C21/26—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 specially adapted for navigation in a road network
    • G01C21/34—Route searching; Route guidance
    • G01C21/36—Input/output arrangements for on-board computers
    • G—PHYSICS
    • G08—SIGNALLING
    • G08G—TRAFFIC CONTROL SYSTEMS
    • G08G1/00—Traffic control systems for road vehicles
    • G08G1/16—Anti-collision systems

Definitions

  • This disclosure relates to a gesture detection device and a gesture detection method.
  • Patent Document 1 proposes a control device that detects information about a user's hand only from a gesture area set based on the area of the driver's face.
  • the gesture detection device detects the occupant's hand based on the image. Therefore, depending on the state of the image, the gesture detection device may detect an object other than the hand as a hand.
  • the present disclosure is for solving the above-mentioned problems, and an object of the present disclosure is to provide a gesture detection device that accurately detects a hand in a gesture of an occupant.
  • the gesture detection device includes a face information acquisition unit, a hand candidate detection unit, and a determination unit.
  • the face information acquisition unit acquires information on the face orientation of the occupant.
  • the face orientation is detected based on the image captured by the image pickup device provided in the vehicle.
  • the hand candidate detection unit detects a hand candidate that is a candidate for the occupant's hand based on the image.
  • the determination unit rejects the hand candidate information so that the hand candidate is not detected as the occupant's hand in the gesture of the occupant to be detected based on a predetermined condition regarding the face orientation.
  • a gesture detection device for accurately detecting a hand in a occupant's gesture is provided.
  • FIG. It is a functional block diagram which shows the structure of the gesture detection apparatus in Embodiment 1.
  • FIG. It is a figure which shows an example of the structure of the processing circuit included in a gesture detection device. It is a figure which shows another example of the structure of the processing circuit included in the gesture detection apparatus.
  • FIG. It is a functional block diagram which shows the structure of the gesture detection apparatus in Embodiment 2.
  • FIG. It is a figure which shows an example of the face orientation of an occupant in Embodiment 2.
  • FIG. It is a flowchart which shows the gesture detection method in Embodiment 2.
  • FIG. It is a figure which shows an example of the frame to be processed. It is a figure which shows the relationship from the 1st frame to the 4th frame in Embodiment 3.
  • FIG. It is a functional block diagram which shows the structure of the gesture detection apparatus in Embodiment 4. It is a flowchart which shows the gesture detection method in Embodiment 4. It is a figure which shows an example of the frame to be processed. It is a block diagram which shows the structure of the gesture detection device and the device which operates in connection with it in Embodiment 5.
  • FIG. 1 is a functional block diagram showing the configuration of the gesture detection device 100 according to the first embodiment. Further, FIG. 1 shows an image pickup device 110 and a face detection unit 10 as devices that operate in connection with the gesture detection device 100.
  • the image pickup device 110 is provided in the vehicle.
  • the image pickup device 110 captures an image of an occupant inside the vehicle.
  • the face detection unit 10 detects the face orientation of the occupant based on the image.
  • the face orientation corresponds to, for example, the direction facing the front of the occupant's face, the direction of the line of sight, and the like.
  • the gesture detection device 100 detects the gesture of the hand of the occupant of the vehicle based on the image taken by the image pickup device 110.
  • the gesture detection device 100 includes a face information acquisition unit 20, a hand candidate detection unit 30, and a determination unit 40.
  • the face information acquisition unit 20 acquires information on the face orientation of the occupant from the face detection unit 10.
  • the hand candidate detection unit 30 detects a hand candidate that is a candidate for the occupant's hand based on the image taken by the image pickup device 110.
  • the hand candidate detection unit 30 detects a hand candidate by, for example, matching a pattern of the shape of an object (information on the luminance distribution) in the image with a predetermined pattern of the shape of the hand.
  • the determination unit 40 rejects the hand candidate information based on a predetermined condition regarding the face orientation.
  • the gesture detection device 100 does not identify the rejected hand candidate as a hand constituting the occupant's gesture.
  • FIG. 2 is a diagram showing an example of the configuration of the processing circuit 90 included in the gesture detection device 100.
  • Each function of the face information acquisition unit 20, the hand candidate detection unit 30, and the determination unit 40 is realized by the processing circuit 90. That is, the processing circuit 90 has a face information acquisition unit 20, a hand candidate detection unit 30, and a determination unit 40.
  • the processing circuit 90 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), or an FPGA (Field). -ProgrammableGateArray), or a circuit that combines these.
  • the functions of the face information acquisition unit 20, the hand candidate detection unit 30, and the determination unit 40 may be individually realized by a plurality of processing circuits, or may be collectively realized by one processing circuit.
  • FIG. 3 is a diagram showing another example of the configuration of the processing circuit included in the gesture detection device 100.
  • the processing circuit includes a processor 91 and a memory 92.
  • each function of the face information acquisition unit 20, the hand candidate detection unit 30, and the determination unit 40 is realized.
  • each function is realized by executing the software or firmware described as a program by the processor 91.
  • the gesture detection device 100 has a memory 92 for storing the program and a processor 91 for executing the program.
  • the program describes a function in which the gesture detection device 100 acquires information on the face orientation of the occupant detected based on the image captured by the image pickup device 110 provided in the vehicle. Further, the program describes a function in which the gesture detection device 100 detects a hand candidate, which is a candidate for a occupant's hand, based on the image. Further, the program describes a function of rejecting the information of the hand candidate so that the hand candidate is not detected as a hand in the gesture of the occupant to be detected based on the predetermined condition regarding the face orientation. As described above, the program causes the computer to execute the procedure or method of the face information acquisition unit 20, the hand candidate detection unit 30, and the determination unit 40.
  • the processor 91 is, for example, a CPU (Central Processing Unit), an arithmetic unit, a microprocessor, a microcomputer, a DSP (Digital Signal Processor), or the like.
  • the memory 92 is, for example, non-volatile or volatile such as RAM (RandomAccessMemory), ROM (ReadOnlyMemory), flash memory, EPROM (ErasableProgrammableReadOnlyMemory), EEPROM (ElectricallyErasableProgrammableReadOnlyMemory). It is a semiconductor memory.
  • the memory 92 may be any storage medium used in the future, such as a magnetic disk, a flexible disk, an optical disk, a compact disk, a mini disk, or a DVD.
  • Each function of the face information acquisition unit 20, the hand candidate detection unit 30, and the determination unit 40 may be partially realized by dedicated hardware and the other part may be realized by software or firmware. In this way, the processing circuit realizes each of the above functions by hardware, software, firmware, or a combination thereof.
  • FIG. 4 is a flowchart showing the gesture detection method in the first embodiment.
  • the face detection unit 10 Prior to step S1 shown in FIG. 4, the face detection unit 10 detects the face orientation of the occupant based on the image taken by the image pickup device 110 provided in the vehicle.
  • step S1 the face information acquisition unit 20 acquires information on the face orientation of the occupant from the face detection unit 10.
  • step S2 the hand candidate detection unit 30 detects a hand candidate that is a candidate for the occupant's hand based on the image captured by the image pickup device 110.
  • step S3 the determination unit 40 determines whether or not to reject the hand candidate information based on a predetermined condition regarding the face orientation.
  • the determination unit 40 rejects the hand candidate information according to the determination result.
  • the rejected hand candidate is not detected as a occupant's hand in the occupant's gesture to be detected. In other words, the gesture detection device 100 does not identify the rejected hand candidate as a hand constituting the occupant's gesture.
  • the gesture detection device 100 in the first embodiment includes a face information acquisition unit 20, a hand candidate detection unit 30, and a determination unit 40.
  • the face information acquisition unit 20 acquires information on the face orientation of the occupant. The face orientation is detected based on the image captured by the image pickup device 110 provided in the vehicle.
  • the hand candidate detection unit 30 detects a hand candidate that is a candidate for the occupant's hand based on the image.
  • the determination unit 40 rejects the hand candidate information so that the hand candidate is not detected as the occupant's hand in the gesture of the occupant to be detected, based on a predetermined condition regarding the face orientation.
  • Such a gesture detection device 100 accurately detects the hand in the gesture of the occupant.
  • the gesture detection method in the first embodiment acquires information on the face orientation of the occupant detected based on the image captured by the image pickup device 110 provided in the vehicle.
  • the gesture detection method detects a hand candidate that is a candidate for the occupant's hand based on the image. Further, the gesture detection method rejects the hand candidate information so that the hand candidate is not detected as a hand in the gesture of the occupant to be detected based on a predetermined condition regarding the face orientation.
  • the occupant's hand in the gesture is accurately detected.
  • the gesture detection device and the gesture detection method according to the second embodiment will be described.
  • the second embodiment is a subordinate concept of the first embodiment, and the gesture detection device according to the second embodiment includes each configuration of the gesture detection device 100 according to the first embodiment. The same configuration and operation as in the first embodiment will be omitted.
  • FIG. 5 is a functional block diagram showing the configuration of the gesture detection device 101 according to the second embodiment. Further, FIG. 5 shows an image pickup device 110 and an in-vehicle device 120 as devices that operate in connection with the gesture detection device 101.
  • the image pickup device 110 is provided in the front center of the vehicle interior.
  • the image pickup apparatus 110 photographs the interior of the vehicle at a wide angle, and photographs both the driver's seat and the passenger seat at the same time.
  • the image pickup device 110 is, for example, a camera that detects infrared rays, a camera that detects visible light, and the like.
  • the gesture detection device 101 according to the second embodiment detects the gesture of the hand of the occupant of the vehicle based on the image captured by the image pickup device 110.
  • the gesture is a gesture for operating the in-vehicle device 120.
  • the in-vehicle device 120 is, for example, an air conditioner, an audio system, or the like.
  • the gesture detected by the gesture detection device 101 controls the temperature of the air conditioner, adjusts the volume of the audio, and the like.
  • the in-vehicle device 120 is not limited to the air conditioner and the audio.
  • the gesture detection device 101 includes a video acquisition unit 50, a face detection unit 10, a storage unit 60, a face information acquisition unit 20, a hand candidate detection unit 30, and a determination unit 40.
  • the image acquisition unit 50 acquires the image captured by the image pickup device 110 for each frame.
  • the face detection unit 10 detects the face and face orientation of the occupant for each frame of the image. For example, the face detection unit 10 detects the face parts of the occupant and detects the face orientation based on the position of the face parts. The face orientation detected based on the position of the face part is the direction facing the front of the occupant's face. Or, for example, the face detection unit 10 detects the line of sight of the occupant and detects the face orientation based on the line of sight. The face orientation detected based on the line of sight is the direction in which the line of sight is facing. That is, the face orientation in the second embodiment includes at least one of the direction facing the front of the occupant's face and the direction of the line of sight.
  • FIG. 6 is a diagram showing an example of the face orientation of the occupant in the second embodiment.
  • Face orientation is represented by Pitch, Yaw, and Roll angles.
  • the pitch angle, yaw angle and roll angle are 0 degrees.
  • the face detection unit 10 detects at least the pitch angle and the yaw angle among the pitch angle, the yaw angle and the roll angle.
  • the face detection unit 10 in the second embodiment detects the head position in the image.
  • the head position detected in the second embodiment is a position in the height direction. In this way, the face detection unit 10 detects the face orientation and the head position of the occupant. The head position can be read as the face position.
  • the storage unit 60 stores information on the face orientation and the head position for each frame.
  • the face information acquisition unit 20 acquires face orientation information for each frame.
  • the face information acquisition unit 20 acquires the face orientation information in the frame to be processed.
  • the face information acquisition unit 20 operates as follows.
  • the frame before the frame to be processed is the first frame
  • the frame to be processed is the second frame.
  • the face orientation of the occupant in the first frame is detected.
  • the occupant's face in the second frame is not detected.
  • the face information acquisition unit 20 acquires the face orientation and head position information in the first frame from the storage unit 60.
  • the second frame is a frame within a predetermined number of frames from the first frame.
  • the predetermined number of frames may be stored in the gesture detection device 101, for example, or may be input from the outside.
  • the first frame is preferably a frame in which the face orientation of the occupant is detected most recently from the second frame.
  • the hand candidate detection unit 30 detects a hand candidate that is a candidate for the occupant's hand for each frame of the image captured by the image pickup device 110.
  • the hand candidate detection unit 30 selects a occupant's hand candidate by, for example, matching a pattern of the shape of an object (information of luminance distribution) in the image with a predetermined pattern of the shape of the hand, that is, by pattern matching processing.
  • the shape of the hand to be detected may be either the shape of the open hand or the shape of the closed hand.
  • the shape of the hand to be detected may be, for example, the shape of the hand indicating the number, the shape of the hand indicating the direction, the shape of the hand indicating the intention of the occupant (OK, Good, etc.), or the like.
  • the determination unit 40 rejects the hand candidate information for each frame based on a predetermined condition regarding the face orientation.
  • the predetermined conditions may be stored in the gesture detection device 101, for example, or may be input from the outside. An example of predetermined conditions will be described later. "Rejecting" may include the determination unit 40 identifying the hand candidate as something other than a hand. Alternatively, "rejecting” may include invalidating the information of the hand candidate by the determination unit 40. In any case, the rejected hand candidate is not detected as the occupant's hand in the occupant's gesture to be detected. In other words, the gesture detection device 101 does not identify the rejected hand candidate as a hand constituting the occupant's gesture.
  • the gesture detection device 101 identifies the hand candidate not rejected by the determination unit 40 as a hand constituting the gesture of the occupant. Based on the gesture by the occupant's hand identified by the gesture detection device 101, the operation process of the in-vehicle device 120 and the like are executed. In the functional block diagram shown in FIG. 5, the functional unit that performs processing between the determination unit 40 and the in-vehicle device 120 is not shown.
  • the determination unit 40 in the second embodiment rejects the hand candidate information when at least one of the pitch angle and the yaw angle indicating the face orientation exceeds a predetermined range. That is, the predetermined condition regarding the face orientation in the second embodiment is that at least one of the pitch angle and the yaw angle representing the face orientation exceeds the predetermined range.
  • the range is, for example, between the angle corresponding to the front direction of the face and the angle corresponding to the oblique direction in which the image pickup apparatus 110 is located. This is because when the occupant makes a gesture, the occupant's face usually faces the image pickup device 110 from the front direction.
  • the range is predetermined for each of the pitch angle and the yaw angle.
  • the predetermined condition regarding the face orientation may be that at least one of the pitch angle and the yaw angle exceeds a predetermined threshold value.
  • the determination unit 40 rejects the hand candidate information of the second frame based on the conditions regarding the face orientation and the head position in the first frame. For example, if at least one of the pitch angle, yaw angle, and head position in the first frame exceeds a predetermined range, the determination unit 40 rejects the hand candidate information in the second frame. That is, the predetermined condition regarding the face orientation is that at least one of the pitch angle, the yaw angle, and the head position in the first frame exceeds the predetermined range.
  • the functions of the face detection unit 10, the face information acquisition unit 20, the hand candidate detection unit 30, the determination unit 40, the image acquisition unit 50, and the storage unit 60 are realized by the processing circuit shown in FIG. 2 or FIG.
  • FIG. 7 is a flowchart showing the gesture detection method in the second embodiment.
  • step S10 the image acquisition unit 50 acquires a frame to be processed in the image captured by the image pickup device 110.
  • step S20 the face detection unit 10 detects the occupant's face, face orientation, and head position in the frame to be processed.
  • step S30 the gesture detection device 101 determines whether or not the face orientation is detected. If the face orientation is detected, step S40 is executed. If no face orientation is detected, step S80 is executed.
  • step S40 the storage unit 60 stores face orientation and head position information for each frame.
  • step S50 the face information acquisition unit 20 acquires face orientation information in the frame to be processed.
  • the face information acquisition unit 20 may acquire the face orientation information from the face detection unit 10 or the storage unit 60.
  • step S60 the hand candidate detection unit 30 detects the occupant's hand candidate in the frame to be processed.
  • step S70 the determination unit 40 determines whether or not the face orientation satisfies a predetermined condition.
  • the predetermined condition is that at least one of the pitch angle and the yaw angle exceeds the predetermined range. If at least one of them exceeds the range, that is, if the condition is satisfied, step S120 is executed. If both are within that range, that is, if the conditions are not met, the gesture detection method ends.
  • FIG. 8 is a diagram showing an example of a frame to be processed.
  • the occupant is gesturing the hand 31 for operating the in-vehicle device 120.
  • the face detection unit 10 detects the occupant's face 11. Further, the face frame 12 is set so as to surround the face 11.
  • the hand candidate detection unit 30 detects the occupant's hand 31 as a hand candidate.
  • the hand candidate frame 32 is set so as to surround the hand candidate.
  • the occupant's face is facing the front. In this case, both the pitch angle and the yaw angle are within a predetermined range.
  • the determination unit 40 determines that the face orientation does not satisfy a predetermined condition. Therefore, in the case of FIG. 8, the gesture detection method ends. That is, the gesture detection device 101 identifies that the hand candidate is the hand 31 that constitutes the gesture of the occupant.
  • FIG. 9 is a diagram showing an example of a frame to be processed.
  • the occupant does not make a hand gesture for operating the in-vehicle device 120.
  • the occupant is looking into the information displayed on the center console of the vehicle.
  • the information is, for example, information about navigation.
  • the face detection unit 10 detects the occupant's face 11.
  • the face frame 12 is set so as to surround the face 11.
  • the hand candidate detection unit 30 erroneously detects the occupant's face 11 as a hand candidate.
  • the hand candidate frame 32 is set so as to include the hand candidate.
  • the hand candidate detection unit 30 may determine that the occupant's face 11 is a closed hand 31 and detect it as a hand candidate. be.
  • step S120 is executed.
  • step S80 the face information acquisition unit 20 determines whether or not the frame to be processed is within a predetermined number of frames from the frame in which the face orientation of the occupant was detected most recently. If the frame to be processed is within a predetermined number of frames, that is, if this condition is satisfied, step S90 is executed. If this condition is not met, the gesture detection method ends.
  • step S90 the face information acquisition unit 20 acquires information on the face orientation and head position in the frame in which the face orientation of the occupant was detected most recently from the storage unit 60.
  • step S100 the hand candidate detection unit 30 detects the occupant's hand candidate in the frame to be processed.
  • step S110 the determination unit 40 determines whether or not the face orientation and the head position satisfy predetermined conditions.
  • the predetermined condition is that at least one of the pitch angle, yaw angle and head position exceeds the predetermined range. If at least one is beyond that range, step S120 is executed. If everything is within that range, the gesture detection method ends.
  • FIG. 10 is a diagram showing an example of a frame to be processed.
  • the frame shown in FIG. 10 is a frame after the frame shown in FIG. 9, and is a frame within a predetermined number of frames.
  • the occupant is looking further in order to confirm the information displayed on the center console of the vehicle in detail.
  • the face detection unit 10 has failed to detect the occupant's face 11, and the face frame 12 is not set.
  • the hand candidate detection unit 30 erroneously detects the occupant's head as a hand candidate. Further, the hand candidate frame 32 is set so as to include the hand candidate.
  • FIG. 10 is a frame within a predetermined number of frames from the frame of FIG.
  • the face information acquisition unit 20 acquires information on the face orientation and the head position in the frame of FIG.
  • the occupant's face is diagonally downward.
  • the pitch angle and yaw angle exceed a predetermined range.
  • the determination unit 40 determines that the face orientation satisfies a predetermined condition. Therefore, step S120 is executed.
  • step S120 the determination unit 40 rejects the hand candidate information.
  • the determination unit 40 identifies the hand candidate as something other than a hand.
  • the determination unit 40 replaces the detection result of the hand candidate with the detection result of an object other than the hand. In this way, the determination unit 40 rejects the hand candidate information based on a predetermined condition regarding the face orientation. This completes the gesture detection method.
  • the gesture detection device 101 performs a hand candidate detection process after performing a face 11 detection process and a face orientation information acquisition process.
  • the gesture detection device 101 may execute the face 11 detection process and the face orientation information acquisition process after the hand candidate detection process.
  • the gesture detection device 101 may execute the hand candidate detection process in parallel with the face 11 detection process and the face orientation information acquisition process.
  • the face detection unit 10 succeeds in detecting the occupant's face 11 and the face orientation in the first frame, and fails to detect the occupant's face 11 in the second frame.
  • the first frame is a frame in which the face orientation of the occupant is detected most recently from the second frame.
  • the frame shown in FIG. 9 corresponds to the first frame
  • the frame shown in FIG. 10 corresponds to the second frame.
  • step S10 the image acquisition unit 50 acquires the second frame in the image captured by the image pickup device 110.
  • step S20 the face detection unit 10 fails to detect the occupant's face 11 in the second frame. Therefore, the face orientation and head position are not detected.
  • step S30 the gesture detection device 101 determines that the face orientation of the occupant has not been detected.
  • Step S80 is executed.
  • step S80 the face information acquisition unit 20 determines whether or not the second frame is within a predetermined number of frames from the first frame in which the face orientation of the occupant is detected most recently. As described above, in order to satisfy this condition for the first frame and the second frame, step S90 is executed.
  • step S90 the face information acquisition unit 20 acquires information on the face orientation and head position in the first frame from the storage unit 60.
  • step S100 the hand candidate detection unit 30 detects the occupant's hand candidate in the second frame.
  • step S110 the determination unit 40 determines whether or not at least one of the pitch angle, yaw angle, and head position in the first frame exceeds a predetermined range. If at least one is beyond that range, step S120 is executed. If everything is within that range, the gesture detection method ends.
  • step S120 the determination unit 40 rejects the hand candidate information in the second frame. This completes the gesture detection method for one processing target frame. After that, step S10 is executed again for the next frame.
  • Such a gesture detection device 101 reduces the identification of objects other than the occupant's hand as the hand 31. That is, the gesture detection device 101 accurately detects the hand 31 that constitutes the gesture of the occupant.
  • the occupant When the occupant operates the in-vehicle device 120, the occupant looks into the information displayed on the display device such as the dashboard of the vehicle and the center console to confirm. In that case, the occupant's head is reflected in the detection range of the hand candidate.
  • the hand candidate detection unit 30 may determine that the occupant's head (or face 11) is a closed hand 31 (such as a thumbs-up hand) and detect it as a hand candidate (for example, FIG. 10).
  • the face orientation of the occupant is included in the diagonal range from the front direction of the vehicle to the position of the image pickup device 110 (for example, FIG. 8).
  • the face orientation is out of the range.
  • the gesture detection device 101 in the second embodiment rejects the information of the hand candidate erroneously detected based on the predetermined condition regarding the face orientation. In other words, when the face orientation of the occupant exceeds a predetermined range, the gesture detection device 101 identifies that the hand candidate is something other than the hand, and rejects the information of the hand candidate. As a result, the gesture detection device 101 accurately detects the hand 31 in the gesture of the occupant.
  • the predetermined conditions regarding face orientation are not limited to the above conditions.
  • the condition is that at least one of the pitch, yaw, and roll angles and the head position of at least one of the lateral head position, the depth head position, and the height head position. It may be a combination of conditions.
  • the gesture detection device 101 in the second embodiment includes a storage unit 60.
  • the storage unit 60 stores information on the face orientation detected for each frame of the image and information on the head position of the occupant.
  • the face information acquisition unit 20 performs the face in the first frame.
  • the orientation information and the head position information are acquired from the storage unit 60.
  • the second frame is a frame within a predetermined number of frames (first predetermined number of frames) from the first frame.
  • the hand candidate detection unit 30 detects the hand candidate in the second frame.
  • the determination unit 40 rejects the hand candidate information in the second frame based on the predetermined conditions regarding the face orientation and the head position in the first frame.
  • the hand candidate detection unit 30 Since the face detection pattern matching process and the hand candidate detection pattern matching process are different from each other, even if the face detection unit 10 fails to detect the occupant's face orientation, the hand candidate detection unit 30 still uses the occupant's face 11. , Head, etc. may be erroneously detected as a hand candidate (for example, FIG. 10).
  • the movement of the occupant from the posture shown in FIG. 9 to the posture shown in FIG. 10, that is, the movement of the occupant looking into the display device is continuous and is performed in a short time. Therefore, even when the determination unit 40 rejects the hand candidate information in the frame to be processed based on the face orientation information in the frame close in time to the frame to be processed, the accuracy of the rejection determination is high. It doesn't get worse.
  • the gesture detection device 101 according to the second embodiment prevents the occupant's face 11, head, and the like from being detected as hand candidates even when the occupant's face orientation is temporarily not detected. As a result, the detection accuracy of the occupant's hand
  • the first frame in the second embodiment is a frame in which the face orientation of the occupant is detected most recently from the second frame.
  • the gesture detection device 101 determines whether or not the most recently detected face orientation satisfies a predetermined condition. Therefore, the gesture detection device 101 accurately detects the occupant's hand 31.
  • the gesture detection device and the gesture detection method according to the third embodiment will be described.
  • the third embodiment is a subordinate concept of the first embodiment, and the gesture detection device according to the third embodiment includes each configuration of the gesture detection device 101 according to the second embodiment. The same configuration and operation as those of the first and second embodiments will be omitted.
  • FIG. 11 is a diagram showing the relationship from the first frame to the fourth frame in the third embodiment.
  • the first frame is the first frame to be processed among the plurality of frames constituting the video.
  • the face orientation of the occupant is detected, and the hand candidate is also detected.
  • the determination unit 40 determines whether or not at least one of the pitch angle and the yaw angle of the face 11 in the first frame exceeds a predetermined range. .. The determination unit 40 rejects the hand candidate information in the first frame based on the determination result.
  • the second frame is a frame after the first frame by a predetermined number of frames.
  • the first predetermined number of frames may be stored in the gesture detection device, for example, or may be input from the outside.
  • the face orientation of the occupant is not detected, but the hand candidate is detected.
  • the first frame is a frame in which the face orientation of the occupant is detected most recently from the second frame.
  • the determination unit 40 has at least one of the pitch angle, yaw angle, and head position in the first frame in advance. Determine if it exceeds the specified range.
  • the determination unit 40 rejects the hand candidate information in each frame based on the determination result. This determination operation is the same as that of the second embodiment.
  • the gesture detection device prevents the occupant's face 11, head, and the like from being detected as hand candidates even when the occupant's face orientation is temporarily not detected.
  • the third frame is a frame after the second frame.
  • the face orientation of the occupant is not detected, but the hand candidate is detected.
  • the determination unit 40 rejects the hand candidate information. In other words, the determination unit 40 rejects the hand candidate information without making a rejection determination regarding the face orientation.
  • step S80 shown in FIG. 7 each frame from the frame next to the second frame to the third frame is determined to be "No". That is, the hand candidate information is not rejected. Therefore, an object other than the hand detected as a hand candidate may be recognized as the hand 31. Therefore, the gesture detection device rejects the information of the hand candidate detected in each frame from the frame next to the second frame to the third frame without making a rejection determination regarding the face orientation. Therefore, the detection accuracy of the occupant's hand 31 is improved.
  • the fourth frame is a frame after the second predetermined number of frames from the frame following the third frame.
  • the second predetermined number of frames may be stored in the gesture detection device, for example, or may be input from the outside.
  • the face orientation of the occupant is continuously detected from the frame following the third frame to the fourth frame.
  • hand candidates are detected.
  • the determination unit 40 rejects the hand candidate information. In other words, the determination unit 40 rejects the hand candidate information without making a rejection determination regarding the face orientation.
  • the detected face orientation may not be accurate.
  • the occupant's face 11 usually faces the direction in which the image pickup apparatus 110 is located from the front direction. Therefore, it is preferable that the rejection determination of the hand candidate information is restarted in a state where the face orientation of the occupant is likely to be included in the predetermined range.
  • the gesture detection device rejects the hand candidate information until the face detection is successful for the second predetermined number of frames or more.
  • the determination unit 40 restarts the determination as to whether or not to reject the hand candidate information. This improves the detection accuracy of the occupant's hand 31.
  • the gesture detection device and the gesture detection method according to the fourth embodiment will be described.
  • the fourth embodiment is a subordinate concept of the first embodiment.
  • the gesture detection device according to the fourth embodiment includes each configuration of the gesture detection device 101 according to the second embodiment. The same configuration and operation as any one of the first to third embodiments will be omitted.
  • FIG. 12 is a functional block diagram showing the configuration of the gesture detection device 102 according to the fourth embodiment.
  • the gesture detection device 102 includes a hand candidate detection unit 30A.
  • the hand candidate detection unit 30A in the fourth embodiment is a modified example of the hand candidate detection unit 30 in the second embodiment.
  • the hand candidate detection unit 30A performs hand candidate detection processing in the hand candidate detection target area set in the video.
  • the detection target area is preset, for example, at a position where the occupant performs a hand gesture. Or, for example, the detection target area is set at a position including the hand candidate frame 32 detected in the frame before the frame to be processed.
  • the hand candidate detection unit 30A in the fourth embodiment narrows the detection target area based on a predetermined condition regarding the face orientation. For example, as a predetermined condition, when at least one of the pitch angle and the yaw angle exceeds the predetermined range, the hand candidate detection unit 30A narrows the detection target area. In other words, the predetermined condition is that at least one of the pitch angle and the yaw angle exceeds the predetermined range.
  • the hand candidate detection unit 30A performs hand candidate detection processing in the narrowed detection target area.
  • the determination of whether or not the face orientation satisfies a predetermined condition is executed by, for example, the hand candidate detection unit 30A.
  • the hand candidate detection unit 30A may acquire the determination result by the determination unit 40.
  • the detection target area corresponds to, for example, a region called a gesture detection region.
  • FIG. 13 is a flowchart showing the gesture detection method according to the fourth embodiment. Steps S10 to S50 are the same as those in the second embodiment. Further, step S80 and step S90 are the same as those in the second embodiment.
  • the hand candidate detection unit 30A detects the occupant's hand candidate in the detection target area.
  • FIG. 14 is a diagram showing an example of a frame to be processed.
  • the occupant does not make a hand gesture for operating the in-vehicle device 120.
  • the occupant operates the rear view mirror 2 provided in the room by hand 31.
  • the occupant is looking at the rear-view mirror 2.
  • the face detection unit 10 detects the occupant's face 11.
  • the face frame 12 is set so as to surround the face 11.
  • the detection target area 33A is set in the frame.
  • the hand candidate detection unit 30A detects the occupant's hand 31 as a hand candidate within the detection target area 33A.
  • the hand candidate frame 32 is set so as to include the hand candidate.
  • the determination unit 40 determines whether or not the face orientation satisfies a predetermined condition.
  • the face orientation of the occupant detected by the face detection unit 10 is diagonally upward. Both the pitch angle and the yaw angle indicating the face orientation exceed a predetermined range.
  • the hand candidate detection unit 30A determines that the face orientation satisfies a predetermined condition. Step S120 is executed.
  • step S120 the hand candidate information is rejected.
  • step S130 is executed after step S120.
  • the hand candidate detection unit 30A narrows the detection target area 33A so that the rejected hand candidate is not detected.
  • the hand candidate detection unit 30A sets, for example, a detection target area 33B in which the size of the detection target area 33A is reduced so as not to include the hand candidate frame 32.
  • the detection target area 33B corresponds to an area in which the upper portion of the detection target area 33A is reduced.
  • the hand candidate detection unit 30A detects the hand candidate in the narrowed detection target area 33B.
  • Such a gesture detection device 102 does not detect the hand 31 that operates the rearview mirror 2 as a hand in the gesture for operating the in-vehicle device 120. More specifically, the gesture detection device 102 rejects the information of the hand candidate detected based on the hand 31 that operates the rearview mirror 2. Further, the gesture detection device 102 narrows the detection target area 33A based on the information on the face orientation of the occupant so that the hand candidate is not detected in the area around the rearview mirror 2.
  • the positions and sizes of the detection target areas 33A and 33B shown in FIG. 14 are examples, and are not limited thereto.
  • the detection target areas 33A and 33B may be not only the central portion between the driver's seat and the passenger seat but also the regions extended in the direction of both seats (left-right direction).
  • the gesture detection device 102 described above executes a process of narrowing the detection target area 33A after the process of rejecting the information of the hand candidate.
  • the timing of the process of narrowing the detection target area 33A may be between steps S50 and S60 and between steps S90 and S100.
  • the hand candidate detection unit 30A narrows the detection target area 33A in the processing target frame when the face orientation in the processing target frame satisfies a predetermined condition.
  • the gesture detection device shown in each of the above embodiments can be applied to a system constructed by appropriately combining a navigation device, a communication terminal, a server, and the functions of applications installed in the navigation device.
  • the navigation device includes, for example, a PND (Portable Navigation Device) and the like.
  • the communication terminal includes, for example, a mobile terminal such as a mobile phone, a smartphone and a tablet.
  • FIG. 15 is a block diagram showing the configuration of the gesture detection device 101 and the device that operates in connection with the gesture detection device 101 in the fifth embodiment.
  • the gesture detection device 101 and the communication device 130 are provided in the server 300.
  • the gesture detection device 101 acquires an image taken by the image pickup device 110 provided in the vehicle 1 via the communication device 140 and the communication device 130.
  • the gesture detection device 101 acquires information on the face orientation of the occupant detected based on the image.
  • the gesture detection device 101 detects a hand candidate based on the image.
  • the gesture detection device 101 rejects the hand candidate information based on a predetermined condition regarding the face orientation.
  • the gesture detection device 101 identifies the hand candidate that has not been rejected as the hand 31 that constitutes the gesture of the occupant. Based on the gesture by the occupant's hand 31 identified by the gesture detection device 101, the operation process of the in-vehicle device 120 and the like are executed.
  • gesture detection device 101 may be provided in the server 300, and the other part may be provided in the vehicle 1 in a distributed manner. The same effect is obtained when the gesture detection device 100 shown in the first embodiment is provided in the server 300.
  • each embodiment can be freely combined, and each embodiment can be appropriately modified or omitted.

Landscapes

  • Engineering & Computer Science (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Remote Sensing (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Automation & Control Theory (AREA)
  • Image Analysis (AREA)
  • User Interface Of Digital Computer (AREA)

Abstract

La présente invention concerne un dispositif de détection de geste détectant avec précision la main dans un geste effectué par un occupant de véhicule. Le dispositif de détection de geste comprend une unité d'acquisition d'informations de visage, une unité de détection de main candidate et une unité de détermination. L'unité d'acquisition d'informations de visage acquiert des informations concernant l'orientation faciale de l'occupant du véhicule. L'orientation faciale est détectée sur la base d'une vidéo prise par un dispositif d'imagerie installé dans un véhicule. L'unité de détection de main candidate détecte une main candidate, qui est une candidate pour la main de l'occupant du véhicule, sur la base de la vidéo. L'unité de détermination rejette, sur la base d'une condition prédéterminée liée à l'orientation faciale, des informations de main candidate de telle sorte que la main candidate n'est pas détectée en tant que main de l'occupant du véhicule qui doit être détectée dans le geste effectué par l'occupant du véhicule.
PCT/JP2020/020828 2020-05-27 2020-05-27 Dispositif de détection de geste et procédé de détection de geste WO2021240668A1 (fr)

Priority Applications (1)

Application Number Priority Date Filing Date Title
PCT/JP2020/020828 WO2021240668A1 (fr) 2020-05-27 2020-05-27 Dispositif de détection de geste et procédé de détection de geste

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2020/020828 WO2021240668A1 (fr) 2020-05-27 2020-05-27 Dispositif de détection de geste et procédé de détection de geste

Publications (1)

Publication Number Publication Date
WO2021240668A1 true WO2021240668A1 (fr) 2021-12-02

Family

ID=78723081

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2020/020828 WO2021240668A1 (fr) 2020-05-27 2020-05-27 Dispositif de détection de geste et procédé de détection de geste

Country Status (1)

Country Link
WO (1) WO2021240668A1 (fr)

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2014048937A (ja) * 2012-08-31 2014-03-17 Omron Corp ジェスチャ認識装置、その制御方法、表示機器、および制御プログラム
JP2014197252A (ja) * 2013-03-29 2014-10-16 パナソニック株式会社 ジェスチャー操作装置、そのプログラム、およびジェスチャー操作装置を搭載した車両
WO2018122891A1 (fr) * 2016-12-26 2018-07-05 三菱電機株式会社 Dispositif d'entrée de panneau tactile, dispositif de détermination de geste tactile, procédé de détermination de geste tactile et programme de détermination de geste tactile
JP2018528536A (ja) * 2015-08-31 2018-09-27 エスアールアイ インターナショナルＳＲＩ Ｉｎｔｅｒｎａｔｉｏｎａｌ 運転行動を監視する方法及びシステム
WO2019229938A1 (fr) * 2018-05-31 2019-12-05 三菱電機株式会社 Dispositif de traitement d'image, procédé de traitement d'image, et système de traitement d'image
JP2019536673A (ja) * 2017-08-10 2019-12-19 ペキン センスタイム テクノロジー ディベロップメント カンパニー リミテッド 運転状態監視方法及び装置、運転者監視システム、並びに車両

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2014048937A (ja) * 2012-08-31 2014-03-17 Omron Corp ジェスチャ認識装置、その制御方法、表示機器、および制御プログラム
JP2014197252A (ja) * 2013-03-29 2014-10-16 パナソニック株式会社 ジェスチャー操作装置、そのプログラム、およびジェスチャー操作装置を搭載した車両
JP2018528536A (ja) * 2015-08-31 2018-09-27 エスアールアイ インターナショナルＳＲＩ Ｉｎｔｅｒｎａｔｉｏｎａｌ 運転行動を監視する方法及びシステム
WO2018122891A1 (fr) * 2016-12-26 2018-07-05 三菱電機株式会社 Dispositif d'entrée de panneau tactile, dispositif de détermination de geste tactile, procédé de détermination de geste tactile et programme de détermination de geste tactile
JP2019536673A (ja) * 2017-08-10 2019-12-19 ペキン センスタイム テクノロジー ディベロップメント カンパニー リミテッド 運転状態監視方法及び装置、運転者監視システム、並びに車両
WO2019229938A1 (fr) * 2018-05-31 2019-12-05 三菱電機株式会社 Dispositif de traitement d'image, procédé de traitement d'image, et système de traitement d'image

Similar Documents

Publication Publication Date Title
CN108958467B (zh) 用于控制全息图的显示器的设备和方法，车辆***
JP3903968B2 (ja) 非接触式情報入力装置
JP6513321B2 (ja) 車両用撮像制御装置、ドライバモニタリング装置、及び、車両用撮像制御方法
WO2019016936A1 (fr) Dispositif et procédé d'aide au fonctionnement
WO2018061413A1 (fr) Dispositif de détection de geste
WO2021240668A1 (fr) Dispositif de détection de geste et procédé de détection de geste
KR102441079B1 (ko) 차량의 디스플레이 제어 장치 및 방법
US10953811B2 (en) Vehicle image controller, system including the same, and method thereof
JP7109649B2 (ja) 覚醒度推定装置、自動運転支援装置および覚醒度推定方法
JP7051014B2 (ja) 顔検出処理装置および顔検出処理方法
WO2021240671A1 (fr) Dispositif de détection de geste et procédé de détection de geste
JP2009113599A (ja) 車載用入力装置
JP2022143854A (ja) 乗員状態判定装置および乗員状態判定方法
JP7175381B2 (ja) 覚醒度推定装置、自動運転支援装置および覚醒度推定方法
WO2021229741A1 (fr) Dispositif de détection de geste et procédé de détection de geste
JP6865906B2 (ja) 表示制御装置及び表示制御方法
JP6847323B2 (ja) 視線検出装置および視線検出方法
JP7258262B2 (ja) 調整装置、調整システム、表示装置、乗員監視装置、および、調整方法
JP7267517B2 (ja) ジェスチャ認識装置およびジェスチャ認識方法
WO2022157880A1 (fr) Dispositif de détection de main, dispositif de reconnaissance de geste et procédé de détection de main
JP6956686B2 (ja) 角度制御装置および角度制御方法
US20240174248A1 (en) Vehicle warning apparatus
CN115206130B (zh) 一种车位检测方法、***、终端及存储介质
US20230234560A1 (en) Method and Apparatus for Autonomous Parking Assist
WO2023170777A1 (fr) Dispositif de surveillance d'occupant de véhicule, procédé de surveillance d'occupant de véhicule, et programme de surveillance d'occupant de véhicule

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20938446

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20938446

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: JP