CN115147926A - Gesture recognition method and head-mounted equipment - Google Patents
Gesture recognition method and head-mounted equipment Download PDFInfo
- Publication number
- CN115147926A CN115147926A CN202210771164.9A CN202210771164A CN115147926A CN 115147926 A CN115147926 A CN 115147926A CN 202210771164 A CN202210771164 A CN 202210771164A CN 115147926 A CN115147926 A CN 115147926A
- Authority
- CN
- China
- Prior art keywords
- wireless communication
- gesture recognition
- module
- communication module
- head
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
- G06V40/28—Recognition of hand or arm movements, e.g. recognition of deaf sign language
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/014—Hand-worn input/output arrangements, e.g. data gloves
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Health & Medical Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- General Health & Medical Sciences (AREA)
- Psychiatry (AREA)
- Social Psychology (AREA)
- Multimedia (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
According to the scheme, an event camera module is controlled to collect continuous images, corresponding image event information is generated when the change value of the pixel values collected twice continuously is detected to meet a preset condition for each pixel point in the images, the image event information is sent to a gesture recognition module, the gesture recognition module determines the gesture action of a target hand based on the image event information, the gesture recognition delay in the head-mounted equipment is reduced, and the gesture recognition speed is improved.
Description
Technical Field
The application relates to the technical field of gesture recognition, in particular to a gesture recognition method and a head-mounted device.
Background
With the continuous development of science and technology, various head-mounted devices come out endlessly. In order to meet the user requirements and achieve user interaction, the head-mounted device is required to perform gesture recognition, so that a corresponding control instruction is generated according to the recognized user gesture to complete control over the head-mounted device. Therefore, how to effectively recognize the gesture of the user by the head-mounted device has become a technical problem to be solved.
Disclosure of Invention
An object of the embodiments of the present application is to provide a gesture recognition method and a head-mounted device, so as to solve the above technical problems.
In order to achieve the above object, the technical solution provided by the present application includes:
in a first aspect, a gesture recognition method is provided, which is applied to a head-mounted device, where the head-mounted device includes an event camera module and a gesture recognition module, and the method includes:
controlling the event camera module to acquire continuous images, generating corresponding image event information when detecting that the change value of pixel values acquired twice continuously meets a preset condition for each pixel point in the images, and sending the image event information to the gesture recognition module; the image event information comprises identification information and timestamp information of the pixel points, and the image event information further comprises at least one of pixel values of the pixel points and pixel value change information of the pixel points;
and controlling the gesture recognition module to determine the gesture action of the target hand based on the image event information.
In the above embodiment, the image event information is generated by the event camera module, and the gesture recognition is performed according to the image event information to determine the gesture motion of the target hand, so that the gesture recognition delay in the head-mounted device is reduced, and the gesture recognition speed is increased.
With reference to the first aspect, in some optional embodiments, before the controlling the event camera module to acquire the successive images, the method includes:
determining a first position of a hand-mounted device relative to the head-mounted device;
and determining that the target hand is located within a preset position range of the head-mounted device according to the first position.
In the above embodiment, after the target hand is determined to be located within the preset position range of the head-mounted device, the image event information is sent to the gesture recognition module, so that the possibility that the gesture recognition module recognizes the gesture is improved.
With reference to the first aspect, in some optional embodiments, the method further comprises:
and when the target hand is determined not to be in the preset position range according to the first position, controlling the event camera module to enter a low power consumption state.
In the above embodiment, when it is determined that the target hand is not within the preset position range, the event camera module is controlled to enter the low power consumption state, so that the overall power consumption of the head-mounted device can be reduced.
With reference to the first aspect, in some optional embodiments, the sending the image event information to the gesture recognition module includes:
determining a second position of the target hand relative to the head-mounted device according to the first position;
determining a target pixel region in the image corresponding to the second position;
and sending the image event information corresponding to each pixel point in the target pixel area to the gesture recognition module.
In the above embodiment, the image event information corresponding to each pixel point in the target pixel region is sent to the gesture recognition module, so that the transmission amount of image data can be reduced, and the data processing amount of the gesture recognition module can also be reduced.
With reference to the first aspect, in some optional embodiments, the method further comprises:
and after the target pixel area is determined, for the pixel points in the image except the target pixel area, the collection of the pixel values of the pixel points is suspended.
In the above embodiment, the acquisition of the pixel values of the pixel points in the image except for the target pixel region is suspended, so that the power consumption of the event camera module can be reduced.
With reference to the first aspect, in some optional embodiments, the gesture recognition module determines a gesture action of the target hand based on the image event information, including:
determining a second position of the target hand relative to the head-mounted device according to the first position;
determining a target pixel region in the image corresponding to the second position;
and determining the gesture action of the target hand based on the image event information corresponding to each pixel point in the target pixel area.
In the above embodiment, the gesture recognition module directly analyzes the image event information corresponding to each pixel point in the target pixel region to determine the target gesture, so that the gesture recognition efficiency of the gesture recognition module can be improved.
With reference to the first aspect, in some optional embodiments, the head-mounted device includes a first display device provided with a first wireless communication module and a second display device provided with a second wireless communication module, the gesture recognition module is located within the first display device, and the event camera module includes a first event camera module located in the first display device and a second event camera module located in the second display device;
the second event camera module sends the generated image event information to the first wireless communication module through the second wireless communication module, and the first wireless communication module sends the image event information to the gesture recognition module.
In the embodiment, the two event camera modules located at different position points are used for acquiring the event images, so that the accuracy of gesture recognition is improved, the transmission of image event information is realized in a wireless communication mode, and the circuit structure between the first display device and the second display device can be simplified.
With reference to the first aspect, in some optional embodiments, the second wireless communication module does not transmit, to the first wireless communication module, image event information corresponding to pixel points in the image other than the target pixel region.
In the above embodiment, the second display device does not need to transmit the related information of the pixel points outside the target pixel region, that is, does not need to transmit the information of the entire image to the first display device, and the data amount of wireless transmission can be reduced.
With reference to the first aspect, in some optional embodiments, the head-mounted device further includes a third wireless communication module and a fourth wireless communication module, and the hand-mounted device includes a fifth wireless communication module;
the determining a first position of a hand-worn device relative to the head-worn device comprises:
determining a first position of the hand-worn device relative to the head-worn device based on communication between the third wireless communication module and the fifth wireless communication module, and communication between the fourth wireless communication module and the fifth wireless communication module.
In the above embodiment, the positioning of the hand-worn device is realized through the communication between the communication module on the head-worn device and the communication module on the hand-worn device.
In a second aspect, the present application further provides a head-mounted device comprising: the system comprises a processor, a memory, an event camera module and a gesture recognition module;
the memory has stored therein a computer program;
the processor executes the computer program to implement any one of the above-mentioned gesture recognition methods.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are required to be used in the embodiments of the present application will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present application and therefore should not be considered as limiting the scope, and that those skilled in the art can also obtain other related drawings based on the drawings without inventive efforts.
Fig. 1 is a schematic overall flow chart of a gesture recognition method according to an embodiment of the present disclosure;
fig. 2 is a schematic flowchart of a first part of a gesture recognition method according to an embodiment of the present application;
fig. 3 is a schematic flow chart of a second part of a gesture recognition method according to an embodiment of the present application;
FIG. 4 is a schematic diagram illustrating a first position determination provided by an embodiment of the present application;
fig. 5 is a schematic structural diagram of a head-mounted device according to an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be described in detail below with reference to the drawings in the embodiments of the present application. It should be noted that the following embodiments may be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.
The embodiment of the application provides a gesture recognition method which can be applied to a head-mounted device comprising an event camera module and a gesture recognition module. Through the method, the accurate recognition of the gesture can be realized, the specific flow of the method can be seen in fig. 1, and the method comprises the following steps:
s11: and controlling the event camera module to acquire continuous images, generating corresponding image event information when detecting that the change value of the pixel values acquired twice continuously meets a preset condition for each pixel point in the images, and sending the image event information to the gesture recognition module.
S12: and controlling the gesture recognition module to determine the gesture action of the target hand based on the image event information.
It should be noted that the head-mounted device in the embodiments of the present application refers to any device that can be worn on the head and includes an event camera module and a gesture recognition module, including but not limited to: intelligent glasses, intelligent helmet and the equipment that fuses to the head dress. Hand-worn devices refer to any device that can be worn on the hand, including but not limited to: smart watch, smart bracelet, smart ring and fuse the equipment to the hand dress.
The image event information corresponding to each pixel point in the embodiment of the present application includes: the image event information further includes at least one of a pixel value of the pixel point and pixel value change information of the pixel point.
First, an event camera module in the embodiment of the present application will be described.
The event camera module in the embodiment of the application can acquire image information in a scene, detect each pixel point in the image, and capture dynamic changes in the scene in an event-driven mode. Unlike conventional camera modules, when a real scene changes, the event camera module generates some pixel-level outputs (i.e., events), and an event can be represented by (t, x, y, p). Where x and y constitute the pixel coordinates of the event in two-dimensional space, i.e. the pixel point identity. t is a timestamp of the event and may characterize the point in time at which the event occurred. p is the polarity of the event, and may represent the current pixel value of the pixel point, and may also represent the change condition of the pixel point, for example, it may be a change value, and may also represent the brightness change of the scene: up or down.
The event camera module may be used for depth information measurement. The event camera module in the embodiments of the present application may employ a DVS (dynamic vision) sensor, or a DAVIS (dynamic active pixel vision) sensor. The event camera module captures dynamic changes in the scene and outputs a sparse, asynchronous event stream at microsecond temporal resolution. And aiming at each pixel point, when the change value of the pixel point acquired twice continuously is detected to meet the preset condition, generating corresponding image event information. For example, when it is detected that a variation value of a pixel value of a certain pixel exceeds a preset threshold, corresponding image event information may be generated. The event camera module has a very high dynamic range, very low latency, high time resolution, and very low requirements on data storage and computational resources.
It is understood that the event camera module is an asynchronous sensor, creating a paradigm shift in the way visual information is acquired. In the embodiment of the application, when the change value of the pixel value of a certain pixel point reaches the set threshold value, the event camera module can generate corresponding image event information, namely, an event can be returned, and in the whole camera view field, an event can be returned as long as the change condition of one pixel value meets the preset condition.
In some optional embodiments, the event camera module may first send the pixel value of each pixel point in the collected first frame image to the gesture recognition module, that is, send the collected first frame image to the gesture recognition module, and then generate corresponding image event information for each pixel point according to the above principle for the subsequently collected image, that is, for the subsequently collected image, the pixel value of each pixel point does not need to be sent to the gesture recognition module, but the image event information of the pixel point whose change value of the pixel points collected twice continuously meets the preset condition may be sent to the gesture recognition module, so that the image transmission amount is greatly reduced. For the gesture recognition module, the image event information and the first frame image can be analyzed, and the gesture action of the user is determined based on the pixel value change condition of each pixel point.
In other optional embodiments, the event camera module may only send the image event information to the gesture recognition module, that is, for the pixel points whose change values of the pixel values do not satisfy the preset condition, the related information of the pixel points (such as the pixel values, the change conditions of the pixel values, and the like) may not be sent to the gesture recognition module, so that the data transmission amount may be reduced, and the gesture recognition efficiency may be improved. For the gesture recognition module, the gesture recognition module can analyze the image event information, and determine the gesture action of the user based on the pixel value change condition of each pixel point.
The gesture recognition module in the embodiment of the present application is explained below.
The gesture recognition module can determine the gesture action of the target hand according to a preset algorithm and the received image event information. In some embodiments, the gesture recognition module may perform feature extraction from the image event information to obtain an image feature vector. In some embodiments, the gesture recognition module may utilize deep learning, neural network approaches to accomplish gesture recognition. The gesture recognition model may be a pre-trained KNN model, a random forest, a lightweight network Mobilenetv2 model, and the like, which is not limited herein.
The gesture recognition module can recognize the gesture motion of the target hand only when the target hand is within the collection view angle range of the event camera module. Therefore, in some embodiments, before step S11, it may be determined whether the target hand is located within the preset position range of the head-mounted device, if so, step S11 is then performed, otherwise, the event camera module is controlled to enter the low power consumption state.
For example, when it is determined that the target hand is not within the preset position range according to the first position, the gesture recognition module may be further controlled to enter a low power consumption state.
In this embodiment of the application, the low power consumption state refers to a state in which all or part of the functions of the event camera module or the gesture recognition module are turned off, and when two event camera modules are included in the head-mounted device, one event camera module on the head-mounted device may be turned off.
The preset position range in the embodiment of the application can be flexibly set by developers, for example, a part or all of the regions in the event camera module collecting view angle range can be set as the preset position range. Illustratively, a partial area of the front surface of the event camera module is taken as a preset position range.
For example, a first position of the hand-mounted device relative to the head-mounted device may be determined, and then whether the target hand is located within a preset position range of the head-mounted device may be determined according to the first position.
The target hand in the embodiment of the application can be flexibly arranged according to the requirement, such as a wrist, an arm, a finger and the like.
When the user uses the hand-worn device, the user needs to wear the hand-worn device correctly on the corresponding position of the hand, so that after the hand-worn device is worn correctly on the hand of the user, the position of the hand-worn device relative to the target hand is basically fixed, and therefore, the second position of the target hand wearing the hand-worn device relative to the head-worn device can be determined according to the first position of the hand-worn device relative to the head-worn device. It should be noted that the first position in the embodiment of the present application may be a position area, or may be an accurate position point. When the first position is a position area, the position area may be determined based on a distance or an angle, which is described in detail below.
In a first alternative embodiment, step S11 may include the sub-steps as shown in fig. 2:
s111: and determining a second position of the target hand part relative to the head-mounted device according to the first position of the hand-mounted device relative to the head-mounted device.
S112: and determining a target pixel area in the acquired image corresponding to the second position.
S113: and sending the image event information corresponding to each pixel point in the target pixel area to a gesture recognition module.
Illustratively, the image event information of the pixel points in the image except the target pixel region may not be sent to the gesture recognition module, so that the data transmission amount can be further reduced, and the data processing amount of the gesture recognition module can be reduced. Or after the target pixel area is determined, the collection of the pixel value of the pixel point is suspended aiming at the pixel point in the image except the target pixel area, so that the power consumption of the event camera module is reduced.
It can be understood that, for convenience of data processing, the positions of the pixels in the image, the target hand and the hand-worn device can be unified into the same coordinate system.
Therefore, for step S112, the second position may be mapped to the acquired image, the position of the target pixel point corresponding to the second position is determined from the image, and according to the position of the target pixel point, the target pixel area is determined from the image according to the preset area determination rule, so that the determined target pixel area includes the target hand, and thus the gesture action of the target hand can be determined. The target pixel area is usually a part of the whole image, for example, it may only occupy 1/10, 1/4, 1/2, etc. of the whole image pixel area.
The region determination rule in the embodiment of the present application may be flexibly set by a developer, for example, the position of a target pixel point may be used as a center, the center is used as a circle center, a preset length R is used as a radius, a circle region is determined, and a collection region formed by the circle regions of the target pixel points is used as a target pixel region. The developer can determine the size of R according to the possible motion trail of the target hand of the user in the process of drawing the corresponding gesture action. The region determination rule provided in the embodiment of the present application does not constitute a limitation to the rule, and in other embodiments, other rules may also be used to determine the target pixel region.
In a second alternative embodiment, step S12 may include the sub-steps shown in fig. 3:
s121: a second position of the target hand relative to the head mounted device is determined based on the first position of the hand mounted device relative to the head mounted device.
S122: a target pixel region in the image corresponding to the second location is determined.
S123: and determining the gesture action of the target hand based on the image event information corresponding to each pixel point in the target pixel area.
In the second embodiment, the gesture recognition module is required to determine the target pixel area, and the gesture recognition module determines the gesture motion of the target hand according to the image event information corresponding to each pixel point in the target pixel area.
Illustratively, the head-mounted device in the embodiment of the present application includes a first display device and a second display device, the gesture recognition module is located in the first display device, and the event camera module includes a first event camera module located in the first display device and a second event camera module located in the second display device.
The first display device and the second display device are arranged on different positions of the head-mounted equipment. For example, when the head-mounted device is smart glasses, the first display device may be disposed on left glasses, and the second display device may be disposed on right glasses. In the embodiment of the present application, the first display device may be used as a master display device, and the second display device may be used as a slave display device.
The image event information generated by the first event camera module and the second event camera module can be sent to the gesture recognition module, and the gesture recognition module combines the image event information generated by the two event camera modules, so that the effect of accurately recognizing gesture actions is achieved.
For example, since the second event camera module and the gesture recognition module are not integrated in one device, to simplify the circuit configuration, the second event camera module in the embodiment of the present application may transmit the image event information to the gesture recognition module by wireless communication. That is, the first display device may be provided with a first wireless communication module, the second display device may be provided with a second wireless communication module, and the second event camera module may transmit the generated image event information to the first wireless communication module through the second wireless communication module and the first wireless communication module transmits the generated image event information to the gesture recognition module.
In some embodiments, the second wireless communication module does not transmit the image event information corresponding to the pixel points in the image except the target pixel region to the first wireless communication module, that is, the information of the whole image does not need to be transmitted, so that the data amount of wireless transmission can be reduced.
Since the first event camera module is integrated with the gesture recognition module in the same display device, the first event camera module may transmit the generated image event information to the gesture recognition module by way of wired communication. It is to be understood that the communication method herein is not limited to the communication method between the event camera module and the gesture recognition module.
Exemplarily, the head-mounted device further comprises a third wireless communication module and a fourth wireless communication module, and the hand-mounted device comprises a fifth wireless communication module; at this time, the first position of the hand-worn device relative to the head-worn device may be determined based on the communication between the third wireless communication module and the fifth wireless communication module, and the communication between the fourth wireless communication module and the fifth wireless communication module.
It is understood that, in the embodiment of the present application, the third wireless communication module and the fourth wireless communication module may be different modules from the first wireless communication module and the second wireless communication module, that is, the first wireless communication module and the second wireless communication module are used to implement transmission of image event information, and the third wireless communication module and the fourth wireless communication module are used to implement positioning of the hand-worn device. Of course, in some other embodiments, the third wireless communication module and the fourth wireless communication module may be the same module as the first wireless communication module and the second wireless communication module, that is, the first wireless communication module and the second wireless communication module are used for realizing the transmission of the image event information and simultaneously realizing the positioning of the hand-worn device.
The first wireless communication module and the second wireless communication module in the embodiment of the present application may adopt a UWB communication module, a bluetooth module, a WIFI module, or the like.
The manner in which the first position of the hand-worn device relative to the head-worn device is determined is described in detail below.
In a first example, a first distance between the third wireless communication module and the fifth wireless communication module and a second distance between the fourth wireless communication module and the fifth wireless communication module may be determined using a TOF (time of flight) method, and a first position of the hand-worn device relative to the head-worn device may be determined based on the first distance, the second distance, and a preset third distance. In the three-dimensional space, in the case where two points are fixed and the lengths of the three lines are known, a plurality of third points (position points of the hand-worn device) satisfying the condition may be determined, and in this example, a region composed of the third points satisfying the condition may be used as the first position, that is, the first position is a position region determined based on the distance.
For ease of understanding, a process of determining the first distance between the third wireless communication module and the fifth wireless communication module will be described herein. In this example, the third wireless communication module may be controlled to transmit a pulse signal to the fifth wireless communication module at time Ta1 and receive a response signal returned by the fifth wireless communication module at time Ta2, and the fifth wireless communication module may receive the pulse signal transmitted by the third wireless communication module at time Tb1 and transmit the response information to the third wireless communication module at time Tb2, where the pulse signal may be transmitted to the fifth wireless communication module at time Ta1 and the response information may be transmitted to the third wireless communication module according to a formula
Calculating the air transmission time T of the pulse signal between the third wireless communication module and the fifth wireless communication module, and then calculating a first distance S between the third wireless communication module and the fifth wireless communication module according to the formula S = C × T, wherein C is the speed of light.
It should be understood that, the above description is given by taking the example that the third wireless communication module actively initiates communication to the fifth wireless communication module, the communication mode does not limit the communication mode between the third wireless communication module and the fifth wireless communication module, and in other embodiments, the fifth wireless communication module may actively initiate communication to the third wireless communication module.
The above description describes a manner of calculating the first distance, and with reference to the same method, the second distance between the fourth wireless communication module and the fifth wireless communication module can be calculated, which is not described herein again.
The third wireless communication module and the fourth wireless communication module in this example may be respectively integrated in different wireless communication apparatuses, for example, the third wireless communication module may be integrated in a first wireless communication apparatus, and the fourth wireless communication module may be integrated in a second wireless communication apparatus. At this time, the distance between the third wireless communication module and the fourth wireless communication module is equal to the distance between the first wireless communication device and the second wireless communication device. It should be noted that in this example, the third distance between the third wireless communication module and the fourth wireless communication module may be increased as much as possible, so that the accuracy of the determined first position may be increased. Thus, for example, the first wireless communication apparatus and the second wireless communication apparatus may be disposed on both sides of the headset. When the head-mounted device is smart glasses, the first wireless communication device and the second wireless communication device may be respectively disposed on left glasses and right glasses of the smart glasses.
UWB (Ultra Wide Band) has a single channel bandwidth exceeding 500MHz, has high positioning accuracy, and can be accurate to centimeter-level accuracy, so that the first wireless communication device and the second wireless communication device can adopt UWB communication devices. Of course, in other embodiments, bluetooth devices, WIFI devices, etc. may also be used.
In a second example, a first position of the hand-worn device relative to the head-worn device may be determined using a carrier phase difference method. Specifically, the wireless frame signals received by the third wireless communication module and the fourth wireless communication module may be subjected to cross-correlation processing, an azimuth angle of the hand-worn device relative to the head-worn device is determined based on a phase difference of a correlation peak time, and a region formed by location points with an azimuth angle meeting requirements is used as the first location, that is, the first location is a location region determined based on the azimuth angle.
The following describes a manner of determining the first position of the hand-worn device with respect to the head-worn device by using the carrier phase difference method.
Referring to fig. 4, assuming that point B is a position of the third wireless communication module, point C is a position of the fourth wireless communication module, and point a is a position of the fifth wireless communication module, the fifth wireless communication module may simultaneously transmit wireless frame signals to the third wireless communication module and the fourth wireless communication module, and may determine a delay of the third wireless communication module and the fourth wireless communication module receiving the wireless frame signals according to a phase difference of the wireless frame signals received by the third wireless communication module and the fourth wireless communication module, and may determine a length difference between AB and AC according to the delay, and may select a point D on a longer line segment (line segment AB in the figure) of AC and AB according to the length difference, where the point D makes the line segment AD equal to the line segment AC. Since the length of the segment BC is usually much smaller than the lengths of the segments AC and AB, the angle BDC can be made a right angle, and since the length of the segment BC is known and the length of the segment BD can be obtained, the angle ABC can be obtained on the basis of the known length. Angle ABC is the azimuth angle of the hand-worn device transmitting the wireless frame signal relative to the third wireless communication module. The angle ACB is an azimuth angle of the hand-worn device transmitting the wireless frame signal relative to the fourth wireless communication module.
In this example, to improve the accuracy of the determined azimuth, for example, the third wireless communication module and the fourth wireless communication module may be disposed on the same side of the head-mounted device. In this case, the third wireless communication module and the fourth wireless communication module may be integrated in one communication device, that is, a third antenna and a fourth antenna may be disposed in one communication device, the third antenna and the corresponding processing circuit may be used as the third wireless communication module, and the fourth antenna and the corresponding processing circuit may be used as the fourth wireless communication module.
In a third example, the first position of the hand-mounted device with respect to the head-mounted device may be determined in combination with the first example and the second example. Specifically, one first location area may be determined in a manner provided by the first example, another first location area may be determined in a manner provided by the second example, and then the precise location of the hand-worn device is determined based on the intersection of the two first location areas, so as to achieve precise location of the hand-worn device. In this example, the position of the hand-worn device needs to be located by combining the two methods, and in order to ensure that the first position region determined by the TOF method and the second position region determined by the carrier phase difference method both have higher reliability, the positions of the third wireless communication module and the fourth wireless communication module need to be reasonably set.
In another alternative embodiment, the head-mounted device may further include a sixth wireless communication module and a seventh wireless communication module on the basis of the third wireless communication module and the fourth wireless communication module, at this time, when the hand-mounted device is located, the sixth wireless communication module and the seventh wireless communication module may be controlled to communicate with the fifth wireless communication module, respectively, and the first position of the hand-mounted device relative to the head-mounted device is determined based on the communication between the third wireless communication module and the fifth wireless communication module, the communication between the fourth wireless communication module and the fifth wireless communication module, the communication between the sixth wireless communication module and the fifth wireless communication module, and the communication between the seventh wireless communication module and the fifth wireless communication module.
In this embodiment, the third wireless communication module and the fourth wireless communication module may be integrated in the first wireless communication apparatus, and the sixth wireless communication module and the seventh wireless communication module may be integrated in the second wireless communication apparatus. In the present embodiment, the third wireless communication module is configured by a third antenna and a corresponding processing circuit, the fourth wireless communication module is configured by a fourth antenna and a corresponding processing circuit, the sixth wireless communication module is configured by a sixth antenna and a corresponding processing circuit, and the seventh wireless communication module is configured by a seventh antenna and a corresponding processing circuit. The first wireless communication device and the second wireless communication device may be respectively disposed on two sides of the head-mounted device, for example, when the head-mounted device is smart glasses, the first wireless communication device and the second wireless communication device may be respectively disposed on left glasses and right glasses. Exemplarily, the first wireless communication device and the second wireless communication device are far away from the middle position, so that the distance between the first wireless communication device and the second wireless communication device is increased, and the accuracy of positioning the hand-worn device is improved.
For example, in this embodiment, the first azimuth angle of the hand-worn device relative to the first wireless communication apparatus and the second azimuth angle of the hand-worn device relative to the second wireless communication apparatus can be measured by the carrier phase difference method described above, and since the first wireless communication apparatus and the second wireless communication apparatus are fixed and known in position, the position of the hand-worn device can be determined according to the distance between the first wireless communication apparatus and the second wireless communication apparatus, the first azimuth angle and the second azimuth angle. In the embodiment, the direction of the hand-wearing type equipment relative to the head-wearing type equipment is measured by the double-antenna carrier phase difference method, so that high measurement accuracy can be achieved. And finally, centimeter-level positioning accuracy can be obtained.
In an actual application scenario, due to environmental interference, or walking of a user, movement of a hand, and the like, it may happen that although a target hand is within a gesture recognition area, a gesture recognition module cannot detect the target hand, or is in a gesture search state for a long time, which seriously affects user experience. Therefore, for example, when it is determined that the gesture recognition module does not detect the gesture motion of the target hand within the preset time length range, the gesture recognition module may be restarted. The information of the first position can also be sent to the gesture recognition module to assist the gesture recognition module in rapidly positioning and recognizing gesture actions according to the information. At the moment, the gesture recognition module is restarted, the detected information of the first position of the hand-worn device relative to the head-worn device is utilized, the gesture recognition module is assisted to rapidly recognize the gesture again, the waiting time of the user can be shortened, and the user experience satisfaction is improved.
For example, in the embodiment of the present application, the image capturing view angle of the event camera module may also be controlled by the first position of the hand-mounted device relative to the head-mounted device. For example, the event camera module can be always aligned to the target hand, so that gesture images can be better collected and gesture recognition can be carried out, and the accuracy and efficiency of gesture recognition are improved.
Illustratively, according to the first position of the hand-worn device relative to the head-worn device, the image acquisition view angle of the event camera module is reduced, part of pixel acquisition functions of the event camera module are turned off, the power consumption of the event camera module is reduced, the size of data volume input to the gesture recognition module is reduced, the operation amount of the gesture recognition module is reduced, and meanwhile the power consumption of the gesture recognition module is also reduced.
In an actual application scenario, when the hands of other users fall within the image capturing view angle range of the event camera module, interference may be generated, and the gesture actions of other users are mistaken for the gesture actions of the user wearing the hand-worn device, so for example, after the image is captured by the event camera module and the gesture actions of the target hand are detected according to the image, it may be determined whether the detected gesture is a false gesture according to the first position of the hand-worn device relative to the head-worn device.
For example, the collected image including the target hand may be analyzed to calculate a third position of the target hand relative to the head-mounted device; by the second position of the target hand calculated above relative to the head-mounted device and the third position of the target hand calculated here relative to the head-mounted device, the distance between the two positions can be determined, when the distance is greater than or equal to a preset distance threshold, the detected gesture can be determined as a false gesture, and when the distance is smaller than the preset distance threshold, the detected gesture can be determined as a true gesture. The preset distance threshold may be flexibly set by a developer, and may be set to 20 cm, 30 cm, or 40 cm, for example.
The head-mounted device in the embodiment of the application can obtain the self pose through synchronous positioning and mapping technology. Accordingly, the position of the hand-wearing device can be obtained from the first position of the hand-wearing device relative to the head-wearing device.
Based on the same inventive concept, the embodiment of the present application further provides a head-mounted device, please refer to fig. 5, which includes a processor 501, a memory 502, an event camera module 503, and a gesture recognition module 504; the memory 502 has stored therein a computer program; a computer program as described by the processor 501 to implement any of the methods described above.
It should be noted that the head mounted device in the embodiments of the present application may also include more or fewer components than shown in fig. 5, or have a different configuration than that shown in fig. 5.
The processor 501 may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware in a processor or instructions in the form of software. The Processor 501 may be a general-purpose Processor, and includes a Central Processing Unit (CPU), a Network Processor (NP), and the like; but may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components. The various methods, steps, and logic blocks disclosed in the embodiments of the present invention may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in connection with the embodiments of the present invention may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module may be located in ram, flash memory, rom, prom, or eprom, registers, etc. storage media as is well known in the art.
The memory 502 includes, but is not limited to, random Access Memory (RAM), read Only Memory (ROM), programmable Read Only Memory (PROM), erasable read only memory (EPROM), electrically erasable read only memory (EEPROM), and the like. In the embodiment of the present application, the memory 502 stores programs required for executing the channel equalization method.
The processor 501 and memory 502 are connected by a communication bus, which may be an ISA bus, PCI bus, EISA bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one double-headed arrow is shown in FIG. 5, but this does not indicate only one bus or one type of bus.
The present embodiment further provides a computer-readable storage medium, such as a floppy disk, an optical disk, a hard disk, a flash memory, a U-disk, a Secure Digital (SD) card, a multimedia data (MMC) card, etc., where one or more programs for implementing the above steps are stored in the computer-readable storage medium, and the one or more programs can be executed by one or more processors to implement the steps of the method in the above embodiments, which is not described herein again.
In the embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. The above-described apparatus embodiments are merely illustrative, and for example, the division of the units into only one type of logical function may be implemented in other ways, and for example, multiple units or components may be combined or integrated into another system, or some features may be omitted, or not implemented. In addition, the shown or discussed coupling or direct coupling or communication connection between each other may be through some communication interfaces, indirect coupling or communication connection between devices or units, and may be in an electrical, mechanical or other form.
In addition, units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional modules in the embodiments of the present application may be integrated together to form an independent part, or each module may exist alone, or two or more modules may be integrated to form an independent part.
In this document, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions.
The above description is only an example of the present application and is not intended to limit the scope of the present application, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims (10)
1. A gesture recognition method is applied to a head-mounted device, the head-mounted device comprises an event camera module and a gesture recognition module, and the method comprises the following steps:
controlling the event camera module to acquire continuous images, generating corresponding image event information when detecting that the change value of pixel values acquired twice continuously meets a preset condition for each pixel point in the images, and sending the image event information to the gesture recognition module; the image event information comprises identification information and timestamp information of the pixel points, and the image event information further comprises at least one of pixel values of the pixel points and pixel value change information of the pixel points;
and controlling the gesture recognition module to determine the gesture action of the target hand based on the image event information.
2. The gesture recognition method of claim 1, wherein prior to the controlling the event camera module to capture successive images, the method comprises:
determining a first position of a hand-worn device relative to the head-worn device;
and determining that the target hand is located within a preset position range of the head-mounted device according to the first position.
3. The gesture recognition method of claim 2, wherein the method further comprises:
and when the target hand is determined not to be in the preset position range according to the first position, controlling the event camera module to enter a low power consumption state.
4. The gesture recognition method of claim 2, wherein the sending the image event information to the gesture recognition module comprises:
determining a second position of the target hand relative to the head mounted device according to the first position;
determining a target pixel region in the image corresponding to the second position;
and sending the image event information corresponding to each pixel point in the target pixel area to the gesture recognition module.
5. The gesture recognition method of claim 4, wherein the method further comprises:
and after the target pixel area is determined, for the pixel points in the image except the target pixel area, the collection of the pixel values of the pixel points is suspended.
6. The gesture recognition method of claim 2, wherein the gesture recognition module determines the gesture action of the target hand based on the image event information, and comprises:
determining a second position of the target hand relative to the head-mounted device according to the first position;
determining a target pixel region in the image corresponding to the second position;
and determining the gesture action of the target hand based on the image event information corresponding to each pixel point in the target pixel area.
7. The gesture recognition method according to claim 4, wherein the head mounted device includes a first display device provided with a first wireless communication module and a second display device provided with a second wireless communication module, the gesture recognition module is located within the first display device, and the event camera module includes a first event camera module located in the first display device and a second event camera module located in the second display device;
the second event camera module sends the generated image event information to the first wireless communication module through the second wireless communication module, and the first wireless communication module sends the image event information to the gesture recognition module.
8. The gesture recognition method according to claim 7, wherein the second wireless communication module does not transmit image event information corresponding to pixel points in the image other than the target pixel region to the first wireless communication module.
9. The gesture recognition method according to any one of claims 2-6, wherein the head mounted device further comprises a third wireless communication module and a fourth wireless communication module, and the hand mounted device comprises a fifth wireless communication module;
the determining a first position of a hand-worn device relative to the head-worn device comprises:
determining a first position of the hand-worn device relative to the head-worn device based on communication between the third wireless communication module and the fifth wireless communication module, and communication between the fourth wireless communication module and the fifth wireless communication module.
10. A head-mounted device, comprising: the system comprises a processor, a memory, an event camera module and a gesture recognition module;
the memory has stored therein a computer program;
the processor executes the computer program to implement the method of any one of claims 1-9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210771164.9A CN115147926A (en) | 2022-06-30 | 2022-06-30 | Gesture recognition method and head-mounted equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210771164.9A CN115147926A (en) | 2022-06-30 | 2022-06-30 | Gesture recognition method and head-mounted equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115147926A true CN115147926A (en) | 2022-10-04 |
Family
ID=83410249
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210771164.9A Pending CN115147926A (en) | 2022-06-30 | 2022-06-30 | Gesture recognition method and head-mounted equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115147926A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115661941A (en) * | 2022-12-09 | 2023-01-31 | 荣耀终端有限公司 | Gesture recognition method and electronic equipment |
-
2022
- 2022-06-30 CN CN202210771164.9A patent/CN115147926A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115661941A (en) * | 2022-12-09 | 2023-01-31 | 荣耀终端有限公司 | Gesture recognition method and electronic equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20170330031A1 (en) | Fusing device and image motion for user identification, tracking and device association | |
| EP2770783B1 (en) | A wearable information system having at least one camera | |
| US20140300746A1 (en) | Image analysis method, camera apparatus, control apparatus, control method and storage medium | |
| US10600206B2 (en) | Tracking system and method thereof | |
| CN112153306B (en) | Image acquisition system, method and device, electronic equipment and wearable equipment | |
| Pundlik et al. | Collision detection for visually impaired from a body-mounted camera | |
| CN115147926A (en) | Gesture recognition method and head-mounted equipment | |
| JP2015118442A (en) | Information processor, information processing method, and program | |
| CN114995628A (en) | Method for recognizing air gesture and related equipment thereof | |
| Liu et al. | A multimodal dynamic hand gesture recognition based on radar–vision fusion | |
| KR20120026956A (en) | Method and apparatus for motion recognition | |
| CN113778255B (en) | Touch recognition method and device | |
| TWI510082B (en) | Image capturing method for image rcognition and system thereof | |
| CN112884803B (en) | Real-time intelligent monitoring target detection method and device based on DSP | |
| Yang et al. | Video analytics-based real-time intelligent crossing detection system (RICDS): Killer app for edge computing | |
| de la Hamette et al. | Fingermouse: A wearable hand tracking system | |
| CN112595728B (en) | Road problem determination method and related device | |
| An et al. | Image-based positioning system using LED Beacon based on IoT central management | |
| CN111242084B (en) | Robot control method, robot control device, robot and computer readable storage medium | |
| CN110213407B (en) | Electronic device, operation method thereof and computer storage medium | |
| US11436818B2 (en) | Interactive method and interactive system | |
| CN111857354A (en) | Unlocking method and device, electronic equipment and storage medium | |
| KR101273634B1 (en) | Tracking Method of Multiple Objects using Mobile Device in Augumented Reality Environment and System Using the same | |
| CN115145394A (en) | Control method and system, head-mounted equipment and hand-mounted equipment | |
| CN112541418A (en) | Method, apparatus, device, medium, and program product for image processing |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |