CN114563018B - Method and apparatus for calibrating head mounted display device - Google Patents
Method and apparatus for calibrating head mounted display device Download PDFInfo
- Publication number
- CN114563018B CN114563018B CN202210212811.2A CN202210212811A CN114563018B CN 114563018 B CN114563018 B CN 114563018B CN 202210212811 A CN202210212811 A CN 202210212811A CN 114563018 B CN114563018 B CN 114563018B
- Authority
- CN
- China
- Prior art keywords
- system state
- head
- mounted display
- display device
- covariance
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 64
- 230000001133 acceleration Effects 0.000 claims abstract description 99
- 230000003068 static effect Effects 0.000 claims abstract description 38
- 230000004044 response Effects 0.000 claims abstract description 20
- 230000036544 posture Effects 0.000 claims description 64
- 238000004590 computer program Methods 0.000 claims description 10
- 239000013598 vector Substances 0.000 claims description 7
- 238000001914 filtration Methods 0.000 claims description 5
- 230000008859 change Effects 0.000 claims description 4
- 230000005484 gravity Effects 0.000 claims description 4
- 238000010606 normalization Methods 0.000 claims description 3
- 230000008569 process Effects 0.000 description 11
- 238000010586 diagram Methods 0.000 description 8
- 230000006870 function Effects 0.000 description 6
- 238000004891 communication Methods 0.000 description 5
- 230000003287 optical effect Effects 0.000 description 4
- 125000004122 cyclic group Chemical group 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 230000006835 compression Effects 0.000 description 2
- 238000007906 compression Methods 0.000 description 2
- 239000000835 fiber Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 239000004984 smart glass Substances 0.000 description 2
- 230000036461 convulsion Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000009877 rendering Methods 0.000 description 1
Abstract
Methods and apparatus for calibrating a head mounted display device are disclosed. One embodiment of the method comprises the following steps: controlling an IMU sensor to acquire acceleration information and angular velocity information, and judging whether the head-mounted display equipment is in a static state or not; if the system is in a static state, executing a system state correction step: determining a system state and a system state covariance of the head-mounted display device; calculating an updated system state and an updated system state covariance based on the acceleration information and/or the angular velocity information, and the system state covariance of the head-mounted display device; obtaining a corrected system state and a corrected system state covariance by using the updated system state and the updated system state covariance; responding to the execution result of the system state correction step not meeting the correction condition, and continuing to execute the system state correction step; and in response to the execution result of the system state correction step meeting the correction condition, calibrating the head-mounted display device with the corrected system state.
Description
Technical Field
The embodiment of the application relates to the technical field of computers, in particular to a method and a device for calibrating head-mounted display equipment.
Background
In the related art, the pose of the head-mounted display device needs to be tracked by adopting data output by the IMU sensor. However, when the IMU sensor works, the gyroscope often has null shift and other phenomena, so that the data output by the IMU sensor is inaccurate, and the head-mounted display equipment cannot be accurately tracked.
Disclosure of Invention
The embodiment of the application provides a method and a device for calibrating head-mounted display equipment.
In a first aspect, embodiments of the present application provide a method for calibrating a head mounted display device, the method comprising: controlling an IMU sensor to acquire acceleration information and angular velocity information so as to judge whether a head-mounted display device worn by a user is in a static state or not; if the head-mounted display device is in a static state, the following system state correction steps are executed: determining a system state and a system state covariance of the head-mounted display device, wherein the system state comprises attitude information of an IMU sensor under an inertial system, and a bias of a gyroscope and/or a bias of an acceleration sensor; calculating an updated system state and an updated system state covariance based on the acceleration information and/or the angular velocity information, and the system state covariance of the head-mounted display device; obtaining a corrected system state and a corrected system state covariance by using the updated system state and the updated system state covariance; in response to the execution result of the system state correction step not meeting the preset correction condition, respectively determining the corrected system state and the corrected system state covariance as the system state and the system state covariance of the head-mounted display device, and continuing to execute the system state correction step; and in response to the execution result of the system state correction step meeting the correction condition, calibrating the head-mounted display device with the corrected system state.
In a second aspect, an embodiment of the present application provides an apparatus for calibrating a head mounted display device, the apparatus comprising: the static state judging unit is configured to control the IMU sensor to acquire acceleration information and angular velocity information so as to judge whether the head-mounted display device worn by the user is in a static state or not; a system state correction step execution unit configured to execute the following system state correction step if the head-mounted display device is in a stationary state: determining a system state and a system state covariance of the head-mounted display device, wherein the system state comprises attitude information of an IMU sensor under an inertial system, and a bias of a gyroscope and/or a bias of an acceleration sensor; calculating an updated system state and an updated system state covariance based on the acceleration information and/or the angular velocity information, and the system state covariance of the head-mounted display device; obtaining a corrected system state and a corrected system state covariance by using the updated system state and the updated system state covariance; a determining unit configured to determine the corrected system state and the corrected system state covariance as the system state and the system state covariance of the head-mounted display device, respectively, in response to the execution result of the system state correction step not satisfying the preset correction condition, and continue to execute the system state correction step; and a calibration unit configured to calibrate the head-mounted display device with the corrected system state in response to the execution result of the system state correction step satisfying the correction condition.
In a third aspect, an embodiment of the present application provides an electronic device, including: one or more processors; and a storage device for storing one or more programs which, when executed by the one or more processors, cause the one or more processors to implement the method as described above.
In a fourth aspect, embodiments of the present application provide a computer readable medium having a computer program stored thereon, wherein the program when executed by a processor implements a method as described above.
The method and the device for calibrating the head-mounted display device provided by the embodiment of the application can control the IMU sensor to acquire the acceleration information and the angle information to judge whether the head-mounted display device in the wearing state of the user is in the static state, and circularly execute the system state correction step under the condition that the head-mounted display device is determined to be in the static state, so that the bias of the gyroscope and/or the accelerometer in the IMU sensor can be corrected. The system state correcting step may include determining a system state and a system state covariance, then calculating an updated system state and an updated system state covariance based on the determined system state and system state covariance, and acceleration information and/or angular velocity information, finally obtaining a corrected system state and a corrected system state covariance using the updated system state and the updated system state covariance, and after determining that the execution result of the system state correcting step satisfies a correction condition, calibrating the head-mounted display device with a bias of a gyroscope and/or a bias of an acceleration sensor in the finally obtained corrected system state. The bias of the gyroscope and/or the accelerometer can be determined through the limited cyclic execution system state step, the determined bias is adopted to calibrate the head-mounted display device worn by the user, the user is not required to take down the head-mounted display device for calibration, the calibration mode is simpler, and the user experience is improved.
Drawings
Other features, objects and advantages of the present application will become more apparent upon reading of the detailed description of non-limiting embodiments, made with reference to the accompanying drawings in which:
FIG. 1 is an exemplary system architecture diagram in which an embodiment of the present application may be applied;
FIG. 2 is a flow chart of one embodiment of a method for calibrating a head mounted display device in accordance with the present application;
FIG. 3 is a flow chart of yet another embodiment of a method for calibrating a head mounted display device in accordance with the present application;
FIG. 4 is a schematic illustration of one application scenario of a method for calibrating a head mounted display device according to the present application;
FIG. 5 is a schematic structural view of one embodiment of an apparatus for calibrating a head mounted display device in accordance with the present application;
fig. 6 is a schematic diagram of a computer system suitable for use in implementing an embodiment of the application.
Detailed Description
The application is described in further detail below with reference to the drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the application and are not limiting of the application. It should be noted that, for convenience of description, only the portions related to the present application are shown in the drawings.
It should be noted that, without conflict, the embodiments of the present application and features of the embodiments may be combined with each other. The application will be described in detail below with reference to the drawings in connection with embodiments.
FIG. 1 illustrates an exemplary system architecture 100 in which embodiments of a method for calibrating a head mounted display device or an apparatus for calibrating a head mounted display device of the present application may be applied.
As shown in fig. 1, the system architecture 100 may include a terminal device 101, a head mounted display device 102. The terminal device 101 and the head mounted display device 102 may be connected by a network, which may include a variety of connections, such as wire, wireless communication links, or fiber optic cables, among others. The terminal device 101 and the head-mounted display device 102 may interact with each other via a network to transmit or receive data information or the like.
A user may interact with the head mounted display device 102 over a network using the terminal device 101 to receive or transmit data messages or the like. The terminal device 101 may have a display screen, a Central Processing Unit (CPU), a memory, an I/O interface, and the like, and may process received data. The head-mounted display device 102 may be an electronic device with an image display function, including but not limited to AR smart glasses and VR smart glasses.
The terminal device 101 may be a variety of electronic devices with a display screen including, but not limited to, smartphones, tablet computers, electronic book readers, MP3 players (Moving Picture Experts Group Audio Layer III, moving picture experts compression standard audio layer 3), MP4 (Moving Picture Experts Group Audio Layer IV, moving picture experts compression standard audio layer 4) players, laptop and desktop computers, and the like.
The terminal apparatus 101 described above can also provide various services. For example, the terminal device may process data collected by the IMU sensor in the head-mounted display device 102 to obtain a corrected system state that may be used to calibrate the head-mounted display device, so that the head-mounted display device may be calibrated.
It should be noted that, the method for calibrating a head-mounted display device provided by the embodiment of the present application is generally performed by the terminal device 101, and accordingly, the apparatus for calibrating a head-mounted display device is generally disposed in the terminal device 101.
It should be understood that the number of terminal devices 101 and head mounted display devices 102 in fig. 1 is merely illustrative. There may be any number of terminal devices 101, and head mounted display devices 102, as desired for implementation. For example, fig. 1 may include two terminal devices 101, where one terminal device 101 may perform processing such as analysis on data acquired by the acquired IMU sensor, calibrate the head-mounted display device based on the processing result, and the other terminal device 101 performs image rendering on the display data and sends the image to the head-mounted display device for display.
It should also be noted that although the scheme disclosed in the present application may be applied to the terminal device 101, this does not exclude that the scheme may also be applied to the head-mounted display device 102. In the case that the scheme disclosed in the present application is applied to the head-mounted display device 102, the head-mounted display device 102 may also process the data collected by the IMU sensor, and calibrate the head-mounted display device by using the processing result. In this case, the method for calibrating the head mounted display device may be performed by the head mounted display device 102, and correspondingly, the means for calibrating the head mounted display device may also be provided in the head mounted display device 102.
With continued reference to fig. 2, a flow 200 of one embodiment of a method for calibrating a head mounted display device in accordance with the present application is shown. The method for calibrating a head mounted display device comprises the steps of:
Step 202, controlling the IMU sensor to collect acceleration information and angular velocity information so as to judge whether the head-mounted display device worn by the user is in a static state.
The IMU sensor can be arranged in the head-mounted display device, and after a user wears the head-mounted display device, the IMU sensor can acquire the acceleration and the angular velocity of the head-mounted display device in space, so that the pose of the head-mounted display device can be determined. Because the IMU sensor has zero offset, the acquired data of the IMU sensor is inaccurate, and the pose of the head-mounted display equipment in space cannot be accurately obtained. To calibrate IMU sensors, it is often necessary for a user to keep the head mounted display device in a stationary state until use. However, when the user uses the head-mounted display device, the head-mounted display device is in a worn state, which often requires calibration of the IMU sensor after the head-mounted device is removed to be in a stationary state with a high degree of rest. However, the wearing of the user needs to be interrupted, the continuous use of the equipment by the user is affected, and the user experience is poor.
In this embodiment, after the head-mounted display device starts to be calibrated, an execution subject of the method for calibrating the head-mounted display device (for example, the terminal device shown in fig. 1) may control the IMU sensor to collect the acceleration information and the angular velocity information by a wired connection manner or a wireless connection manner. The gyroscope in the IMU sensor can acquire angular velocity information of the head-mounted display device, and the accelerometer in the IMU sensor can acquire acceleration information of the head-mounted display device. Then, the executing body may process the collected acceleration information and angular velocity information in various manners, so as to determine that the implementation of the head-mounted display device worn by the user is in a stationary state. As an example, for the acceleration and angular velocity collected in the sliding time window from T- Δt to T, the acceleration and angular velocity collected first may be taken as the reference acceleration and reference angular velocity, and if the differences between the other accelerations and the reference acceleration are both smaller than the first preset threshold value, the differences between the other angular velocities and the reference angular velocity are both smaller than the second preset threshold value, it may be determined that the head-mounted display device worn by the user is in a stationary state.
It should be noted that, in the process of executing the method for calibrating the head-mounted display device disclosed in the present embodiment, the user always wears the head-mounted display device, so the IMU sensor is in a state of being worn by the user when acquiring the acceleration information and the angular velocity information. It will be appreciated that in the present embodiment, since the head-mounted display device is in a state of being worn by the user, the stationary state in this scheme refers to a stationary state of the head-mounted display device in a state of being worn by the user, rather than a stationary state in a case where the head-mounted device is placed on, for example, a desk or the like in the prior art scheme, the stationary degree of the stationary state in this scheme significantly fluctuates relatively much.
In some optional implementations of this embodiment, the foregoing embodiment may determine whether the head-mounted display device worn by the user is in a stationary state by: controlling an IMU sensor to acquire acceleration information and angular velocity information in real time in a preset time sliding window; and judging whether the head-mounted display device worn by the user is in a static state or not based on the change of the acceleration information and the angular velocity information in the sliding window of the preset time. It will be appreciated that the degree of change in the acceleration information and the angular velocity information within the preset time sliding window may be represented by a variance or standard deviation of the acceleration value and the angular velocity value acquired within the time sliding window. Specifically, the execution subject may calculate a variance or standard deviation of the acceleration value and the angular velocity value acquired within the time sliding window, compare the calculated variance or standard deviation with a given value, determine that the head-mounted display device is in a stationary state if the calculated variance or standard deviation is smaller than the given value, and determine that the head-mounted display device is not in a stationary state if the calculated variance or standard deviation is greater than or equal to the given value.
Step 204, if the head-mounted display device is in a static state, a system state correction step is performed.
In this embodiment, if it is determined that the head-mounted display device worn by the user is in a stationary state, the system state correction steps 2042 to 2046 may be performed using the acceleration information and/or the angular velocity information acquired by the IMU.
In step 2042, a system state and a system state covariance of the head mounted display device are determined.
In this embodiment, various types of system parameters may be employed to represent the system state of the head-mounted display device. The system states may include attitude information of the IMU sensor under inertial frame and bias of the gyroscope. Or the system state may include attitude information of the IMU sensor under inertial frame and the accelerometer bias. Or the system state may include attitude information of the IMU sensor under inertial frame, bias of the gyroscope, and bias of the accelerometer. Here, the type of parameters included in the system state may be set by those skilled in the art according to actual requirements, for example, the method disclosed in this embodiment is used to overcome zero bias of the gyroscope, where the system state may include attitude information of the IMU sensor under the inertial system and bias of the gyroscope.
The execution subject may determine the system state (or, IMU system state) of the head mounted display device and the covariance of the system in various ways. In some optional implementations, if the executing body determines that the system state correction step is performed for the first time, or may also be understood as determining that the calibration of the IMU in the head-mounted display device has not been initialized by the system, the executing body may acquire the initialized system state and the initialized system state covariance, and determine the initialized system state and the initialized system state covariance as the system state and the system state covariance of the head-mounted display device.
In some alternative implementations, the system state may be initialized by: acquiring acceleration information acquired by an IMU sensor when head-mounted display equipment worn by a user is in a static state during system initialization; then, based on the gravity acceleration and the acquired acceleration information, determining initial attitude information of the IMU sensor under an inertial system; finally, initializing a system state based on the determined attitude information of the IMU sensor under the inertial frame, the preset bias of the gyroscope and/or the acceleration jerk bias. When the execution body determines that the head-mounted display device is in a static state, the acceleration value a m in the static state can be combined with the gravity g= [0, -9.8,0] T to calculate the posture GRI of the IMU sensor under the inertial system, and the initial value of the bias b g of the gyroscope is set to be [0, 0] T, so that the initial system state X= [ GRI,bg]T can be obtained. It will be appreciated that if the scheme presented in this implementation can also be used to correct zero-bias of the accelerometer, the system state can also include the bias of the accelerometer. Further, the initialized system state covariance p=p 0,P0 may be an empirically set value. The system state may also be initialized according to other means, without limitation.
If the execution subject does not execute the system state correction step for the first time, the system state and the system state covariance of the head-mounted display device may be corrected system state and corrected system state covariance obtained by executing the system state correction step last time.
Step 2044, calculating an updated system state and an updated system state covariance based on the acceleration information and/or the angular velocity information, and the system state covariance of the head-mounted display device.
In this embodiment, the executing body may process the system state and the system state covariance of the head-mounted display device determined based on step 2042 in various manners, and the angular velocity information newly acquired by the gyroscope and/or the acceleration information newly acquired by the accelerometer, so that an updated system state and an updated system state covariance may be calculated. For example, for the latest acquired angular velocity information and the system state of the head-mounted display device determined in step 2042, the updated system state may be obtained by the first-order longgnus-base tower method, and the updated system state covariance may be calculated using the updated system state and the system state covariance of the head-mounted display device determined in step 2042. It will be appreciated that the person skilled in the art may also process the latest acquired acceleration information and/or angular velocity information, as well as the system state and system state covariance of the head mounted display device, to obtain an updated system state and an updated system state covariance, without limitation.
Step 2046, using the updated system state and the updated system state covariance, obtains a corrected system state and a corrected system state covariance.
In this embodiment, the execution body may process the updated system state and the updated system state covariance obtained in step 2044 in various manners, so as to modify the updated system state and the updated system state covariance, and thus obtain a modified system state and a modified system state covariance. As an example, the above-described execution bodies may process the updated system state and the updated system state covariance in a nonlinear optimization manner to obtain an optimized system state (i.e., a corrected system state) and an optimized system state covariance (i.e., a corrected system state covariance).
In some optional implementations of this embodiment, the executing entity may correct the updated system state and the updated system state covariance by using a kalman filtering algorithm to obtain a corrected system state and a corrected system state covariance. Here, when the updated system state and the updated system state covariance are corrected, the corresponding constraint condition may be set to be that the IMU is stationary, that is, the constraint angular velocity and the linear acceleration are satisfied to be 0, and then the updated system state and the updated system state covariance are corrected by adopting a kalman filtering algorithm, so that the corrected system state and the corrected system state covariance may be obtained.
And step 206, in response to the execution result of the system state correction step not meeting the preset correction condition, determining the corrected system state and the corrected system state covariance as the system state and the system state covariance of the head-mounted display device respectively, and continuing to execute the system state correction step.
In this embodiment, after the execution body executes the system state correction step, it may be determined whether the execution result of the system state correction step satisfies a preset correction condition. In the case where it is determined that the execution result does not satisfy the correction condition, the execution subject may determine to continue to execute the system state correction step. At this time, the corrected system state and the corrected system state covariance obtained by the last execution of the system state correction step may be determined as the system state and the system state covariance of the head-mounted display device used when the system state correction step is executed again, respectively, and the execution of step 2042 may be continued. It is understood that, in the case where it is determined that the execution result satisfies the correction condition, the execution subject may execute step 208.
In some optional implementations of this embodiment, the correction condition may be that a total execution duration of the system state correction step is greater than a preset duration. For example, if the preset duration is 10 seconds, the execution body may continue to execute the system state correction step in a cyclic manner if it is determined that the total duration of the cyclic execution of the system state correction step is less than or equal to 10 seconds. In the implementation manner, the head-mounted display device is calibrated under the condition of being worn by a user, the preset time length is set as the cycle end condition for the system state correction step, the time length required for accurate control and calibration can be realized, and the user experience is improved.
In some optional implementations, the above correction condition may further be that the corrected system state covariance is smaller than a preset system state covariance; or the correction condition may be that the total number of execution times of the system state correction step is greater than a preset number of times. It will be appreciated that the person skilled in the art may set the above-described correction conditions according to actual requirements, without being particularly limited herein.
And step 208, in response to the execution result of the system state correction step meeting the correction condition, calibrating the head-mounted display device by using the corrected system state.
In this embodiment, if the execution result of the system state correction step satisfies the preset correction condition, the cycle of the system state correction step may be ended, and the corrected system state obtained by executing the system state correction step for the last time may be obtained. The bias of the gyroscope and/or the bias of the accelerometer is then obtained from the corrected system state, and the IMU sensor in the head mounted display device may be calibrated using the obtained bias of the gyroscope and/or the bias of the accelerometer.
In some optional embodiments, the execution conditions of the method for calibrating the head-mounted display device of the present embodiment may be preset, and the execution subject may execute the calibration method provided in the present embodiment after determining that the head-mounted display device meets the execution conditions. The above execution condition may be that the head-mounted display device worn by the user is in a specified posture, or the execution condition may also be that the head-mounted display device worn by the user is in a stationary state in the specified posture. In some alternative implementations, before performing step 202, the executing entity may send indication information to the user when receiving the request for calibration of the head-mounted display device, for example, the indication information may "please raise the head by 30 °" or "please lower the head by 30 °". The user wearing the head-mounted display device can control the head-mounted display device to be in a specified posture according to the indication of the indication information, so that the execution body can execute the calibration method disclosed by the application.
It will be appreciated that after the IMU sensor collects the acceleration information and the angular velocity information, it may be determined whether the head mounted display device worn by the user is in a specified pose. And executing the calibration method of the embodiment after determining that the head-mounted display device is in the specified posture, judging whether the head-mounted display device is in a static state, and executing the system state correction step by adopting the acquired acceleration information and/or angular velocity information. In some optional implementation manners, after determining that the head-mounted display device is in the specified posture, the calibration method of the embodiment may be directly executed, that is, after the IMU sensor starts to collect information, the calibration method of the embodiment may be parallel to the IMU collecting information, so that the calibration speed of the head-mounted display device is improved.
According to the method for calibrating the head-mounted display device, which is provided by the embodiment of the application, under the condition that the user wears the head-mounted display device, the IMU sensor can acquire acceleration information and angular velocity information, and under the condition that the fact that the head-mounted display device is static is determined, the system state correction step is circularly executed based on the acquired information to calibrate the head-mounted display device worn by the user.
With continued reference to fig. 3, a flow 300 of another embodiment of a method for calibrating a head mounted display device in accordance with the present application is shown. The method for calibrating a head mounted display device comprises the steps of:
step 302, controlling an IMU sensor in a head-mounted display device worn by a user to acquire acceleration information and angular velocity information in at least two postures.
In this embodiment, the IMU sensor may collect acceleration information and angular velocity information of the head-mounted display device in at least two different postures, and calibrate the head-mounted display device worn by the user using the collected acceleration information and angular velocity information in the at least two postures. Accordingly, an execution subject of a method for calibrating a head-mounted display device (e.g., a terminal device shown in fig. 1) can control an IMU sensor in the head-mounted display device worn by a user to acquire acceleration information and angular velocity information in at least two poses. At least one of the at least two gestures may include a first gesture and a second gesture different from the first gesture. It should be noted that, if the first gesture and the second gesture satisfy the following conditions, it may be determined that the second gesture is different from the first gesture: the acceleration acquired in the stationary state in the first posture and the acceleration acquired in the stationary state in the second posture are normalized to obtain corresponding vectors, and an included angle between the two vectors is larger than a threshold (for example, 20 °). Calibrating the head-mounted display device by adopting at least two gestures including the first gesture and the second gesture can improve the accuracy of the calibration of the scheme. For example, the first gesture and the second gesture may be gestures corresponding to a user wearing the head-mounted display device raising his head by 30 degrees and lowering his head by 30 degrees, respectively.
In some optional implementations, the at least two gestures may further include a third gesture that is different from both the first gesture and the second gesture. It should be noted that, if the first posture, the second posture and the third posture satisfy the following conditions, it may be determined that the first posture, the second posture and the third posture are different from each other in pairs: and the included angle between two vectors obtained after acceleration normalization processing of any two gestures in the first gesture, the second gesture and the third gesture in a static state is larger than a threshold (for example, 20 degrees). It can be understood that the calibration of the head-mounted display device by adopting at least two gestures including the first gesture, the second gesture and the third gesture can further improve the accuracy of the calibration of the scheme, and particularly when the accelerometer and the gyroscope are calibrated by using the scheme, the implementation mode can obtain better calibration results.
Step 304, based on the collected acceleration information and angular velocity information, it is determined whether the head-mounted display device is in a stationary state in at least two postures.
In this embodiment, the execution subject may determine whether the head-mounted display device is in a stationary state in each posture based on the acceleration information and the angular velocity information acquired in step 302.
In some optional implementations of this embodiment, a user wearing the head-mounted display device may control the head-mounted display device to be in the first posture, and at this time, the executing body may determine whether the head-mounted display device is in the stationary state through the acceleration information and the angular velocity information in the first posture. And performing the following system state correction step based on the acceleration and/or angular velocity information in the first posture when it is determined that the head-mounted display device is in the stationary state. Similarly, the user wearing the head-mounted display device may also control the head-mounted display device to be in the second posture, at which time the execution body may determine whether the head-mounted display device is in the stationary state by the acceleration information and the angular velocity information in the second posture. And performing the following system state correction step based on the acceleration and/or angular velocity information in the second posture when it is determined that the head-mounted display device is in the stationary state.
Step 306, if the head-mounted display device is in a static state, performing the following system state correction steps:
Step 3062, determining a system state and a system state covariance of the head-mounted display device.
Step 3064, calculating an updated system state and an updated system state covariance based on the acceleration information and/or angular velocity information, and the system state and system state covariance of the head-mounted display device.
Step 3066, using the updated system state and the updated system state covariance, obtaining a corrected system state and a corrected system state covariance.
Step 308, in response to the execution result of the system state correction step not meeting the preset correction condition, determining the corrected system state and the corrected system state covariance as the system state and the system state covariance of the head-mounted display device, respectively, and continuing to execute the system state correction step.
Step 310, in response to the execution result of the system state correction step meeting the correction condition, calibrating the head-mounted display device with the corrected system state.
In this embodiment, the steps 306 to 310 are substantially the same as the steps 204 to 208 in the above embodiment, and are not repeated here.
Compared with the embodiment shown in fig. 2, in the method provided in this embodiment, the IMU sensor may collect acceleration information and angular velocity information of the head-mounted display device in different postures, and the execution subject may calculate the corrected system state and the corrected system state covariance by using the acceleration information and/or the angular velocity information in different postures when the execution subject circularly executes the system state correction step. If the correction condition of the system state correction step circulation is the circulation execution times or the circulation duration, the solution provided by the embodiment can make the convergence effect of the system state covariance better, and the finally obtained corrected system state is more accurate, thereby improving the calibration effect of the head-mounted display device. Or if the correction condition of the system state correction step circulation is the corrected system state covariance, the solution provided by the embodiment enables the convergence effect of the system state covariance to be better, and the system state which can be used for correcting the head-mounted display device can be obtained by rapidly exiting the circulation, so that the correction speed of the head-mounted display device is improved.
With continued reference to fig. 4, fig. 4 is a schematic diagram of an application scenario of the method for calibrating a head-mounted display device according to the present embodiment. In the application scenario of fig. 4, the head-mounted display device displays indication information "raise 30 ° and keep stationary for 5s", and a user wearing the head-mounted display device may control the head-mounted display device to be in a first posture corresponding to the posture of "raise 30 °; then, the IMU sensor can collect the acceleration and the angular velocity at the moment and judge whether the head-mounted display equipment is in a static state or not; if so, a system state correction step may be performed based on the acceleration and/or angular velocity acquired in the first pose; after the user meets the preset condition of '30 DEG head lifting is kept still for 5 s', the head-mounted display device can display indication information '30 DEG head lowering is kept still for 5 s', and the user wearing the head-mounted display device can control the head-mounted display device to be in a second posture corresponding to the posture of '30 DEG head lowering' according to the indication information; similarly, the IMU sensor may collect acceleration and angular velocity at this time and determine whether the head mounted display device is in a stationary state; if so, the system state correction step can be continuously executed based on the acceleration and/or angular velocity acquired under the second gesture until a corrected system state meeting the correction condition is output; finally, the head mounted display device may be calibrated with the last outputted corrected system state.
With further reference to fig. 5, as an implementation of the method shown in the above figures, the present application provides an embodiment of an apparatus for calibrating a head-mounted display device, which corresponds to the method embodiment shown in fig. 2, and which is particularly applicable to various electronic devices.
As shown in fig. 5, an apparatus 500 for calibrating a head-mounted display device of the present embodiment includes: a stationary state judgment unit 502, a system state correction step execution unit 504, a determination unit 506, and a calibration unit 508. The static state judging unit 502 is configured to control the IMU sensor to acquire acceleration information and angular velocity information so as to judge whether the head-mounted display device worn by the user is in a static state or not; a system state correction step execution unit 504 configured to execute the following system state correction steps if the head-mounted display device is in a stationary state: determining a system state and a system state covariance of the head-mounted display device, wherein the system state comprises attitude information of an IMU sensor under an inertial system, and a bias of a gyroscope and/or a bias of an acceleration sensor; calculating an updated system state and an updated system state covariance based on the acceleration information and/or the angular velocity information, and the system state covariance of the head-mounted display device; obtaining a corrected system state and a corrected system state covariance by using the updated system state and the updated system state covariance; a determining unit 506 configured to determine the corrected system state and the corrected system state covariance as the system state and the system state covariance of the head-mounted display device, respectively, in response to the execution result of the system state correction step not satisfying the preset correction condition, and continue to execute the system state correction step; and a calibration unit 508 configured to calibrate the head-mounted display device with the corrected system state in response to the execution result of the system state correction step satisfying the correction condition.
In some optional implementations of the present embodiment, the stationary state determining unit 502 is further configured to: controlling an IMU sensor in a head-mounted display device worn by a user to acquire acceleration information and angular velocity information in at least two postures, wherein the at least two postures comprise a first posture and a second posture different from the first posture; based on the acquired acceleration information and angular velocity information, it is determined whether the head-mounted display device is in a stationary state in at least two poses.
In some optional implementations of the present embodiment, the system state correction step execution unit 504 is further configured to: under the condition that the head-mounted display equipment worn by a user is in a first posture, if the head-mounted display equipment is determined to be in a static state, acquiring acceleration information and/or angular velocity information under the first posture so as to execute a system state correction step; and under the condition that the head-mounted display equipment worn by the user is in the second posture, if the head-mounted display equipment is determined to be in the static state, acquiring acceleration information and/or angular velocity information under the second posture so as to execute a system state correction step.
In some optional implementations of the present embodiment, the stationary state determining unit 502 is further configured to: controlling an IMU sensor to acquire acceleration information and angular velocity information in real time in a preset time sliding window; and judging whether the head-mounted display device worn by the user is in a static state or not based on the change of the acceleration information and the angular velocity information in the sliding window of the preset time.
In some optional implementations of the present embodiment, in the system state correction step, determining a system state and a system state covariance of the head-mounted display device includes: in response to determining that the system state correction step is performed for the first time, the initialized system state and the initialized system state covariance are determined as the system state and the system state covariance of the head-mounted display device.
In some alternative implementations of the present embodiment, the system state is initialized by: acquiring acceleration information acquired by an IMU sensor when head-mounted display equipment worn by a user is in a static state during system initialization; determining initial attitude information of the IMU sensor under an inertial system based on the gravity acceleration and the acquired acceleration information; and initializing a system state based on the determined attitude information of the IMU sensor under the inertial frame, the preset bias of the gyroscope and/or the acceleration plus bias.
In some optional implementations of the present embodiment, in the system state correction step, using the updated system state and the updated system state covariance, obtaining a corrected system state and a corrected system state covariance includes: and correcting the updated system state and the updated system state covariance by adopting a Kalman filtering algorithm to obtain a corrected system state and a corrected system state covariance.
In some alternative implementations of the present embodiment, the correction conditions include at least one of: the corrected system state covariance is smaller than the preset system state covariance; the total execution times of the system state correction step are larger than the preset times; the total execution duration of the system state correction step is longer than the preset duration.
In some optional implementations of this embodiment, the apparatus 500 further includes: and the starting unit is configured to send indication information to a user in response to receiving a calibration request of the head-mounted display device, wherein the indication information is used for indicating that the user wearing the head-mounted display device controls the head-mounted display device to be in one of at least two postures.
The elements recited in apparatus 500 correspond to the various steps in the methods described with reference to fig. 2 and 3. Thus, the operations and features described above with respect to the method are equally applicable to the apparatus 500 and the units contained therein, and are not described in detail herein.
Referring now to FIG. 6, there is illustrated a schematic diagram of a computer system 600 suitable for use in implementing an electronic device of an embodiment of the present application. The electronic device shown in fig. 6 is only an example and should not be construed as limiting the functionality and scope of use of the embodiments of the application.
As shown in fig. 6, the computer system 600 includes a Central Processing Unit (CPU) 601, which can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 602 or a program loaded from a storage section 608 into a Random Access Memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the system 600 are also stored. The CPU 601, ROM 602, and RAM 603 are connected to each other through a bus 604. An input/output (I/O) interface 605 is also connected to bus 604.
The following components are connected to the I/O interface 605: an input portion 606 including a keyboard, mouse, etc.; an output portion 607 including a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, a speaker, and the like; a storage section 608 including a hard disk and the like; and a communication section 609 including a network interface card such as a LAN card, a modem, or the like. The communication section 609 performs communication processing via a network such as the internet. The drive 610 is also connected to the I/O interface 605 as needed. Removable media 611 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is installed as needed on drive 610 so that a computer program read therefrom is installed as needed into storage section 608.
In particular, according to embodiments of the present disclosure, the processes described above with reference to flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method shown in the flowcharts. In such an embodiment, the computer program may be downloaded and installed from a network through the communication portion 609, and/or installed from the removable medium 611. The above-described functions defined in the method of the present application are performed when the computer program is executed by a Central Processing Unit (CPU) 601. The computer readable medium of the present application may be a computer readable signal medium or a computer readable storage medium, or any combination of the two. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present application, however, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, with the computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations of the present application may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, smalltalk, C ++ and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (for example, through the Internet using an Internet service provider).
The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units involved in the embodiments of the present application may be implemented in software or in hardware. The described units may also be provided in a processor, for example, described as: a processor includes a stationary state judging unit, a system state correction step executing unit, a determining unit, and a calibrating unit. The names of these units do not constitute limitations on the unit itself in some cases, and for example, the stationary state judgment unit may also be described as "a unit that controls the IMU sensor to collect acceleration information and angular velocity information to judge whether or not the head-mounted display device worn by the user is in a stationary state".
As another aspect, the present application also provides a computer-readable medium that may be contained in the apparatus described in the above embodiments; or may be present alone without being fitted into the device. The computer readable medium carries one or more programs which, when executed by the apparatus, cause the apparatus to: controlling an IMU sensor to acquire acceleration information and angular velocity information so as to judge whether a head-mounted display device worn by a user is in a static state or not; if the head-mounted display device is in a static state, the following system state correction steps are executed: determining a system state and a system state covariance of the head-mounted display device, wherein the system state comprises attitude information of an IMU sensor under an inertial system, and a bias of a gyroscope and/or a bias of an acceleration sensor; calculating an updated system state and an updated system state covariance based on the acceleration information and/or the angular velocity information, and the system state covariance of the head-mounted display device; obtaining a corrected system state and a corrected system state covariance by using the updated system state and the updated system state covariance; in response to the execution result of the system state correction step not meeting the preset correction condition, respectively determining the corrected system state and the corrected system state covariance as the system state and the system state covariance of the head-mounted display device, and continuing to execute the system state correction step; and in response to the execution result of the system state correction step meeting the correction condition, calibrating the head-mounted display device with the corrected system state.
The above description is only illustrative of the preferred embodiments of the present application and of the principles of the technology employed. It will be appreciated by persons skilled in the art that the scope of the application referred to in the present application is not limited to the specific combinations of the technical features described above, but also covers other technical features formed by any combination of the technical features described above or their equivalents without departing from the inventive concept described above. Such as the above-mentioned features and the technical features disclosed in the present application (but not limited to) having similar functions are replaced with each other.
Claims (10)
1. A method for calibrating a head mounted display device, wherein the head mounted display device comprises an IMU sensor, the method comprising:
Controlling an IMU sensor in the head-mounted display device worn by a user to acquire acceleration information and angular velocity information under at least two postures; judging whether the head-mounted display equipment is in a wearing static state under the at least two postures or not based on the acquired acceleration information and angular velocity information; the at least two postures comprise a first posture and a second posture which is different from the first posture, the acceleration collected in the static state under the first posture and the acceleration collected in the static state under the second posture are subjected to normalization processing to obtain corresponding vectors, and the included angle between the two vectors is larger than a threshold value;
And under the state that the head-mounted display equipment is in the first posture or the second posture, if the head-mounted display equipment is in a static state, executing the following system state correction steps: determining a system state and a system state covariance of the head-mounted display device, wherein the system state comprises attitude information of the IMU sensor under an inertial frame, and a bias of a gyroscope and/or a bias of an acceleration sensor; calculating an updated system state and an updated system state covariance based on acceleration information and/or angular velocity information, and a system state covariance of the head-mounted display device; correcting the updated system state and the updated system state covariance by adopting a Kalman filtering algorithm to obtain a corrected system state and a corrected system state covariance;
in response to the execution result of the system state correction step not meeting a preset correction condition, determining the corrected system state and the corrected system state covariance as the system state and the system state covariance of the head-mounted display device respectively, and continuing to execute the system state correction step;
And responding to the execution result of the system state correction step to meet the correction condition, and calibrating the head-mounted display device by adopting the corrected system state.
2. The method of claim 1, wherein the performing the system state correction step if the head mounted display device is in a stationary state comprises:
If the head-mounted display device worn by the user is in the first posture, acquiring acceleration information and/or angular velocity information in the first posture if the head-mounted display device is determined to be in a static state, so as to execute the system state correction step; and
And under the condition that the head-mounted display equipment worn by the user is in the second posture, if the head-mounted display equipment is determined to be in a static state, acquiring acceleration information and/or angular velocity information in the second posture so as to execute the system state correction step.
3. The method of claim 1, wherein the controlling acceleration information and angular velocity information acquired by the IMU sensor to determine whether the head mounted display device worn by the user is in a stationary state comprises:
Controlling the IMU sensor to collect acceleration information and angular velocity information in real time in a preset time sliding window;
And judging whether the head-mounted display equipment worn by the user is in a static state or not based on the change of the acceleration information and the angular velocity information in the sliding window of the preset time.
4. The method of claim 1, wherein the determining the system state and system state covariance of the head-mounted display device comprises:
and in response to determining that the system state correction step is performed for the first time, determining an initialized system state and an initialized system state covariance as the system state and the system state covariance of the head-mounted display device.
5. The method of claim 4, wherein the system state is initialized by:
acquiring acceleration information acquired by an IMU sensor when head-mounted display equipment worn by a user is in a static state during system initialization;
determining initial attitude information of the IMU sensor under an inertial system based on the gravity acceleration and the acquired acceleration information;
The system state is initialized based on the determined attitude information of the IMU sensor under the inertial frame, the preset bias of the gyroscope and/or the acceleration plus bias.
6. The method of claim 1, wherein the correction condition comprises at least one of:
the corrected system state covariance is smaller than a preset system state covariance;
The total execution times of the system state correction step are larger than the preset times;
And the total execution duration of the system state correction step is longer than the preset duration.
7. The method of claim 1, wherein prior to controlling an IMU sensor in the head mounted display device worn by a user to collect acceleration information and angular velocity information in at least two poses, the method comprises:
And sending indication information to a user in response to receiving the calibration request of the head-mounted display device, wherein the indication information is used for indicating that the user wearing the head-mounted display device controls the head-mounted display device to be in one of the at least two postures.
8. An apparatus for calibrating a head mounted display device, wherein the head mounted display device includes an IMU sensor, the apparatus comprising:
A stationary state judging unit configured to control an IMU sensor in the head-mounted display device worn by a user to acquire acceleration information and angular velocity information in at least two postures; judging whether the head-mounted display equipment is in a static state under the at least two postures or not based on the acquired acceleration information and angular velocity information; the at least two postures comprise a first posture and a second posture which is different from the first posture, the acceleration collected in the static state under the first posture and the acceleration collected in the static state under the second posture are subjected to normalization processing to obtain corresponding vectors, and the included angle between the two vectors is larger than a threshold value;
A system state correction step execution unit configured to execute the following system state correction step if the head-mounted display device is in a stationary state: determining a system state and a system state covariance of the head-mounted display device, wherein the system state comprises attitude information of the IMU sensor under an inertial frame, and a bias of a gyroscope and/or a bias of an acceleration sensor; calculating an updated system state and an updated system state covariance based on acceleration information and/or angular velocity information, and a system state covariance of the head-mounted display device; correcting the updated system state and the updated system state covariance by adopting a Kalman filtering algorithm to obtain a corrected system state and a corrected system state covariance;
A determining unit configured to determine the corrected system state and the corrected system state covariance as the system state and the system state covariance of the head-mounted display device, respectively, in response to the execution result of the system state correction step not satisfying a preset correction condition, and continue to execute the system state correction step;
and a calibration unit configured to calibrate the head-mounted display device with the corrected system state in response to the execution result of the system state correction step satisfying the correction condition.
9. An electronic device, comprising:
one or more processors;
Storage means for storing one or more programs,
When executed by the one or more processors, causes the one or more processors to implement the method of any of claims 1-7.
10. A computer readable medium having stored thereon a computer program, wherein the program when executed by a processor implements the method of any of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210212811.2A CN114563018B (en) | 2022-03-04 | Method and apparatus for calibrating head mounted display device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210212811.2A CN114563018B (en) | 2022-03-04 | Method and apparatus for calibrating head mounted display device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114563018A CN114563018A (en) | 2022-05-31 |
| CN114563018B true CN114563018B (en) | 2024-06-25 |
Family
ID=
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109238277A (en) * | 2018-08-01 | 2019-01-18 | 清华大学 | The localization method and device of vision inertial data depth integration |
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109238277A (en) * | 2018-08-01 | 2019-01-18 | 清华大学 | The localization method and device of vision inertial data depth integration |
Non-Patent Citations (2)
| Title |
|---|
| 张复建 等.级联卡尔曼滤波在初始对准中的应用.《计算机测量与控制》.2018,(第9期),287-291. * |
| 级联卡尔曼滤波在初始对准中的应用;张复建 等;《计算机测量与控制》(第9期);287-291 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109858445B (en) | Method and apparatus for generating a model | |
| KR20180081356A (en) | Method for processing distortion of fingerprint image and apparatus thereof | |
| US20200348138A1 (en) | System and method for low latency, high performance pose fusion | |
| US20190347763A1 (en) | Foveated domain storage and processing | |
| US20190041978A1 (en) | User defined head gestures methods and apparatus | |
| KR20170010638A (en) | Measuring method of signal and electronic device thereof | |
| CN110059623B (en) | Method and apparatus for generating information | |
| CN114710555A (en) | Infant monitoring method and device | |
| CN112818898B (en) | Model training method and device and electronic equipment | |
| KR20180093558A (en) | Method and electronic device for providing an interface to acquire an image of a subject | |
| CN114022614A (en) | Method and system for estimating confidence of three-dimensional reconstruction target position | |
| CN114563018B (en) | Method and apparatus for calibrating head mounted display device | |
| CN106168850B (en) | Electronic device for displaying screen and control method thereof | |
| CN111345928A (en) | Head posture monitoring method and device, storage medium and electronic equipment | |
| US11543485B2 (en) | Determining location or orientation based on environment information | |
| CN113407045B (en) | Cursor control method and device, electronic equipment and storage medium | |
| CN110096134B (en) | VR handle ray jitter correction method, device, terminal and medium | |
| CN109993767B (en) | Image processing method and system | |
| CN115495038A (en) | Audio adjusting method and device, electronic equipment and readable storage medium | |
| CN114563018A (en) | Method and apparatus for calibrating head mounted display device | |
| US10831992B2 (en) | Determining a reading speed based on user behavior | |
| CN114882587A (en) | Method, apparatus, electronic device, and medium for generating countermeasure sample | |
| CN114863650B (en) | Obstacle early warning method and device, electronic equipment and readable storage medium | |
| CN113515201B (en) | Cursor position updating method and device and electronic equipment | |
| CN113672137A (en) | Cursor position updating method and device and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |