WO2024012268A1 - 虚拟操作方法、装置、电子设备及可读存储介质 - Google Patents
虚拟操作方法、装置、电子设备及可读存储介质 Download PDFInfo
- Publication number
- WO2024012268A1 WO2024012268A1 PCT/CN2023/104823 CN2023104823W WO2024012268A1 WO 2024012268 A1 WO2024012268 A1 WO 2024012268A1 CN 2023104823 W CN2023104823 W CN 2023104823W WO 2024012268 A1 WO2024012268 A1 WO 2024012268A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- plane
- point cloud
- target
- image
- physical space
- Prior art date
Links
- 238000000034 method Methods 0.000 title claims abstract description 81
- 238000004364 calculation method Methods 0.000 claims description 25
- 210000002478 hand joint Anatomy 0.000 claims description 17
- 239000013598 vector Substances 0.000 claims description 10
- 230000004927 fusion Effects 0.000 claims description 5
- 238000004590 computer program Methods 0.000 claims description 2
- 230000008569 process Effects 0.000 description 19
- 238000006243 chemical reaction Methods 0.000 description 15
- 230000002093 peripheral effect Effects 0.000 description 13
- 230000006870 function Effects 0.000 description 10
- 238000004422 calculation algorithm Methods 0.000 description 9
- 238000001514 detection method Methods 0.000 description 9
- 238000010586 diagram Methods 0.000 description 9
- 238000006073 displacement reaction Methods 0.000 description 7
- 238000012545 processing Methods 0.000 description 6
- 230000001360 synchronised effect Effects 0.000 description 5
- 238000005457 optimization Methods 0.000 description 4
- 206010049565 Muscle fatigue Diseases 0.000 description 3
- 238000004891 communication Methods 0.000 description 3
- 230000003321 amplification Effects 0.000 description 2
- 238000013528 artificial neural network Methods 0.000 description 2
- 230000006399 behavior Effects 0.000 description 2
- 238000013135 deep learning Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000002452 interceptive effect Effects 0.000 description 2
- 238000003199 nucleic acid amplification method Methods 0.000 description 2
- 238000007781 pre-processing Methods 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 241000699670 Mus sp. Species 0.000 description 1
- 230000003190 augmentative effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000007599 discharging Methods 0.000 description 1
- 230000009977 dual effect Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 230000004807 localization Effects 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 230000002441 reversible effect Effects 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0487—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
- G06F3/0488—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures
- G06F3/04883—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures for inputting data by handwriting, e.g. gesture or text
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0484—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
- G06F3/04847—Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
Definitions
- This application belongs to the field of mixed reality technology, and specifically relates to an operation execution method, device, electronic equipment and readable storage medium.
- the user can interact with the XR device through a peripheral device. Specifically, if the user needs to trigger the XR device to perform a certain operation, the user can first trigger the XR device to establish a connection with the peripheral device, and then perform mobile input on the peripheral device, so that the XR device can obtain the peripheral device and the XR device The relative position information between them, and based on the relative position information, perform the certain operation.
- XR extended reality
- the purpose of the embodiments of the present application is to provide a virtual operation method, device, electronic device and readable storage medium, which can solve the problem of poor accuracy of XR equipment in performing operations required by users.
- embodiments of the present application provide a virtual operation method, which method includes: displaying a virtual touch interface that is associated with the position of a target plane in the physical space where the electronic device is located; and identifying the user's gesture information. ; When the gesture information and the position information of the first area in the physical space in the virtual touch interface meet the preset conditions, perform the target operation.
- a virtual operating device which includes: a display module, an identification module, and an execution module.
- the display module is used to display a virtual touch interface, and the virtual touch interface is associated with the position of the target plane in the physical space where the electronic device is located.
- the recognition module is used to recognize the user's gesture information.
- the execution module is configured to execute the target operation when the gesture information recognized by the recognition module and the position information of the first area in the physical space in the virtual touch interface displayed by the display module meet preset conditions.
- inventions of the present application provide an electronic device.
- the electronic device includes a processor and a memory.
- the memory stores programs or instructions that can be run on the processor.
- the programs or instructions are processed by the processor.
- the processor is executed, the steps of the method described in the first aspect are implemented.
- embodiments of the present application provide a readable storage medium.
- Programs or instructions are stored on the readable storage medium.
- the steps of the method described in the first aspect are implemented. .
- inventions of the present application provide a chip.
- the chip includes a processor and a communication interface.
- the communication interface is coupled to the processor.
- the processor is used to run programs or instructions to implement the first aspect. steps of the method.
- embodiments of the present application provide a computer program product, the program product is stored in a storage medium, and the program product is executed by at least one processor to implement the steps of the method described in the first aspect.
- the electronic device can display a virtual touch interface associated with the position of the target plane in the physical space where the electronic device is located, and recognize the user's gesture information, so that the first area between the gesture information and the virtual touch interface When the location information of the physical space meets the preset conditions, the target operation is performed. Since electronic devices can display the same physical space as the electronic device, The virtual touch interface is associated with the position of the target plane and recognizes the user's gesture information, so that the user can directly perform separation between the gesture information and the position information of the first area in the physical space that meets the preset conditions on the virtual touch interface. Empty gestures enable the electronic device to perform the target operation without first triggering the XR device to connect to the peripheral device and then making mobile input to the peripheral device. Therefore, the time-consuming process of triggering the electronic device to perform the target operation can be reduced, so , which can improve the convenience of use of electronic devices.
- Figure 1 is one of the flow diagrams of the virtual operation method provided by the embodiment of the present application.
- Figure 2 is the second schematic flowchart of the virtual operation method provided by the embodiment of the present application.
- Figure 3 is a schematic diagram of detecting joint points of a user's hand provided by an embodiment of the present application.
- Figure 4 is the third schematic flowchart of the virtual operation method provided by the embodiment of the present application.
- Figure 5 is the fourth schematic flowchart of the virtual operation method provided by the embodiment of the present application.
- Figure 6 is a schematic diagram of the target coordinate range provided by the embodiment of the present application.
- Figure 7 is the fifth schematic flowchart of the virtual operation method provided by the embodiment of the present application.
- Figure 8 is a schematic diagram of a virtual touch interface provided by an embodiment of the present application.
- Figure 9 is a schematic flowchart No. 6 of the virtual operation method provided by the embodiment of the present application.
- Figure 10 is the seventh flowchart of the virtual operation method provided by the embodiment of the present application.
- Figure 11 is a schematic structural diagram of a virtual operating device provided by an embodiment of the present application.
- Figure 12 is a schematic structural diagram of an electronic device provided by an embodiment of the present application.
- Figure 13 is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present application.
- first, second, etc. in the description and claims of this application are used to distinguish similar objects and are not used to describe a specific order or sequence. It is to be understood that the figures so used are interchangeable under appropriate circumstances so that the embodiments of the present application can be practiced in orders other than those illustrated or described herein, and that "first,” “second,” etc. are distinguished Objects are usually of one type, and the number of objects is not limited. For example, the first object can be one or multiple.
- “and/or” in the description and claims indicates at least one of the connected objects, and the character “/" generally indicates that the related objects are in an "or” relationship.
- Figure 1 shows a flow chart of a virtual operation method provided by an embodiment of the present application.
- a virtual operation method provided by an embodiment of the present application may include the following steps 101 to 103.
- Step 101 The electronic device displays a virtual touch interface.
- the above-mentioned virtual touch interface is associated with the position of the target plane in the physical space where the electronic device is located.
- the electronic device may be an extended reality (extended reality, XR) device.
- the XR equipment can be any of the following: see-through display XR equipment, non-see-through display XR equipment.
- see-through display XR device can be understood as: an XR device with a see-through display screen, that is, the see-through display XR device can display virtual objects on the see-through display screen so that the user can While observing the surrounding physical space environment where the user is located through the see-through display screen, observe the virtual objects on the see-through display screen.
- non-see-through display XR device can be understood as: Objects, so that the user can observe the surrounding physical space environment and virtual objects where the user is located on the opaque display screen.
- the electronic device has one or more sets of camera modules that tilt downward at a target angle; wherein the target angle may be determined by the position of the camera module on the electronic device, or by Electronic equipment is determined by meeting the specific usage requirements of the user.
- the camera module can provide one or more of image data (color or monochrome) and depth data, and the field of view (field) of the two shooting modules of each set of camera modules of view (FOV) overlap rate is greater than a certain overlap rate threshold, where the certain overlap rate threshold can be 90%, so that the two shooting modules can form an FOV greater than a certain field of view threshold, where the certain overlap rate threshold can be 90%.
- the field of view angle threshold can be 120°.
- the above-mentioned camera module may be composed of an image sensor and a depth sensor (such as structured light or a TOF camera using Time of Flight (TOF)).
- a depth sensor such as structured light or a TOF camera using Time of Flight (TOF)
- the above-mentioned camera module may be composed of two image sensors placed separately.
- the sensor data of the camera module will be generated in real time at a specific frequency and transmitted to the system, and the system will perform corresponding calculations for each frame of data.
- the specific calculation process is detailed below.
- the electronic device when the electronic device is in a working state, the electronic device can display a virtual image on the target plane of the physical space where the electronic device is currently located based on the user's click input on the physical keys of the electronic device. Touch interface.
- current physical space can be understood as: the surrounding space environment in which the user is using the electronic device.
- the positional association between the virtual touch interface and the target plane in the physical space where the electronic device is located can be understood as the virtual touch interface displayed by the electronic device can find the corresponding realistic target plane in the surrounding space environment where the electronic device is located. .
- the target plane may be any one of the following: a plane on the target object in the environment where the electronic device is currently located, or a plane perpendicular to the line of sight direction of the electronic device.
- the target object may be: a scene, a user's body part (for example, the user's palm part), etc.
- the electronic device can use the depth information obtained by the depth sensor or image sensor in the camera module to perform plane detection to determine the target plane.
- the electronic device pre-processes the obtained depth information and converts it into coordinate information required in subsequent steps.
- the electronic device detects a target plane in the spatial environment that can be used as a virtual touch interface based on the coordinate information.
- the electronic device can superimpose and display a virtual touch panel on the target plane, and the virtual touch panel includes a virtual touch interface.
- Step 102 The electronic device recognizes the user's gesture information.
- the electronic device can turn on the camera of the electronic device to collect at least one frame of images, and perform image recognition on the at least one frame of images to identify the user's gesture information relative to the electronic device.
- the electronic device can use the target information to identify the user's gesture information relative to the electronic device.
- the target information may be at least one of the following: depth information and image information.
- Step 103 The electronic device performs the target operation when the gesture information and the position information of the first area in the physical space in the virtual touch interface meet the preset conditions.
- the above-mentioned target operation is: an operation corresponding to gesture information that satisfies preset conditions.
- the electronic device when the electronic device recognizes the user's gesture information relative to the electronic device, it can determine the spatial position relationship between the gesture information and the virtual touch interface to determine whether the gesture information is a target gesture. information.
- the electronic device determines to recognize the target gesture information.
- the target gesture information is the gesture information required to perform the target operation.
- the target gesture information may be at least one of the following: gesture trajectory and gesture type.
- the above-mentioned gesture types may include any of the following: single-finger gesture types, two-finger gesture types, and multi-finger gesture types.
- the single-finger gesture type may include at least one of the following: single-finger click type, single-finger double-click type, single-finger triple-click type, single-finger sliding type, single-finger long press type, etc.
- the two-finger gesture type may include at least the following One item: two-finger click type, two-finger swipe up type, two-finger slide down type, two-finger left swipe type, two-finger right swipe type, two-finger long press type, two-finger zoom-out type, two-finger zoom-in type; multi-finger
- the gesture type may include at least one of the following: a multi-finger left sliding type, a multi-finger right sliding type, a multi-finger sliding up type, and a multi-finger sliding down type.
- two-finger reduction type may specifically be: a gesture type in which the user moves two fingers toward each other to reduce the distance between the two fingers.
- two-finger amplification type may be specifically: a gesture type in which the user moves two fingers away from each other to increase the distance between the two fingers.
- the electronic device can first determine the target gesture information, and then determine the operation corresponding to the target gesture information (ie, the target operation) from multiple corresponding relationships based on the target gesture information, so that the electronic device The device can perform the target operation.
- the target gesture information ie, the target operation
- Each of the multiple correspondences is a correspondence between a target gesture information and an operation.
- the virtual operation method provided by the embodiment of the present application may also include the following steps 201 to 202.
- Step 201 The electronic device acquires a fourth image of the physical space.
- the fourth image includes an image of the user's hand.
- the above “obtaining the fourth image of the physical space” can be understood as: the electronic device continuously acquires at least one frame of images of the physical space where the electronic device is located.
- At least one frame of the fourth image includes depth map data information and image data information of the physical space where the electronic device is located.
- the multiple frames of the fourth image are used to convert the physical space around the electronic device and the image information of the user's hand into point cloud information in the first coordinate system.
- the first coordinate system is the coordinate system of the space corresponding to the environment where the electronic device is currently located.
- the coordinate system of the above-mentioned environment corresponding space may specifically be a world coordinate system.
- the electronic device converts the image information of at least one frame of the fourth image into point cloud information in the world coordinate system
- Step 202 The electronic device performs user gesture recognition on the fourth image to obtain at least one hand joint point information.
- the electronic device can detect the joint points of the user's hand based on at least one fourth frame of the image.
- the detection of hand joint points you can refer to the typical algorithm based on deep learning.
- the hand image data and hand depth map data included in the fourth image are input to the neural network, and the number of joint points of the hand is predicted in the second
- the coordinates in the coordinate system are converted into coordinates in the first coordinate system.
- the first coordinate system may be a world coordinate system; the second coordinate system may be a camera coordinate system.
- the electronic device can select M target joint points from the hand joint points.
- the M target joint points may be fingertips of the user's hand.
- the electronic device can obtain M first coordinate information of the M target joint points in the second coordinate system.
- M first coordinate information corresponds to M target joint points one-to-one.
- the electronic device can convert M pieces of first coordinate information into M pieces of second coordinate information in the first coordinate system.
- the M pieces of second coordinate information and the M pieces of first coordinate information correspond one to one.
- each hand contains 21 joint points, and each joint point has an index, such as the numbers 0, 1, 2, 3, 4...20 in Figure 3.
- the electronic device uses a deep learning algorithm to input hand image data and hand depth map data into the neural network.
- the network can predict the coordinates of the hand joint points in the camera coordinate system.
- the coordinates of the hand joint points in the camera coordinate system can be converted into coordinates in the world coordinate system.
- the specific conversion process is as follows. The relevant descriptions in the embodiments will not be repeated here.
- the electronic device can determine the fingertip coordinates from the coordinates of the joint points of the user's hands. For example, the coordinates of the five hand joint points numbered 16, 17, 18, 19, and 20 in the camera coordinate system are determined, and the coordinates are
- the specific implementation method for converting from the camera coordinate system to the world coordinate system is the same as mentioned above.
- the electronic device determines M target spatial distances based on M pieces of second coordinate information and target interface position information respectively.
- the above-mentioned target interface position information is: the position information of the virtual touch interface in the first coordinate system.
- M target space distances and M second coordinate information correspond one to one.
- the M target spatial distances may specifically be the spatial distances between the fingertips of the user's hands and the virtual touch interface in the first coordinate system.
- the electronic device determines that the target gesture information is recognized.
- the first preset spatial distance may be a spatial distance threshold preset by the electronic device.
- the electronic device calculates the spatial distance between the fingertip point coordinates and the virtual touch interface.
- the spatial distance between the fingertip point coordinates and the virtual touch interface is within the set
- the target gesture information is generated.
- one or more target gesture information can be determined according to M target spatial distances, and each target gesture information can be represented by the relative coordinates of the target gesture information in the virtual touch interface.
- the touch behavior of the M target gesture information on the virtual touch interface includes touch time, touch frequency and touch displacement. If the touch time of the touch point is short, it can be judged as a click; if the touch time is long and the touch displacement does not occur, it can be judged as a long press; if the touch time is long and the touch displacement occurs, it can be judged as a slide, and the touch frequency can be Used to determine the number of clicks; a longer touch time and a change in the distance between two touch points can be determined as zooming in or zooming out.
- one of the M touch points when one of the M touch points has a short touch time within one second on the virtual touch interface, it can be determined as a single-finger click in the gesture parameter; when the M touch points When two of the touch points have a shorter touch time on the virtual touch interface within one second, it can be determined as a two-finger click in the gesture parameter; when one of the M touch points A touch point on the virtual touch interface that has a long touch time and no touch displacement can be determined as a single-finger long press in the gesture parameters; when one of the M touch points is touched A touch that takes a long time and causes touch displacement on the virtual touch interface can be determined as a single-finger sliding in the gesture parameters, and the sliding direction can be determined based on the displacement direction; when M touch points If the two touch points on the virtual touch interface are touched for a long time, touch displacement occurs, and the distance between the two touch points becomes larger, it can be determined that the two-finger amplification in the gesture parameters is used.
- the electronic device can obtain the touch behavior of the user's fingers on the virtual touch interface, and then map the corresponding interactive control to the interaction logic of the physical touch panel, so that the user can obtain a touch experience similar to that of a notebook when using the electronic device. interactive experience of the control panel, and improve the accuracy of electronic devices in performing operations required by users.
- the electronic device can display a virtual touch interface associated with the position of the target plane in the physical space where the electronic device is located, and recognize the user's gesture information, so that between the gesture information and the virtual touch interface
- the first area performs the target operation when the location information of the physical space meets the preset conditions. Since the electronic device can display a virtual touch interface associated with the position of the target plane in the physical space where the electronic device is located, the user can directly perform air gestures relative to the electronic device on the virtual touch interface without having to perform relative gestures in the air.
- the air gesture of the electronic device can therefore avoid the situation where the user's hand muscle fatigue causes the user's air gesture to be inaccurate relative to the electronic device.
- the electronic device can accurately recognize the user's gesture information and accurately perform the target operation corresponding to the gesture information. In this way, the accuracy of the electronic device in performing the user's required operations can be improved.
- the electronic device Before the electronic device obtains the target plane of the current environment and displays the virtual touch interface on the target plane, The electronic device needs to perform corresponding preprocessing on the image data of the current environment collected by the sensor to determine the target plane. Next, how the electronic device preprocesses the image data collected by the sensor is explained.
- the virtual operation method provided by the embodiment of the present application may also include the following steps 301 to 304.
- Step 301 The electronic device acquires the first image.
- the first image is an image of the physical space where the electronic device is located.
- the image of the physical space where the electronic device is located includes a color or monochrome image and a depth image.
- the image of the physical space where the electronic device is located may be provided by an image sensor or a depth sensor.
- the depth sensor may be a structured light or TOF sensor.
- image data corresponding to the color or monochrome image obtained by the electronic device only needs to be processed by conventional ISP.
- Step 302 The electronic device calculates the first point cloud of the physical space in the first coordinate system based on the first image.
- the first point cloud may be point cloud data of the physical space obtained by the electronic device according to the first image in the first coordinate system.
- the above-mentioned first image is a depth image
- the depth image includes N pieces of first depth information and N pieces of third coordinate information corresponding to N first pixels.
- the above step 302 can be implemented through the following steps 302a to 302b.
- Step 302a The electronic device calculates the second point cloud of the physical space in the second coordinate system based on the N pieces of first depth information and the N pieces of third coordinate information.
- each first pixel of the N first pixels respectively corresponds to a piece of first depth information and a piece of third coordinate information.
- the N first depth information of the N first pixels may be obtained by a depth sensor, or may be obtained by dual cameras in a stereo matching manner.
- the electronic device can optionally use filter optimization processing for the N first depth information, thereby achieving an optimization effect of reducing noise and maintaining clear boundaries.
- the N pieces of third coordinate information are N pieces of pixel coordinate information of the depth image.
- the second point cloud may specifically be the point cloud information of the N pixel coordinate information of the depth image in the second coordinate system.
- the electronic device may convert the pixel coordinate information of the depth image into point cloud information in the second coordinate system.
- the electronic device can obtain each pixel coordinate (m, n) of the depth map.
- camera_factor, camera_fx, camera_fy, camera_cx, camera_cy are camera internal parameters.
- the electronic device calculates the point cloud information (X, Y, Z) corresponding to the depth image pixel point (m, n) in the second coordinate system.
- the electronic device performs conversion processing on each pixel in the depth map to obtain point cloud information of the first image in the second coordinate system at the current moment.
- Step 302b The electronic device converts the second point cloud into the first point cloud.
- converting the second point cloud into the first point cloud can be understood as: the electronic device converts the point cloud information of N third coordinate information in the second coordinate system into N third coordinate information. Point cloud information in the first coordinate system.
- the point cloud information of the N third coordinate information in the second coordinate system can be converted into the point cloud information of the N third coordinate information in the first coordinate system through the relevant coordinate system conversion calculation formula.
- the conversion calculation method from the second coordinate system to the first coordinate system is Twc.
- Pc X, Y, Z
- Pw Twc*Pc
- the conversion calculation method Twc may refer to the existing conversion calculation method, which will not be described in detail here in the embodiment of the present application.
- the electronic device performs coordinate system conversion calculation on each point cloud point in the second point cloud to obtain point cloud information of the second point cloud in the first coordinate system.
- the electronic device can calculate the second point cloud of the physical space in the second coordinate system based on the N pieces of first depth information and N pieces of third coordinate information, and convert the second point cloud through the coordinate system conversion calculation formula.
- the first point cloud is obtained through conversion.
- the electronic device performs corresponding preprocessing on the image data information collected by the sensor in the current physical space, and converts it into image data information in the data format required in subsequent steps.
- the first image when the sensor of the electronic device does not include a depth sensor, the first image may also be acquired only by the image sensor.
- the electronic device may calculate the first point cloud based on at least one frame of the second image.
- the second image is an image of the physical space where the electronic device is located.
- the image of the physical space where the electronic device is located may be a depth image including a sparse point cloud.
- the above-mentioned sparse point cloud can be generated by using the second image through a simultaneous localization and mapping (SLAM) algorithm.
- SLAM simultaneous localization and mapping
- Step 303 The electronic device determines the target point cloud located within the target coordinate range from the first point cloud.
- the above target coordinate range is determined based on the coordinate information of the electronic device in the first coordinate system.
- the target coordinate range may be a point cloud coordinate range surrounded by a cube coordinate set.
- the length W, width D, and height H2 of the cube can be determined according to the conditions for the user to use the virtual touch panel.
- the length W of the touchpad is required to be at least 100 centimeters (cm)
- the width D is required to be at least 30cm
- the field of view is required to be within 30cm to 100cm
- the target point cloud is the first point cloud within the target coordinate range.
- Step 304 The electronic device determines the target plane based on the target point cloud.
- the first coordinate system is a coordinate system corresponding to the physical space.
- the coordinate system corresponding to the physical space may specifically be a world coordinate system.
- the electronic device can obtain at least one plane through a plane detection method for the target point cloud.
- the plane detection method may be a region growing method or a plane fitting method.
- the electronic device may determine the target plane from the at least one plane that satisfies the first preset condition.
- step 304 can be specifically implemented through the following steps 304a and 304c.
- Step 304a The electronic device determines at least one first plane based on the target point cloud.
- each first plane includes point cloud information constituting the plane.
- the at least one first plane may be a plane with different or the same normal vectors.
- the electronic device can use a region growing method to detect the plane on the target point cloud, that is, randomly select multiple point cloud points as plane seeds in the target point cloud area. , loop to determine whether the points in the seed neighborhood belong to the plane, and if so, add the plane and expand the neighborhood search.
- the region growing algorithm is a public algorithm and will not be described in further detail here.
- the electronic device can use plane fitting to perform plane detection on the target point cloud. That is, the electronic device performs clustering processing on the target point cloud and uses typical The horizontal plane fitting algorithm performs plane fitting and fuses the plane with the existing reference plane.
- the plane fitting algorithm is a public algorithm and will not be described in further detail here.
- the following will take the electronic device's plane detection through dense depth information as an example to illustrate how the electronic device obtains the target plane and generates a virtual touch panel.
- Step 304b The electronic device determines the plane that satisfies the first preset condition among the at least one first plane as the second plane.
- the first preset condition includes at least one of the following: at least one first plane in which the angle between the normal vector and the horizontal plane of the physical space is less than a preset value; at least one first plane in which the angle between the normal vector and the physical space horizontal plane is perpendicular to the target direction. plane; at least one first plane in which the plane size is within a preset range.
- the preset angle value can be determined according to the user's specific use.
- the electronic device may remove the first plane whose angle value between the normal vector and the normal vector of the horizontal plane is greater than the preset angle value according to the preset angle value.
- the point cloud information of each first plane can determine the normal vector, circumscribed rectangle, and plane size of the plane.
- the electronic device may determine, based on the circumscribed rectangle of the target point cloud, a first plane in which at least one side of the circumscribed rectangle is perpendicular to the target direction of the electronic device.
- the target direction is the direction of the user's line of sight when using the electronic device.
- the electronic device may determine a first plane whose plane size is within a preset range according to the plane size of the first plane.
- the preset range may be determined by the target coordinate range.
- Step 304c The electronic device determines the corresponding position of the point cloud of the second plane in the physical space as the target plane.
- the electronic device may determine at least one first plane that meets the first preset condition as the second plane.
- the electronic device may determine the location area in the physical space corresponding to the point cloud constituting the second plane as the target plane.
- the dotted rectangular frame is the second plane, and the point cloud making up the second plane corresponds to the rectangular plane in the physical space.
- An area with a preset length M and width N can be generated as a virtual touch.
- the following will take the electronic device's plane detection through sparse depth information as an example to illustrate how the electronic device obtains the target plane and generates a virtual touch panel.
- the target coordinate range includes at least one first coordinate range.
- Step 401 The electronic device determines at least one point cloud subset in the target point cloud based on at least one first coordinate range.
- each point cloud subset corresponds to each first coordinate range, and the point cloud subset includes at least one point cloud.
- the at least one point cloud subset corresponds to the at least one first coordinate range, and each point cloud subset includes at least one point cloud in the target point cloud.
- the height H2 of the rectangular coordinate set corresponding to the above target coordinate range is equally divided into at least one interval, and the target point cloud is assigned to at least one interval according to the value of the Y axis in the first coordinate system. Within, at least one point cloud subset is obtained.
- Step 402 The electronic device determines at least one first plane according to the target point cloud subset in at least one point cloud subset.
- the target point cloud point subset is a point cloud subset in which the number of point clouds in at least one point cloud subset is greater than or equal to a preset number.
- a typical horizontal plane fitting method can be used to calculate a horizontal plane composed of at least one point cloud subset to obtain at least one point cloud of the current frame.
- the at least one first plane corresponds to the at least one point cloud subset in a one-to-one manner.
- Step 403 The electronic device fuses the third plane and the fourth plane in at least one first plane to obtain the target plane.
- the fourth plane is a plane that has been stored by the electronic device, and the spatial positions of the third plane and the fourth plane in the physical space satisfy the second preset condition.
- the third plane is a plane obtained by clustering the target point cloud in the point cloud point subset, and using a horizontal plane fitting method to perform plane detection.
- the fourth plane is a plane determined and stored based on the previous frame of at least one second image.
- the above-mentioned second preset condition includes at least one of the following: overlap with the fourth plane in the first direction, and the spatial distance from the fourth plane in the second direction is less than or equal to the first 2. Preset spatial distance; the first direction is perpendicular to the second direction.
- the electronic device plane -fuses the third plane with the fourth plane stored in the electronic device system.
- the electronic device can also process the discrete point cloud in the current frame.
- the electronic device can also process the discrete point cloud in the current frame.
- the electronic device can also process the discrete point cloud in the current frame. Among them, for each discrete point cloud in at least one point cloud subset, if the distance between the discrete point cloud and the fourth plane of the system in the horizontal direction is less than a certain set threshold, and the height difference in the vertical direction is within a certain threshold, then Fusion of this discrete point cloud and the fourth plane.
- the electronic device can obtain at least one second image without relying on a specific depth module, and obtain the target plane based on the at least one second image, thereby generating a virtual touch interface, avoiding the need for the user to raise his hand relative to the Perform air gestures of the gesture type corresponding to a certain operation on the XR device. Therefore, the time-consuming process of triggering the electronic device to perform the target operation can be reduced, and the user can avoid hand muscle fatigue that affects the gesture operation. Accuracy, in this way, can improve the ease of use and accuracy of XR equipment.
- the electronic device can use one hand as a virtual touch pad and then operate it with the other hand.
- An example will be given below to illustrate this situation.
- the virtual operation method provided by the embodiment of the present application can also be implemented through the following steps 501 to 504.
- Step 501 The electronic device acquires a third image of the physical space.
- the third image includes an image of the user's hand.
- the above-mentioned third image is an image of the physical space where the electronic device is located, and each frame of the third image includes the user's hand.
- the third image includes depth map data information and image data information of the physical space of the electronic device, and image data information or depth map data information of the joint points of the user's hand.
- Step 502 The electronic device calculates a third point cloud of the physical space in the first coordinate system based on the third image.
- the third point cloud includes user hand position information.
- At least one frame of the third image acquired by the electronic device is used to convert the image information of the user's hand into a third point cloud in the first coordinate system.
- the above-mentioned hand posture information is information about the joint points of the user's hand and the position and state of the palm.
- the third point cloud is the point cloud coordinates of the joint points of the user's hand and the palm in the first coordinate system.
- the electronic device detects the hand joint points of each frame of the third image data, and can obtain the five fingers and palm posture of one hand, as shown in Figure 3, from which 11 joint points are selected: five fingers ( 16, 17, 18, 19, 20) and palms (0, 1, 4, 7, 10, 14).
- the electronic device converts the coordinates of the selected 11 joint points in the camera coordinate system into coordinates in the world coordinate system. For the specific conversion process, refer to the relevant descriptions in the above embodiments, which will not be described again here.
- Step 503 The electronic device determines the third plane in the third point cloud based on the user's hand position information.
- the electronic device can obtain the circumscribed rectangle of the third point cloud through the fitting plane method.
- the third plane corresponding to the circumscribed rectangle includes the normal direction of the plane and the coordinate data of the vertices constituting the plane.
- the electronic device can obtain the area of the circumscribed rectangle based on the vertex data of the circumscribed rectangle.
- the area of the circumscribed rectangle that is greater than or equal to the preset area may be determined as the target circumscribed rectangle.
- the electronic device may determine the target circumscribed rectangle as the third plane.
- Step 504 The electronic device determines the screen corresponding to the third plane in the physical space as the target plane.
- the first coordinate system is the coordinate system corresponding to the physical space.
- the electronic device may determine the location area in the physical space corresponding to the point cloud constituting the third plane as the target plane.
- the electronic device can recognize the user's hands, determine one hand as the virtual touch interface, and recognize the gesture information of the other hand relative to the electronic device, without the need to perform gestures relative to the electronic device in the air. Therefore, , can reduce the time-consuming process in triggering the electronic device to perform the target operation, and avoid the user's hand muscle fatigue, which affects the accuracy of the gesture operation. In this way, the convenience and accuracy of using the XR device can be improved.
- the execution subject may be a device.
- the device execution method is taken as an example to describe the device provided by the embodiments of this application.
- FIG. 11 shows a possible structural diagram of a virtual operating device involved in the above embodiment.
- the virtual operating device 60 provided by the embodiment of the present application may include: a display module 61 , an identification module 62 and an execution module 63 .
- the display module 61 is used to display a virtual touch interface, and the virtual touch interface is associated with the position of the target plane in the physical space where the electronic device is located.
- the recognition module 62 is used to recognize the user's gesture information.
- the execution module 63 is configured to execute the target operation when the gesture information recognized by the recognition module and the position information of the first area in the physical space in the virtual touch interface displayed by the display module meet preset conditions.
- the above-mentioned virtual operating device 60 further includes: an acquisition module, a calculation module, and a determination module.
- the acquisition module is used to acquire a first image, where the first image is an image of the physical space where the electronic device is located.
- the calculation module is used to calculate the first point cloud of the physical space in the first coordinate system according to the first image obtained by the acquisition module.
- the determination module is used to determine the target point cloud located within the target coordinate range from the first point cloud calculated by the calculation module.
- the target coordinate range is determined based on the coordinate information of the electronic device in the first coordinate system, and is based on The target point cloud determines the target plane.
- the first coordinate system is the coordinate system corresponding to the physical space.
- the first image is a depth image
- the depth image includes N first depth information and N third coordinate information corresponding to N first pixels
- the above calculation module includes: a conversion submodule.
- the calculation module is used to calculate the second point cloud of the physical space in the second coordinate system based on the N pieces of first depth information and the N pieces of third coordinate information.
- the conversion submodule is used to convert the second point cloud obtained by the calculation module into the first point cloud.
- the second coordinate system is the image sensor of the virtual operating device the corresponding coordinate system.
- the above-mentioned determination module is specifically configured to determine at least one first plane based on the target point cloud, and determine the plane that satisfies the first preset condition among the at least one first plane as the second plane, and determining the corresponding position of the point cloud of the second plane in the physical space as the target plane.
- the first preset condition includes at least one of the following: at least one plane in the first plane whose normal vector angle value with the horizontal plane of the physical space is less than a preset value, at least one plane in the first plane that is perpendicular to the target direction, at least A plane whose plane size is within a preset range in the first plane.
- the target coordinate range includes at least one first coordinate range
- the above-mentioned determination module includes: a fusion sub-module.
- the determination module is specifically configured to determine at least one point cloud subset in the target point cloud according to at least one first coordinate range.
- Each point cloud subset corresponds to each first coordinate range
- the point cloud subset includes at least one point cloud, and determine at least one first plane based on a target point cloud subset in the at least one point cloud subset.
- the target point cloud point subset is a point cloud number in the at least one point cloud subset that is greater than or equal to a preset number.
- the fusion submodule is used to fuse the third plane and the fourth plane in at least one first plane to obtain the target plane.
- the fourth plane is a plane that has been stored by the electronic device.
- the third plane and the fourth plane are in the physical space. The spatial location satisfies the preset conditions.
- the above-mentioned operation execution device further includes: an acquisition module, a calculation module, and a determination module.
- An acquisition module is configured to acquire a third image of the physical space, where the third image includes an image of the user's hand.
- the calculation module is configured to calculate a third point cloud of the physical space in the first coordinate system based on the third image obtained by the acquisition module.
- the third point cloud includes user hand position and posture information.
- the determination module is configured to determine the third plane in the third point cloud calculated by the calculation module based on the user's hand posture information, and determine the screen corresponding to the third plane determined by the determination module in the physical space as the target plane.
- the first coordinate system is the coordinate system corresponding to the physical space.
- the above-mentioned identification module 62 includes: an acquisition sub-module.
- the acquisition sub-module is used to acquire a fourth image of the physical space, where the fourth image includes an image of the user's hand.
- the recognition module is used to perform user gesture recognition on the fourth image to obtain at least one hand joint point information.
- At least one hand joint point information includes at least one first coordinate of at least one hand joint in the second coordinate system.
- the above execution module 63 includes: determination sub-module.
- the determination sub-module is used to determine the first area of the virtual touch interface when the spatial distance between the at least one first coordinate and the virtual touch interface in the physical space meets a preset condition.
- the execution module is specifically used to execute the target operation according to the operation information corresponding to the first area.
- the virtual operating device provided by the embodiment of the present application can display a virtual touch interface associated with the position of the target plane in the physical space where the virtual operating device is located, and recognize the user's gesture information, so that the user can directly touch the virtual operating device.
- On the control interface perform an air gesture whose gesture information and the position information of the first area in the physical space meet the preset conditions, so that the virtual operating device can perform the target operation without first triggering the XR device to connect to the peripheral device and then connecting the peripheral device.
- the device performs mobile input. Therefore, the time-consuming process of triggering the virtual operating device to perform the target operation can be reduced. In this way, the usability of the virtual operating device can be improved.
- the virtual operating device in the embodiment of the present application may be an electronic device or a component in the electronic device, such as an integrated circuit or a chip.
- the electronic device may be a terminal or other devices other than the terminal.
- the electronic device may be a mobile phone, a tablet computer, a notebook computer, a handheld computer, a vehicle-mounted electronic device, a mobile internet device (MID), or augmented reality (AR)/virtual reality (VR).
- MID mobile internet device
- AR augmented reality
- VR virtual reality
- the virtual operating device in the embodiment of the present application may be a device with an operating system.
- the operating system can be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiments of this application.
- the virtual operating device provided by the embodiment of the present application can implement each process implemented by the method embodiments of Figures 1 to 10. In order to avoid To avoid repetition, I won’t go into details here.
- this embodiment of the present application also provides an electronic device 80, which includes a processor 81 and a memory 82.
- the memory 82 stores information that can be run on the processor 81.
- the program or instruction when executed by the processor 81, implements each process step of the above-mentioned shooting method embodiment, and can achieve the same technical effect. To avoid duplication, it will not be described again here.
- the electronic devices in the embodiments of the present application include the above-mentioned mobile electronic devices and non-mobile electronic devices.
- FIG. 13 is a schematic diagram of the hardware structure of an electronic device implementing an embodiment of the present application.
- the electronic device 100 includes but is not limited to: radio frequency unit 1001, network module 1002, audio output unit 1003, input unit 1004, sensor 1005, display unit 1006, user input unit 1007, interface unit 1008, memory 1009, processor 110, etc. part.
- the electronic device 100 may also include a power supply (such as a battery) that supplies power to various components.
- the power supply may be logically connected to the processor 110 through a power management system, thereby managing charging, discharging, and function through the power management system. Consumption management and other functions.
- the structure of the electronic device shown in Figure 13 does not constitute a limitation of the electronic device.
- the electronic device may include more or less components than shown in the figure, or combine certain components, or arrange different components, which will not be described again here. .
- the processor 110 is specifically configured to display a virtual touch interface on the target plane of the environment where the electronic device is currently located; and to recognize the user's air gesture with respect to the electronic device; and to recognize that the user is in the virtual touch interface.
- a target operation is performed in response to the first air gesture; the target operation is: an operation corresponding to the gesture parameter of the first air gesture.
- the electronic device provided by the embodiment of the present application can display a virtual touch interface associated with the position of the target plane in the physical space where the electronic device is located, and recognize the user's gesture information, so that the user can directly touch the virtual touch interface , perform an air gesture in which the gesture information and the position information of the first area in the physical space meet the preset conditions, so that the electronic device can perform the target operation without first triggering the connection between the XR device and the peripheral device and then performing mobile input on the peripheral device. , Therefore, the time-consuming process of triggering the electronic device to perform the target operation can be reduced, and thus the convenience of use of the electronic device can be improved.
- the processor 110 is specifically configured to acquire a first image, which is an image of the physical space where the electronic device is located; and calculate the physical space in the first coordinate system based on the first image.
- the first point cloud from the first point cloud, determine the target point cloud located within the target coordinate range.
- the target coordinate range is determined based on the coordinate information of the electronic device in the first coordinate system; based on the target point cloud, determine Target plane; where the first coordinate system is the coordinate system corresponding to the physical space.
- the processor 110 is specifically configured to calculate the second point cloud of the physical space in the second coordinate system based on N pieces of first depth information and N pieces of third coordinate information; convert the second point The cloud is converted into a first point cloud; where the second coordinate system is the coordinate system corresponding to the image sensor of the electronic device.
- the processor 110 is specifically configured to determine at least one first plane based on the target point cloud; determine the plane that satisfies the first preset condition among the at least one first plane as the second plane; Determine the position corresponding to the point cloud of the second plane in the physical space as the target plane; wherein the first preset condition includes at least one of the following: the angle between the normal vector of the at least one first plane and the horizontal plane of the physical space is less than the preset a plane of values; at least one first plane perpendicular to the target direction; at least one first plane with a plane size within a preset range.
- the processor 110 is specifically configured to determine at least one point cloud subset in the target point cloud according to at least one first coordinate range, and each point cloud subset is related to each first coordinate range.
- the point cloud subset includes at least one point cloud; at least one first plane is determined according to the target point cloud subset in the at least one point cloud subset, and the target point cloud point subset is the point in the at least one point cloud subset A subset of point clouds whose number is greater than or equal to the preset number; fuse the third plane and the fourth plane in at least one first plane to obtain the target plane, the fourth plane is a plane that has been stored by the electronic device, and the third plane
- the spatial positions of the three planes and the fourth plane in the physical space satisfy preset conditions.
- the processor 110 is specifically configured to obtain a third image of the physical space, where the third image includes an image of the user's hand; and calculate the physical space in the first coordinate system based on the third image.
- the third point cloud includes the user's hand posture information; determines the third plane in the third point cloud based on the user's hand posture information; places the third plane in the physical space
- the corresponding screen is determined as the target plane; where the first coordinate system is the coordinate system corresponding to the physical space.
- the processor 110 is specifically configured to obtain a fourth image of the physical space, where the fourth image includes an image of the user's hand; perform user gesture recognition on the fourth image to obtain at least one hand joint Click for information.
- the processor 110 is specifically configured to determine the first coordinate of the virtual touch interface when the spatial distance between at least one first coordinate and the virtual touch interface in the physical space meets a preset condition. area; perform the target operation based on the operation information corresponding to the first area.
- the input unit 1004 may include a graphics processing unit (GPU) 1041 and a microphone 1042.
- the graphics processor 1041 is responsible for the image capture device (GPU) in the video capture mode or the image capture mode. Process the image data of still pictures or videos obtained by cameras (such as cameras).
- the display unit 1006 may include a display panel 1061, which may be configured in the form of a liquid crystal display, an organic light emitting diode, or the like.
- the user input unit 1007 includes a touch panel 1071 and at least one of other input devices 1072 . Touch panel 1071 is also called a touch screen.
- the touch panel 1071 may include two parts: a touch detection device and a touch controller.
- Other input devices 1072 may include, but are not limited to, physical keyboards, function keys (such as volume control keys, switch keys, etc.), trackballs, mice, and joysticks, which will not be described again here.
- Memory 1009 may be used to store software programs as well as various data.
- the memory 1009 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, Image playback function, etc.) etc.
- memory 1009 may include volatile memory or nonvolatile memory, or memory 1009 may include both volatile and nonvolatile memory.
- non-volatile memory can be read-only memory (ROM), programmable ROM (PROM), erasable programmable read-only memory (erasable PROM, EPROM), electrically removable memory.
- Volatile memory can be random access memory (RAM), static random access memory (static RAM, SRAM), dynamic random access memory (dynamic RAM, DRAM), synchronous dynamic random access memory (synchronous DRAM, SDRAM), double data rate synchronous dynamic random access memory (double data rate SDRAM, DDRSDRAM), enhanced synchronous dynamic random access memory (enhanced SDRAM, ESDRAM), synchronous link dynamic random access memory (synch link DRAM) , SLDRAM) and direct memory bus random access memory (direct rambus RAM, DRRAM).
- RAM random access memory
- static random access memory static random access memory
- DRAM dynamic random access memory
- DRAM dynamic random access memory
- SDRAM synchronous dynamic random access memory
- double data rate SDRAM double data rate SDRAM
- DDRSDRAM double data rate SDRAM
- ESDRAM enhanced synchronous dynamic random access memory
- synchronous link dynamic random access memory synchronous link dynamic random access memory
- SLDRAM direct memory bus random access memory
- direct rambus RAM direct rambus RAM
- the processor 110 may include one or more processing units; optionally, the processor 110 integrates an application processor and a modem processor, where the application processor mainly handles operations related to the operating system, user interface, application programs, etc., Modem processors mainly process wireless communication signals, such as baseband processors. It can be understood that the above modem processor may not be integrated into the processor 110 .
- the methods of the above embodiments can be implemented by means of software plus the necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases the former is better. implementation.
- the technical solution of the present application can be embodied in the form of a computer software product that is essentially or contributes to the existing technology.
- the computer software product is stored in a storage medium (such as ROM/RAM, disk , CD), including several instructions to cause a terminal (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of various embodiments of the present application.
Landscapes
- Engineering & Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Position Input By Displaying (AREA)
Abstract
本申请公开了一种虚拟操作方法、装置、电子设备及可读存储介质,属于混合现实技术领域。其中,该方法包括:显示虚拟触控界面,该虚拟触控界面与电子设备所处物理空间中目标平面的位置关联;识别用户的手势信息;在手势信息与虚拟触控界面中第一区域在物理空间的位置信息满足预设条件的情况下,执行目标操作。
Description
相关申请的交叉引用
本申请主张在2022年07月14日在中国提交的中国专利申请号202210834498.6的优先权,其全部内容通过引用包含于此。
本申请属于混合现实技术领域,具体涉及一种操作执行方法、装置、电子设备及可读存储介质。
通常,在用户使用扩展现实(extended reality,XR)设备的场景下,用户可以通过外设装置与该XR设备进行交互。具体地,若用户需求触发XR设备执行某个操作,则用户可以先触发XR设备与外设装置建立连接,然后再对外设装置进行移动输入,以使得XR设备可以获取该外设装置与XR设备间的相对位置信息,并基于该相对位置信息,执行该某个操作。
但是,由于在用户通过外设装置与XR设备进行交互时,用户需要先触发XR设备与外设装置建立连接,然后再对外设装置进行移动输入,才能触发XR设备执行某个操作,因此,导致在触发XR设备执行某个操作的过程中的耗时较长。
如此,导致XR设备的使用便捷性较差。
发明内容
本申请实施例的目的是提供一种虚拟操作方法、装置、电子设备及可读存储介质,能够解决XR设备执行用户需求操作的准确性较差的问题。
第一方面,本申请实施例提供了一种虚拟操作方法,该方法包括:显示虚拟触控界面,该虚拟触控界面与电子设备所处物理空间中目标平面的位置关联;识别用户的手势信息;在手势信息与虚拟触控界面中第一区域在物理空间的位置信息满足预设条件的情况下,执行目标操作。
第二方面,本申请实施例提供了一种虚拟操作装置,该虚拟操作装置包括:显示模块、识别模块以及执行模块。其中,显示模块,用于显示虚拟触控界面,该虚拟触控界面与电子设备所处物理空间中目标平面的位置关联。识别模块,用于识别用户的手势信息。执行模块,用于在识别模块识别的手势信息与显示模块显示的虚拟触控界面中第一区域在物理空间的位置信息满足预设条件的情况下,执行目标操作。
第三方面,本申请实施例提供了一种电子设备,该电子设备包括处理器和存储器,所述存储器存储可在所述处理器上运行的程序或指令,所述程序或指令被所述处理器执行时实现如第一方面所述的方法的步骤。
第四方面,本申请实施例提供了一种可读存储介质,所述可读存储介质上存储程序或指令,所述程序或指令被处理器执行时实现如第一方面所述的方法的步骤。
第五方面,本申请实施例提供了一种芯片,所述芯片包括处理器和通信接口,所述通信接口和所述处理器耦合,所述处理器用于运行程序或指令,实现如第一方面所述的方法的步骤。
第六方面,本申请实施例提供一种计算机程序产品,该程序产品被存储在存储介质中,该程序产品被至少一个处理器执行以实现如第一方面所述的方法的步骤。
在本申请实施例中,电子设备可以显示与电子设备所处物理空间中目标平面的位置关联的虚拟触控界面,并识别用户的手势信息,从而在手势信息与虚拟触控界面中第一区域在物理空间的位置信息满足预设条件的情况下,执行目标操作。由于电子设备可以显示与电子设备所处物理空
间中目标平面的位置关联的虚拟触控界面,并识别用户的手势信息,这样用户可以直接在虚拟触控界面上,进行手势信息与第一区域在物理空间的位置信息满足预设条件的隔空手势,以使得电子设备执行目标操作,而无需先触发XR设备与外设装置连接,再对外设装置进行移动输入,因此,可以减少在触发电子设备执行目标操作的过程中的耗时,如此,可以提高电子设备的使用便捷性。
图1是本申请实施例提供的虚拟操作方法的流程示意图之一;
图2是本申请实施例提供的虚拟操作方法的流程示意图之二;
图3是本申请实施例提供的用户手部的关节点进行检测示意图;
图4是本申请实施例提供的虚拟操作方法的流程示意图之三;
图5是本申请实施例提供的虚拟操作方法的流程示意图之四;
图6是本申请实施例提供的目标坐标范围示意图;
图7是本申请实施例提供的虚拟操作方法的流程示意图之五;
图8是本申请实施例提供的虚拟触控界面示意图;
图9是本申请实施例提供的虚拟操作方法的流程示意图之六;
图10是本申请实施例提供的虚拟操作方法的流程示意图之七;
图11是本申请实施例提供的虚拟操作装置的结构示意图;
图12是本申请实施例提供的电子设备的结构示意图;
图13是本申请实施例提供的电子设备的硬件结构示意图。
下面将结合本申请实施例中的附图,对本申请实施例中的技术方案进行清楚地描述,显然,所描述的实施例是本申请一部分实施例,而不是全部的实施例。基于本申请中的实施例,本领域普通技术人员获得的所有其他实施例,都属于本申请保护的范围。
本申请的说明书和权利要求书中的术语“第一”、“第二”等是用于区别类似的对象,而不用于描述特定的顺序或先后次序。应该理解这样使用的数据在适当情况下可以互换,以便本申请的实施例能够以除了在这里图示或描述的那些以外的顺序实施,且“第一”、“第二”等所区分的对象通常为一类,并不限定对象的个数,例如第一对象可以是一个,也可以是多个。此外,说明书以及权利要求中“和/或”表示所连接对象的至少其中之一,字符“/”,一般表示前后关联对象是一种“或”的关系。
下面结合附图,通过具体的实施例及其应用场景对本申请实施例提供的虚拟操作方法、装置、电子设备及可读存储介质进行详细地说明。
图1示出了本申请实施例提供的一种虚拟操作方法的流程图。如图1所示,本申请实施例提供的一种虚拟操作方法可以包括下述的步骤101至步骤103。
步骤101、电子设备显示虚拟触控界面。
本申请实施例中,上述虚拟触控界面与电子设备所处物理空间中目标平面的位置关联。
可选地,本申请实施例中,电子设备具体可以为扩展现实(extended reality,XR)设备。其中,该XR设备可以是以下任意一种:透视型显示XR设备、非透视型显示XR设备。
需要说明的是,上述“透视型显示XR设备”可以理解为:具有可透视的显示屏的XR设备,即该透视型显示XR设备可以在可透视的显示屏上显示虚拟对象,以使得用户可以在透过该可透视的显示屏观察用户所处的周围物理空间环境的同时,观察该可透视的显示屏上的虚拟对象。
上述“非透视型显示XR设备”可以理解为:具有不可透视的显示屏的XR设备,即该非透视型显示XR设备可以在不可透视的显示屏上显示用户所处的周围物理空间环境和虚拟对象,以使得用户可以在该不可透视的显示屏上观察用户所处的周围物理空间环境和虚拟对象。
可选地,本申请实施例中,电子设备具有一组或多组向下倾斜目标角度的相机模组;其中,该目标角度可以是由相机模组在电子设备上的位置确定的,或者由电子设备满足用户具体使用要求来确定的。
进一步可选地,本申请实施例中,相机模组能够提供图像数据(彩色或单色)和深度数据的一种或多种,每组相机模组的两个拍摄模块的视场角(field of view,FOV)重叠率大于某个重叠率阈值,其中,该某个重叠率阈值可以为90%,从而该两个拍摄模块能够形成大于某个视场角阈值的FOV,其中,该某个视场角阈值可以为120°。
在一种示例中,上述相机模组可以由一个图像传感器和一个深度传感器(例如结构光或者利用飞行时间法(Time of flight,TOF)的TOF相机)构成。
在另一种示例中,上述相机模组可以由两个分开放置的图像传感器构成。
需要说明的是,当***启动后,相机模组的传感器数据将以特定的频率实时生成并传输到***中,***则针对每帧数据进行相应的计算,具体计算过程在下面详述。
可选地,本申请实施例中,在电子设备处于工作状态的情况下,电子设备可以根据用户对电子设备的物理按键的点击输入,在电子设备当前所处物理空间的目标平面上,显示虚拟触控界面。
需要说明的是,上述“当前所处物理空间”可以理解为:用户使用电子设备时所处的周围空间环境。“虚拟触控界面与电子设备所处物理空间中目标平面的位置关联”可以理解为电子设备显示的虚拟触控界面在电子设备所处的周围空间环境里可以找到与之对应的现实的目标平面。
可选地,本申请实施例中,上述目标平面可以为以下任一项:电子设备当前所处环境中的目标物体上的平面、与电子设备的视线方向垂直的平面。
其中,目标物体可以为:景物、用户的身体部位(例如用户的手掌部位)等。
可选地,本申请实施例中,电子设备可以使用相机模组中的深度传感器或者图像传感器获得的深度信息来进行平面检测,以确定目标平面。
进一步可选地,本申请实施例中,电子设备将获得的深度信息进行预处理,转换为后续步骤需要的坐标信息。
进一步可选地,本申请实施例中,电子设备基于该坐标信息,检测空间环境中可作为虚拟触控界面的目标平面。
进一步可选地,本申请实施例中,电子设备可以在目标平面上,叠加显示虚拟触控板,该虚拟触控板上包括虚拟触控界面。
步骤102、电子设备识别用户的手势信息。
可选地,本申请实施例中,电子设备可以开启电子设备的摄像头采集至少一帧图像,并对该至少一帧图像进行图像识别,以识别用户相对于电子设备的手势信息。
可选地,本申请实施例中,电子设备可以使用目标信息,识别用户相对于电子设备的手势信息。其中,目标信息可以为以下至少一项:深度信息、图像信息。
步骤103、电子设备在手势信息与虚拟触控界面中第一区域在物理空间的位置信息满足预设条件的情况下,执行目标操作。
本申请实施例中,上述目标操作为:与满足预设条件的手势信息对应的操作。
可选地,本申请实施例中,电子设备在识别到用户相对电子设备的手势信息的情况下,可以确定该手势信息与虚拟触控界面的空间位置关系,以确定该手势信息是否为目标手势信息。
进一步可选地,本申请实施例中,在确定用户相对电子设备的手势信息与虚拟触控界面的空间距离小于或等于预设空间距离(例如下述实施例中的第一预设空间距离)的情况下,电子设备确定识别到目标手势信息。
进一步可选地,本申请实施例中,目标手势信息为需要执行目标操作的手势信息。
可选地,本申请实施例中,上述目标手势信息可以为以下至少一项:手势轨迹、手势类型。
其中,上述手势类型可以包括以下任一项:单指手势类型、双指手势类型、多指手势类型。
这里,单指手势类型可以包括以下至少一项:单指单击类型、单指双击类型、单指三击类型、单指滑动类型、单指长按类型等;双指手势类型可以包括以下至少一项:双指单击类型、双指上滑类型、双指下滑类型、双指左滑类型、双指右滑类型、双指长按类型、双指缩小类型、双指放大类型;多指手势类型可以包括以下至少一项:多指左滑类型、多指右滑类型、多指上滑类型、多指下滑类型。
需要说明的是,上述“双指缩小类型”具体可以为:用户双指相互朝向移动,以使得该双指间的距离缩小的手势类型。上述“双指放大类型”具体可以为:用户双指相互远离移动,以使得该双指间的距离增大的手势类型。
可选地,本申请实施例中,电子设备可以先确定目标手势信息,然后再根据该目标手势信息,从多个对应关系中确定与该目标手势信息对应的操作(即目标操作),从而电子设备可以执行目标操作。
其中,多个对应关系中的每个对应关系分别为一个目标手势信息和一个操作间的对应关系。
以下将具体说明,电子设备是如何确定目标手势信息的。
可选地,本申请实施例中,结合图1,如图2所示,在上述步骤103之前,本申请实施例提供的虚拟操作方法还可以包括下述的步骤201至步骤202。
步骤201、电子设备获取物理空间的第四图像。
本申请实施例中,第四图像包括用户手部图像。
需要说明的是,上述“获取物理空间的第四图像”可以理解为:电子设备连续采集电子设备所处物理空间的至少一帧图像。
可选地,本申请实施例中,至少一帧第四图像包括电子设备所处的物理空间的深度图数据信息和图像数据信息。
可以理解,多帧第四图像用于将电子设备周围的物理空间和用户手部的图像信息转化为在第一坐标系下的点云信息。
可选地,本申请实施例中,第一坐标系为电子设备当前所处环境对应空间的坐标系。
进一步可选地,本申请实施例中,上述环境对应空间的坐标系具体可以为世界坐标系。
需要说明的是,电子设备将至少一帧第四图像的图像信息转换为在世界坐标系下的点云信息可以参考下述实施例中转换方法,在此不再赘述。
步骤202、电子设备对第四图像进行用户手势识别,得到至少一个手部关节点信息。
需要说明的是,电子设备可以根据至少一帧第四图像对用户手部关节点进行检测。关于手部关节点的检测可以参考典型的基于深度学习的算法,将第四图像中包括的手部图像数据和手部深度图数据输入到神经网络,预测得到手部若干关节点的在第二坐标系下的坐标并转换为在第一坐标系下的坐标。
可选地,本申请实施例中,第一坐标系具体可以为世界坐标系;第二坐标系具体可以为相机坐标系。
可选地,本申请实施例中,电子设备可以从手部关节点中选出M个目标关节点。
进一步可选地,本申请实施例中,上述M个目标关节点,可以为用户手部的指尖。
进一步可选地,本申请实施例中,电子设备可以获取M个目标关节点在第二坐标系下的M个第一坐标信息。
其中,M个第一坐标信息和M个目标关节点一一对应。
进一步可选地,本申请实施例中,电子设备可以将M个第一坐标信息,转换为在第一坐标系下的M个第二坐标信息。
其中,该M个第二坐标信息和该M个第一坐标信息一一对应。
举例说明,如图3所示,每只手包含21个关节点,每个关节点有索引,如图3中的数字0、1、2、3、4…..20。电子设备基于深度学习的算法,将手部图像数据和手部深度图数据输入到神经
网络,可以预测得到手部关节点的在相机坐标系下的坐标,进一步地,可以将手部关节点在相机坐标系下的坐标转换为在世界坐标系中的坐标,具体转换过程参考下述实施例中相关说明,在此不再赘述。电子设备可以从用户双手关节点坐标中确定出指尖坐标,如确定出编号为16、17、18、19、20的五个手部关节点在相机坐标系下对应的坐标,并将该坐标由相机坐标系转换到世界坐标系,具体实施方法与前述相同。
可选地,本申请实施例中,电子设备分别根据M个第二坐标信息和目标界面位置信息,确定M个目标空间距离。
进一步,可选地,本申请实施例中,上述目标界面位置信息为:虚拟触控界面在第一坐标系下的位置信息。
其中,M个目标空间距离和M个第二坐标信息一一对应。
进一步,可选地,本申请实施例中,M个目标空间距离具体可以为用户手部指尖点和虚拟触控界面在第一坐标系下的空间距离。
可选地,本申请实施例中,在M个目标空间距离中包括小于或等于第一预设空间距离的目标空间距离的情况下,电子设备确定识别到目标手势信息。
可选地,本申请实施例中,第一预设空间距离可以为电子设备预先设定的空间距离阈值。
可选地,本申请实施例中,在第一坐标系下,电子设备计算指尖点坐标和虚拟触控界面的空间距离,当指尖点坐标与虚拟触控界面的空间距离在设定的空间阈值内时,即产生目标手势信息。
进一步可选地,本申请实施例中,根据M个目标空间距离可以确定一个或多个目标手势信息,每个目标手势信息可以用该目标手势信息在虚拟触控界面内的相对坐标表示。
可以理解,M个目标手势信息在虚拟触控界面发生的触控行为包括触控时间、触控频率和触控位移。触控点的触控时间较短可以判定为点击;触控时间较长且触控位移未发生可以判定为长按;触控时间较长且触控位移发生可以判定为滑动,触控频率可以用来判定点击次数;触控时间较长且两个触控点之间的距离发生改变可以判定为缩小或放大。
举例说明,当M个触控点中的一个触控点在虚拟触控界面一秒内发生触控时间较短的一次触控,可以判定为该手势参数中的单指单击;当M个触控点中的两个触控点在虚拟触控界面一秒内发生触控时间较短的一次触控,可以判定为该手势参数中的双指单击;当M个触控点中的一个触控点在虚拟触控界面发生触控时间较长且未发生触控位移的一次触控,可以判定为该手势参数中的单指长按;当M个触控点中的一个触控点在虚拟触控界面发生触控时间较长且发生触控位移的一次触控,可以判定为该手势参数中的单指滑动,并且根据位移方向可以判定滑动方向;当M个触控点中的两个触控点在虚拟触控界面发生触控时间较长、触控位移发生且两个触控点之间的距离变大,可以判定为该手势参数中的双指放大。第一隔空手势的手势参数中的其他参数可以参考举例判定。
如此可知,电子设备可以获取用户手指在虚拟触控界面上的触控行为,进而将相应的交互控制映射到物理触控板的交互逻辑上,使得用户在使用电子设备时,可以获得类似笔记本触控板的交互体验,并且提升电子设备执行用户需求操作的准确性。
本申请实施例提供的虚拟操作方法,电子设备可以显示与电子设备所处物理空间中目标平面的位置关联的虚拟触控界面,并识别用户的手势信息,从而在手势信息与虚拟触控界面中第一区域在物理空间的位置信息满足预设条件的情况下,执行目标操作。由于电子设备可以显示与电子设备所处物理空间中目标平面的位置关联的虚拟触控界面,这样用户可以直接在虚拟触控界面上进行相对于电子设备的隔空手势,而无需凌空进行相对于电子设备的隔空手势,因此,可以避免出现因用户手部肌肉疲劳,而导致用户相对于电子设备进行的隔空手势不准确的情况。且电子设备可以准确地识别用户的手势信息,并准确地执行与该手势信息对应的目标操作,如此,可以提高电子设备执行用户需求操作的准确性。
当然,在电子设备获取当前所处环境的目标平面,并且在目标平面上显示虚拟触控界面之前,
电子设备需要通过对传感器采集当前所处环境的图像数据进行相应的预处理,以确定目标平面。接下来对电子设备如何预处理传感器采集的图像数据进行说明。
可选地,本申请实施例中,结合图1,如图4所示,在上述步骤101之前,本申请实施例提供的虚拟操作方法还可以包括下述的步骤301至步骤304。
步骤301、电子设备获取第一图像。
本申请实施例中,上述第一图像为该电子设备所处物理空间的图像。
可选地,本申请实施例中,该电子设备所处的物理空间的图像包括彩色或单色图像和深度图像。
可选地,本申请实施例中,该电子设备所处的物理空间的图像可以为图像传感器或深度传感器提供的。
进一步可选地,本申请实施例中,该深度传感器具体可以为结构光或者TOF传感器。
需要说明的是,电子设备获得的彩色或单色图像对应的图像数据仅需经过常规ISP处理。
步骤302、电子设备根据第一图像,计算第一坐标系下的物理空间的第一点云。
可选地,本申请实施例中,第一点云可以为电子设备根据第一图像在第一坐标系下得到的物理空间的点云数据。
可选地,本申请实施例中,上述第一图像为深度图像,该深度图像包括N个第一像素对应的N个第一深度信息和N个第三坐标信息。具体地,结合图4,如图5所示,上述步骤302具体可以通过下述的步骤302a至步骤302b实现。
步骤302a、电子设备根据N个第一深度信息和N个第三坐标信息,计算第二坐标系下所述物理空间的第二点云。
本申请实施例中,N个第一像素的每个第一像素分别对应一个第一深度信息和一个第三坐标信息。
可选地,本申请实施例中,N个第一像素的N个第一深度信息可以是由深度传感器获取,也可以由双摄像头以立体匹配的方式获取。
需要说明的是,电子设备对N个第一深度信息可选择使用滤波优化处理,从而达到降噪且保持清晰边界的优化效果。
针对滤波优化处理的说明,可以参考现有滤波优化处理的技术方案,本申请实施例在此不予赘述。
可选地,本申请实施例中,N个第三坐标信息为该深度图像的N个像素坐标信息。
本申请实施例中,该第二点云具体可以为该深度图像的N个像素坐标信息在第二坐标系下的点云信息。
进一可选地,本申请实施例中,电子设备可以将深度图像的像素坐标信息转换为第二坐标系下的点云信息。
其中,电子设备可以获取深度图的每一个像素坐标(m,n),针对有效的深度值depth_mn,使用像素坐标系到第二坐标系的转换公式:
Z=depth_mn/camera_factor;
X=(n-camera_cx)*Z/camera_fx;
Y=(m–camera_cy)*Z/camera_fy;
Z=depth_mn/camera_factor;
X=(n-camera_cx)*Z/camera_fx;
Y=(m–camera_cy)*Z/camera_fy;
式中,camera_factor,camera_fx,camera_fy,camera_cx,camera_cy为相机内参。
进一可选地,本申请实施例中,电子设备计算得到深度图像素点(m,n)在第二坐标系下对应的点云信息(X,Y,Z)。
可选地,本申请实施例中,电子设备对深度图中的每一像素点进行转换处理,即可得到当前时刻第一图像在第二坐标系下的点云信息。
步骤302b、电子设备将第二点云,转换为第一点云。
需要说明的是,“将第二点云,转换为第一点云”可以理解为:电子设备将N个第三坐标信息在第二坐标系下的点云信息转换为N个第三坐标信息在第一坐标系下的点云信息。
本申请实施例中,N个第三坐标信息在第二坐标系下的点云信息可以通过相关坐标系转换计算公式转换为N个第三坐标信息在第一坐标系下的点云信息。
举例说明,假定第一坐标系为W,第二坐标系为C,以及第二坐标系到第一坐标系的转换计算方式为Twc。取第二点云中的一个点云点坐标为Pc(X,Y,Z),使用如下计算公式得到该点云点转换到第一坐标系后的坐标Pw:
Pw=Twc*Pc
Pw=Twc*Pc
其中,转换计算方式Twc可以参考现有的转换计算方式,本申请实施例在此不予赘述。
可选地,本申请实施例中,电子设备对第二点云中的每一个点云点进行坐标系转换计算,得到第二点云在第一坐标系下的点云信息。
如此可知,由于电子设备可以根据N个第一深度信息和N个第三坐标信息,计算第二坐标系下所述物理空间的第二点云,并将第二点云通过坐标系转换计算公式转换得到第一点云,这样,电子设备通过对传感器采集当前所处物理空间的图像数据信息进行相应的预处理,并转换为后续步骤需要的数据格式的图像数据信息。
本申请实施例中,当电子设备的传感器中不包括深度传感器时,第一图像也可以仅由图像传感器获取。
可选地,本申请实施例中,电子设备可以根据至少一帧第二图像,计算得到第一点云。
进一步可选地,本申请实施例中,上述第二图像为电子设备所处的物理空间的的图像。
进一步可选地,本申请实施例中,电子设备所处的物理空间的的图像可以为包括稀疏点云的深度图像。
进一步可选地,本申请实施例中,上述稀疏点云具体可以通过同时定位和地图构建(simultaneous localization and mapping,SLAM)算法利用该第二图像生成。
步骤303、电子设备从第一点云中,确定位于目标坐标范围内的目标点云。
本申请实施例中,上述目标坐标范围是基于电子设备在第一坐标系下的坐标信息确定的。
可选地,本申请实施例中,该目标坐标范围具体可以为由一个立方体坐标集所包围的点云坐标范围。
进一步可选地,本申请实施例中,如图6所示,该立方体的长W、宽D、高H2可以根据满足用户使用虚拟触控板的条件所确定。
举例说明,假定满足用户使用虚拟触控板的条件为:触控板长W要求至少为100厘米(cm),宽D要求至少为30cm,视场范围要求在30cm至100cm内,则该立方体的长W可以确定为100cm,宽D可以确定为30cm,高H2可以确定为70cm。
可选地,本申请实施例中,该目标点云为处于该目标坐标范围内的第一点云。
步骤304、电子设备基于目标点云,确定目标平面。
本申请实施例中,第一坐标系为物理空间对应的坐标系。
可选地,本申请实施例中,该物理空间对应的坐标系具体可以为世界坐标系。
可选地,本申请实施例中,电子设备对目标点云通过平面检测的方法可以获得至少一个平面。
进一步可选地,本申请实施例中,该平面检测的方法具体可以为区域增长方法或平面拟合法。
进一步可选地,本申请实施例中,电子设备可以从该至少一个平面中满足第一预设条件的平面当中确定目标平面。
可选地,本申请实施例中,结合图4,如图7所示,上述步骤304具体可以通过下述的步骤304a和步骤304c实现。
步骤304a、电子设备基于目标点云,确定至少一个第一平面。
可选地,本申请实施例中,每个第一平面都包含构成该平面的点云信息。
进一步可选地,本申请实施例中,该至少一个第一平面可以为法向量不同或者相同的平面。
在一种示例中,针对目标点云为稠密点云时,电子设备对目标点云可以采用区域增长的方式来进行平面检测,即在目标点云区域中随机选择多个点云点作为平面种子,循环判断种子邻域中的点是否归属平面,是则加入平面并扩大邻域搜索,区域增长算法为公开算法,在此不做进一步详述。
在另一种示例中,针对目标点云为稀疏点云时,电子设备对目标点云可以采用平面拟合的方式来进行平面检测,即电子设备对目标点云进行聚类处理,并使用典型的水平平面拟合算法进行平面拟合,以及对平面和已有参考平面进行融合。平面拟合算法为公开算法,在此不做进一步详述。
以下将以电子设备通过稠密深度信息来进行平面检测为例,举例说明电子设备是如何获得目标平面并生成虚拟触控板的。
步骤304b、电子设备将至少一个第一平面中满足第一预设条件的平面确定为第二平面。
本申请实施例中,第一预设条件包括以下至少一项:至少一个第一平面中与物理空间水平面的法向量夹角值小于预设值的平面;至少一个第一平面中与目标方向垂直的平面;至少一个第一平面中平面尺寸处于预设范围的平面。
可选地,本申请实施例中,该预设角度值可以根据用户具体使用来确定。
进一步可选地,本申请实施例中,电子设备可以根据该预设角度值去除法向量与水平面的法向量的夹角值大于预设角度值的第一平面。
可选地,本申请实施例中,每个第一平面的点云信息可以确定该平面的法向量、外接矩形以及平面尺寸。
进一步可选地,本申请实施例中,电子设备可以根据目标点云的外接矩形,确定该外接矩形的至少一条边与该电子设备的目标方向垂直的第一平面。
进一步可选地,本申请实施例中,目标方向为用户使用电子设备时的视线方向。
进一步可选地,本申请实施例中,电子设备可以根据该第一平面的平面尺寸确定平面尺寸在预设范围内的第一平面。
进一步可选地,本申请实施例中,该预设范围可以由目标坐标范围确定。
步骤304c、电子设备将第二平面的点云在物理空间对应的位置确定为目标平面。
本申请实施例中,电子设备可以将至少一个满足第一预设条件的第一平面确定为第二平面。
可选地,本申请实施例中,电子设备可以将组成第二平面的点云对应在物理空间的位置区域确定为目标平面。
举例说明,如图8所示,其中虚线矩形框为第二平面,组成第二平面的点云对应在物理空间的矩形平面内,可以生成一个预设长M、宽N的区域作为虚拟触控板的目标平面。
需要说明的是,若***中存在多个目标平面,则针对这些目标平面均生成虚拟触控板区域,再在电子设备的屏幕上显示出来,并交由用户选择。电子设备可以对生成的虚拟触控板进行追踪定位,针对虚拟触控板的追踪定位无需特殊处理,仅需实时计算电子设备的空间位置即可,该部分由SLAM算法解决,本申请实施例不再赘述。
以下将以电子设备通过稀疏深度信息来进行平面检测为例,举例说明电子设备是如何获得目标平面并生成虚拟触控板的。
可选地,本申请实施例中,目标坐标范围包括至少一个第一坐标范围。
本申请实施例中,结合图4,如图9所示,还可以通过下述步骤401至步骤403来实现:
步骤401、电子设备根据至少一个第一坐标范围,在目标点云中确定至少一个点云子集。
本申请实施例中,每个点云子集与每个第一坐标范围对应,该点云子集中包括至少一个点云。
本申请实施例中,该至少一个点云子集与该至少一个第一坐标范围一一对应,每个点云子集中包括目标点云中的至少一个点云。
进一步可选地,本申请实施例中,将上述目标坐标范围对应的长方形坐标集的高H2等分为至少一个区间,根据目标点云在第一坐标系中Y轴的值分配到至少一个区间内,得到至少一个点云子集。
步骤402、电子设备根据至少一个点云子集中的目标点云子集,确定至少一个第一平面。
本申请实施例中,目标点云点子集为至少一个点云子集中点云数量大于或等于预设数量的点云子集。
可选地,本申请实施例中,针对至少一个点云子集中的目标点云,可以使用典型的水平平面拟合法计算由至少一个点云子集所构成的水平平面,得到当前帧的至少一个第一平面和若干未构成第一平面的离散点云。
本申请实施例中,该至少一个第一平面与该至少一个点云子集一一对应。
步骤403、电子设备将至少一个第一平面中的第三平面与第四平面进行融合,得到目标平面。
本申请实施例中,第四平面为电子设备已存储的平面,第三平面与第四平面在物理空间的空间位置满足第二预设条件。
可选地,本申请实施例中,第三平面为通过对点云点子集中的目标点云进行聚类处理,并采用水平平面拟合的方法来进行平面检测得到的平面。
可选地,本申请实施例中,该第四平面是根据至少一帧第二图像的上一帧图像确定并存储的平面。
可选地,本申请实施例中,上述第二预设条件包括以下至少一项:与第四平面在第一方向上有重叠、与第四平面在第二方向上的空间距离小于或等于第二预设空间距离;第一方向垂直于第二方向。
可选地,本申请实施例中,电子设备将第三平面与电子设备***内存储的第四平面进行平面融合。
可选地,本申请实施例中,电子设备在当前帧还可以对离散点云进行处理。其中,针对至少一个点云子集中的每个离散点云,若该离散点云与***的第四平面在水平方向距离小于某个设定阈值,且在垂直方向高度差在一定阈值内,则融合该离散点云和第四平面。
如此可知,由于电子设备可以不依赖于具体深度模块获得至少一张第二图像,并且根据至少一张第二图像获得目标平面,从而生成虚拟触控界面,避免了用户长时间举起手部相对于XR设备进行与某个操作对应的手势类型的隔空手势,因此,可以减少在触发电子设备执行目标操作的过程中的耗时,并且避免用户因手部肌肉疲劳,而导致影响手势操作的精度,如此,可以提高XR设备的使用便捷性和准确性。
当然,可能会出现用户需要在移动时使用该电子设备的情况,此时,电子设备可以将一只手作为虚拟触控版,然后再用另一只手进行操作。以下将针对该情况进行举例说明。
可选地,本申请实施例中,结合图1,如图10所示,在上述步骤101之前,本申请实施例提供的虚拟操作方法具体还可以通过下述的步骤501至步骤504实现。
步骤501、电子设备获取物理空间的第三图像。
本申请实施例中,第三图像包括用户手部图像。
可选地,本申请实施例中,上述第三图像为电子设备所处物理空间的图像,该第三图像中的每帧第三图像中包括用户手部。
进一步可选地,本申请实施例中,第三图像包括电子设备的物理空间的深度图数据信息和图像数据信息,以及用户手部关节点的图像数据信息或深度图数据信息。
步骤502、电子设备根据第三图像,计算物理空间在第一坐标系下的第三点云。
本申请实施例中,第三点云包括用户手部位姿信息。
可以理解,电子设备获取的至少一帧第三图像用于将用户手部的图像信息转化为在第一坐标系下的第三点云。
可选地,本申请实施例中,上述手部位姿信息为用户手部关节点以及手掌的位置和状态的信息。
可选地,本申请实施例中,第三点云为用户手部关节点以及手掌在第一坐标系下的点云坐标。
可选地,本申请实施例中,电子设备将至少一帧第三图像的图像信息转化为在第一坐标系下的点云信息可以参考本实施例上述的坐标系转换方法。
举例说明,电子设备对每一帧第三图像数据进行手部关节点的检测,可以获得一只手的五指以及手掌位姿,如图3所示,从中选取11个关节点分别为:五指(16、17、18、19、20)与手掌(0、1、4、7、10、14)。电子设备将选取的11个关节点的在相机坐标系下的坐标,转换为在世界坐标系中的坐标,具体转换过程参考上述实施例中相关说明,在此不再赘述。
步骤503、电子设备根据用户手部位姿信息,在第三点云中确定第三平面。
可选地,本申请实施例中,根据第三点云,电子设备可以通过拟合平面法得到该第三点云的外接矩形。
进一步可选地,本申请实施例中,该外接矩形对应的第三平面包括该平面的法向和构成该平面的顶点坐标数据。
可选地,本申请实施例中,电子设备根据外接矩形的顶点数据可以得到该外接矩形的面积。
进一步可选地,本申请实施例中,外接矩形中面积大于或等于预设面积的可以确定为目标外接矩形。
进一步可选地,本申请实施例中,电子设备可以将该目标外接矩形确定为第三平面。
步骤504、电子设备将第三平面在物理空间对应的屏幕确定为目标平面。
本申请实施例中,第一坐标系为所述物理空间对应的坐标系。
可选地,本申请实施例中,电子设备可以将组成第三平面的点云对应在物理空间的位置区域确定为目标平面。
如此可知,电子设备可以识别用户的手部,将一只手确定为虚拟触控界面,并识别另一只手相对于电子设备的手势信息,而无需凌空进行相对于电子设备的隔空手势,因此,可以减少在触发电子设备执行目标操作的过程中的耗时,并且避免用户因手部肌肉疲劳,而导致影响手势操作的精度,如此,可以提高XR设备的使用便捷性和准确性。
本申请实施例提供的方法,执行主体可以为装置。本申请实施例中以装置执行方法为例,说明本申请实施例提供的装置的。
图11示出了上述实施例中所涉及的一种虚拟操作装置的一种可能的结构示意图。如图11所示,本申请实施例提供的虚拟操作装置60可以包括:显示模块61、识别模块62以及执行模块63。其中,其中,显示模块61,用于显示虚拟触控界面,该虚拟触控界面与电子设备所处物理空间中目标平面的位置关联。识别模块62,用于识别用户的手势信息。执行模块63,用于在识别模块识别的手势信息与显示模块显示的虚拟触控界面中第一区域在物理空间的位置信息满足预设条件的情况下,执行目标操作。
在一种可能的实现方式中,上述虚拟操作装置60还包括:获取模块、计算模块以及确定模块。其中,获取模块,用于获取第一图像,该第一图像为电子设备所处物理空间的图像。计算模块,用于根据获取模块获取的第一图像,计算第一坐标系下的物理空间的第一点云。确定模块,用于从计算模块计算得到的第一点云中,确定位于目标坐标范围内的目标点云,该目标坐标范围是基于电子设备在第一坐标系下的坐标信息确定的,并基于所述目标点云,确定目标平面。其中,第一坐标系为物理空间对应的坐标系。
在一种可能的实现方式中,第一图像为深度图像,该深度图像包括N个第一像素对应的N个第一深度信息和N个第三坐标信息,上述计算模块包括:转换子模块。计算模块,用于根据N个第一深度信息和N个第三坐标信息,计算第二坐标系下物理空间的第二点云。转换子模块,用于将计算模块得到的第二点云,转换为第一点云。其中,第二坐标系为虚拟操作装置的图像传感器
对应的坐标系。
在一种可能的实现方式中,上述确定模块具体用于基于目标点云,确定至少一个第一平面,并将该至少一个第一平面中满足第一预设条件的平面确定为第二平面,以及将该第二平面的点云在物理空间对应的位置确定为目标平面。其中,第一预设条件包括以下至少一项:至少一个第一平面中与物理空间水平面的法向量夹角值小于预设值的平面,至少一个第一平面中与目标方向垂直的平面,至少一个第一平面中平面尺寸处于预设范围的平面。
在一种可能的实现方式中,目标坐标范围包括至少一个第一坐标范围,上述确定模块包括:融合子模块。确定模块,具体用于根据至少一个第一坐标范围,在目标点云中确定至少一个点云子集,每个点云子集与每个第一坐标范围对应,该点云子集中包括至少一个点云,并根据该至少一个点云子集中的目标点云子集,确定至少一个第一平面,该目标点云点子集为至少一个点云子集中点云数量大于或等于预设数量的点云子集。融合子模块,用于将至少一个第一平面中的第三平面与第四平面进行融合,得到目标平面,该第四平面为电子设备已存储的平面,第三平面与第四平面在物理空间的空间位置满足预设条件。
在一种可能的实现方式中,上述操作执行装置还包括:获取模块、计算模块以及确定模块。获取模块,用于获取物理空间的第三图像,该第三图像包括用户手部图像。计算模块,用于根据获取模块获取的第三图像,计算物理空间在第一坐标系下的第三点云,该第三点云包括用户手部位姿信息。确定模块,用于根据用户手部位姿信息,在计算模块计算得到的第三点云中确定第三平面,并将该确定模块确定的第三平面在物理空间对应的屏幕确定为目标平面。其中,第一坐标系为所述物理空间对应的坐标系。
在一种可能的实现方式中,上述识别模块62包括:获取子模块。获取子模块用于获取所述物理空间的第四图像,该第四图像包括用户手部图像。识别模块用于对第四图像进行用户手势识别,得到至少一个手部关节点信息。
在一种可能的实现方式中,至少一个手部关节点信息包括至少一个手部关节在第二坐标系下的至少一个第一坐标。上述执行模块63包括:确定子模块。确定子模块用于至少一个第一坐标与虚拟触控界面在物理空间的空间距离满足预设条件的情况下,确定虚拟触控界面的第一区域。执行模块,具体用于根据第一区域对应操作信息,执行目标操作。
本申请实施例提供的虚拟操作装置,由于虚拟操作装置可以显示与虚拟操作装置所处物理空间中目标平面的位置关联的虚拟触控界面,并识别用户的手势信息,这样用户可以直接在虚拟触控界面上,进行手势信息与第一区域在物理空间的位置信息满足预设条件的隔空手势,以使得虚拟操作装置执行目标操作,而无需先触发XR设备与外设装置连接,再对外设装置进行移动输入,因此,可以减少在触发虚拟操作装置执行目标操作的过程中的耗时,如此,可以提高虚拟操作装置的使用便捷性。
本申请实施例中的虚拟操作装置可以是电子设备,也可以是电子设备中的部件,例如集成电路或芯片。该电子设备可以是终端,也可以为除终端之外的其他设备。示例性的,电子设备可以为手机、平板电脑、笔记本电脑、掌上电脑、车载电子设备、移动上网装置(mobile internet device,MID)、增强现实(augmented reality,AR)/虚拟现实(virtual reality,VR)设备、机器人、可穿戴设备、超级移动个人计算机(ultra-mobile personal computer,UMPC)、上网本或者个人数字助理(personal digital assistant,PDA)等,还可以为服务器、网络附属存储器(network attached storage,NAS)、个人计算机(personal computer,PC)、电视机(television,TV)、柜员机或者自助机等,本申请实施例不作具体限定。
本申请实施例中的虚拟操作装置可以为具有操作***的装置。该操作***可以为安卓(Android)操作***,可以为iOS操作***,还可以为其他可能的操作***,本申请实施例不作具体限定。
本申请实施例提供的虚拟操作装置能够实现图1至图10的方法实施例实现的各个过程,为避
免重复,这里不再赘述。
可选地,本申请实施例中,如图12所示,本申请实施例还提供一种电子设备80,包括处理器81和存储器82,存储器82上存储有可在所述处理器81上运行的程序或指令,该程序或指令被处理器81执行时实现上述拍摄方法实施例的各个过程步骤,且能达到相同的技术效果,为避免重复,这里不再赘述。
需要说明的是,本申请实施例中的电子设备包括上述所述的移动电子设备和非移动电子设备。
图13为实现本申请实施例的一种电子设备的硬件结构示意图。
该电子设备100包括但不限于:射频单元1001、网络模块1002、音频输出单元1003、输入单元1004、传感器1005、显示单元1006、用户输入单元1007、接口单元1008、存储器1009、以及处理器110等部件。
本领域技术人员可以理解,电子设备100还可以包括给各个部件供电的电源(比如电池),电源可以通过电源管理***与处理器110逻辑相连,从而通过电源管理***实现管理充电、放电、以及功耗管理等功能。图13中示出的电子设备结构并不构成对电子设备的限定,电子设备可以包括比图示更多或更少的部件,或者组合某些部件,或者不同的部件布置,在此不再赘述。
其中,处理器110,具体用于在电子设备当前所处环境的目标平面上,显示虚拟触控界面;并识别用户相对于该电子设备的隔空手势;以及在识别到用户在该虚拟触控界面上相对于该电子设备的第一隔空手势的情况下,响应于该第一隔空手势,执行目标操作;该目标操作为:与该第一隔空手势的手势参数对应的操作。
本申请实施例提供的电子设备,由于电子设备可以显示与电子设备所处物理空间中目标平面的位置关联的虚拟触控界面,并识别用户的手势信息,这样用户可以直接在虚拟触控界面上,进行手势信息与第一区域在物理空间的位置信息满足预设条件的隔空手势,以使得电子设备执行目标操作,而无需先触发XR设备与外设装置连接,再对外设装置进行移动输入,因此,可以减少在触发电子设备执行目标操作的过程中的耗时,如此,可以提高电子设备的使用便捷性。
可选地,本申请实施例中,处理器110,具体用于获取第一图像,该第一图像为电子设备所处物理空间的图像;根据第一图像,计算第一坐标系下的物理空间的第一点云;从第一点云中,确定位于目标坐标范围内的目标点云,该目标坐标范围是基于电子设备在第一坐标系下的坐标信息确定的;基于目标点云,确定目标平面;其中,第一坐标系为物理空间对应的坐标系。
可选地,本申请实施例中,处理器110,具体用于根据N个第一深度信息和N个第三坐标信息,计算第二坐标系下物理空间的第二点云;将第二点云,转换为第一点云;其中,第二坐标系为电子设备的图像传感器对应的坐标系。
可选地,本申请实施例中,处理器110,具体用于基于目标点云,确定至少一个第一平面;将至少一个第一平面中满足第一预设条件的平面确定为第二平面;将第二平面的点云在物理空间对应的位置确定为目标平面;其中,第一预设条件包括以下至少一项:至少一个第一平面中与物理空间水平面的法向量夹角值小于预设值的平面;至少一个第一平面中与目标方向垂直的平面;至少一个第一平面中平面尺寸处于预设范围的平面。
可选地,本申请实施例中,处理器110,具体用于根据至少一个第一坐标范围,在目标点云中确定至少一个点云子集,每个点云子集与每个第一坐标范围对应,该点云子集中包括至少一个点云;根据至少一个点云子集中的目标点云子集,确定至少一个第一平面,该目标点云点子集为该至少一个点云子集中点云数量大于或等于预设数量的点云子集;将至少一个第一平面中的第三平面与第四平面进行融合,得到目标平面,该第四平面为电子设备已存储的平面,该第三平面与该第四平面在物理空间的空间位置满足预设条件。
可选地,本申请实施例中,处理器110,具体用于获取物理空间的第三图像,该第三图像包括用户手部图像;根据第三图像,计算物理空间在第一坐标系下的第三点云,该第三点云包括用户手部位姿信息;根据用户手部位姿信息,在第三点云中确定第三平面;将第三平面在物理空间
对应的屏幕确定为目标平面;其中,第一坐标系为物理空间对应的坐标系。
可选地,本申请实施例中,处理器110,具体用于获取物理空间的第四图像,该第四图像包括用户手部图像;对第四图像进行用户手势识别,得到至少一个手部关节点信息。
可选地,本申请实施例中,处理器110,具体用于至少一个第一坐标与虚拟触控界面在物理空间的空间距离满足预设条件的情况下,确定该虚拟触控界面的第一区域;根据第一区域对应操作信息,执行目标操作。
应理解的是,本申请实施例中,输入单元1004可以包括图形处理器(graphics processing unit,GPU)1041和麦克风1042,图形处理器1041对在视频捕获模式或图像捕获模式中由图像捕获装置(如摄像头)获得的静态图片或视频的图像数据进行处理。显示单元1006可包括显示面板1061,可以采用液晶显示器、有机发光二极管等形式来配置显示面板1061。用户输入单元1007包括触控面板1071以及其他输入设备1072中的至少一种。触控面板1071,也称为触摸屏。触控面板1071可包括触摸检测装置和触摸控制器两个部分。其他输入设备1072可以包括但不限于物理键盘、功能键(比如音量控制按键、开关按键等)、轨迹球、鼠标、操作杆,在此不再赘述。
存储器1009可用于存储软件程序以及各种数据。存储器1009可主要包括存储程序或指令的第一存储区和存储数据的第二存储区,其中,第一存储区可存储操作***、至少一个功能所需的应用程序或指令(比如声音播放功能、图像播放功能等)等。此外,存储器1009可以包括易失性存储器或非易失性存储器,或者,存储器1009可以包括易失性和非易失性存储器两者。其中,非易失性存储器可以是只读存储器(read-only memory,ROM)、可编程只读存储器(programmable ROM,PROM)、可擦除可编程只读存储器(erasable PROM,EPROM)、电可擦除可编程只读存储器(electrically EPROM,EEPROM)或闪存。易失性存储器可以是随机存取存储器(random access memory,RAM),静态随机存取存储器(static RAM,SRAM)、动态随机存取存储器(dynamic RAM,DRAM)、同步动态随机存取存储器(synchronous DRAM,SDRAM)、双倍数据速率同步动态随机存取存储器(double data rate SDRAM,DDRSDRAM)、增强型同步动态随机存取存储器(enhanced SDRAM,ESDRAM)、同步连接动态随机存取存储器(synch link DRAM,SLDRAM)和直接内存总线随机存取存储器(direct rambus RAM,DRRAM)。本申请实施例中的存储器1009包括但不限于这些和任意其它适合类型的存储器。
处理器110可包括一个或多个处理单元;可选的,处理器110集成应用处理器和调制解调处理器,其中,应用处理器主要处理涉及操作***、用户界面和应用程序等的操作,调制解调处理器主要处理无线通信信号,如基带处理器。可以理解的是,上述调制解调处理器也可以不集成到处理器110中。
需要说明的是,在本文中,术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的包含,从而使得包括一系列要素的过程、方法、物品或者装置不仅包括那些要素,而且还包括没有明确列出的其他要素,或者是还包括为这种过程、方法、物品或者装置所固有的要素。在没有更多限制的情况下,由语句“包括一个……”限定的要素,并不排除在包括该要素的过程、方法、物品或者装置中还存在另外的相同要素。此外,需要指出的是,本申请实施方式中的方法和装置的范围不限按示出或讨论的顺序来执行功能,还可包括根据所涉及的功能按基本同时的方式或按相反的顺序来执行功能,例如,可以按不同于所描述的次序来执行所描述的方法,并且还可以添加、省去、或组合各种步骤。另外,参照某些示例所描述的特征可在其他示例中被组合。
通过以上的实施方式的描述,本领域的技术人员可以清楚地了解到上述实施例方法可借助软件加必需的通用硬件平台的方式来实现,当然也可以通过硬件,但很多情况下前者是更佳的实施方式。基于这样的理解,本申请的技术方案本质上或者说对现有技术做出贡献的部分可以以计算机软件产品的形式体现出来,该计算机软件产品存储在一个存储介质(如ROM/RAM、磁碟、光盘)中,包括若干指令用以使得一台终端(可以是手机,计算机,服务器,或者网络设备等)执行本申请各个实施例该的方法。
上面结合附图对本申请的实施例进行了描述,但是本申请并不局限于上述的具体实施方式,上述的具体实施方式仅仅是示意性的,而不是限制性的,本领域的普通技术人员在本申请的启示下,在不脱离本申请宗旨和权利要求所保护的范围情况下,还可做出很多形式,均属于本申请的保护之内。
Claims (20)
- 一种虚拟操作方法,所述方法包括:显示虚拟触控界面,所述虚拟触控界面与电子设备所处物理空间中目标平面的位置关联;识别用户的手势信息;在所述手势信息与所述虚拟触控界面中第一区域在所述物理空间的位置信息满足预设条件的情况下,执行目标操作。
- 根据权利要求1所述的方法,其中,所述显示虚拟触控界面之前,所述方法还包括:获取第一图像,所述第一图像为电子设备所处物理空间的图像;根据所述第一图像,计算第一坐标系下的所述物理空间的第一点云;从所述第一点云中,确定位于目标坐标范围内的目标点云,所述目标坐标范围是基于所述电子设备在所述第一坐标系下的坐标信息确定的;基于所述目标点云,确定目标平面;其中,所述第一坐标系为所述物理空间对应的坐标系。
- 根据权利要求2所述的方法,其中,所述第一图像为深度图像,所述深度图像包括N个第一像素对应的N个第一深度信息和N个第三坐标信息;所述根据所述第一图像,计算第一坐标系下的所述物理空间的第一点云包括:根据所述N个第一深度信息和所述N个第三坐标信息,计算第二坐标系下所述物理空间的第二点云;将所述第二点云,转换为所述第一点云;其中,所述第二坐标系为所述电子设备的图像传感器对应的坐标系。
- 根据权利要求2或3所述的方法,其中,所述基于所述目标点云,确定目标平面,包括:基于所述目标点云,确定至少一个第一平面;将所述至少一个第一平面中满足第一预设条件的平面确定为第二平面;将所述第二平面的点云在所述物理空间对应的位置确定为所述目标平面;其中,所述第一预设条件包括以下至少一项:所述至少一个第一平面中与所述物理空间水平面的法向量夹角值小于预设值的平面;所述至少一个第一平面中与目标方向垂直的平面;所述至少一个第一平面中平面尺寸处于预设范围的平面。
- 根据权利要求2所述的方法,其中,所述目标坐标范围包括至少一个第一坐标范围;所述基于所述目标点云,确定至少一个第一平面,并根据所述至少一个第一平面,确定所述目标平面,包括:根据所述至少一个第一坐标范围,在所述目标点云中确定至少一个点云子集,每个所述点云子集与每个所述第一坐标范围对应,所述点云子集中包括至少一个点云;根据所述至少一个点云子集中的目标点云子集,确定所述至少一个第一平面,所述目标点云点子集为所述至少一个点云子集中点云数量大于或等于预设数量的点云子集;将所述至少一个第一平面中的第三平面与第四平面进行融合,得到所述目标平面,所述第四平面为所述电子设备已存储的平面,所述第三平面与所述第四平面在所述物理空间的空间位置满足第二预设条件。
- 根据权利要求1所述的方法,其中,所述显示虚拟触控界面之前,所述方法还包括:获取物理空间的第三图像,所述第三图像包括用户手部图像;根据所述第三图像,计算所述物理空间在第一坐标系下的第三点云,所述第三点云包括用户手部位姿信息;根据所述用户手部位姿信息,在所述第三点云中确定第三平面;将所述第三平面在所述物理空间对应的屏幕确定为目标平面;其中,所述第一坐标系为所述物理空间对应的坐标系。
- 根据权利要求1所述的方法,其中,所述识别用户的手势信息包括:获取所述物理空间的第四图像,所述第四图像包括用户手部图像;对所述第四图像进行用户手势识别,得到至少一个手部关节点信息。
- 根据权利要求7所述的方法,其中,所述至少一个手部关节点信息包括至少一个手部关节在第二坐标系下的至少一个第一坐标;所述在所述手势信息与所述虚拟触控界面中第一区域在所述物理空间的位置信息满足预设条件的情况下,执行目标操作包括:所述至少一个第一坐标与所述虚拟触控界面在所述物理空间的空间距离满足预设条件的情况下,确定所述虚拟触控界面的第一区域;根据所述第一区域对应操作信息,执行目标操作。
- 一种虚拟操作装置,所述虚拟操作装置包括:显示模块、识别模块以及执行模块;所述显示模块,用于显示虚拟触控界面,所述虚拟触控界面与电子设备所处物理空间中目标平面的位置关联;所述识别模块,用于识别用户的手势信息;所述执行模块,用于在所述识别模块识别的手势信息与所述显示模块显示的虚拟触控界面中第一区域在所述物理空间的位置信息满足预设条件的情况下,执行目标操作。
- 根据权利要求9所述的虚拟操作装置,其中,所述虚拟操作装置还包括:获取模块、计算模块以及确定模块;所述获取模块,用于获取第一图像,所述第一图像为电子设备所处物理空间的图像;所述计算模块,用于根据所述获取模块获取的第一图像,计算第一坐标系下的所述物理空间的第一点云;所述确定模块,用于从所述计算模块计算得到的第一点云中,确定位于目标坐标范围内的目标点云,所述目标坐标范围是基于所述电子设备在所述第一坐标系下的坐标信息确定的;并基于所述目标点云,确定目标平面;其中,所述第一坐标系为所述物理空间对应的坐标系。
- 根据权利要求9所述的操作执行装置,其中,所述操作执行装置还包括:获取模块、计算模块以及确定模块;所述获取模块,用于获取物理空间的第三图像,所述第三图像包括用户手部图像;所述计算模块,用于根据所述获取模块获取的所述第三图像,计算所述物理空间在第一坐标系下的第三点云,所述第三点云包括用户手部位姿信息;所述确定模块,用于根据所述用户手部位姿信息,在所述计算模块计算得到的第三点云中确定第三平面;并将所述确定模块确定的第三平面在所述物理空间对应的屏幕确定为目标平面;其中,所述第一坐标系为所述物理空间对应的坐标系。
- 根据权利要求10或11所述的操作执行装置,其中,所述确定模块,具体用于基于所述目标点云,确定至少一个第一平面;并将所述至少一个第一平面中满足第一预设条件的平面确定为第二平面;以及将所述第二平面的点云在所述物理空间对应的位置确定为所述目标平面;其中,所述第一预设条件包括以下至少一项:所述至少一个第一平面中与所述物理空间水平面的法向量夹角值小于预设值的平面;所述至少一个第一平面中与目标方向垂直的平面;所述至少一个第一平面中平面尺寸处于预设范围的平面。
- 根据权利要求10所述的操作执行装置,其中,所述目标坐标范围包括至少一个第一坐标范围;所述确定模块包括:融合子模块;所述确定模块,具体用于根据所述至少一个第一坐标范围,在所述目标点云中确定至少一个点云子集,每个所述点云子集与每个所述第一坐标范围对应,所述点云子集中包括至少一个点云; 并根据所述至少一个点云子集中的目标点云子集,确定所述至少一个第一平面,所述目标点云点子集为所述至少一个点云子集中点云数量大于或等于预设数量的点云子集;所述融合子模块,用于将所述至少一个第一平面中的第三平面与第四平面进行融合,得到所述目标平面,所述第四平面为所述电子设备已存储的平面,所述第三平面与所述第四平面在所述物理空间的空间位置满足第二预设条件。
- 根据权利要求9所述的操作执行装置,其中,所述操作执行装置还包括:获取模块、计算模块以及确定模块;所述获取模块,用于所述显示虚拟触控界面之前,获取物理空间的第三图像,所述第三图像包括用户手部图像;所述计算模块,用于根据所述第三图像,计算所述物理空间在第一坐标系下的第三点云,所述第三点云包括用户手部位姿信息;所述确定模块,用于根据所述用户手部位姿信息,在所述第三点云中确定第三平面;并将所述第三平面在所述物理空间对应的屏幕确定为目标平面;其中,所述第一坐标系为所述物理空间对应的坐标系。
- 根据权利要求9所述的虚拟操作装置,其中,所述识别模块包括:获取子模块;所述获取子模块用于获取所述物理空间的第四图像,所述第四图像包括用户手部图像;所述识别模块用于对所述第四图像进行用户手势识别,得到至少一个手部关节点信息。
- 根据权利要求15所述的虚拟操作装置,其中,所述至少一个手部关节点信息包括至少一个手部关节在第二坐标系下的至少一个第一坐标;所述执行模块包括:确定子模块;所述确定子模块,用于所述至少一个第一坐标与所述虚拟触控界面在所述物理空间的空间距离满足预设条件的情况下,确定所述虚拟触控界面的第一区域;所述执行模块,具体用于根据所述第一区域对应操作信息,执行目标操作。
- 一种电子设备,其特征在于,包括处理器和存储器,所述存储器存储可在所述处理器上运行的程序或指令,所述程序或指令被所述处理器执行时实现如权利要求1至8中任一项所述的虚拟操作执行方法的步骤。
- 一种可读存储介质,其特征在于,所述可读存储介质上存储程序或指令,所述程序或指令被处理器执行时实现如权利要求1至8中任一项所述的虚拟操作执行方法的步骤。
- 一种计算机程序产品,所述程序产品被至少一个处理器执行以实现如权利要求1至8中任一项所述的虚拟操作方法。
- 一种用户设备UE,包括所述UE被配置成用于执行如权利要求1至8中任一项所述的虚拟操作方法。
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202210834498.6 | 2022-07-14 | ||
CN202210834498.6A CN115220636B (zh) | 2022-07-14 | 2022-07-14 | 虚拟操作方法、装置、电子设备及可读存储介质 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2024012268A1 true WO2024012268A1 (zh) | 2024-01-18 |
Family
ID=83611311
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2023/104823 WO2024012268A1 (zh) | 2022-07-14 | 2023-06-30 | 虚拟操作方法、装置、电子设备及可读存储介质 |
Country Status (2)
Country | Link |
---|---|
CN (1) | CN115220636B (zh) |
WO (1) | WO2024012268A1 (zh) |
Families Citing this family (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN115220636B (zh) * | 2022-07-14 | 2024-04-26 | 维沃移动通信有限公司 | 虚拟操作方法、装置、电子设备及可读存储介质 |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20180114349A1 (en) * | 2016-10-21 | 2018-04-26 | Institute For Information Industry | Augmented reality system and method |
CN108519817A (zh) * | 2018-03-26 | 2018-09-11 | 广东欧珀移动通信有限公司 | 基于增强现实的交互方法、装置、存储介质及电子设备 |
CN111399654A (zh) * | 2020-03-25 | 2020-07-10 | Oppo广东移动通信有限公司 | 信息处理方法、装置、电子设备及存储介质 |
CN115220636A (zh) * | 2022-07-14 | 2022-10-21 | 维沃移动通信有限公司 | 虚拟操作方法、装置、电子设备及可读存储介质 |
Family Cites Families (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP3567453B1 (en) * | 2018-05-07 | 2021-07-21 | Apple Inc. | Devices and methods for measuring using augmented reality |
CN109949422B (zh) * | 2018-10-15 | 2020-12-15 | 华为技术有限公司 | 用于虚拟场景的数据处理方法以及设备 |
CN109901733A (zh) * | 2019-01-03 | 2019-06-18 | 合肥龙图腾信息技术有限公司 | 一种触控方法、触控设备及计算机可读存储介质 |
CN111913564B (zh) * | 2019-05-07 | 2023-07-18 | 广东虚拟现实科技有限公司 | 虚拟内容的操控方法、装置、***、终端设备及存储介质 |
CN113269089B (zh) * | 2021-05-25 | 2023-07-18 | 上海人工智能研究院有限公司 | 基于深度学习的实时手势识别方法及*** |
-
2022
- 2022-07-14 CN CN202210834498.6A patent/CN115220636B/zh active Active
-
2023
- 2023-06-30 WO PCT/CN2023/104823 patent/WO2024012268A1/zh unknown
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20180114349A1 (en) * | 2016-10-21 | 2018-04-26 | Institute For Information Industry | Augmented reality system and method |
CN108519817A (zh) * | 2018-03-26 | 2018-09-11 | 广东欧珀移动通信有限公司 | 基于增强现实的交互方法、装置、存储介质及电子设备 |
CN111399654A (zh) * | 2020-03-25 | 2020-07-10 | Oppo广东移动通信有限公司 | 信息处理方法、装置、电子设备及存储介质 |
CN115220636A (zh) * | 2022-07-14 | 2022-10-21 | 维沃移动通信有限公司 | 虚拟操作方法、装置、电子设备及可读存储介质 |
Also Published As
Publication number | Publication date |
---|---|
CN115220636B (zh) | 2024-04-26 |
CN115220636A (zh) | 2022-10-21 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
TWI690842B (zh) | 基於手勢辨認的互動顯示方法和裝置 | |
US20140300542A1 (en) | Portable device and method for providing non-contact interface | |
US20130335324A1 (en) | Computer vision based two hand control of content | |
WO2013118458A1 (en) | Image processing device, and computer program product | |
CN112506340B (zh) | 设备控制方法、装置、电子设备及存储介质 | |
JP2004246578A (ja) | 自己画像表示を用いたインタフェース方法、装置、およびプログラム | |
CN112068698A (zh) | 一种交互方法、装置及电子设备、计算机存储介质 | |
WO2011146070A1 (en) | System and method for reporting data in a computer vision system | |
CN114138121B (zh) | 用户手势识别方法、装置、***、存储介质及计算设备 | |
WO2024012268A1 (zh) | 虚拟操作方法、装置、电子设备及可读存储介质 | |
JP2012238293A (ja) | 入力装置 | |
CN110944139A (zh) | 一种显示控制方法及电子设备 | |
Liang et al. | Turn any display into a touch screen using infrared optical technique | |
TW201539252A (zh) | 觸控系統 | |
CN115529405A (zh) | 前置摄像头的图像显示方法及装置 | |
CN111766947A (zh) | 一种显示方法、装置、可穿戴设备及介质 | |
US9983685B2 (en) | Electronic apparatuses and methods for providing a man-machine interface (MMI) | |
CN103793053A (zh) | 一种便携式终端的手势投影方法及装置 | |
Mishra et al. | Virtual Mouse Input Control using Hand Gestures | |
WO2023246381A1 (zh) | 一种用于光场显示器的交互方法、装置、光场显示器*** | |
Pullan et al. | High Resolution Touch Screen Module | |
CN116841397A (zh) | 操作执行方法、装置、电子设备及可读存储介质 | |
CN113556606A (zh) | 基于手势的电视机控制方法、装置、设备及存储介质 | |
CN116027908A (zh) | 颜色获取方法、装置、电子设备及存储介质 | |
CN114090144A (zh) | 内容显示方法和装置 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 23838760 Country of ref document: EP Kind code of ref document: A1 |