CN117950489A

CN117950489A - Somatosensory interaction method, electronic equipment, system and readable storage medium

Info

Publication number: CN117950489A
Application number: CN202211338653.1A
Authority: CN
Inventors: 陈霄汉; 马春晖; 刘航; 黄磊; 赵杰
Original assignee: Huawei Technologies Co Ltd
Current assignee: Huawei Technologies Co Ltd
Priority date: 2022-10-28
Filing date: 2022-10-28
Publication date: 2024-04-30
Also published as: WO2024088073A1

Abstract

The application provides a somatosensory interaction method, electronic equipment, a system and a readable storage medium, and relates to the technical field of terminals, wherein the method is applied to the electronic equipment provided with a display screen and a camera, and comprises the following steps: when a user performs somatosensory actions, acquiring a first image comprising the user through a camera; determining a first image region in a first image; determining a mapping relation between a first image area and an operation area of a user on a display screen; according to the mapping relation, interactive control operation is executed, and the problem that a user cannot control the whole operation area when the occupied area in the image acquired by the camera is small can be solved.

Description

Somatosensory interaction method, electronic equipment, system and readable storage medium

Technical Field

The present application relates to the field of terminal technologies, and in particular, to a somatosensory interaction method, an electronic device, a system, and a readable storage medium.

Background

The somatosensory interaction technology can enable a user to interact with peripheral electronic equipment through limb actions without using handheld equipment or contacting the electronic equipment, so that the application in the electronic equipment is controlled. The somatosensory interaction technology is widely applied to the fields of motion monitoring, somatosensory games, 3D virtual reality and the like, for example, in the somatosensory game field, an electronic device can acquire a user image through a camera so as to detect limb actions of a user through the image, and therefore the limb actions of the user are mapped to limb actions of a virtual player in a game operation area displayed on a display screen of a terminal device, and the game is controlled.

In the existing somatosensory interaction technology, a mapping relation is preset between a pixel point of an image acquired by a general camera and a pixel point of an operation area displayed on a display screen, and based on the mapping relation, terminal equipment can respond to a corresponding position in the operation area according to the position of a limb of a user in the image, so that somatosensory interaction is realized. However, the distance between the user and the camera of the electronic device may affect the area occupied by the user in the image, and when the distance between the user and the camera is relatively long, the area occupied by the user in the image is relatively small, so that the user cannot control the operation area entirely, thereby affecting the control experience of the user.

Disclosure of Invention

The application provides a somatosensory interaction method, electronic equipment, a system and a readable storage medium, which solve the problem that a user cannot control the whole operation area when the occupied area in an image is smaller to a certain extent.

In order to achieve the above purpose, the application adopts the following technical scheme:

In a first aspect, the present application provides a somatosensory interaction method, applied to an electronic device, where the electronic device includes a display screen and a camera, the method includes:

When a user performs somatosensory actions, acquiring a first image comprising the user through a camera; determining a first image region in a first image; determining a mapping relation between a first image area and an operation area of a user on a display screen; and executing interactive control operation according to the mapping relation.

According to the somatosensory interaction method provided by the application, the first image when the user executes the somatosensory action is obtained through the camera of the electronic device, the electronic device can determine the first image area when the user executes the somatosensory action according to the area occupied by the user in the first image, and determine the mapping relation between the pixel points in the first image area of the user and the pixel points in the operation area corresponding to the user and displayed on the display screen, and based on the mapping relation, the electronic device can respond to the operation area on the display screen according to the position of the limb of the user in the first image area, so that the whole operation area can be controlled when the user executes the somatosensory action in the first image area, and the control experience of the user is improved.

In a possible implementation manner of the first aspect, determining a first image area in the first image includes:

detecting first skeletal node data of a user in a first image; and determining a first image area according to the first bone node data and the preset initial bone node data.

In a possible implementation manner of the first aspect, the method further includes: when a user executes a first preset action, acquiring an initial image comprising the user through a camera; initial skeletal node data of a user in an initial image is detected.

In a possible implementation manner of the first aspect, determining the first image area according to the first bone node data and the preset initial bone node data includes:

Determining a first length of a first limb and a second length of a first reference limb in the initial image according to the initial bone node data; determining a third length of the first reference limb in the first image from the first bone node data; determining the maximum length of the first limb in the first image according to the ratio of the first length to the second length and the third length; the first image area is determined based on the maximum length.

In a possible implementation manner of the first aspect, determining a mapping relationship between the first image area and an operation area of the user on the display screen includes:

Determining an indication line corresponding to a first limb in the first image according to the first bone node data; and determining a mapping area corresponding to the first image area in the operation area according to the indication line and the maximum length.

Based on the possible mode, the electronic device can determine the scaling of the maximum length of the first limb of the user in the first image relative to the initial image through the first bone node data and the second bone node data, so that the first image area is determined, the first image area indicated by the first limb of the user for controlling the area is mapped into the operation area, the user can control the whole corresponding operation area when performing somatosensory actions through the first limb, and the control experience of the user is improved.

In a possible implementation manner of the first aspect, the method further includes:

Determining an initial image area in the initial image according to the initial bone node data;

Determining a first image area according to the first bone node data and preset initial bone node data, including: a first image region is determined based on the first bone node data, the initial bone node data, and the initial image region.

In a possible implementation manner of the first aspect, determining an initial image area in the initial image according to the initial bone node data includes:

Determining a fifth length of the second limb in the initial image and an initial origin of the initial image area according to the initial bone node data; determining an initial active length according to the fifth length; an initial image region is determined based on the initial active length and the initial origin.

In a possible implementation manner of the first aspect, determining the first image area according to the first bone node data, the initial bone node data and the initial image area includes:

Determining a sixth length of the second reference limb in the initial image according to the initial bone node data; determining a seventh length of the second reference limb in the first image and a first origin of the first image region according to the first bone node data; determining a first active length according to the initial active length, the sixth length and the seventh length; a first image region is determined based on the first active length and the first origin.

In a possible implementation manner of the first aspect, the first image area is a square area with a side length of a first active length in the first image, and the first origin is a center point of the first image area.

Based on the possible mode, the electronic device can determine the scaling of the limbs of the user in the first image relative to the initial image through the first bone node data and the second bone node data, so that a first image area surrounded by the maximum moving range of the user in each direction of the image coordinate system when the somatosensory action is executed is determined, the first image area is mapped to the whole operation area, the electronic device controls the corresponding operation area according to the position of the limbs of the user in the first image area, and the control experience of the user is improved.

In a possible implementation manner of the first aspect, the method further includes: determining a first indication frame according to a preset first scaling and the size of a first image area; a first indication frame is displayed in an image display area on the display screen, and the first indication frame is used for representing the first image area.

In a possible implementation manner of the first aspect, the method further includes: when the user executes a second preset action, acquiring an initial image sequence of the user through the camera; detecting the maximum displacement of a user in the target direction in the initial image sequence; determining an initial active length according to the maximum displacement; determining an initial origin of the initial image region according to the initial bone node data; an initial image region is determined based on the initial active length and the initial origin.

In a possible implementation manner of the first aspect, the initial image area is an initial image area with a length of an initial active length in the initial image, and the initial origin is a side end point or a center point of the initial image area.

In a possible implementation manner of the first aspect, the operation area includes n sub-operation areas, where n is an integer greater than or equal to 2;

determining a mapping relationship between a first image area and an operation area of a user on a display screen, including:

Dividing the first image area into n first sub-image areas corresponding to the n sub-operation areas one by one; and determining the mapping relation between each first sub-image area and the corresponding sub-operation area according to the lengths of the first sub-image areas and the lengths of the sub-operation areas.

In the optional mode, the electronic device can determine the maximum displacement of the user in the target direction in the first image by detecting the first bone node data in the first image acquired by the camera in real time, so as to determine the length of each first sub-image area in the first image area of the user, and timely adjust the mapping relation between each first sub-image area and the corresponding sub-operation area.

In a possible implementation manner of the first aspect, the method further includes: determining a second indication frame according to a preset second scaling and the size of the first image area; and displaying a second indication frame in the image display area on the display screen, wherein the second indication frame is used for indicating the first image area.

Based on the second scale, the image display area on the display screen may display the first skeletal node data and the second indicator frame to guide the user to jump a fixed distance in the target direction during the somatosensory interaction by the relative position between the displayed first skeletal node data and the second indicator frame.

In a possible implementation manner of the first aspect, the method further includes: identifying an obstacle in the first image; determining a first distance between the obstacle and the first image area; if the first distance is smaller than the preset distance, the collision prompt message is played.

Based on the possible implementation manner, the electronic device can play collision prompt information by detecting the relative size of the first distance between the first image area of the user and the obstacle in the first image and the preset distance, so that collision early warning is provided for the user, the user is prevented from colliding with the obstacle in the environment when performing somatosensory interaction with the electronic device, and the control experience of the user is improved.

In one possible implementation of the first aspect, the first image comprises at least two users; and when the second distance between the first image areas of the two users is detected to be smaller than the preset distance, playing collision prompt information.

Based on the possible implementation mode, the electronic equipment can play collision prompt information through detecting the relative size of the second distance between the first image areas of any two users in the first image and the preset distance, can provide collision early warning for the users, avoids collision between the users and the electronic equipment when the users and the electronic equipment perform somatosensory interaction simultaneously, and improves the control experience of the users.

In a possible implementation manner of the first aspect, playing collision prompt information includes: the method comprises the steps of voice prompt, text prompt display on a display screen or flash display of a first indication frame or a second indication frame on the display screen.

In a second aspect, the present application provides an electronic device comprising: a processor for running a computer program stored in a memory to implement the method in any one of the possible implementations of the first aspect.

In a third aspect, the application provides a chip system comprising a processor executing a computer program stored in a memory to implement the method of any one of the possible implementations of the first aspect.

In a fourth aspect, the present application provides a computer readable storage medium storing a computer program which when executed by a processor implements the method of any one of the possible implementations of the first aspect.

In a fifth aspect, the application provides a computer program product for, when run on an electronic device, causing the electronic device to perform the method of any one of the possible implementations of the first aspect.

Technical effects of the second to fifth aspects provided by the present application may be referred to technical effects of each possible implementation manner of the first aspect, which are not described herein.

Drawings

Fig. 1 is a schematic structural diagram of an electronic device according to an embodiment of the present application;

fig. 2 is a schematic software structure of an electronic device according to an embodiment of the present application;

Fig. 3 is a schematic diagram of an application scenario one of somatosensory interaction between an electronic device and a user according to an embodiment of the present application;

Fig. 4 is a schematic diagram of an application scenario two of somatosensory interaction between an electronic device and a user according to an embodiment of the present application;

FIG. 5 is a flowchart of a somatosensory interaction method according to an embodiment of the present application;

FIG. 6 is a schematic diagram of a bone node in a first image according to an embodiment of the present application;

Fig. 7 is a schematic diagram of a display interface of an electronic device according to an embodiment of the present application;

fig. 8 is a schematic diagram of an initial image acquired by a camera according to an embodiment of the present application;

fig. 9 is a schematic diagram two of an initial image acquired by a camera according to an embodiment of the present application;

Fig. 10 is a schematic diagram III of an initial image obtained by a camera according to an embodiment of the present application;

FIG. 11 is a flowchart illustrating a method for determining a first image area according to an embodiment of the present application;

FIG. 12 is a schematic diagram showing a first embodiment of determining a mapping relationship between a first image area and an operation area;

FIG. 13 is a flowchart II of a method for determining a first image area according to an embodiment of the present application;

fig. 14 is a schematic diagram ii of determining a mapping relationship between a first image area and an operation area according to an embodiment of the present application;

fig. 15 is a second schematic diagram of a display interface of an electronic device according to an embodiment of the present application;

fig. 16 is a schematic diagram III of a display interface of an electronic device according to an embodiment of the present application;

FIG. 17 is a third schematic diagram for determining a mapping relationship between a first image area and an operation area according to an embodiment of the present application;

fig. 18 is a schematic diagram of a display interface of an electronic device according to an embodiment of the present application;

FIG. 19 is a diagram showing a mapping relationship between a first image area and an operation area according to an embodiment of the present application;

fig. 20 is a schematic diagram of a display interface of an electronic device according to an embodiment of the present application;

fig. 21 is a schematic diagram of a display interface of an electronic device according to an embodiment of the present application;

Fig. 22 is a flowchart of another somatosensory interaction method according to an embodiment of the present application.

Detailed Description

The somatosensory interaction technology can enable a user to interact with peripheral electronic equipment through limb actions without using handheld equipment or contacting the electronic equipment, so that the application in the electronic equipment is controlled. Somatosensory interaction technology is widely applied to the fields of motion monitoring, somatosensory games, 3D virtual reality and the like.

In the process of somatosensory interaction between the electronic equipment and a user based on the existing somatosensory interaction technology, generally, a camera is used for acquiring an image when the user executes somatosensory action, a mapping relation is preset between pixel points in the image and pixel points of an operation area displayed on a display screen, and based on the mapping relation, the electronic equipment can respond to corresponding positions in the operation area according to the positions of limbs of the user in the image, so that the somatosensory interaction is realized. The distance between the user and the camera arranged in the electronic equipment can influence the size of the area occupied by the user in the image acquired by the camera, specifically, the farther the distance between the user and the camera of the electronic equipment is, the smaller the area occupied by the user in the acquired image is, the closer the distance between the user and the camera of the electronic equipment is, and the larger the area occupied by the user in the acquired image is. If the distance between the user and the camera is far in the process of somatosensory interaction between the user and the electronic equipment, the electronic equipment still maps the pixel points of the limbs of the user in the image to the pixel points in the operation area based on the fixed mapping relation, so that the user cannot control part of the area in the operation area, and the control experience of the user is affected.

In addition, the sizes of the different users are different, and accordingly, the areas occupied by the different users in the images acquired by the cameras of the electronic equipment are also different, and when a plurality of users with different sizes control the same operation area, the users with smaller occupied areas in the images cannot operate the whole operation area corresponding to the user because of the existing method for mapping the pixel points in the images containing the motion of the user body to the pixel points in the operation area in a one-to-one correspondence manner.

Therefore, aiming at the problem that the whole operation area cannot be controlled when the area occupied by the user in the image shot by the camera of the electronic device is small by the existing somatosensory interaction technology, the application provides a somatosensory interaction method, electronic device, system and readable storage medium.

The following describes the technical solution in the embodiment of the present application with reference to the drawings and related embodiments in the embodiment of the present application. In the description of embodiments of the application, the terminology used in the embodiments below is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in the specification of the present application and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include, for example, "one or more" such forms of expression, unless the context clearly indicates to the contrary. It should also be understood that in the following embodiments of the present application, "at least one", "one or more" means one or more than two (including two). The term "and/or" is used to describe an association relationship of associated objects, meaning that there may be three relationships; for example, a and/or B may represent: a alone, a and B together, and B alone, wherein A, B may be singular or plural. The character "/" generally indicates that the context-dependent object is an "or" relationship.

Reference in the specification to "one embodiment" or "some embodiments" or the like means that a particular feature, structure, or characteristic described in connection with the embodiment is included in one or more embodiments of the application. Thus, appearances of the phrases "in one embodiment," "in some embodiments," "in other embodiments," and the like in the specification are not necessarily all referring to the same embodiment, but mean "one or more but not all embodiments" unless expressly specified otherwise. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless expressly specified otherwise. The term "coupled" includes both direct and indirect connections, unless stated otherwise. The terms "first," "second," and the like are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated.

In embodiments of the application, words such as "exemplary" or "such as" are used to mean serving as an example, instance, or illustration. Any embodiment or design described herein as "exemplary" or "for example" is not necessarily to be construed as preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "such as" is intended to present related concepts in a concrete fashion.

The somatosensory interaction method provided by the application is applied to electronic equipment provided with a display screen and a camera, wherein the electronic equipment can be a smart television, a smart screen, a mobile phone, a tablet personal computer, a notebook computer, a desktop computer, an augmented reality (augmented reality, AR)/Virtual Reality (VR) equipment, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (personal DIGITAL ASSISTANT, PDA) and the like.

Referring to fig. 1, a schematic structural diagram of an electronic device 100 according to the present application is provided. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 131, a universal serial bus (universal serial bus, USB) interface 130, a charge management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, keys 190, a motor 191, an indicator 192, a camera 193, a display 194, and a subscriber identity module (subscriber identification module, SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyro sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, and the like.

It should be understood that the illustrated structure of the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the application, electronic device 100 may include more or fewer components than shown, or certain components may be combined, or certain components may be split, or different arrangements of components. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

For example, when the electronic device 100 is a mobile phone or a tablet computer, all the components in the illustration may be included, or only some of the components in the illustration may be included.

The processor 110 may include one or more processing units, such as: the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor (IMAGE SIGNAL processor, ISP), a controller, a memory, a video codec, a digital signal processor (DIGITAL SIGNAL processor, DSP), a baseband processor, and/or a neural Network Processor (NPU), etc. Wherein the different processing units may be separate devices or may be integrated in one or more processors.

The controller may be a neural hub and a command center of the electronic device 100, among others. The controller can generate operation control signals according to the instruction operation codes and the time sequence signals to finish the control of instruction fetching and instruction execution.

A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may hold instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to reuse the instruction or data, it may be called directly from memory. Repeated accesses are avoided and the latency of the processor 110 is reduced, thereby improving the efficiency of the system.

In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an integrated circuit (inter-INTEGRATED CIRCUIT, I2C) interface, an integrated circuit built-in audio (inter-INTEGRATED CIRCUIT SOUND, I2S) interface, a pulse code modulation (pulse code modulation, PCM) interface, a universal asynchronous receiver transmitter (universal asynchronous receiver/transmitter, UART) interface, a mobile industry processor interface (mobile industry processor interface, MIPI), a general-purpose input/output (GPIO) interface, a subscriber identity module (subscriber identity module, SIM) interface, and/or a universal serial bus (universal serial bus, USB) interface, among others.

The I2C interface is a bi-directional synchronous serial bus comprising a serial data line (SERIAL DATA LINE, SDA) and a serial clock line (derail clock line, SCL). In some embodiments, the processor 110 may contain multiple sets of I2C buses. The processor 110 may be coupled to the touch sensor 180K, charger, flash, camera 193, etc., respectively, through different I2C bus interfaces. For example: the processor 110 may be coupled to the touch sensor 180K through an I2C interface, such that the processor 110 communicates with the touch sensor 180K through an I1C bus interface to implement a touch function of the electronic device 100.

The I1S interface may be used for audio communication. In some embodiments, the processor 110 may contain multiple sets of I2S buses. The processor 110 may be coupled to the audio module 170 via an I1S bus to enable communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 may communicate audio signals to the wireless communication module 160 through an I1S interface.

PCM interfaces may also be used for audio communication to sample, quantize and encode analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 may be coupled through a PCM bus interface.

In some embodiments, the audio module 170 may also communicate audio signals to the wireless communication module 160 through a PCM interface. Both the I2S interface and the PCM interface may be used for audio communication.

The UART interface is a universal serial data bus for asynchronous communications. The bus may be a bi-directional communication bus. It converts the data to be transmitted between and among parallel communications.

In some embodiments, a UART interface is typically used to connect the processor 110 with the wireless communication module 160. For example: the processor 110 communicates with a bluetooth module in the wireless communication module 160 through a UART interface to implement a bluetooth function. In some embodiments, the audio module 170 may transmit an audio signal to the wireless communication module 160 through a UART interface, to implement a function of playing music through a bluetooth headset.

The MIPI interface may be used to connect the processor 110 to peripheral devices such as a display 194, a camera 193, and the like. The MIPI interfaces include camera serial interfaces (CAMERA SERIAL INTERFACE, CSI), display serial interfaces (DISPLAY SERIAL INTERFACE, DSI), and the like. In some embodiments, processor 110 and camera 193 communicate through a CSI interface to implement the photographing functions of electronic device 100. The processor 110 and the display 194 communicate via a DSI interface to implement the display functionality of the electronic device 100.

The GPIO interface may be configured by software. The GPIO interface may be configured as a control signal or as a data signal. In some embodiments, a GPIO interface may be used to connect the processor 110 with the camera 193, the display 194, the wireless communication module 160, the audio module 170, the sensor module 180, and the like. The GPIO interface may also be configured as an I2C interface, an I2S interface, a UART interface, an MIPI interface, etc.

The USB interface 130 is an interface conforming to the USB standard specification, and may specifically be a Mini USB interface, a Micro USB interface, a USB Type C interface, or the like. The USB interface 130 may be used to connect a charger to charge the electronic device 100, and may also be used to transfer data between the electronic device 100 and a peripheral device. And can also be used for connecting with a headset, and playing audio through the headset. The interface may also be used to connect other electronic devices, such as AR devices, etc.

It should be understood that the interfacing relationship between the modules illustrated in the embodiments of the present application is only illustrative, and is not meant to limit the structure of the electronic device 100. In other embodiments of the present application, the electronic device 100 may also employ different interfacing manners in the above embodiments, or a combination of multiple interfacing manners.

The charge management module 140 is configured to receive a charge input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charge management module 140 may receive a charging input of a wired charger through the USB interface 130. In some wireless charging embodiments, the charge management module 140 may receive wireless charging input through a wireless charging coil of the electronic device 100. The charging management module 140 may also supply power to the electronic device through the power management module 141 while charging the battery 142.

The power management module 141 is used for connecting the battery 142, and the charge management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and/or the charge management module 140 to power the processor 110, the internal memory 131, the external memory interface 120, the display 194, the camera 193, the wireless communication module 160, and the like. The power management module 141 may also be configured to monitor battery capacity, battery cycle number, battery health (leakage, impedance) and other parameters.

In other embodiments, the power management module 141 may also be provided in the processor 110. In other embodiments, the power management module 141 and the charge management module 140 may be disposed in the same device.

The wireless communication function of the electronic device 100 may be implemented by the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, a modem processor, a baseband processor, and the like.

The antennas 1 and 2 are used for transmitting and receiving electromagnetic wave signals. Each antenna in the electronic device 100 may be used to cover a single or multiple communication bands. Different antennas may also be multiplexed to improve the utilization of the antennas. For example: the antenna 1 may be multiplexed into a diversity antenna of a wireless local area network. In other embodiments, the antenna may be used in conjunction with a tuning switch.

The mobile communication module 150 may provide a solution for wireless communication including 2G/3G/4G/5G, etc., applied to the electronic device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (low noise amplifier, LNA), etc. The mobile communication module 150 may receive electromagnetic waves from the antenna 1, perform processes such as filtering, amplifying, and the like on the received electromagnetic waves, and transmit the processed electromagnetic waves to the modem processor for demodulation. The mobile communication module 150 can amplify the signal modulated by the modem processor, and convert the signal into electromagnetic waves through the antenna 1 to radiate.

In some embodiments, at least some of the functional modules of the mobile communication module 150 may be disposed in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 may be provided in the same device as at least some of the modules of the processor 110.

The modem processor may include a modulator and a demodulator. The modulator is used for modulating the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used for demodulating the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low frequency baseband signal to the baseband processor for processing. The low frequency baseband signal is processed by the baseband processor and then transferred to the application processor. The application processor outputs sound signals through an audio device (not limited to the speaker 170A, the receiver 170B, etc.), or displays images or video through the display screen 194. In some embodiments, the modem processor may be a stand-alone device. In other embodiments, the modem processor may be provided in the same device as the mobile communication module 150 or other functional module, independent of the processor 110.

The wireless communication module 160 may provide solutions for wireless communication including wireless local area network (wireless local area networks, WLAN) (e.g., wireless fidelity (WIRELESS FIDELITY, wi-Fi) network), bluetooth (BT), global navigation satellite system (global navigation SATELLITE SYSTEM, GNSS), frequency modulation (frequency modulation, FM), near field communication (NEAR FIELD communication, NFC), infrared (IR), etc., applied to the electronic device 100. The wireless communication module 160 may be one or more devices that integrate at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the electromagnetic wave signals, filters the electromagnetic wave signals, and transmits the processed signals to the processor 110. The wireless communication module 160 may also receive a signal to be transmitted from the processor 110, frequency modulate it, amplify it, and convert it to electromagnetic waves for radiation via the antenna 2.

In some embodiments, antenna 1 and mobile communication module 150 of electronic device 100 are coupled, and antenna 2 and wireless communication module 160 are coupled, such that electronic device 100 may communicate with a network and other devices through wireless communication techniques. The wireless communication techniques can include a global system for mobile communications (global system for mobile communications, GSM), general packet radio service (GENERAL PACKET radio service, GPRS), code division multiple access (code division multiple access, CDMA), wideband code division multiple access (wideband code division multiple access, WCDMA), time division code division multiple access (time-division code division multiple access, TD-SCDMA), long term evolution (long term evolution, LTE), BT, GNSS, WLAN, NFC, FM, and/or IR techniques, among others. The GNSS may include a global satellite positioning system (global positioning system, GPS), a global navigation satellite system (global navigation SATELLITE SYSTEM, GLONASS), a beidou satellite navigation system (beidou navigation SATELLITE SYSTEM, BDS), a quasi zenith satellite system (quasi-zenith SATELLITE SYSTEM, QZSS) and/or a satellite based augmentation system (SATELLITE BASED AUGMENTATION SYSTEMS, SBAS).

The electronic device 100 implements display functions through a GPU, a display screen 194, an application processor, and the like. The GPU is a microprocessor for image processing, and is connected to the display 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

The display screen 194 is used to display images, videos, and the like. Such as a first illumination pattern, a second illumination pattern, etc., in embodiments of the present application. The display 194 includes a display panel. The display panel may employ a Liquid Crystal Display (LCD) CRYSTAL DISPLAY, an organic light-emitting diode (OLED), an active-matrix organic LIGHT EMITTING diode (AMOLED), a flexible light-emitting diode (FLED), miniled, microLed, micro-oLed, a quantum dot LIGHT EMITTING diode (QLED), or the like. In some embodiments, the electronic device 100 may include 1 or N display screens 194, N being a positive integer greater than 1.

The electronic device 100 may implement photographing functions through an ISP, a camera 193, a video codec, a GPU, a display screen 194, an application processor, and the like.

The ISP is used to process data fed back by the camera 193. For example, when photographing, the shutter is opened, light is transmitted to the camera photosensitive element through the lens, the optical signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing, so that the electrical signal is converted into an image visible to naked eyes. ISP can also perform algorithm optimization on noise and brightness of the image. The ISP can also optimize parameters such as exposure, color temperature and the like of a shooting scene. In some embodiments, the ISP may be provided in the camera 193.

The camera 193 is used to capture still images or video. The object generates an optical image through the lens and projects the optical image onto the photosensitive element. The focal length of the lens can be used to represent the viewing range of the camera, and a small focal length Duan Yue of the lens represents a larger viewing range of the lens. The photosensitive element may be a charge coupled device (charge coupled device, CCD) or a Complementary Metal Oxide Semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, which is then transferred to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV, or the like format.

In the present application, the electronic device 100 may include 2 or more cameras 193 of the focal length.

The digital signal processor is used for processing digital signals, and can process other digital signals besides digital image signals. For example, when the electronic device 100 selects a frequency bin, the digital signal processor is used to fourier transform the frequency bin energy, or the like.

Video codecs are used to compress or decompress digital video. The electronic device 100 may support one or more video codecs. In this way, the electronic device 100 may play or record video in a variety of encoding formats, such as: dynamic picture experts group (moving picture experts group, MPEG) 1, MPEG3, MPEG4, etc.

The NPU is a neural-network (NN) computing processor, and can rapidly process input information by referencing a biological neural network structure, for example, referencing a transmission mode between human brain neurons, and can also continuously perform self-learning. Applications such as intelligent awareness of the electronic device 100 may be implemented through the NPU, for example: image recognition, face recognition, speech recognition, text understanding, etc.

In an embodiment of the present application, the NPU or other processor may be configured to perform operations such as analysis and processing on images in video stored by the electronic device 100.

The external memory interface 120 may be used to connect an external memory card, such as a Micro SD card, to enable expansion of the memory capabilities of the electronic device 100. The external memory card communicates with the processor 110 through an external memory interface 120 to implement data storage functions. For example, files such as music, video, etc. are stored in an external memory card.

The internal memory 131 may be used to store computer executable program code that includes instructions. The processor 110 executes various functional applications of the electronic device 100 and data processing by executing instructions stored in the internal memory 131. The internal memory 131 may include a storage program area and a storage data area. The storage program area may store application programs (such as a sound playing function, an image playing function, etc.) required for at least one function of the operating system. The storage data area may store data (e.g., audio data, phonebook, etc.) created during use of the electronic device 100.

In addition, the internal memory 131 may include a high-speed random access memory, and may further include a nonvolatile memory such as at least one magnetic disk storage device, a flash memory device, a universal flash memory (universal flash storage, UFS), and the like.

The electronic device 100 may implement audio functions through an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, an application processor, and the like.

The audio module 170 is used to convert digital audio signals to analog audio signal outputs and also to convert analog audio inputs to digital audio signals. The audio module 170 may also be used to encode and decode audio signals. In some embodiments, the audio module 170 may be disposed in the processor 110, or a portion of the functional modules of the audio module 170 may be disposed in the processor 110.

The speaker 170A, also referred to as a "horn," is used to convert audio electrical signals into sound signals. The electronic device 100 may listen to music through the speaker 170A or to a hands-free conversation, for example, the speaker may play the comparison analysis provided by embodiments of the present application.

A receiver 170B, also referred to as a "earpiece", is used to convert the audio electrical signal into a sound signal. When electronic device 100 is answering a telephone call or voice message, voice may be received by placing receiver 170B in close proximity to the human ear.

Microphone 170C, also referred to as a "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or transmitting voice information, the user can sound near the microphone 170C through the mouth, inputting a sound signal to the microphone 170C. The electronic device 100 may be provided with at least one microphone 170C. In other embodiments, the electronic device 100 may be provided with two microphones 170C, and may implement a noise reduction function in addition to collecting sound signals. In other embodiments, the electronic device 100 may also be provided with three, four, or more microphones 170C to enable collection of sound signals, noise reduction, identification of sound sources, directional recording functions, etc.

The earphone interface 170D is used to connect a wired earphone. The headset interface 170D may be a USB interface 130 or a 3.5mm open mobile electronic device platform (open mobile terminal platform, OMTP) standard interface, a american cellular telecommunications industry association (cellular telecommunications industry association of the USA, CTIA) standard interface.

The pressure sensor 180A is used to sense a pressure signal, and may convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A may be disposed on the display screen 194. The pressure sensor 180A is of various types, such as a resistive pressure sensor, an inductive pressure sensor, a capacitive pressure sensor, and the like. The capacitive pressure sensor may be a capacitive pressure sensor comprising at least two parallel plates with conductive material. The capacitance between the electrodes changes when a force is applied to the pressure sensor 180A. The electronic device 100 determines the strength of the pressure from the change in capacitance. When a touch operation is applied to the display screen 194, the electronic apparatus 100 detects the touch operation intensity according to the pressure sensor 180A. The electronic device 100 may also calculate the location of the touch based on the detection signal of the pressure sensor 180A. In some embodiments, touch operations that act on the same touch location, but at different touch operation strengths, may correspond to different operation instructions.

The gyro sensor 180B may be used to determine a motion gesture of the electronic device 100. In some embodiments, the angular velocity of electronic device 100 about three axes (i.e., x, y, and z axes) may be determined by gyro sensor 180B. The gyro sensor 180B may be used for photographing anti-shake. For example, when the shutter is pressed, the gyro sensor 180B detects the shake angle of the electronic device 100, calculates the distance to be compensated by the lens module according to the angle, and makes the lens counteract the shake of the electronic device 100 through the reverse motion, so as to realize anti-shake. The gyro sensor 180B may also be used for navigating, somatosensory game scenes.

The air pressure sensor 180C is used to measure air pressure. In some embodiments, electronic device 100 calculates altitude from barometric pressure values measured by barometric pressure sensor 180C, aiding in positioning and navigation.

The magnetic sensor 180D includes a hall sensor. The electronic device 100 may detect the opening and closing of the flip cover using the magnetic sensor 180D. In some embodiments, when the electronic device 100 is a flip machine, the electronic device 100 may detect the opening and closing of the flip according to the magnetic sensor 180D. And then according to the detected opening and closing state of the leather sheath or the opening and closing state of the flip, the characteristics of automatic unlocking of the flip and the like are set.

The acceleration sensor 180E may detect the magnitude of acceleration of the electronic device 100 in various directions (typically three axes). The magnitude and direction of gravity may be detected when the electronic device 100 is stationary. The electronic equipment gesture recognition method can also be used for recognizing the gesture of the electronic equipment, and is applied to horizontal and vertical screen switching, pedometers and other applications.

A distance sensor 180F for measuring a distance. The electronic device 100 may measure the distance by infrared or laser. In some embodiments, the electronic device 100 may range using the distance sensor 180F to achieve quick focus.

The proximity light sensor 180G may include, for example, a Light Emitting Diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The electronic device 100 emits infrared light outward through the light emitting diode. The electronic device 100 detects infrared reflected light from nearby objects using a photodiode. When sufficient reflected light is detected, it may be determined that there is an object in the vicinity of the electronic device 100. When insufficient reflected light is detected, the electronic device 100 may determine that there is no object in the vicinity of the electronic device 100. The electronic device 100 can detect that the user holds the electronic device 100 close to the ear by using the proximity light sensor 180G, so as to automatically extinguish the screen for the purpose of saving power. The proximity light sensor 180G may also be used in holster mode, pocket mode to automatically unlock and lock the screen.

The ambient light sensor 180L is used to sense ambient light level. The electronic device 100 may adaptively adjust the brightness of the display 194 based on the perceived ambient light level. The ambient light sensor 180L may also be used to automatically adjust white balance when taking a photograph. Ambient light sensor 180L may also cooperate with proximity light sensor 180G to detect whether electronic device 100 is in a pocket to prevent false touches.

The fingerprint sensor 180H is used to collect a fingerprint. The electronic device 100 may utilize the collected fingerprint feature to unlock the fingerprint, access the application lock, photograph the fingerprint, answer the incoming call, etc.

The temperature sensor 180J is for detecting temperature. In some embodiments, the electronic device 100 performs a temperature processing strategy using the temperature detected by the temperature sensor 180J. For example, when the temperature reported by temperature sensor 180J exceeds a threshold, electronic device 100 performs a reduction in the performance of a processor located in the vicinity of temperature sensor 180J in order to reduce power consumption to implement thermal protection. In other embodiments, when the temperature is below another threshold, the electronic device 100 heats the battery 142 to avoid the low temperature causing the electronic device 100 to be abnormally shut down. In other embodiments, when the temperature is below a further threshold, the electronic device 100 performs boosting of the output voltage of the battery 142 to avoid abnormal shutdown caused by low temperatures.

The touch sensor 180K, also referred to as a "touch panel". The touch sensor 180K may be disposed on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, which is also called a "touch screen". The touch sensor 180K is for detecting a touch operation acting thereon or thereabout. The touch sensor may communicate the detected touch operation to the application processor to determine the touch event type. Visual output related to touch operations may be provided through the display 194. In other embodiments, the touch sensor 180K may also be disposed on the surface of the electronic device 100 at a different location than the display 194.

The bone conduction sensor 180M may acquire a vibration signal. In some embodiments, bone conduction sensor 180M may acquire a vibration signal of a human vocal tract vibrating bone pieces. The bone conduction sensor 180M may also contact the pulse of the human body to receive the blood pressure pulsation signal.

In some embodiments, bone conduction sensor 180M may also be provided in a headset, in combination with an osteoinductive headset. The audio module 170 may parse out a voice signal based on the vibration signal of the vocal part vibration bone piece obtained by the bone conduction sensor 180M, and implement a voice function. The application processor can analyze heart rate information based on the blood pressure beat signals acquired by the bone conduction sensor 180M, so that a heart rate detection function is realized.

The keys 190 include a power-on key, a volume key, etc. The keys 190 may be mechanical keys. Or may be a touch key. The electronic device 100 may receive key inputs, generating key signal inputs related to user settings and function controls of the electronic device 100.

The motor 191 may generate a vibration cue. The motor 191 may be used for incoming call vibration alerting as well as for touch vibration feedback. For example, touch operations acting on different applications (e.g., photographing, audio playing, etc.) may correspond to different vibration feedback effects. The motor 191 may also correspond to different vibration feedback effects by touching different areas of the display screen 194. Different application scenarios (such as time reminding, receiving information, alarm clock, game, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect may also support customization.

The indicator 192 may be an indicator light, may be used to indicate a state of charge, a change in charge, a message indicating a missed call, a notification, etc.

The SIM card interface 195 is used to connect a SIM card. The SIM card may be inserted into the SIM card interface 195, or removed from the SIM card interface 195 to enable contact and separation with the electronic device 100. The electronic device 100 may support 1 or N SIM card interfaces, N being a positive integer greater than 1. The SIM card interface 195 may support Nano SIM cards, micro SIM cards, and the like. The same SIM card interface 195 may be used to insert multiple cards simultaneously. The types of the plurality of cards may be the same or different. The SIM card interface 195 may also be compatible with different types of SIM cards. The SIM card interface 195 may also be compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to realize functions such as communication and data communication. In some embodiments, the electronic device 100 employs esims, i.e.: an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

Referring to fig. 2, a software structure of an electronic device according to an embodiment of the application is shown. The operating system in the electronic device may be an Android system, a microsoft Windows system (Windows), an apple mobile operating system (iOS) or a hong system (Harmony OS), etc. Here, an operation system of an electronic device will be described as an example of a hong system.

In some embodiments, the hong-and-Monte-Care system may be divided into four layers, including a kernel layer, a system services layer, a framework layer, and an application layer, with the layers communicating via software interfaces.

As shown in fig. 2, the kernel layer includes a kernel abstraction layer (Kernel Abstract Layer, KAL) and a driver subsystem. The KAL comprises a plurality of kernels, such as a Linux Kernel of a Linux system, a lightweight internet of things system Kernel LiteOS and the like. The drive subsystem may then include a hardware drive framework (HARDWARE DRIVER Foundation, HDF). The hardware driver framework can provide unified peripheral access capability and driver development and management framework. The kernel layer of the multi-kernel can select corresponding kernels for processing according to the requirements of the system.

The system service layer is a core capability set of the hong Monte system, and provides service for application programs through the framework layer. The layer may include a set of system basic capability subsystems, a set of base software service subsystems, a set of enhanced software service subsystems, and a set of hardware service subsystems.

The system basic capability subsystem set provides basic capabilities for running, scheduling, migrating, etc. operations of distributed applications on devices of the hong system. Subsystems such as distributed soft buses, distributed data management, distributed task scheduling, ark multi-lingual runtime, public base library, multi-modal input, graphics, security, artificial intelligence (ARTIFICIAL INTELLIGENCE, AI), user program framework, etc., may be included. Wherein, the ark multi-language runtime provides the C or C++ or JavaScript (JS) multi-language runtime and basic system class library, and can also provide the runtime for Java programs (i.e. application programs or parts of framework layers developed in Java language) which are statically built by using the ark compiler.

The basic set of software services subsystems provides common, generic software services for the hong system. Subsystems such as event notification, telephony, multimedia, design For X (DFX), MSDP & DV, etc. may be included.

The enhanced software services subsystem set provides differentiated capability enhanced software services for different devices for the hong Monte System. May include smart screen proprietary services, wearable proprietary services, internet of things (Internet of Things, ioT) proprietary services subsystem components.

The hardware service subsystem set provides hardware services for the hong system. Subsystems such as location services, biometric identification, wearable proprietary hardware services, ioT proprietary hardware services, and the like may be included.

The framework layer provides Java, C, C++, JS and other multi-language User program frameworks and capability (Ability) frameworks for the HongMong system application development, two User Interface (UI) frameworks (comprising Java UI frameworks applicable to Java languages and JS UI frameworks applicable to JS languages) and multi-language framework application program interfaces (Application Programming Interface, API) with various software and hardware services open to the outside. The APIs supported by the hong system devices will also vary depending on the degree of componentization clipping of the system.

The application layer includes system applications and third party applications (or referred to as extension applications). The system applications may include applications installed by default for electronic devices such as desktops, control boxes, settings, telephones, and the like. The extended application may be an application designed by the manufacturer of the electronic device, such as an application program for an electronic device manager, a switch migration, a note, weather, etc. While third party non-system applications may be developed by other vendors, applications may be run in hong-and-Monte systems, such as gaming, navigation, social or shopping applications.

Providing the ability to run tasks in the background and unified data access abstraction. The PA primarily provides support for the FA, for example, as a background service providing computing power, or as a data repository providing data access capability. The application developed based on the FA or the PA can realize specific service functions, support cross-device scheduling and distribution, and provide consistent and efficient application experience for users.

Hardware interaction and resource sharing can be realized among a plurality of electronic devices running the buddha system through a distributed soft bus, distributed device virtualization, distributed data management and distributed task scheduling.

The embodiment of the application provides a somatosensory interaction method, which is exemplified by taking electronic equipment as an intelligent electricity as an example and combining an application scene of somatosensory interaction between the electronic equipment and a user.

Referring to fig. 3, a schematic diagram of a somatosensory interaction application scenario one is shown. The user may face the electronic device and control various applications (apps) displayed on the display screen of the electronic device through his own limb. The electronic equipment can guide the user to move by running application programs such as fitness class, game class and the like, and can also enable the user to browse commodities, share information such as pictures, video and the like by simply operating APP such as commodity class, video and the like through turning pages, sliding, confirming and the like by hand, so that somatosensory interaction is carried out between the electronic equipment and the user.

Taking an application scenario shown in fig. 3 as an example, a display screen of the electronic device includes an image display area and an operation area corresponding to a user. Specifically, in the process of running the game APP, the electronic equipment is used for displaying a game interface, and the image display area is used for displaying images of the user, which are acquired by the camera in real time when the user performs motion sensing actions. For example, the electronic device may display an image including a limb form of a user when performing somatosensory actions on the image display region based on a preset scaling; the electronic device may also display, in the image display area, bone node data corresponding to the user when performing somatosensory actions based on a preset scaling after detecting bone node data in the image of the user according to a human bone node algorithm.

When a plurality of users simultaneously interact with the electronic device in a somatosensory manner, the camera of the electronic device can acquire images including somatosensory actions of the plurality of users in real time, and meanwhile, according to different application scenes, the plurality of users can control the same operation area by executing the somatosensory actions, and the plurality of users can also control the operation areas corresponding to the users by executing the somatosensory actions.

For example, refer to fig. 4 for a schematic diagram of a second application scenario of somatosensory interaction. When a plurality of users on the electronic equipment interact with the electronic equipment in a somatosensory way, an operation area corresponding to each user can be displayed on a display screen of the electronic equipment, namely the electronic equipment can control the operation area corresponding to each user according to the somatosensory action executed by each user, and simultaneously, images containing the plurality of users and acquired by a camera are displayed in an image display area in real time. Taking the game APP shown in fig. 4 as an example, when the game APP running on the electronic device is in the two-person game mode, one display image area and two operation areas (i.e., the operation area 1 and the operation area 2 in fig. 4) may be displayed on the display screen of the electronic device, the two operation areas correspond to the two users (i.e., the user 1 and the user 2 in fig. 4) one by one, the electronic device may display, on the image display area, an image, obtained by the camera, of a bone node corresponding to the two users when simultaneously performing somatosensory actions, where a distance between the user 1 and the camera of the electronic device is smaller than a distance between the user 2 and the camera of the electronic device, so, when the user 1 and the user 2 perform somatosensory actions, in the image obtained by the camera of the electronic device, an area occupied by the user 1 in the image is larger than an area occupied by the user 2 in the image, and correspondingly, an area occupied by the bone node data 1 corresponding to the user 1 in the image display area displayed by the image display area is larger than an area occupied by the bone node data corresponding to the user 2 in the image display area.

Referring to fig. 5, a flowchart of an embodiment of a somatosensory interaction method provided by the present application specifically includes the following:

S501, when a user performs a somatosensory action, a first image including the user is acquired by a camera.

The first image refers to an image acquired by the camera when one or more users perform somatosensory actions in the process of somatosensory interaction with the electronic equipment, and the first image can comprise one or more users.

S502, determining a first image area in the first image.

In the embodiment of the application, the electronic equipment can detect the first bone node data of each user in the first image through detection algorithms such as a gesture detection algorithm, a bone node detection algorithm, a three-dimensional reconstruction algorithm and the like, and further determine the first image area of each user in the first image according to the first bone node data and preset initial bone node data. Wherein the first bone node data of the user in the first image comprises a plurality of bone nodes of the user and position coordinates of each bone node in the first image.

The first image area represents the maximum movement range when the user performs somatosensory actions under the image coordinate system corresponding to the camera, namely the boundary which can be touched by the limbs of the user under the image coordinate system corresponding to the camera.

It can be understood that if the type of the camera set on the electronic device is a common camera for acquiring a two-dimensional image, an image coordinate system corresponding to the type of the camera is a two-dimensional coordinate system, and correspondingly, the first image area may be a plane area; if the type of the camera arranged on the electronic equipment is a depth camera for acquiring a three-dimensional image, an image coordinate system corresponding to the type of the camera is a three-dimensional coordinate system, and correspondingly, the first image area can be a three-dimensional space area. The electronic device may determine the position coordinates of each skeletal node of the user in the image coordinate system corresponding to the camera, thereby determining the length of each limb of the user in the image from the position coordinates of the skeletal nodes.

For example, referring to a schematic diagram of a first image acquired by a camera shown in fig. 6, it is assumed that the first image of a user acquired by the camera of the electronic device is a two-dimensional image, and a size of the first image is w×h, where W is a length of the first image and H is a height of the first image. The first image includes a user and the electronic device detects a plurality of skeletal nodes of the user in the first image including, but not limited to: neck skeletal node a, left wrist skeletal node B, right wrist skeletal node C, head skeletal node D, left ear skeletal node E, left eye skeletal node F, right eye skeletal node G, right ear skeletal node H, nose skeletal node I, left shoulder skeletal node J, right shoulder skeletal node K, left elbow skeletal node M, right elbow skeletal node N, crotch skeletal node L, left knee skeletal node R, right knee skeletal node S, left foot skeletal node U, right foot skeletal node V, and the like. The electronic device may establish a two-dimensional coordinate system based on the size of the first image, that is, the coordinate of the top left vertex of the first image is (0, 0) and the coordinate of the bottom right vertex of the image is (W, H), and then the electronic device may determine the position coordinate (x, y) of each bone node of the user in the first image, thereby determining the length of the limb consisting of two adjacent bone nodes and the image area occupied by the limb in the first image. For example, if the position coordinate of the left wrist bone node B in the first image is (x _B,y_B) and the position coordinate of the left elbow bone node M in the first image is (x _M,y_M), it can be determined that the left forearm length of the user in the first image isThe image area occupied by the left forearm of the user in the first image is an image area composed of a set of pixels on the line between the left wrist bone node B and the left elbow bone node M.

It will be appreciated that the skeletal node data enumerated by the present application is merely exemplary, and that the electronic device may detect more or less skeletal node data than shown by the present application based on actual somatosensory interaction scenarios.

In an embodiment of the present application, reference is made to a schematic display interface of an electronic device shown in fig. 7. Before somatosensory interaction is carried out between the electronic equipment and the user, the display screen can display text prompt information of 'please execute the following actions' and a first preset action so as to prompt the user to execute the first preset action facing the electronic equipment. When the user performs the first preset action, the camera of the electronic device may acquire an initial image including the user, and detect initial skeletal node data of the user in the initial image, so as to detect a maximum extension length of each limb of the user in the initial image according to the initial skeletal node data, thereby determining an initial image area based on the maximum extension length of the limb. For example, as shown in fig. 7, the first preset motion may be standing upright while the arms straighten and are parallel to the ground.

The types of the first image area and the initial image area (collectively referred to as image areas) of the user determined by the electronic device are also different for different somatosensory interaction scenes. In one example, the electronic device performs the interactive control operation in the corresponding operation area according to the position of the first limb of the user in the image, and the image area of the user is used for representing the image area corresponding to the maximum extension length of the first limb in a certain direction in the image acquired by the camera when the user performs the somatosensory action. For example, assuming that the first limb is a right arm, the image area of the user is an image area occupied by a maximum extension length of a right wrist bone node of the right arm in a certain direction relative to a right shoulder bone node in an image acquired by the camera, the camera of the electronic device may acquire an initial image of the user when performing a first preset action shown in fig. 7, detect position coordinates of the right wrist bone node C, the right elbow bone node N and the right shoulder bone node K of the user in the initial image, respectively, in the initial image, determine a length of a right forearm and a length of a right big arm, and determine a sum of the length of the right forearm and the length of the right big arm as a maximum extension length of the right arm of the user in the initial image, thereby determining that the initial image area of the user is an image area corresponding to the maximum extension length of the right arm of the user in the initial image.

In another example, the electronic device performs the interactive control operation in the corresponding operation area according to the position of the second limb of the user in the image, where the image area of the user is used to represent an image area surrounded by the maximum movement range of the second limb in each direction in the image acquired by the camera when the user performs the somatosensory action, for example, the maximum movement range of the limb of the user in one direction in the image acquired by the camera may be the maximum extension length of the left wrist iliac node of the two arms in one direction relative to the right wrist iliac node in the image acquired by the camera when the user performs the somatosensory interaction with the electronic device.

In this example, the electronic device may determine a fifth length of the second limb in the initial image and an initial origin of the initial image region from the initial skeletal node data; determining an initial active length according to the fifth length; and determining an initial image area according to the initial activity length and the initial origin, wherein the initial image area of the user is a square area taking the initial origin as a center point and taking the initial activity length as a side length. It should be noted that, the electronic device may also determine the shape of the image area of the user according to the shape of the operation area displayed on the display screen, for example, if the shape of the operation area is a matrix, the shape of the image area may be a square or a matrix; if the shape of the operation region is circular, the shape of the image region may be circular.

Alternatively, the electronic device may determine the initial image region based on a different second limb and a different manner. In a first aspect, the electronic device may determine an initial image area of the user in the initial image according to a maximum extension length of two arms of the user in the initial image, for example, referring to fig. 8, where the second limb may be two arms of the user, the electronic device may determine an initial active length of the initial image area of the user to be α×length according to detected position coordinates of a right wrist bone node C, a right elbow bone node N, a right shoulder bone node K, a neck bone node a, a left wrist bone node B, a left elbow bone node M, and a left shoulder bone node J in the initial image, respectively, where the initial active length of the initial image area of the user is determined to be α×length, that is, a square area where the initial image area of the user is α×length and the width in the initial image may be a neck bone node a of the user, and the initial origin of the initial image area may be a coefficient of α is equal to or greater than 1.

In a second mode, the electronic device may determine an initial image area of the user in the initial image according to the height of the user in the initial image, for example, referring to fig. 9, which shows a schematic diagram of the initial image, the second limb may be a limb range from the head bone node D to the left foot bone node U of the user in the initial image, and the electronic device may determine height=y _U-y_D of the user in the initial image according to the detected position coordinates (x _D,y_D) of the head bone node D and the position coordinates (x _U,y_U) of the left foot bone node U in the initial bone node data in the initial image; or the electronic device may determine that the height of the user in the initial image is equal to or less than 1.5 (y _U-y_D) according to the detected position coordinates (x _D,y_D) of the head bone node D and the position coordinates (x _L,y_L) of the crotch bone node L in the initial bone node data in the initial image, then it may determine that the initial active length of the initial image area of the user is equal to or less than β height, that is, a square area in the initial image area with both length and width being equal to or less than β height, and the initial origin of the initial image area may be the crotch bone node L of the user, where β is a coefficient greater than or equal to 1.

In other examples, the electronic device performs the interactive control operation in the corresponding operation region by detecting the displacement amount of the user in the target direction in the image sequence, and the image region of the user is used to represent the maximum displacement of the user in the target direction in the image coordinate system corresponding to the camera. In this example, the electronic device can determine the initial image area of the user by:

In a first mode, referring to a schematic diagram three of an initial image shown in fig. 10, the electronic device may determine a maximum extension length of two arms of a user in the initial image according to detected initial bone node data in the initial image, so as to determine a maximum displacement of the length of the user in a left-right direction of an initial image area of the user, where a center point of the initial image area is a coordinate point located between a left-foot bone node U and a right-foot bone node V in the initial image, and α is a coefficient greater than or equal to 1.

In a second mode, when a user executes a second preset action, acquiring an initial image sequence of the user through a camera, wherein the initial image sequence comprises initial images of continuous multiframes; detecting the maximum displacement of a user in the target direction in the initial image sequence; determining an initial active length according to the maximum displacement; determining an initial origin of an initial image area according to initial skeleton node data in an initial image including a first preset action executed by a user, determining the initial image area according to an initial activity length and the initial origin, wherein the initial image area is an initial image area with the length of the initial activity length in the initial image, and the initial origin is a side end point or a center point of the initial image area.

Specifically, before somatosensory interaction is performed between the electronic device and the user, prompt information can be displayed on the display screen so as to guide the user to jump leftwards from the first position to the second position or jump rightwards from the first position to the third position. In the process of jumping of the user, the camera can acquire an initial image sequence, and the electronic equipment can determine the maximum displacement of the skeleton node of the user in the left-right direction in the initial image sequence when the user finishes jumping once according to an image processing algorithm, so that the initial image area of the user is determined according to the maximum displacement. For example, a display screen may display prompt information for prompting the user to complete one left Jump, and the camera may acquire an initial image sequence in a process that the user jumps left from the first position to the second position, and if the electronic device determines that the maximum displacement of the user in the left-right direction is Jump according to the initial image sequence, the initial activity length of the initial image area of the user may be determined as gamma, where gamma is a coefficient greater than or equal to 2.

Further, the display screen of the electronic device may display a prompt message multiple times to guide the user to repeatedly jump left and right multiple times, and the electronic device may determine an average value of a plurality of maximum displacements detected by the user when the user completes jumping multiple times as an initial active length of an initial image area of the user.

It should be noted that, in an example, the preset initial skeletal node data and the preset initial image area may be initial skeletal node data and an initial image area pre-stored in the electronic device and corresponding to account information of the user. In another example, there may be different users performing somatosensory interaction with the electronic device, so the preset initial skeletal node data and the preset initial image area may also be that the electronic device obtains, through the camera, an initial image of the current user when performing a preset action before each somatosensory interaction with the user, so as to detect the initial skeletal node data of the current user in the initial image and determine the initial image area of the current user.

It will be appreciated that during somatosensory interaction with an electronic device, a user may move back and forth relative to the camera of the electronic device, and the area occupied by the user in the image acquired by the camera may change with the change of the distance between the user and the camera, but the scaling between the same limbs of the user in different images is fixed. For example, assuming that a camera on an electronic device acquires an initial image of a user at a first position and a first image of the user at a second position, wherein a distance between the user at the first position and the camera is smaller than a distance between the user at the second position and the camera, an area occupied by the user in the initial image is larger than an area occupied by the user in the first image, and a ratio between a length between a head bone node D and a neck bone node a of the user in the initial image and a length between the head bone node D and the neck bone node a of the user in the first image is equal to a ratio between a length between a left ear bone node E and a right ear bone node F of the user in the initial image and a length between a left ear bone node E and a right ear bone node F of the user in the first image. Therefore, in one embodiment, the scaling of the limb of the user in the first image relative to the limb of the user in the initial image may be determined according to the first bone node data of the user and the preset initial bone node data, and the preset initial image area may be scaled according to the scaling, so as to determine the first image area of the user in the first image.

In another embodiment, the first image region of the user in the first image may also be determined based on the first bone node data of the user in the first image. For example, assuming that a first image area of a user in the first image is a square area, according to first skeletal node data of the user in the first image, it may be determined whether the user is facing the display screen of the electronic device, if so, it may be determined that a side length of the first image area in the first image is length' =4 (l _KA+l_AJ) and a first origin of the first image area is neck skeletal node a according to position coordinates of a right shoulder skeletal node K, a neck skeletal node a and a left shoulder skeletal node J in the first image. Further, whether the user faces the display screen of the electronic device may be determined according to the length l _KA between the right shoulder bone node K and the neck bone node a and the length l _AJ between the neck bone node a and the left shoulder bone node J in the first image, and specifically, if l _KA＝l_AJ, it indicates that the user faces the display screen of the electronic device in the forward direction.

S503, determining a mapping relation between the first image area and an operation area of a user on a display screen.

The operation area is an area which is displayed on the display screen and can respond according to the somatosensory action of the user when the user executes the somatosensory action.

In the embodiment of the application, based on different types of first image areas, different mapping methods can be adopted to determine the mapping relation between the first image areas and the operation areas, so as to determine the corresponding mapping point of each pixel point in the first image area in the image in the operation area.

S504, executing interactive control operation according to the mapping relation.

In the process of the somatosensory interaction between the electronic equipment and the user, the interaction control can be performed on the operation area based on the mapping relation between the first image area determined by the first image area of the fixed type and the operation area corresponding to the user and displayed on the display screen; and the mapping relation between the first image areas of different types and the operation areas corresponding to the users and displayed on the display screen can be switched in different application scenes so as to interactively control the operation areas.

Assuming that a camera of the electronic device is a common camera for acquiring a two-dimensional image, an image coordinate system corresponding to the camera is a two-dimensional coordinate system, taking a game APP running in the electronic device as an example, an exemplary description is made on a method for performing somatosensory interaction between the electronic device and a user by combining different game items and different types of first image areas, and the method mainly relates to a specific process that the electronic device determines the first image areas according to first skeleton node data and preset initial skeleton node data and determines a mapping relation between the first image areas and an operation area.

Example one: the game item is that the electronic device controls an operation area of the user on the display screen by detecting somatosensory actions performed by a first limb of the user, and the first image area of the user is an area determined based on the maximum length of the first limb of the user in the first image.

The first limb may be the whole left arm, the whole right arm, the left forearm or the right forearm, provided that the electronic device performs the somatosensory interaction operation on the corresponding position in the operation area by detecting the position coordinates of the left hand or the right hand of the user in the first image acquired by the camera.

Referring to a method flowchart of determining a first image region shown in fig. 11, in an example one, a method of determining a first image region from first bone node data and preset initial bone node data includes the steps of:

S1101, determining a first length of a first limb and a second length of a first reference limb in an initial image according to initial bone node data.

It should be noted that, if the first limb of the user in the image includes a plurality of sub-limbs, the length of each sub-limb may be determined according to the position coordinates of the skeletal nodes at two ends of each sub-limb in the image acquired by the camera, so as to determine the sum of the lengths of the plurality of sub-limbs as the length of the first limb.

The first reference limb is a limb whose length is not easily changed by the motion of the user, that is, the length of the first reference limb of the user in different images is equal or similar in length in different images when the camera is located at the same position and performs different motion of the user, where the similar length means that the difference in length of the first reference limb of the user in different images is smaller than a preset threshold.

The first reference limb may be any one of a limb between the head bone node D and the neck bone node a, a limb between the neck bone node a and the left shoulder bone node J or the right shoulder bone node J, a limb between the left ear bone node E and the left eye bone node F, a limb between the right eye bone node G and the right ear bone node H, a limb between the left ear bone node E to the right ear bone node H, and a limb between the neck bone node a and the collapsed bone node L, by way of example.

The second length of the first reference limb in the initial image can be determined according to the position coordinates of the bone nodes at the two ends of the first reference limb in the initial image.

S1102, determining a third length of the first reference limb in the first image according to the first bone node data.

And determining the third length of the first reference limb in the first image according to the position coordinates of the bone nodes at the two ends of the first reference limb in the first image.

S1103, determining the maximum length of the first limb in the first image according to the ratio of the first length to the second length and the third length.

The ratio between the first length of the first limb in the initial image and the second length of the first reference limb is equal to the ratio of the maximum length of the first limb in the first image and the third length of the first reference limb.

It will be appreciated that if the user is performing somatosensory actions, the first limb is not parallel to the display screen, and in the first image obtained by the camera, the actual length of the first limb in the first image, which is determined according to the skeletal node of the first limb in the first image, is not the maximum length of the first limb.

S1104, determining the first image area according to the maximum length.

Specifically, if the first limb includes a first bone node and a second bone node, and the first bone node is a reference bone node, that is, when the user performs somatosensory actions at the same position, the position coordinates in the images acquired by the first bone node at different times are basically unchanged, and the position coordinates in the images acquired by the second bone node at different times are changed along with the somatosensory actions of the user. The electronic device may determine, according to the position coordinates of the first bone node and the second bone node in the first image, an extension direction of the first limb in the first image along the first bone node to the second bone node, and determine, according to the extension direction and the maximum length, the position coordinates of a first pixel point with the maximum distance between the second bone node and the first bone node in the first image, and then determine, as the first image area, an image area where a pixel point set between the first bone node and the first pixel point is located in the first image area of the user in the first image.

In an example one, the method for determining a mapping relationship between the first image area and an operation area corresponding to the user displayed on the display screen includes the steps of:

Step one, determining an indication line corresponding to a first limb in a first image according to first bone node data.

Specifically, according to the first bone node data, the position coordinates of the first bone node and the second bone node of the first limb in the first image can be determined, and a connecting line between the first bone node and the second bone node in the first image is an indication line corresponding to the first limb. If the first limb comprises a plurality of sub-limbs, the indication lines corresponding to the first limb comprise a plurality of indication lines corresponding to the sub-limbs one by one.

And step two, determining a mapping area corresponding to the first image area in the operation area according to the indication line and the maximum length.

Specifically, a first skeletal node of a first limb may be initially mapped to a center of an operating region; the maximum mapping length of the maximum length of the first limb in the operation region is determined, so that the mapping region corresponding to the first image region in the operation region is determined according to the maximum mapping length.

By way of example and not limitation, assuming that the first reference limb is a limb between the head skeletal node D and the neck skeletal node a, the first limb being the entire right arm, the first skeletal node in the first limb is the right shoulder skeletal node K, and the second skeletal node is the right wrist skeletal node C, an exemplary method of determining a mapping region corresponding to the first image region in the operation region will be described with reference to the schematic diagram shown in fig. 12.

Referring to the first image shown in fig. 12, a third length of the first reference limb in the first image may be represented as l _DA, a length of the right arm l _KC＝l_KN+l_NC, where l _KN is a length between the right shoulder bone node K and the right elbow bone node N in the first image, and l _NC is a length between the right elbow bone node N and the right carpal bone node C in the first image. If the first length of the right arm in the initial image is length _KC and the second length of the first reference limb in the initial image is length _DA, the maximum length l _KC'＝(length_KC/length_DA)*l_DA of the right arm in the first image is the maximum length l _KC'＝(length_KC/length_DA)*l_DA. In the first image shown in fig. 12, point C' is the maximum movement position coordinate of the right wrist iliac node in the first image with respect to the right shoulder bone node K when the right arm in the first image is at the maximum extension length in the direction from point K to point C.

Referring to an operation region corresponding to a user displayed on the display screen shown in fig. 12, it is assumed that the size of the operation region is equal to the size of the display screen, and the operation region has a length of X and a width of Y, and a two-dimensional coordinate system can be established with an upper left vertex in the operation region as an origin. Firstly, translating and mapping an indication line (namely a connecting line between a right shoulder bone node K and a right wrist bone node C) of a first limb in a first image into an operation area according to an initial mapping proportion of 1:1, so that the right shoulder bone node K is mapped to a central point of the operation area, and then the position coordinates of the right shoulder bone node K in the operation area are (X/2, Y/2). And extending a connecting line between the right shoulder bone node K and the right wrist bone node C to an edge point O of the operation area along the direction from the point K to the point C, wherein a straight line KO is used for representing the mapping length of the right arm in the operation area when the right arm in the first image is at the maximum length KC 'along the direction from the point K to the point C, and a pixel point set corresponding to the straight line KO in the operation area is a mapping area of the first image area of the user in the first image in the operation area, wherein the edge point O is a mapping point corresponding to the right wrist bone node in the first image in the operation area when the right wrist bone node in the first image is at the C'.

Further, in an example one, the electronic device may determine, according to a mapping relationship between the operation area and the first image area, a mapping point corresponding to each pixel point in the first image area in the first image in the operation area. For example, the electronic device may determine a mapping point of the right wrist iliac node C (i.e. the second skeletal node) in the first limb in the first image in the operation region according to a ratio of a maximum length of the first limb to a maximum mapping length, and the fourth length of the first limb in the first image, and specifically, the electronic device may determine a mapping length l _KC",l_KC"＝(l_KO/l_KC')*l_KC of the length l _KC of the right arm in the first image in the operation region according to a ratio between a length of the straight line KC' and a length of the straight line KO, where l _KC" is a length between a midpoint K of the operation region and a point c″ of the right wrist iliac node C in the first image, and a set of pixels corresponding to the straight line kc″ in the operation region is a mapping region of the right arm in the first image in the operation region.

The electronic device may determine the length of the straight line KO in the operation region and the position coordinates of the edge point O in the operation region based on the principle of the similar triangle. Specifically, a vertical line passing through the edge point O intersects a horizontal straight line passing through the point K at a point Q, and a vertical line passing through the point C intersects a straight line KQ at a point P. From the angle between the straight line KC and the straight line KQ and the length l _KC of the straight line KC, the position coordinates of the right wrist iliac node C in the first image in the operation region can be determined, thereby determining the length of the straight line KP and the length of the straight line CP. The length of the straight line KQ can be determined according to the similarity between the right triangle KPC and the right triangle KQO, so that the length of KO and the position coordinates of the edge O can be determined according to the length of the straight line KQ and the length Y/2 of the straight line OQ.

It will be appreciated that if the first limb is the entire left arm, the first skeletal node in the first limb is the left shoulder skeletal node J, and the second skeletal node is the left carpal skeletal node B, then, correspondingly, the left shoulder skeletal node J is mapped to the center point of the operating region, and then, the mapping region of the left arm in the operating region may be determined by referring to the above-described method for determining the mapping region of the right arm in the operating region.

Based on the method provided in the first example, according to initial bone node data of the user in the initial image and first bone node data of the user in the first image obtained by the camera, the moving range of the first limb in the first image when the user is at the current position can be determined; when the extending directions of the first limbs in the first image obtained by the camera are different, the lengths of the mapping areas corresponding to the first image areas of the user in the first image are also different, and accordingly, the scaling values between the maximum lengths of the first limbs in the first image and the maximum mapping lengths in the corresponding operation areas are also different, and according to the scaling values between the maximum lengths of the first limbs in the first image and the maximum mapping lengths in the corresponding operation areas and the indication lines of the first limbs in the first image, the actual mapping areas of the first limbs in the first image in the operation areas can be determined, so that when the user performs body sensing actions, the electronic equipment can control the operation areas corresponding to the user, which are displayed on the display screen, by detecting the positions of the first limbs of the user in the first image obtained by the camera.

Example two: the game item is that the electronic equipment controls an operation area of a user on a display screen by detecting somatosensory actions executed by a first limb of the user in a first image, and the first image area of the user is a square area in the first image.

In an example two, in the process of performing somatosensory interaction between a user and the electronic device, the camera can acquire the first image in real time, and the electronic device also updates the initial image area according to the initial skeleton node data in the initial image and the first skeleton node data in the first image, so as to determine the first image area in the first image, thereby determining the mapping relationship between the first image area and the operation area. That is, the size of the first image area in the first image is different from the size of the initial image area, and the position of the first image area in the first image is also different from the position of the initial image area in the initial image, so that in the process of performing somatosensory interaction between the user and the electronic device, the electronic device needs to update the mapping relationship between the first image area and the operation area in real time, and perform interactive control operation in the operation area based on the mapping relationship updated in real time.

In addition, the electronic device may determine a preset mapping relationship between an initial image area in the initial image and an operation area after determining the initial image area in the initial image, in the process of performing somatosensory interaction between a user and the electronic device, the camera may acquire the first image in real time, determine that the size of the first image area in the first image is the same as the size of the initial image area, and the position of the first image area in the first image is identical to the position of the initial image area in the initial image, that is, in the process of performing somatosensory interaction between the user and the electronic device, the electronic device may perform interactive control operation in the operation area based on the fixed preset mapping relationship.

A method of determining a corresponding mapping point of each pixel point in the square first image area in the operation area will be exemplarily described below with respect to two different types of square first image areas.

In the first embodiment, the size of the first image area in the first image is different from the size of the initial image area, and the position of the first image area in the first image is also different from the position of the initial image area in the initial image.

The electronic device controls the corresponding position in the operation area by detecting the position of the hand of the user in the first image acquired by the camera, and then the initial image area of the user represents the maximum movement range of the two arms of the user in each direction in the initial image acquired by the camera, and the first image area of the user represents the maximum movement range of the two arms of the user in each direction in the first image acquired by the camera. The user's arms may include left wrist skeletal node B, left elbow skeletal node M, left shoulder skeletal node J, neck skeletal node a, right shoulder skeletal node K, right elbow skeletal node N, and right wrist skeletal node C. The position coordinate of the center point of the initial image area in the initial image is the position coordinate of the neck bone node A of the user in the initial image, and the initial active length of the initial image area determined by the electronic equipment according to the initial bone node data in the initial image acquired by the camera is the length of the double arms of the user in the initial image. The length length_CB＝length_CN+length_NK+length_KA+length_AJ+length_JM+length_MB, of the two arms in the initial image, wherein length _CN is the length between the right wrist bone node C and the right elbow bone node N of the user in the initial image; length _NK is the length between the right elbow bone node C to the right shoulder bone node K of the user in the initial image; length _KA is the length between the right shoulder bone node K to the neck bone node a of the user in the initial image; length _AJ is the length between the neck bone node a to the left shoulder bone node J of the user in the initial image; length _JM is the length between the left shoulder bone node J to the left elbow bone node M of the user in the initial image; length _MB is the length between the user's left elbow skeletal node M to the left wrist iliac node B in the initial image.

In a first embodiment, referring to a flowchart of a method for determining a first image area shown in fig. 13, a method for determining a first image area according to first bone node data, initial bone node data, and initial image area includes the steps of:

S1301, determining a sixth length of the second reference limb in the initial image according to the initial bone node data.

The second reference limb is a limb whose length is not easily changed by the motion of the user, that is, the length of the second reference limb of the user in different images is equal or similar in length in different images obtained when the camera is located at the same position and performs different motion of the user, where the similar length means that the difference in length of the second reference limb of the user in different images is smaller than a preset threshold.

The second reference limb may be any one of a limb between the head bone node D and the neck bone node a, a limb between the neck bone node a and the left shoulder bone node J or the right shoulder bone node J, a limb between the left ear bone node E and the left eye bone node F, a limb between the right eye bone node G and the right ear bone node H, a limb between the left ear bone node E to the right ear bone node H, and a limb between the neck bone node a and the collapsed bone node L, by way of example.

The sixth length of the second reference limb in the initial image can be determined according to the position coordinates of the bone nodes at the two ends of the second reference limb in the initial image. If the second reference limb includes a plurality of sub-limbs, the length of the second reference limb is the sum of the lengths of the plurality of sub-limbs.

S1302, determining a seventh length of the second reference limb in the first image and a first origin of the first image area according to the first bone node data.

The seventh length of the second reference limb in the first image can be determined according to the position coordinates of the bone nodes at the two ends of the second reference limb in the first image. The first origin of the first image area is the position of the user's neck skeletal node a in the first image.

S1303, determining a first active length of the first image area according to the size of the initial image area, the sixth length and the seventh length.

Specifically, the electronic device may determine, according to first bone node data in the first image obtained by the camera, whether the user stands facing the screen at the current position, and further determine a first activity length of the first image area. For example, if the electronic device detects that the length between the neck joint point a and the left shoulder joint point J of the user and the length between the left shoulder joint point J and the right shoulder joint point K of the user are equal or the difference is smaller than the first threshold, it may be determined that the user is not standing facing the screen. If the user stands facing the screen, the electronic device may determine the size of the first active length according to a ratio between a sixth length of the second reference limb in the initial image and a seventh length of the second reference limb in the first image being equal to a ratio between the length of the initial image area and the first active length of the first image area; if the user is not standing facing the screen, the first active length of the first image area may be determined to be 6 times the length between the user's neck joint point a and the left shoulder joint point J in the first image.

S1304, determining a first image region according to the first activity length and the first origin.

Specifically, the first image area is a square area with a first origin as a center point and a side length of a first active length in the first image.

In a first embodiment, the method for determining the mapping relationship between the first image area and the operation area according to the size of the first image area and the size of the operation area corresponding to the user displayed on the display screen includes the following steps:

Mapping a first origin of a first image area to a center point of an operation area corresponding to a user and displayed on a display screen;

And step two, determining a mapping point corresponding to each pixel point in the first image area in the operation area according to a first ratio between the first active length of the first image area and the width of the operation area and a second ratio between the first active length of the first image area and the length of the operation area.

By way of example and not limitation, assuming that the second reference limb is a limb between the head skeletal node D and the neck skeletal node a, a sixth length of the second reference limb in the initial image is denoted as length _DA, a seventh length of the second reference limb in the first image is denoted as l _DA, an initial active length of the initial image area is denoted as length, and length is equal to a length of both arms in the initial image, length _CB, a method of determining a corresponding mapping point of each pixel point in the first image area in the operation area is exemplarily described with reference to the schematic diagram shown in fig. 14.

Referring to the first image shown in fig. 14, if the electronic device determines that the user stands facing the display screen at the current position according to the first bone node data in the first image, it may determine a first activity length '= (length _DA/l_DA) ×length of the first image area of the user in the first image, and further determine that the first image area in the first image is a square area with the neck bone node a in the first image as a center point and a side length of length'.

Referring to an operation region corresponding to a user displayed on the display screen shown in fig. 14, the size of the operation region is equal to the size of the display screen, and the length of the operation region is X and the width is Y, and a two-dimensional coordinate system can be established with the upper left vertex in the operation region as the origin. And translating and initially mapping the first image area in the first image into the operation area according to an initial mapping proportion of 1:1, so that the center point of the first image area is mapped to the center point of the operation area. Assuming that the position coordinates of the coordinate points in the first image area in the two-dimensional coordinate system of the operation area are (x, y), a first ratio between the side length of the first image area and the width of the operation area can be determined, and the first ratio is a scaling ratio corresponding to the x value in the position coordinates of each pixel point in the first image area; and determining a second ratio between the side length of the first image area and the length of the operation area, wherein the second ratio is a scaling corresponding to the y value in the position coordinate of each pixel point in the first image area.

For example, as shown in fig. 14, if the position coordinate of the user's right-hand skeletal node C in the two-dimensional coordinate system of the operation region in the first image region is (x _C,y_C), the position coordinate of the mapping point c″ of the user's right-hand skeletal node C in the operation region in the first image region is (x _C",y_C"), x _C"＝(X/length')*x_C,y_C"＝(Y/length')*y_C.

In a first embodiment, the electronic device may preset a first scaling according to the acquired size of the first image and the size of the image display area on the display screen, as shown in fig. 15, in a process of performing somatosensory interaction with a user, the electronic device may determine a side length of the first indication frame according to the preset first scaling and a side length of the first image area in the first image, and display the first indication frame on the image display area on the display screen based on the determined side length of the first indication frame, and may display first skeletal node data of the user detected from the first image in the image display area according to the preset first scaling, where an area in the first indication frame is used for representing the first image area determined from the first image. As shown in fig. 15, only a part of the first skeletal node data of the user displayed in the image display area on the display screen is located in the first indication frame.

As shown in fig. 16, when a game running in the electronic device is in a two-player game mode, two operation areas corresponding to two users, namely, operation area 1 and operation area 2 in fig. 16, respectively, may be displayed on the display screen of the electronic device. Assuming that two users interact with the electronic device in a somatosensory manner, namely, a user 1 and a user 2, and the size of the user 1 is smaller than that of the user 2, the first image acquired by the camera comprises two users, and the area occupied by the user 1 in the first image is smaller than that occupied by the user 2 in the first image. Accordingly, the first bone node data 1 corresponding to the user 1 and the second bone node data 2 corresponding to the user 2 in the first image may be detected, the first image area 1 of the user 1 in the first image may be determined according to the first bone node data 1, and the mapping relationship between the first image area 1 and the operation area 1 may be determined based on the mapping method in the foregoing embodiment. A first image area 2 of the user 2 in the first image is determined from the first bone node data 2 and a mapping relation between the first image area 2 and the operation area 2 is determined based on the mapping method in the previous embodiment.

Further, as shown in fig. 16, the image display area on the display screen may simultaneously display the first bone node data 1 corresponding to the user 1 and the first indication frame 1 for indicating the first image area 1, and the first bone node data 2 corresponding to the user 2 and the first indication frame 2 for indicating the first image area 2, and the area occupied by the first indication frame 1 in the image display area is smaller than the area occupied by the first indication frame 2.

In the second embodiment, the size of the first image area in the first image is the same as the size of the initial image area, and the position of the first image area in the first image is the same as the position of the initial image area in the initial image.

By the same position, it is meant that the position coordinates of the center point of the first image area in the first image are the same as the position coordinates of the center point of the initial image area in the initial image.

By way of example and not limitation, in the second embodiment, referring to the initial image shown in fig. 9, the initial image area may be a square area centered on the crotch bone node L determined based on the height of the user in the initial image, and then all the initial skeletal node data may be included in the initial image area. The electronic device may determine that the initial active length of the initial image area is length=y _U-y_D according to the position coordinate (x _D,y_D) of the head bone node D to the position coordinate (x _U,y_U) of the left foot bone node U in the initial bone node data, that is, the initial image area of the user is a square area with the crotch bone node L in the initial image being located (x _L,y_L) as a center and the side length being length, and correspondingly, the first image area of the user in the first image is a square area with the (x _L,y_L) as a center and the side length being length.

In one example, the method for determining a preset mapping relationship between an initial active region of a user and an operation region corresponding to the user in an initial image includes:

Firstly, translating and initially mapping an initial image area in an initial image into an operation area according to an initial mapping proportion of 1:1, so that a center point of the initial image area is mapped to a center point of the operation area corresponding to a user;

And step two, determining a preset mapping relation between the initial image area and the operation area corresponding to the user according to a first ratio between the width of the initial image area and the width of the operation area and a second ratio between the side length of the initial image area and the length of the operation area.

Specifically, assuming that the position coordinates of the coordinate point in the initial image area in the two-dimensional coordinate system of the operation area are (x, y), a first ratio between the side length of the initial image area and the width of the operation area can be determined, and the first ratio is a scaling ratio corresponding to the x value in the position coordinates of each pixel point in the initial image area and the first image area; and determining a second ratio between the side length of the initial image area and the length of the operation area, wherein the second ratio is a scaling corresponding to a y value in the position coordinates of each pixel point in the initial image area and the first image area.

In the second embodiment, the preset mapping relationship between the initial image area and the operation area corresponding to the user is the mapping relationship between the first image area of the user and the operation area corresponding to the user in the first image. For example, referring to the first image shown in fig. 17, if the length of the operation region is X and the width is Y, the position coordinate of the user's right-hand skeletal node C in the two-dimensional coordinate system of the operation region in the first image region is (X _C,y_C), and the position coordinate of the user's right-hand skeletal node C in the operation region is (X _C",y_C"), X _C"＝(X/length)*x_C,y_C"＝(Y/length)*y_C.

In the second embodiment, the electronic device may preset a first scaling according to the acquired size of the first image and the size of the image display area on the display screen, determine the side length of the first indication frame according to the first scaling and the side length of the first image area in the first image, and display the first indication frame in the image display area on the display screen based on the determined side length of the first indication frame, and simultaneously display the first bone node data of the user detected from the first image in the image display area according to the preset first scaling, where the area in the first indication frame is used to represent the first image area determined from the first image, and as illustrated in fig. 18, the first bone node data of the user displayed in the image display area on the display screen is located in the first indication frame.

Optionally, if the position coordinate of the crotch node of the user in the first image is determined to deviate from the center of the first image area according to the detected first skeletal node data in the first image during the somatosensory interaction between the user and the electronic device, a prompt message may be displayed on the display screen to prompt the user to move the position, so that the position coordinate of the crotch node of the user coincides with the center of the first image area.

For example, the prompt information may be to flash and display the center point of the first indication frame in the image display area, play a voice prompt, and/or display a text prompt information that "position has shifted" on the display screen.

Example three: the game item is that the electronic equipment controls the virtual character of the user in the operation area of the display screen by detecting the left-right jumping action of the user, and the first image area of the user represents the maximum displacement of the user along the target direction under the image coordinate system corresponding to the camera.

Referring to the schematic diagram of fig. 19, if the game running in the electronic device is a running game, the virtual character and the runways may be displayed in the operation area on the display screen, the operation area includes a plurality of sub-operation areas corresponding to the runways one by one, and the display width of the runways in the horizontal direction is the length of the corresponding sub-operation area. The first image area in the first image acquired by the camera may be an area indicated by a bar frame in the first image in fig. 19, where the first image area includes a plurality of first sub-image areas corresponding to a plurality of sub-operation areas in the operation area one by one, the first image area represents a maximum displacement of a user in the first image along a horizontal direction, and correspondingly, the initial image area detected by the camera from the initial image includes a plurality of initial sub-image areas corresponding to a plurality of sub-operation areas in the operation area one by one, and the initial image area represents a maximum displacement of the user in the initial image along the horizontal direction.

In example three, the first image region is an image region in the first image having a first active length and a side or center point is a first origin. The method for determining the first image area according to the first bone node data, the preset initial bone node data and the preset initial image area may refer to a second flowchart shown in fig. 13, and a description of specific implementation manners of the steps S1301 to S1303 in the second example will not be repeated here.

As an example and not by way of limitation, assuming that the second reference limb is a limb between the neck skeletal node a and the crotch joint point L, the sixth length of the second reference limb in the initial image is denoted as length _DA, the seventh length of the second reference limb in the first image is denoted as L _DA, the length of the initial image region is denoted as length, and length is equal to the length of both arms in the initial image, length 'of the first image region in the first image is equal to length _CB, and the length of each first sub-image region in the first image region is equal to length' = (length _DA/l_DA) x length if the first image region includes n first sub-image regions.

It should be noted that, in the third example, the electronic device may determine, according to the length of the first sub-image areas and the length of the sub-operation areas, a mapping relationship between each first sub-image area and the corresponding sub-operation area. As shown in fig. 19 and 20, if the running game is in the single game mode, only one operation area may be displayed on the display screen of the electronic device, three runways are displayed in the operation area, and the length of the operation area in the horizontal direction is X, and accordingly, the operation area includes three sub-operation areas corresponding to the three runways one by one, namely, sub-operation area 1, sub-operation area 2, and sub-operation area 3, and each sub-operation area has a length of X/3. The first image area includes three first sub-image areas, namely, a first sub-image area 1, a first sub-image area 2 and a first sub-image area 3 in turn from left to right, wherein the length l 'of each first sub-image area in the first image is length'/3, and the electronic device can map each first sub-image area to a corresponding sub-operation area according to the length of the first sub-image area and the length of the sub-operation area, that is, map the first sub-image area 1 to the sub-operation area 1, map the first sub-image area 2 to the sub-operation area 2 and map the first sub-image area 3 to the sub-operation area 3.

In an example three, the electronic device may preset a second scaling according to the acquired size of the first image and the size of the image display area on the display screen, as shown in fig. 20, in the process of performing somatosensory interaction with the user, the electronic device may determine a side length of the first indication frame according to the preset second scaling and the length of the first image area determined from the first image, and display the second indication frame on the image display area on the display screen based on the determined side length of the first indication frame, and may display the first skeletal node data of the user detected from the first image in the image display area according to the preset second scaling, where the second indication frame is used to represent n first sub-image areas in the first image area determined from the first image.

Optionally, for n first sub-image areas in the first image areas displayed in the image display area in the display screen, the n first sub-image areas may be displayed in different colors, so as to guide the user to jump a fixed distance according to the relative positional relationship between the first skeletal node data displayed in the image display area and the first image areas, so that the electronic device controls the virtual character in the game to jump by detecting the distance that the user jumps.

Further, the electronic device may determine a relative positional relationship between the virtual character displayed in the operation region and the plurality of sub-operation regions according to a relative positional relationship between the user in the first image and the plurality of first sub-image regions in the first image region and a mapping relationship between each first sub-image region and the corresponding self-operation region. Specifically, the electronic device may determine initial relative positions of the user and the n initial sub-image areas in the initial image; then, according to the first bone node data and the initial bone node data, determining the left-right movement distance of the user in the first image relative to the initial image; and determining the current relative positions of the user and the n first sub-image areas in the first image according to the left-right movement distance and the initial relative positions so as to control the virtual character in the game to jump among the n runways in the operation area.

Illustratively, it is assumed that the electronic device detects a length of each of initial sub-image areas in the initial image area of the initial image from a position coordinate (x _L,y_L) of the user's crotch iliac node L in the initial image in the initial skeletal node data, a position coordinate (x' _L,y'_L) of the user's crotch iliac node L in the first image in the first skeletal node data, and determines a displacement Δx=x' _L-x_L of the user in the first image in the horizontal direction with respect to the initial image.

In one alternative, the electronic device may control the virtual character in the game to jump between the three sub-operation areas according to the relative position of the user in the first image with respect to the three initial sub-image areas (i.e., initial sub-image area 1, initial sub-image area 2, and initial sub-image area 3), the magnitude of Δx, and the length l' of the first sub-image area.

For example, assuming that when a game in the electronic device starts to run, the virtual character in the game is located at the center of the sub-operation area 2, correspondingly, the user in the initial image is located at the center of the initial sub-image area 2, during the game, when the electronic device detects Δx <0 according to the initial image and the first image, the user is indicated to jump leftwards, if |Δx| < length '/3, the electronic device does not control the virtual character in the game to jump leftwards, if |Δx| > length'/3, the electronic device may control the virtual character in the game to jump leftwards by a preset fixed distance X/3, namely, control the virtual character to jump leftwards from the sub-operation area 2 to the sub-operation area 1, correspondingly, the first skeletal node data of the user displayed in the image display area is located at the center of the first sub-image area 1; when the electronic device detects Δx >0 according to the initial image and the first image, the electronic device may determine that the user jumps right, if |Δx| < length '/3, the electronic device does not control the virtual character in the game to jump right, if |Δx| > length'/3, the electronic device may control the virtual character in the game to jump right by a preset fixed distance X/3, that is, control the virtual character to jump right from the sub-operation area 2 to the sub-operation area 3, and correspondingly, the first skeletal node data of the user displayed in the image display area is located in the center of the first sub-image area 3.

Further, if |Δx| > length'/2, that is, the user has exceeded the first image area in the first image, the current relative position of the first image area with respect to the user in the first image may be updated according to the horizontal distance between the position coordinate of the crotch node L of the user in the first image and the side edge of the first image area, and the position of the first image area displayed in the image display area may be adjusted such that the first bone node data of the user displayed in the image display area is located in the adjusted first image area.

In another alternative, the electronic device may determine the relative positions of skeletal nodes with respect to three first sub-image areas (i.e., the first sub-image area 1, the first sub-image area 2, and the first sub-image area 3) in the first image based on the position coordinates of skeletal nodes of the user in the first image, such as skeletal nodes of the crotch skeletal node L, the left foot skeletal node U, the right foot skeletal node V, the skull node D, or the neck skeletal node a, and the like, to control the virtual character in the game to jump between the three sub-operation areas.

Illustratively, taking the crotch joint point L of the user in the first image as an example, it is assumed that the position coordinate of the crotch joint point L in the first image is (x _L,y_L). If it is detected that the coordinate range of the first image area in the horizontal direction, that is, the x coordinate axis direction, in the first image is x ₁ to x ₂, the coordinate range of the first sub-image area 1 in the horizontal direction in the first image is x ₁ to x ₁ +length '/3, the coordinate range of the first sub-image area 2 in the horizontal direction is x ₁ +length'/3 to x ₁ +2x length '/3, and the coordinate range of the first sub-image area 3 in the horizontal direction is x ₁ +2x length'/3 to x ₂. If the position coordinates of the crotch joint point L of the user in the first image are detected to meet the condition of x ₁≤x_L≤x₁ +length'/3, the electronic equipment can control the virtual character in the game to jump to the sub-operation area 1; if the position coordinates of the crotch skeleton node L of the user in the first image are detected to meet the condition of x ₁+length'/3<x_L<x₁ +2x length'/3, the electronic device may control the virtual character in the game to jump to the sub-operation area 2; if it is detected that the position coordinates of the crotch node L of the user in the first image satisfy the condition x ₁+2*length'/3≤x_L≤x₂, the electronic device may control the virtual character in the game to jump to the sub-operation area 3.

If the game running in the electronic device is in the multiplayer game mode, the electronic device may display an operation area corresponding to each user on the display screen according to the number of users, and the size of each operation area may be the same or may not be different. Accordingly, the electronic device may display, in the image display area on the display screen, the plurality of first skeletal node data corresponding to the plurality of users and the first indication frame or the second indication frame corresponding to the first image area corresponding to each user in the detected first image.

As shown in fig. 21, when the running game is in the two-player game mode, two operation areas (i.e., the operation area 1 and the operation area 2 in fig. 21) corresponding to two users and two virtual characters corresponding to the two users are displayed on the display screen of the electronic device, three runways are displayed in each operation area, and the length of each operation area in the horizontal direction is X/2, and accordingly, each operation area includes three sub-operation areas corresponding to the three runways one by one, namely, the sub-operation area 1, the sub-operation area 2 and the sub-operation area 3. The camera can acquire first images of two users when performing somatosensory actions in the game process, the electronic equipment can control the virtual characters in the operation area 1 to jump among the corresponding three sub-operation areas by detecting the relative position relationship between the user 1 and the first image area corresponding to the user 1 in the first images, and can control the virtual characters in the operation area 2 to jump among the corresponding three sub-operation areas by detecting the relative position relationship between the user 2 and the first image area corresponding to the user 2 in the first images acquired by the camera. In addition, as shown in fig. 21, the electronic device may display, in an image display area on the display screen, first bone node data 1 of user 1, first bone node data 2 of user 2, a second instruction frame corresponding to user 1, and a second instruction frame corresponding to user 2 in the first image according to the first image that the camera may acquire.

Fig. 22 provides a flow chart of a somatosensory interaction method, which mainly relates to a specific process that an electronic device avoids collision between a user and surrounding obstacles or between a plurality of users according to a determined first image area in a somatosensory interaction process of the user and the electronic device. As shown in fig. 22, the somatosensory interaction method provided by the embodiment of the present application further includes the following steps:

s2201, identifies an obstacle in the first image.

For example, the electronic device may detect one or more obstacles in the first image acquired by the camera through an image recognition algorithm.

S2202, determining a first distance between the obstacle and the first image area.

Specifically, when the electronic device performs somatosensory interaction with the user, the electronic device may determine a first image area of the user in the first image through the method provided in the embodiment, and further determine a first distance between an obstacle in the first image and a frame of the first image area of the user. The first image area may be a square area in the first image or an image area including n first sub-image areas.

In this embodiment, if the first image acquired by the camera of the electronic device is a two-dimensional image, a first distance between the obstacle in the first image detected by the electronic device and the frame of the first image area of the user is a horizontal distance between the obstacle and the frame of the first image area. Further, the camera of the electronic device may be a depth camera, through which not only the first image of the user when performing the somatosensory action can be obtained, but also depth information of the user and obstacles in the surrounding environment of the user can be determined, and further, according to the depth information and the actual height of the user pre-stored in the electronic device, a first distance between the obstacles in the first image and the first image area of the user in the depth direction is determined.

For example, a method of determining a first distance in a depth direction between an obstacle in a first image and a first image area of a user comprises: determining a first depth difference between the user and an obstacle in the surrounding environment from the depth information; determining a second depth difference value between the user and the obstacle in the first image according to a ratio between the actual height of the user pre-stored in the electronic device and the height of the user in the first image and the first depth difference value; a first distance in the depth direction between the obstacle in the first image and the first image area may be determined from the second depth difference and the size of the first image area of the user in the first image.

S2203, if the first distance is smaller than the preset distance, playing collision prompt information.

For example, the collision prompt information may be displayed on a display screen or the first indication frame or the second indication frame may be displayed on the display screen in a flashing manner, or a voice prompt may be played.

Further, if the electronic device and the plurality of users perform the somatosensory interaction at the same time, the first image obtained by the camera includes at least two users, and the electronic device may determine a first image area corresponding to each user in the first image by using the method provided in the above embodiment, so as to determine a first distance between an obstacle in the first image and a frame of the first image area of each user, and a second distance between frames of the first image areas of any two users. If the camera of the electronic device is a depth camera, the first image acquired by the camera includes at least two users, and the electronic device can determine a second distance between any two users in the depth direction according to the distance between each user and the electronic device acquired by the depth camera. If the second distance between the frames of the first image areas of two adjacent users is smaller than the preset distance, collision prompt information can be played, for example, voice collision prompt is carried out, bone node data of the two users are displayed in a flashing mode in an image display area, the first prompt frame or the second prompt frame of the two users are displayed in the flashing mode in the image display area, or text prompt information is displayed on a display screen.

Optionally, when the first distance between the obstacle and the first image area of the user in the first image is smaller than the preset distance, or the second distance between the rims of the first image areas of the two adjacent users is smaller than the preset distance, the electronic device may guide the user away from the obstacle or increase the distance between the two users by changing the game content and changing the position of the user displayed in the image display area in the first image area.

As an example and not by way of limitation, assuming that when a fruit-cutting game is executed in an electronic device, user 1 and user 2 interact with the electronic device simultaneously in a sense of body, two operation areas corresponding to two users one by one are displayed on a display screen of the electronic device, the electronic device detects that user 1 is located at the left side of the display screen through a first image acquired by a camera, user 2 is located at the right side of the display screen, a second distance between a first image area of user 1 and a first image area of user 2 in the first image is smaller than a preset distance, and a distance between the first image area of user 2 and the right side edge of the first image in the first image is larger than a left side edge of the first image area of user 1 and the first image, the electronic device can increase the display probability of fruit on the right side in the operation area corresponding to user 2 on the display screen, and simultaneously adjust the position of the first image area to the left by a preset displacement so as to display the adjusted first image area on the display screen, thereby guiding the user to move to the right by a larger distance until the fruit is marked on the right side of the display screen. Specifically, if the position coordinate of the center of the first image area of the user 2 in the first image is (X, y), the adjusted position coordinate of the center of the first image area in the first image is (X-shift, y), where shift=x/T (T-1) +x/t×2-X, X represents the width of the display screen, T represents the total number of users, and T represents the T-th user of the T users.

According to the somatosensory interaction method provided by the embodiment of the application, the electronic equipment can detect the first skeleton node data of the user in the first image acquired by the camera, so that the area occupied by the user in the first image is determined according to the first skeleton node data, and based on different game items operated in the electronic equipment, in the somatosensory interaction process of the electronic equipment and at least one user, the electronic equipment can determine the corresponding first image area according to the area occupied by each user in the first image, so as to represent the maximum image area corresponding to each user when the somatosensory action is executed under the image coordinate system corresponding to the camera, and therefore, the mapping relation between the maximum image area and the operation area corresponding to the user displayed on the display screen is determined, and the user can control the whole operation area corresponding to the user displayed on the display screen through the somatosensory action in the interaction process of the user and the electronic equipment even if the distance between the user and the electronic equipment is far, so that the user's control experience is improved to a certain extent.

It should be understood that the sequence number of each step in the foregoing embodiment does not mean that the execution sequence of each process should be determined by the function and the internal logic, and should not limit the implementation process of the embodiment of the present application.

Corresponding to the somatosensory interaction method provided by the embodiment, the embodiment of the application also provides an embodiment of the device such as the electronic equipment. It should be understood that the description of the embodiments of the apparatus may refer to the above description of the embodiments of the method for detecting an electronic device and a camera, and the implementation principle and technical effects are similar to those of the embodiments of the method, which are not repeated herein.

Based on the somatosensory interaction method provided by each embodiment, the embodiment of the application also provides the following contents:

The present embodiment provides a computer program product including a program which, when executed by an electronic device, causes the electronic device to perform the somatosensory interaction method shown in the above embodiments.

Embodiments of the present application provide a computer-readable storage medium storing a computer program which, when executed by a processor, implements the somatosensory interaction method shown in the respective embodiments described above.

An embodiment of the present application provides a chip including a memory and a processor that executes a computer program stored in the memory to implement controlling the electronic device to execute the somatosensory interaction method shown in the foregoing embodiments.

It should be appreciated that the processor referred to in the embodiments of the present application may be a central processing unit (central processing unit, CPU), but may also be other general purpose processors, digital signal processors (DIGITAL SIGNAL processors, DSPs), application Specific Integrated Circuits (ASICs), off-the-shelf programmable gate arrays (field programmable GATE ARRAY, FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

It should also be understood that the memory referred to in embodiments of the present application may be volatile memory or nonvolatile memory, or may include both volatile and nonvolatile memory. The nonvolatile memory may be a read-only memory (ROM), a Programmable ROM (PROM), an erasable programmable ROM (erasable PROM), an electrically erasable programmable EPROM (EEPROM), or a flash memory. The volatile memory may be random access memory (random access memory, RAM) which acts as external cache memory. By way of example, and not limitation, many forms of RAM are available, such as static random access memory (STATIC RAM, SRAM), dynamic random access memory (DYNAMIC RAM, DRAM), synchronous Dynamic Random Access Memory (SDRAM), double data rate synchronous dynamic random access memory (double DATA RATE SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (ENHANCED SDRAM, ESDRAM), synchronous link dynamic random access memory (SYNCHLINK DRAM, SLDRAM), and direct memory bus random access memory (direct rambus RAM, DR RAM).

It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-described division of the functional units and modules is illustrated, and in practical application, the above-described functional distribution may be performed by different functional units and modules according to needs, i.e. the internal structure of the apparatus is divided into different functional units or modules to perform all or part of the above-described functions. The functional units and modules in the embodiment may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit, where the integrated units may be implemented in a form of hardware or a form of a software functional unit. In addition, the specific names of the functional units and modules are only for distinguishing from each other, and are not used for limiting the protection scope of the present application. The specific working process of the units and modules in the above system may refer to the corresponding process in the foregoing method embodiment, which is not described herein again.

In the foregoing embodiments, the descriptions of the embodiments are emphasized, and in part, not described or illustrated in any particular embodiment, reference is made to the related descriptions of other embodiments.

Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

In the embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other manners. For example, the system embodiments described above are merely illustrative, e.g., the division of the modules or units is merely a logical functional division, and there may be additional divisions when actually implemented, e.g., multiple units or components may be combined or integrated into another system, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection via interfaces, devices or units, which may be in electrical, mechanical or other forms.

The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in the embodiments of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.

The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the present application may implement all or part of the flow of the method of the above embodiments, and may be implemented by a computer program to instruct related hardware, where the computer program may be stored in a computer readable storage medium, and when the computer program is executed by a processor, the computer program may implement the steps of each of the method embodiments described above. Wherein the computer program comprises computer program code which may be in source code form, object code form, executable file or some intermediate form etc. The computer readable medium may include at least: any entity or device capable of carrying computer program code to a large screen apparatus, a recording medium, a computer Memory, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), an electrical carrier signal, a telecommunications signal, and a software distribution medium. Such as a U-disk, removable hard disk, magnetic or optical disk, etc. In some jurisdictions, computer readable media may not be electrical carrier signals and telecommunications signals in accordance with legislation and patent practice.

Finally, it should be noted that: the foregoing is merely illustrative of specific embodiments of the present application, and the scope of the present application is not limited thereto, but any changes or substitutions within the technical scope of the present application should be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A somatosensory interaction method applied to electronic equipment, which is characterized by comprising a display screen and a camera, wherein the method comprises the following steps:

When a user executes somatosensory actions, acquiring a first image comprising the user through the camera;

Determining a first image area in the first image;

determining a mapping relation between the first image area and an operation area of the user on the display screen;

and executing interactive control operation according to the mapping relation.

2. The somatosensory interaction method according to claim 1, wherein the determining a first image region in the first image comprises:

detecting first skeletal node data of the user in the first image;

and determining the first image area according to the first bone node data and preset initial bone node data.

3. The somatosensory interaction method according to claim 2, wherein the method further comprises:

when the user executes a first preset action, acquiring an initial image comprising the user through the camera;

the initial skeletal node data of the user in the initial image is detected.

4. A somatosensory interaction method according to claim 3, wherein the determining the first image region according to the first skeletal node data and preset initial skeletal node data comprises:

Determining a first length of a first limb and a second length of a first reference limb in the initial image according to the initial bone node data;

Determining a third length of the first reference limb in the first image from the first bone node data;

determining a maximum length of the first limb in the first image according to the ratio of the first length to the second length and the third length;

And determining the first image area according to the maximum length.

5. The somatosensory interaction method according to claim 4, wherein the determining a mapping relationship between the first image region and an operation region of the user on the display screen comprises:

Determining an indication line corresponding to the first limb in the first image according to the first bone node data;

and determining a mapping area corresponding to the first image area in the operation area according to the indication line and the maximum length.

6. A somatosensory interaction method according to claim 3, wherein the method further comprises:

the determining the first image area according to the first bone node data and the preset initial bone node data includes:

And determining the first image area according to the first bone node data, the initial bone node data and the initial image area.

7. The somatosensory interaction method according to claim 6, wherein the determining an initial image region in the initial image according to the initial skeletal node data comprises:

Determining a fifth length of a second limb in the initial image and an initial origin of the initial image area according to the initial bone node data;

determining an initial active length according to the fifth length;

And determining the initial image area according to the initial activity length and the initial origin.

8. The somatosensory interaction method according to claim 7, wherein the determining the first image region according to the first skeletal node data, the initial skeletal node data, and the initial image region comprises:

Determining a sixth length of a second reference limb in the initial image according to the initial bone node data;

Determining a seventh length of the second reference limb in the first image and a first origin of the first image region from the first bone node data;

determining a first active length of the first image region according to the initial active length, the sixth length and the seventh length;

and determining the first image area according to the first activity length and the first origin.

9. The somatosensory interaction method according to claim 8, wherein the first image region is a square region with a side length of the first active length in the first image, and the first origin is a center point of the first image region.

10. A somatosensory interaction method according to any of claims 7 to 9, wherein the method further comprises:

determining a first indication frame according to a preset first scaling and the size of the first image area;

And displaying the first indication frame in an image display area on the display screen, wherein the first indication frame is used for representing the first image area.

11. A somatosensory interaction method according to claim 3, wherein the method further comprises:

When the user executes a second preset action, acquiring an initial image sequence of the user through the camera;

Detecting the maximum displacement of the user in the target direction in the initial image sequence;

Determining an initial active length according to the maximum displacement;

Determining an initial origin of the initial image region according to the initial bone node data;

12. The somatosensory interaction method according to claim 8 or 11, wherein the initial image region is an image region of which the length is the initial active length in the initial image, and the initial origin is a side end point or a center point of the initial image region.

13. The somatosensory interaction method according to claim 12, wherein the operation region comprises n sub-operation regions, n being an integer greater than or equal to 2;

The determining the mapping relation between the first image area and the operation area of the user on the display screen comprises the following steps:

Dividing the first image area into n first sub-image areas corresponding to the n sub-operation areas one by one;

and determining the mapping relation between each first sub-image area and the corresponding sub-operation area according to the length of the first sub-image area and the length of the sub-operation area.

14. The somatosensory interaction method according to claim 12 or 13, wherein the method further comprises:

determining a second indication frame according to a preset second scaling and the size of the first image area;

and displaying the second indication frame in an image display area on the display screen, wherein the second indication frame is used for representing the first image area.

15. A somatosensory interaction method according to any of claims 6 to 14, wherein the method further comprises:

identifying an obstacle in the first image;

determining a first distance between the obstacle and the first image area;

and if the first distance is smaller than the preset distance, playing collision prompt information.

16. A somatosensory interaction method according to any of claims 6 to 15, wherein the first image comprises at least two users;

And when the second distance between the first image areas of the two users is detected to be smaller than the preset distance, playing collision prompt information.

17. The somatosensory interaction method according to claim 15 or 16, wherein playing collision prompt information comprises:

the voice prompt, the text prompt and/or the first indication frame or the second indication frame are/is displayed on the display screen in a flashing mode.

18. An electronic device, comprising: a processor for running a computer program stored in a memory to implement the method of any one of claims 1 to 17.

19. A chip system comprising a processor executing a computer program stored in a memory to implement the method of any one of claims 1 to 17.

20. A computer readable storage medium, characterized in that the computer readable storage medium stores a computer program which, when executed by a processor, implements the method according to any of claims 1 to 17.