CN118118717A

CN118118717A - Screen sharing method, device, equipment and medium

Info

Publication number: CN118118717A
Application number: CN202211477797.5A
Authority: CN
Inventors: 李蕾; 崔新宇; 段孝超
Original assignee: Beijing Zitiao Network Technology Co Ltd
Current assignee: Beijing Zitiao Network Technology Co Ltd
Priority date: 2022-11-23
Filing date: 2022-11-23
Publication date: 2024-05-31

Abstract

The application provides a screen sharing method, a device, equipment and a medium, wherein the method comprises the following steps: responding to sensor data sent by a host side, and determining a screen sharing mode selected by the host side; if the screen sharing mode selected by the anchor terminal is the target sharing mode, performing debouncing processing on the sensor data to obtain optimized sensor data; rendering the display picture according to the optimized sensor data to obtain a target image picture; and sending the target image frames to the anchor side and the audience side so that the anchor side and the audience side display the target image frames. The application can reduce or even avoid discomfort such as dizziness and the like caused by image shake to human eyes.

Description

Screen sharing method, device, equipment and medium

Technical Field

The embodiment of the application relates to the technical field of data processing, in particular to a screen sharing method, device, equipment and medium.

Background

With the development and popularization of Extended Reality (XR) devices, when a user uses the XR device, the screen of the XR device can be shared with other users in a screen sharing manner, so that the experience of the XR device in a multi-user interaction process is improved. The screen sharing is to share the display content of the screen to other people as it is as the name implies.

Currently, when sharing the screen of the XR device to other users, the users wear the XR device, and upload relevant data of the screen content of the XR device to a cloud server by using a terminal device serving as a screen sharing client. And the cloud server calls the graphic processing resource to perform operations such as rendering, audio and video coding and the like based on the received data. And then, the cloud server transmits the processed data to the user and other users, so that the display equipment of each user side performs operations such as decoding, rendering and the like according to the data transmitted by the cloud server, and displays corresponding image frames, audio data and the like on a screen, and a specific flow can be seen in fig. 1.

However, when the image frame of the user XR device screen is subject to high frequency shaking such as pitching, rolling (tumbling), yawing, etc. in three-dimensional space, such shaking may cause the image frame to be subject to unnatural shaking. And the jittering pictures can be delivered to users and other users in real time after being rendered by the cloud server without difference. Therefore, after the user and other users watch the jittering picture, the user is affected by the motion sickness caused by the mismatch of the vestibular vision, and discomfort such as dizziness is obviously felt.

Disclosure of Invention

The application provides a screen sharing method, a device, equipment and a medium, which can reduce or even avoid discomfort such as dizziness and the like caused by image shake to human eyes.

In a first aspect, an embodiment of the present application provides a screen sharing method, including:

responding to sensor data sent by a host, and determining a screen sharing mode selected by the host;

If the screen sharing mode selected by the anchor terminal is a target sharing mode, performing debouncing processing on the sensor data to obtain optimized sensor data;

Rendering the display picture according to the optimized sensor data to obtain a target image picture;

And sending the target image frames to the anchor side and the audience side so that the anchor side and the audience side display the target image frames.

In a second aspect, an embodiment of the present application provides a screen sharing apparatus, including:

The mode determining module is used for responding to the sensor data sent by the anchor terminal and determining a screen sharing mode selected by the anchor terminal;

the debouncing processing module is used for debouncing the sensor data to obtain optimized sensor data if the screen sharing mode selected by the anchor terminal is a target sharing mode;

the image rendering module is used for rendering the display picture according to the optimized sensor data to obtain a target image picture;

And the image distribution module is used for sending the target image frames to the anchor side and the audience side so as to enable the anchor side and the audience side to display the target image frames.

In a third aspect, an embodiment of the present application provides an electronic device, including:

A processor and a memory, the memory is used for storing a computer program, and the processor is used for calling and running the computer program stored in the memory to execute the screen sharing method in the embodiment of the first aspect or various implementation manners thereof.

In a fourth aspect, embodiments of the present application provide a computer-readable storage medium storing a computer program that causes a computer to execute the screen sharing method as described in the first aspect embodiment or implementations thereof.

In a fifth aspect, embodiments of the present application provide a computer program product comprising program instructions which, when run on an electronic device, cause the electronic device to perform a screen sharing method as described in the embodiments of the first aspect or implementations thereof.

The technical scheme disclosed by the embodiment of the application has at least the following beneficial effects:

When receiving sensor data sent by a anchor terminal, determining a screen sharing mode selected by the anchor terminal, if the screen sharing mode is a target sharing mode, performing debouncing processing on the sensor data to obtain optimized sensor data, performing rendering processing on a display picture based on the optimized sensor data to obtain a target image picture, and then sending the target image picture to the anchor terminal and an audience terminal to enable the anchor terminal and the audience terminal to display the target image picture. According to the application, the screen sharing mode selected by the anchor end is determined during screen sharing, and when the screen sharing mode is the target sharing mode, debouncing processing is performed on sensor data, so that fine and high-frequency unnatural jitters in unstable sensor data are filtered out, stable sensor data are obtained, and further, when rendering processing is performed on a display picture based on the stable sensor data, a stable image picture can be rendered, so that discomfort such as dizziness caused to eyes due to image jittering can be reduced or even avoided based on the stable image picture.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for the description of the embodiments will be briefly described below, and it is apparent that the drawings in the following description are only some embodiments of the present application, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

Fig. 1 is a schematic diagram of a screen sharing method provided in the related art;

fig. 2 is a schematic flow chart of a screen sharing method according to an embodiment of the present application;

FIG. 3 is a schematic diagram of a sharing mode selection interface according to an embodiment of the present application;

FIG. 4 is a schematic view of a user looking outward in a spherical model provided by an embodiment of the present application;

FIG. 5 is a flowchart of another screen sharing method according to an embodiment of the present application;

FIG. 6 is a schematic diagram of a debounce process for 6DoF data according to an embodiment of the present application;

fig. 7 is a schematic block diagram of a screen sharing apparatus according to an embodiment of the present application;

Fig. 8 is a schematic block diagram of an electronic device according to an embodiment of the present application.

Detailed Description

The following description of the embodiments of the present application will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present application, but not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are within the scope of the present disclosure.

It should be noted that the terms "first," "second," and the like in the description and the claims of the present application and the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the application described herein may be implemented in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or server that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed or inherent to such process, method, article, or apparatus, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

The method is suitable for an XR equipment screen sharing scene, and when the image picture of the XR equipment screen is shared currently, the image picture can shake in a three-dimensional space due to the movement of a user, such as pitching, rolling, yawing and the like, and the image picture can be subjected to unnatural shake due to the shake, and the shake picture can be rendered through a cloud server and then delivered to the user without difference, so that after the user views the shake picture, the user is influenced by motion sickness caused by the mismatch of vestibular vision, and discomfort such as dizziness and the like can be obviously felt. Therefore, the application designs a screen sharing scheme, so that discomfort such as dizziness and the like caused by image shake to human eyes can be reduced or even avoided through the scheme.

In order to facilitate understanding of the embodiments of the present application, before describing the embodiments of the present application, some concepts related to all embodiments of the present application are explained appropriately, specifically as follows:

1) Virtual Reality (VR), a technology of creating and experiencing a Virtual world, generating a Virtual environment by calculation, is a multi-source information (the Virtual Reality mentioned herein at least comprises visual perception, and may further comprise auditory perception, tactile perception, motion perception, and even further comprises gustatory perception, olfactory perception, etc.), realizes the simulation of a fused and interactive three-dimensional dynamic view and entity behavior of the Virtual environment, immerses a user into the simulated Virtual Reality environment, and realizes application in various Virtual environments such as a map, a game, a video, education, medical treatment, simulation, collaborative training, sales, assistance in manufacturing, maintenance, repair, and the like.

2) The virtual reality device (VR device) may be provided in the form of glasses, a head mounted display (Head Mount Display, HMD), or a contact lens, for realizing visual perception and other forms of perception, but the form of the virtual reality device is not limited thereto, and may be further miniaturized or enlarged according to actual needs.

Optionally, the virtual reality device described in the embodiments of the present application may include, but is not limited to, the following types:

2.1 Computer end virtual reality (PCVR) equipment, the PC end is utilized to carry out the relevant calculation of the virtual reality function and data output, and the external computer end virtual reality equipment utilizes the data output by the PC end to realize the effect of virtual reality.

2.2 Mobile virtual reality device, supporting the setting of a mobile terminal (e.g., a smart phone) in various ways (e.g., a head mounted display provided with a dedicated card slot), performing related calculations of virtual reality functions by the mobile terminal through wired or wireless connection with the mobile terminal, and outputting data to the mobile virtual reality device, e.g., viewing virtual reality video through the APP of the mobile terminal.

2.3 The integrated virtual reality device has a processor for performing the related computation of the virtual function, so that the integrated virtual reality device has independent virtual reality input and output functions, does not need to be connected with a PC end or a mobile terminal, and has high use freedom.

3) Augmented reality (Augmented Reality, AR): a technique for calculating camera pose parameters of a camera in a real world (or three-dimensional world, real world) in real time during image acquisition by the camera, and adding virtual elements on the image acquired by the camera according to the camera pose parameters. Virtual elements include, but are not limited to: images, videos, and three-dimensional models. The goal of AR technology is to socket the virtual world over the real world on the screen for interaction.

4) Mixed Reality (MR): a simulated scenery integrating computer-created sensory input (e.g., a virtual object) with sensory input from a physical scenery or a representation thereof, in some MR sceneries, the computer-created sensory input may be adapted to changes in sensory input from the physical scenery. In addition, some electronic systems for rendering MR scenes may monitor orientation and/or position relative to the physical scene to enable virtual objects to interact with real objects (i.e., physical elements from the physical scene or representations thereof). For example, the system may monitor movement such that the virtual plants appear to be stationary relative to the physical building.

5) Extended Reality (XR) refers to all real and virtual combined environments and human-machine interactions generated by computer technology and wearable devices, which include multiple forms of Virtual Reality (VR), augmented Reality (AR), and Mixed Reality (MR).

Having described some of the concepts to which embodiments of the present application relate, a specific description of a screen sharing method provided by embodiments of the present application is provided below with reference to the accompanying drawings.

Fig. 2 is a flowchart of a screen sharing method according to an embodiment of the present application. The embodiment of the application is suitable for a scene of screen sharing by using an XR device, and the screen sharing method can be executed by a screen sharing device. The screen sharing means may be composed of hardware and/or software and may be integrated in the electronic device. In the embodiment of the present application, the electronic device may be selected as a server, such as a cloud server, and the like, which is not limited herein.

As shown in fig. 2, the method may include the steps of:

S101, responding to sensor data sent by a host, and determining a screen sharing mode selected by the host.

In the embodiment of the application, the anchor refers to an anchor device, and the anchor device is controlled by an anchor user. The main playing device is an XR device, and the XR device may be a VR device, an AR device, or an MR device, which is not particularly limited in the present application.

It is contemplated that the XR device will typically be used with a handheld device so that a user may interact with the XR device using the handheld device. The handheld device can be a handle or a hand controller. Thus, the anchor devices used by anchor users in the present application may be XR devices and handheld devices. Of course, if the XR device has other interactive functions, such as a gesture interactive function, a voice interactive function, or an eye tracking function, the host device that the host user can selectively use in the present application is only the XR device, which is not specifically limited herein.

It should be understood that the sensor data in the present application is specifically 6 degrees of freedom data.

The degree of freedom refers to the number of directions in which a user can move in the 3D space, and the number of directions is the degree of freedom. The number of directions representing degrees of freedom in the present application is 6 in total.

In the present application, 6 degrees of freedom (Degree of Freedom, abbreviated as DoF) means that a user has the ability to rotate in the X, Y, and Z axes as well as the ability to move in the X, Y, and Z axes. Namely, 6DoF includes: translational degrees of freedom and rotational degrees of freedom. Among them, the translational degrees of freedom are divided into 3 types of forward/backward, up/down, and left/right, and the rotational degrees of freedom are divided into 3 types of Pitch (Pitch), roll (Roll), and Yaw (Yaw). In other words, the 6 degrees of freedom may be composed of 3 types of translational degrees of freedom and 3 types of rotational degrees of freedom.

Considering that human motions can be roughly divided into two major classes of rotation and translation, whereas 6DoF is precisely composed of rotational and translational degrees of freedom, an XR device supporting 6DoF can simulate almost all head motions of a user, etc.

That is, whatever the complexity of any possible movement of the object, it can be represented by a combination of translation and rotation, i.e. it can be expressed by 6DoF data.

As described above, when the anchor device used by the anchor user is an XR device and a handheld device, the sensor data in the present application is 6DoF data of the XR device and 6DoF data of the handheld device. When the anchor device used by the anchor user is an XR device, the sensor data in the application is 6DoF data of the XR device.

Alternatively, when the anchor user needs to share the anchor device screen with other users, a virtual room for screen sharing may be created first, and then other users as spectators invited to the virtual room to lay a foundation for subsequent screen sharing.

The spectator user invited here is in fact the spectator-side device. In this embodiment, the viewer device may be any hardware device with a display screen, such as an XR device, a personal computer, a smart phone, or a notebook computer, which is not limited herein.

In an embodiment of the application, creating a virtual room includes the following:

In the first case, if the screen sharing client is installed on the anchor device, the anchor user may control the anchor device to send an application instruction for creating a virtual room to the server using the screen sharing client. And when the server receives the application instruction sent by the anchor device, a virtual room is allocated to the anchor device, and the virtual room identification is returned to the anchor device. And the host device enters the virtual room based on the virtual room identification returned by the server to complete the creation of the virtual room. Where the virtual room identification refers to information capable of uniquely determining a virtual room, such as a virtual room number or a virtual room ID, etc., which is not limited herein.

In the second case, if the screen sharing client is not installed on the anchor device, the anchor user may control the anchor device to transmit an application instruction for creating a virtual room to the server using the terminal device as the screen sharing client. And when the server receives the application instruction sent by the anchor device, a virtual room is allocated to the anchor device, and the virtual room identification is returned to the anchor device. And the host device enters the virtual room based on the virtual room identification returned by the server to complete the creation of the virtual room.

It should be understood that the terminal device may be selected from a personal computer, a smart phone, a notebook computer, or other smart terminals.

Furthermore, the anchor user can share the virtual room identification to the audience terminal equipment controlled by other users based on the anchor terminal equipment so that the audience terminal equipment enters the created virtual room based on the virtual room identification; or the anchor user actively invites the audience end equipment controlled by other users to enter the created virtual room based on the anchor end equipment.

After the anchor device and the viewer device enter the same virtual room, the anchor user may control the anchor device to perform a screen sharing operation in the virtual room.

Optionally, before starting screen sharing, the hosting side device first displays a sharing mode selection interface in the screen of the hosting side device according to the detected screen sharing instruction, where the sharing mode selection interface includes: prompt information, a target sharing mode control and a common sharing mode control, which are shown in fig. 3. The prompt message may be selected as "please select a screen sharing mode you want". When the fact that the anchor user triggers the target sharing mode control is detected, the fact that the anchor user completes the setting operation of the screen sharing mode is indicated, and at the moment, the anchor device closes the sharing mode selection interface. Or when the designated display duration is reached and the trigger instruction of the anchor user is not detected, selecting a target sharing mode for the anchor user by default, and closing a sharing mode selection interface.

It should be understood that the target sharing mode in the present application refers to a debounce sharing mode. The debounce sharing mode is mainly used for filtering out unnatural jitter existing in the 6DoF data to obtain stable 6DoF data.

Furthermore, the anchor user can upload the shared content, such as lectures or live videos, to the server by using the anchor device according to the guide information, so that the server can perform a screen sharing operation to the viewer device in the same virtual room based on the shared content.

After the upload is completed, the hosting end device may begin sharing the screen. When sharing the screen, the anchor user adjusts the image displayed on the screen by controlling the anchor device. Meanwhile, the anchor terminal equipment can acquire the 6DoF data of the anchor terminal equipment in real time and send the acquired 6DoF data to the server in real time, so that the server can execute image rendering and image sharing operation based on the 6DoF data, and screen sharing is realized.

In the embodiment of the application, when the host side equipment acquires the self 6DoF data, the host side equipment can acquire the self 6DoF data in real time by controlling the sensor in the host side equipment.

It should be appreciated that the sensor in the host end device may be a sensor, such as a nine-axis sensor, or an inertial measurement unit (Inertial measurement unit, IMU), etc., without specific limitation.

In addition, when the host device sends the acquired 6DoF data to the server in real time, if the host device is an XR device and a handheld device, the handheld device sends the 6DoF data to the XR device first, and then the XR device sends the 6DoF data of itself and the 6DoF data of the handheld device to the server together according to a preset transmission format. If the master device is an XR device, the XR device sends the 6DoF data of the master device to the server according to a pre-designated transmission format.

The predetermined transmission format may be as follows:

Wherein TransQuaternion rotation represents a quaternion data, transVector position represents a three-dimensional coordinate data, as follows:

considering that in practical use, it is generally necessary to convert quaternions into euler angle matrix data, the present application can convert the quaternions into euler angle matrix data by the following formula:

Wherein q ₀,q₁,q₂ and q ₃ are quaternions, which correspond to x, y, z, and w in TransQuaternion rotation, respectively.

In practical use, when the anchor user controls the anchor device, the body can unconsciously move, and the body movement can cause the image picture of the screen of the anchor device to shake in three-dimensional space, and the shake can cause the image picture to shake unnaturally. Therefore, in order to reduce or even avoid the simulated motion sickness caused by image shake, which causes discomfort such as dizziness to eyes when a user views an image picture displayed by the device, a host user usually selects a debounce sharing mode when sharing a screen, so that the server performs debounce processing based on the debounce sharing mode, and further performs rendering processing on the display picture based on debounce data, so as to obtain a stable image picture, and share the image picture to different user sides.

Based on the above reasons, after the server receives the 6DoF data sent by the anchor device, it can determine whether the screen sharing mode selected by the anchor device is the debouncing sharing mode or the normal sharing mode, and then perform corresponding operations such as image rendering and image sharing according to the determined screen sharing mode.

When the server determines the screen sharing mode selected by the anchor device, the server can determine the screen sharing mode identifier corresponding to the anchor device by querying the data list. And further, according to the screen sharing mode identification, determining the screen sharing mode selected by the anchor device. Wherein the screen sharing mode identifies a screen sharing mode for characterizing the selection of the anchor user.

For example, assuming that a screen sharing mode identifier "true" is used to identify a debounce sharing mode and a screen sharing mode identifier "false" is used to identify a normal sharing mode, when it is determined that the screen sharing mode identifier corresponding to the anchor device is "true", it may be determined that the anchor device has selected the debounce sharing mode. Conversely, when it is determined that the screen sharing mode corresponding to the anchor device is identified as "false", it may be determined that the anchor device selects the normal sharing mode.

The above-mentioned simulated motion sickness is caused by that the state visually observed by the user is not consistent with the real state of the body. For example, when a user sits or stands to manipulate a character to move by means of a handle, the user visually gets information that "me is moving", while the middle ear vestibular organ responsible for sensing the physical state gives the brain a "me not moving" signal, which contradiction signals would make the brain consider "itself" to be in an abnormal and dangerous state. The brain will then have a strong sense of dizziness, thereby alerting the user to the current state as soon as possible.

S102, if the screen sharing mode selected by the anchor terminal is the target sharing mode, performing debouncing processing on the sensor data to obtain optimized sensor data.

When the screen sharing mode selected by the anchor device is determined to be the target sharing mode (debounce sharing mode), the server can perform debounce processing on the 6DoF data sent by the anchor device so as to filter out unnatural jitter in the 6DoF data, and stable 6DoF data, namely optimized sensor data, is obtained, so that conditions are provided for obtaining stable image pictures later.

Optionally, the server in the application can adopt a preset debouncing processing rule to perform debouncing processing on the 6DoF data to obtain optimized 6DoF data. For example, the data smoothing algorithm may be used to smooth the 6DoF data to filter out fine and high-frequency unnatural jitter in the 6DoF data, so as to obtain optimized 6DoF data.

The data smoothing algorithm may be selected from a sliding average method, an exponential sliding average method, a convolution smoothing algorithm (Savitzky-Golay filtering, abbreviated as SG filtering), and the like, which is not particularly limited herein.

And S103, rendering the display picture according to the optimized sensor data to obtain a target image picture.

It should be understood that the target image frame may be selected as a texture image in the present application.

Optionally, the server may import the optimized 6DoF data into a rendering resource to perform a rendering operation of the specified viewport image. Or the server may also import the optimized 6DoF data into the cloud rendering server to perform rendering operations of the specified viewport image, which is not limited herein.

Typically, the image displayed in virtual space is a spherical model, which the user looks outward at, corresponding to standing at the center of the sphere, and because of the limited viewing angle of the human eye, only a small portion of a 360 degree sphere is visible to the user at the same time. When the user rotates the viewing angle, the other part of the image of the sphere can be seen, as shown in fig. 4. The part of the 360-degree sphere seen by the user at the same time is the viewport, and the image displayed on the viewport is the viewport image.

It should be appreciated that since the 6DoF data may determine the location and angle of the anchor user's gaze, the server may determine the gaze location and angle based on the 6DoF data, and thus the corresponding viewport, i.e., the designated viewport, based on the gaze location and angle. Therefore, the server can perform image rendering processing on the display picture corresponding to the appointed view port to obtain the appointed view port image picture.

In some possible implementations, when the server performs rendering processing on the optimized 6DoF data to obtain the target image frame, the server may call the application interface opened by the Steamvr platform, such as the openvr interface, so as to import the optimized 6DoF data according to the import rule of the interface. And then, the Steamvr platform performs rendering operation on the display picture corresponding to the viewport according to the imported optimized 6DoF data to obtain a texture image corresponding to the viewport.

The DriverPose _t data structure externally provided by the openvr interface includes the following information:

The optimized 6DoF data is imported into the following fields in the data structure:

double vecPosition[3]；

double vecVelocity[3]；

double vecAcceleration[3]；

vr::HmdQuaternion_t qRotation；

double vecAngularVelocity[3]；

double vecAngularAcceleration[3]；

In addition, after the optimized 6DoF data is imported into the openvr interface, conversion of the optimized 6DoF data (Rotation and Position) into a Rotation matrix needs to be implemented in the SubmitLayer interface of openvr.

In the application, the rotation matrix can be obtained by calling glm:quat_ cast (rotationMatrix); this general function implementation.

And S104, transmitting the target image frames to a main broadcasting end and a spectator end so that the main broadcasting end and the spectator end display the target image frames.

Optionally, the server of the application can perform processing such as encoding and packaging on the target image picture obtained by rendering, and further synchronously forwards the packaged data to the anchor terminal equipment and the audience terminal equipment in the same virtual room in a SFU (Selective Forwarding Unit) data forwarding mode, so that after the anchor terminal equipment and the audience terminal equipment receive the packaged data sent by the server, the anchor terminal equipment and the audience terminal equipment display stable image pictures in the anchor terminal equipment and the audience terminal equipment by performing operations such as decoding and rendering on the packaged data, and discomfort such as dizziness and the like caused to eyes due to image jitter can be reduced or even avoided based on the stable image pictures.

In the embodiment of the application, the SFU is a server program for routing and forwarding the audio and video data stream of a webpage instant communication (Web Real-Time Communication, webRTC) client. The most core function of the SFU server is to establish a link with each WebRTC Peer client, receive audio and video data from them, and implement one-to-management capability (i.e., forward the stream of one client to the other WebRTC Peer clients).

In another implementation scenario of the present application, if after receiving the 6DoF data sent by the anchor device, it is determined that the screen sharing mode selected by the anchor device is a normal sharing mode. Then, the server performs rendering processing based on the received 6DoF data according to the existing screen sharing mode to obtain a target image picture, and sends the target image picture to the anchor side device and the audience side device.

It can be understood that the application adds the debounce sharing mode based on the existing screen sharing scheme, so as to use the debounce sharing mode to carry out smoothing processing on the 6DoF data with unnatural jitter uploaded by the anchor device, so as to filter out fine and high-frequency jitter existing in the 6DoF data, and obtain stable 6DoF data. Furthermore, after the stable 6DoF data is used for rendering the display picture, a stable image picture can be obtained, so that discomfort such as dizziness caused when a user watches the image picture displayed by the equipment due to simulated motion sickness caused by image shaking can be reduced or even avoided when a host user and a viewer user watch the stable image picture displayed by the equipment.

According to the screen sharing method provided by the embodiment of the application, when the sensor data sent by the anchor terminal is received, the screen sharing mode selected by the anchor terminal is determined, if the screen sharing mode selected by the anchor terminal is determined to be the target sharing mode, the sensor data is subjected to debouncing processing to obtain optimized sensor data, a display picture is subjected to rendering processing to obtain a target image picture based on the optimized sensor data, and then the target image picture is sent to the anchor terminal and the audience terminal, so that the anchor terminal and the audience terminal display the target image picture. According to the application, the screen sharing mode selected by the anchor end is determined during screen sharing, and when the screen sharing mode is the target sharing mode, debouncing processing is performed on sensor data, so that fine and high-frequency unnatural jitters in unstable sensor data are filtered out, stable sensor data are obtained, and further, when rendering processing is performed on a display picture based on the stable sensor data, a stable image picture can be rendered, so that discomfort such as dizziness caused to eyes due to image jittering can be reduced or even avoided based on the stable image picture. In addition, when the audience end and/or the anchor end are XR equipment capable of providing the virtual space, the application can also prolong the immersion time of the user in the virtual space and improve the reuse rate of the user on the basis of reducing or eliminating the dizziness of the user.

According to the method, the device and the system, the debounce processing is carried out on the sensor data according to the debounce sharing mode selected by the anchor end, and then the stable target image picture is rendered based on the optimized sensor data, so that the phenomenon that the rendered image picture is subjected to the debounce is reduced, and the dizziness of a user caused by the image jitter is reduced or even avoided.

On the basis of the above embodiment, the process of debouncing 6DoF data according to the present application will be further explained with reference to fig. 5,

As shown in fig. 5, the method may include the steps of:

S201, determining a screen sharing mode selected by the anchor in response to the sensor data sent by the anchor.

S202, if the screen sharing mode selected by the anchor terminal is a target sharing mode, a target smoothing algorithm is acquired.

S203, determining a target threshold range corresponding to the target smoothing algorithm.

The target smoothing algorithm in the application can be selected from any one of conventional data smoothing algorithms, such as a sliding average method, an exponential sliding average method or a convolution smoothing algorithm (Savitzky-Golay filtering, abbreviated as SG filtering), and the like.

That is, the server may randomly select one algorithm from all the conventional data smoothing algorithms configured as the target smoothing algorithm.

In consideration of the difference between the threshold ranges according to which different smoothing algorithms perform smoothing processing, the server needs to determine a target threshold range corresponding to a target smoothing algorithm after selecting the target smoothing algorithm from the conventional data smoothing algorithms.

Optionally, the application can determine the target threshold range corresponding to the target smoothing algorithm through the corresponding relation between the query smoothing algorithm and the threshold range. In the correspondence between the smoothing algorithm and the threshold range, the threshold range may be a default threshold range of the smoothing algorithm, or may also be a threshold range that is set by a user, such as a producer, based on debounce precision in a custom manner, which is not specifically limited herein.

And S204, performing debouncing processing on the sensor data according to the target smoothing algorithm and the target threshold range to obtain optimized sensor data.

Optionally, the server of the present application first compares the received 6DoF data with a target threshold range to determine whether the 6DoF data is within the target threshold range. If the 6DoF data is determined to be within the target threshold range, the 6DoF data is not naturally dithered, and the 6DoF data needs to be optimized. If the 6DoF data is determined not to be in the target threshold range, the 6DoF data is indicated to be stable data, and at the moment, the 6DoF data is not required to be optimized, so that the view port position is kept unchanged.

When the application carries out debouncing processing on the 6DoF data with unnatural jitter, firstly, an average value is calculated according to a target threshold range, and then the average value is used as a smooth value, so that a server can carry out debouncing processing on the 6DoF data with unnatural jitter by utilizing a target smoothing algorithm based on the smooth value, and stable 6DoF data is obtained.

When the target smoothing algorithm is used for performing debouncing processing on the 6DoF data with unnatural jitter based on the smoothed value, the magnitude relation between the 6DoF data and the smoothed value can be determined first, and if the 6DoF data is greater than the smoothed value, the 6DoF data is adjusted to be a first value. If the 6DoF data is less than or equal to the smoothed value, the 6DoF data is adjusted to a second value.

Wherein the first value is optionally 1 and the second value is optionally 0.

Illustratively, assuming the target threshold range is [ a, b ], and the 6DoF data is k, when k is not located at [ a, b ], the server calculates an average value based on [ a, b ] as: Further, compare k with/> When k is greater than/>When k is smaller than/>, 6DoF data is adjusted to 1At this time, the 6DoF data is adjusted to 0, as shown in fig. 6.

That is, when the application carries out debouncing processing on the 6DoF data based on the smoothed value calculated in the target threshold range, the 6DoF data larger than the smoothed value is adjusted to the first value, and the 6DoF data smaller than or equal to the smoothed value is adjusted to the second value, so that the optimized 6DoF data can keep the same direction instead of the data effect of back and forth adjustment in the positive and negative directions, thus the view port image can be kept in a stable range as much as possible, and the dizziness brought to the eyes of the user due to the shaking of the image picture caused by the small-range motion shaking of the user can be reduced or even avoided.

And S205, rendering the display picture according to the optimized sensor data to obtain a target image picture.

And S206, the target image frames are sent to the anchor side and the audience side so that the anchor side and the audience side display the target image frames.

A screen sharing apparatus according to an embodiment of the present application will be described in detail with reference to fig. 7. Fig. 7 is a schematic block diagram of a screen sharing apparatus according to an embodiment of the present application.

As shown in fig. 7, the screen sharing apparatus 300 includes: a mode determination module 310, a debounce processing module 320, an image rendering module 330, and an image distribution module 340.

The mode determining module 310 is configured to determine, in response to sensor data sent by a anchor, a screen sharing mode selected by the anchor;

The debounce processing module 320 is configured to perform debounce processing on the sensor data to obtain optimized sensor data if the screen sharing mode selected by the anchor is a target sharing mode;

the image rendering module 330 is configured to perform rendering processing on the display screen according to the optimized sensor data, so as to obtain a target image screen;

the image distribution module 340 is configured to send the target image frame to the anchor side and the audience side, so that the anchor side and the audience side display the target image frame.

An optional implementation manner of the embodiment of the present application, the debounce processing module 320 includes:

an algorithm acquisition unit that acquires a target smoothing algorithm;

The range determining unit is used for determining a target threshold range corresponding to the target smoothing algorithm;

And the debouncing unit is used for debouncing the sensor data according to the target smoothing algorithm and the target threshold range to obtain optimized sensor data.

An optional implementation manner of the embodiment of the present application is a debounce unit, specifically configured to:

Determining whether the sensor data is within the target threshold range;

if the value is within the target threshold range, determining a smooth value according to the target threshold range;

And controlling the target smoothing algorithm to perform debouncing processing on the sensor data by using the smoothing value to obtain optimized sensor data.

An optional implementation manner of the embodiment of the present application, the debounce unit is further configured to:

and determining an average value according to the target threshold range, and determining a smooth value from the average value.

If the sensor data is greater than the smoothed value, adjusting the sensor data to a first value;

and if the sensor data is less than or equal to the smoothed value, adjusting the sensor data to a second value.

An optional implementation manner of the embodiment of the present application, the mode determining module 310 is specifically configured to:

determining a screen sharing mode selected by the anchor terminal according to the recorded screen sharing mode identification;

Wherein the screen sharing mode identification is used for representing the screen sharing mode selected by the anchor.

In an optional implementation manner of the embodiment of the present application, the sensor data is 6-degree-of-freedom data.

When receiving sensor data sent by a anchor terminal, the screen sharing device provided by the embodiment of the application determines a screen sharing mode selected by the anchor terminal, if the screen sharing mode selected by the anchor terminal is determined to be a target sharing mode, the sensor data is subjected to debouncing processing to obtain optimized sensor data, a display picture is subjected to rendering processing to obtain a target image picture based on the optimized sensor data, and then the target image picture is sent to the anchor terminal and the audience terminal, so that the anchor terminal and the audience terminal display the target image picture. According to the application, the screen sharing mode selected by the anchor end is determined during screen sharing, and when the screen sharing mode is the target sharing mode, debouncing processing is performed on sensor data, so that fine and high-frequency unnatural jitters in unstable sensor data are filtered out, stable sensor data are obtained, and further, when rendering processing is performed on a display picture based on the stable sensor data, a stable image picture can be rendered, so that discomfort such as dizziness caused to eyes due to image jittering can be reduced or even avoided based on the stable image picture. In addition, when the audience end and/or the anchor end are XR equipment capable of providing the virtual space, the application can also prolong the immersion time of the user in the virtual space and improve the reuse rate of the user on the basis of reducing or eliminating the dizziness of the user.

It should be understood that apparatus embodiments and the foregoing method embodiments may correspond to each other, and similar descriptions may refer to the method embodiments. To avoid repetition, no further description is provided here. Specifically, the apparatus 300 shown in fig. 7 may perform the method embodiment corresponding to fig. 2, and the foregoing and other operations and/or functions of each module in the apparatus 300 are respectively for implementing the corresponding flow in each method in fig. 2, and are not further described herein for brevity.

The apparatus 300 of the embodiment of the present application is described above in terms of functional modules in conjunction with the accompanying drawings. It should be understood that the functional module may be implemented in hardware, or may be implemented by instructions in software, or may be implemented by a combination of hardware and software modules. Specifically, each step of the method embodiment of the first aspect in the embodiment of the present application may be implemented by an integrated logic circuit of hardware in a processor and/or an instruction in software, and the steps of the method of the first aspect disclosed in connection with the embodiment of the present application may be directly implemented as a hardware decoding processor or implemented by a combination of hardware and software modules in the decoding processor. Alternatively, the software modules may be located in a well-established storage medium in the art such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, and the like. The storage medium is located in a memory, and the processor reads information in the memory, and in combination with hardware, performs the steps in the method embodiment of the first aspect.

Fig. 8 is a schematic block diagram of an electronic device according to an embodiment of the present application. As shown in fig. 8, the electronic device 800 may include:

A memory 410 and a processor 420, the memory 410 being adapted to store a computer program and to transfer the program code to the processor 420. In other words, the processor 420 may call and run a computer program from the memory 410 to implement the screen sharing method in the embodiment of the present application.

For example, the processor 420 may be used to perform the screen sharing method embodiments described above in accordance with instructions in the computer program.

In some embodiments of the application, the processor 420 may include, but is not limited to:

A general purpose Processor, digital signal Processor (DIGITAL SIGNAL Processor, DSP), application SPECIFIC INTEGRATED Circuit (ASIC), field programmable gate array (Field Programmable GATE ARRAY, FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, or the like.

In some embodiments of the application, the memory 410 includes, but is not limited to:

Volatile memory and/or nonvolatile memory. The nonvolatile Memory may be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically Erasable EPROM (EEPROM), or a flash Memory. The volatile memory may be random access memory (Random Access Memory, RAM) which acts as external cache memory. By way of example, and not limitation, many forms of RAM are available, such as static random access memory (STATIC RAM, SRAM), dynamic random access memory (DYNAMIC RAM, DRAM), synchronous Dynamic Random Access Memory (SDRAM), double data rate Synchronous dynamic random access memory (Double DATA RATE SDRAM, DDR SDRAM), enhanced Synchronous dynamic random access memory (ENHANCED SDRAM, ESDRAM), synchronous link dynamic random access memory (SYNCH LINK DRAM, SLDRAM), and Direct memory bus RAM (DR RAM).

In some embodiments of the present application, the computer program may be divided into one or more modules, which are stored in the memory 410 and executed by the processor 420 to perform the screen sharing method provided by the present application. The one or more modules may be a series of computer program instruction segments capable of performing the specified functions, which are used to describe the execution of the computer program in the electronic device.

As shown in fig. 8, the electronic device 400 may further include:

A transceiver 430, the transceiver 430 being connectable to the processor 420 or the memory 410.

The processor 420 may control the transceiver 430 to communicate with other devices, and in particular, may send information or data to other devices or receive information or data sent by other devices. Transceiver 430 may include a transmitter and a receiver. Transceiver 430 may further include antennas, the number of which may be one or more.

It will be appreciated that the various components in the electronic device are connected by a bus system that includes, in addition to a data bus, a power bus, a control bus, and a status signal bus.

The present application also provides a computer storage medium having stored thereon a computer program which, when executed by a computer, enables the computer to perform the method of the above-described method embodiments.

The embodiment of the application also provides a computer program product comprising program instructions which, when run on an electronic device, cause the electronic device to perform the method of the above-described method embodiment.

When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, produces a flow or function in accordance with embodiments of the application, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by a wired (e.g., coaxial cable, fiber optic, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that contains an integration of one or more available media. The usable medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a digital video disc (digital video disc, DVD)), or a semiconductor medium (e.g., a Solid State Drive (SSD)), or the like.

Those of ordinary skill in the art will appreciate that the various illustrative modules and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

In the several embodiments provided by the present application, it should be understood that the disclosed systems, devices, and methods may be implemented in other manners. For example, the apparatus embodiments described above are merely illustrative, and for example, the division of the modules is merely a logical function division, and there may be additional divisions when actually implemented, for example, multiple modules or components may be combined or integrated into another system, or some features may be omitted or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or modules, which may be in electrical, mechanical, or other forms.

The modules illustrated as separate components may or may not be physically separate, and components shown as modules may or may not be physical modules, i.e., may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. For example, functional modules in various embodiments of the present application may be integrated into one processing module, or each module may exist alone physically, or two or more modules may be integrated into one module.

The foregoing is merely illustrative of the present application, and the present application is not limited thereto, and any person skilled in the art will readily recognize that variations or substitutions are within the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A screen sharing method, comprising:

2. The method of claim 1, wherein the debouncing the sensor data to obtain optimized sensor data comprises:

acquiring a target smoothing algorithm;

determining a target threshold range corresponding to the target smoothing algorithm;

And performing debouncing processing on the sensor data according to the target smoothing algorithm and the target threshold range to obtain optimized sensor data.

3. The method according to claim 2, wherein the performing a de-dithering process on the sensor data according to the target smoothing algorithm and the target threshold range to obtain optimized sensor data includes:

Determining whether the sensor data is within the target threshold range;

4. A method according to claim 3, wherein determining a smoothed value from the target threshold range comprises:

5. A method according to claim 3, wherein controlling the target smoothing algorithm to perform a de-dithering process on the sensor data using the smoothed value to obtain optimized sensor data comprises:

6. The method of claim 1, wherein the determining the screen sharing mode selected by the anchor comprises:

7. The method of any one of claims 1-6, wherein the sensor data is 6 degrees of freedom data.

8. A screen sharing apparatus, comprising:

9. An electronic device, comprising:

A processor and a memory for storing a computer program, the processor for invoking and running the computer program stored in the memory to perform the screen sharing method of any of claims 1 to 7.

10. A computer-readable storage medium storing a computer program that causes a computer to execute the screen sharing method according to any one of claims 1 to 7.

11. A computer program product comprising program instructions which, when run on an electronic device, cause the electronic device to perform the screen sharing method of any of claims 1 to 7.