WO2024061243A1

WO2024061243A1 - Live stream interactive method, device, apparatus and storage medium

Info

Publication number: WO2024061243A1
Application number: PCT/CN2023/119866
Authority: WO
Inventors: Zixiong LUO
Original assignee: Beijing Source Technology Co., Ltd.
Priority date: 2022-09-19
Filing date: 2023-09-19
Publication date: 2024-03-28
Also published as: CN117221615A; CN115695831A

Abstract

The present application discloses a live stream interactive method, a device, an apparatus and a storage medium. The method includes: acquiring a request to join live streamer game sent by a first user terminal, where the request includes a user terminal identifier, a virtual interactive room identifier, and a game identifier; parsing the request to join the live streamer game sent by the first user terminal and forwarding the request to a live streamer terminal; and acquiring feedback on the request to join the live streamer game from the live streamer terminal, and based on the feedback on the request and a live streamer game type, instructing to run a user terminal game corresponding to a live streamer terminal game and sending real-time game images and/or sounds of the user terminal game to the first user terminal. In the present application, through the live stream server, the command processing and distribution of live stream content is unified, which enhances the interactivity between the audience and the live streamer.

Description

LIVE STREAM INTERACTIVE METHOD, DEVICE, APPARATUS AND STORAGE MEDIUM

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application claims priority to Chinese Patent Application No. 202211147872.1, filed on September 19, 2022, entitled “Live Stream Interactive Method, Device, Apparatus and Storage Medium” , which is incorporated herein by reference in its entirety.

TECHNICAL FIELD

The present application relates to the field of live stream, in particular to a live stream interactive method, a device, an apparatus and a storage medium.

BACKGROUND

With the development of Internet technology, the network virtual interactive room attracts more and more users with its characteristics of novelty, real-time performance and interactivity.

As a mainstream form of live stream, game live has developed rapidly in recent years. At present, in the process of game live, usually the live streamer plays the game, while the users watch the live streamer play the game. The interaction between the live streamer and the users is relatively simple, and the interaction needs to be improved, since the experience of interaction between the live streamer and the users is affected.

Therefore, it is urgent to solve the problems in the related art.

SUMMARY

The present application provides a live stream interactive method, a device, an apparatus and a storage medium to provide a new interactive mode of game live and improve the experience of interaction between the live streamer and the users.

According to a first aspect, embodiments of the present application provide a live stream interactive method, applied to a live stream server, the method including:

acquiring a request to join live streamer game sent by a first user terminal, where the request includes a user terminal identifier, a virtual interactive room identifier, and a game identifier;

parsing the request to join the live streamer game sent by the first user terminal and forwarding the request to a live streamer terminal; and

acquiring feedback on the request to join the live streamer game from the live streamer terminal, and based on the feedback on the request and a live streamer game type, instructing to run a user terminal game corresponding to a live streamer terminal game and sending real-time game images and/or sounds of the user terminal game to the first user terminal.

In an embodiment, the instructing to run a user terminal game corresponding to a live streamer terminal game and sending real-time game images and/or sounds of the first user terminal game to the first user terminal, further includes:

generating cloud game information based on the request to join live streamer game sent by the first user terminal and sending the cloud game information to a cloud game server, and when the live streamer terminal agrees to the first user terminal joining the live streamer game and the live streamer game is a cloud game,

instructing the cloud game server to run the user terminal game corresponding to the live streamer terminal game based on the cloud game information; and

instructing the cloud game server to send the real-time game images and/or sounds of the user terminal game to the first user terminal.

In an embodiment, the method further includes:

the live stream server acquiring images and/or sounds under current live stream status from the cloud game server;

the live stream server acquiring images and/or sounds under the current live stream status from the live streamer terminal;

the live stream server mixing and processing the images and/or sounds from the cloud game server

and/or

the images and/or sounds from the live streamer terminal to generate live stream distribution images and/or sounds; and

the live stream server distributing the live stream distribution images and/or sounds to a second user terminal.

In an embodiment, the images from the cloud game server acquired by the live stream server under the current live stream status are low-frame-rate images processed by frame extraction and compression.

In an embodiment, the method further includes:

optimizing the sounds from the cloud game server and the live streamer terminal under the current live stream status acquired by the live stream server using a mixing and noise reduction algorithm.

In an embodiment, the method further includes:

the live stream server distributing acquired images from the cloud game server under the current live stream status to a third user terminal, where the images from the cloud game server under the current live stream status are high-frame-rate images originally rendered.

In an embodiment, the method further includes:

when the first user terminal does not match the live streamer game, instructing the cloud game server to send reference data for generating operation simulator corresponding to the live streamer game to the first user terminal, to allow the first user terminal to generate the operation simulator based on the reference data for converting game operation signals between the first user terminal and the cloud game server.

In an embodiment, the instructing to run a user terminal game corresponding to a live streamer terminal game and sending real-time game images and/or sounds of the user terminal game to the first user terminal further includes:

when the live streamer terminal agrees to the user terminal joining the live streamer game, and the live streamer game is a non-cloud game,

connecting the first user terminal to the live streamer terminal as a sub-input device; and

acquiring and sending the real-time images and/or sounds of the live streamer terminal game to the first user terminal.

In an embodiment, the method further includes:

when the first user terminal does not match the live streamer game, sending the reference data for generating operation simulator corresponding to the live streamer game to the first user terminal, to allow the first user terminal to generate the operation simulator based on the reference data for converting game operation signals between the first user terminal and the live streamer terminal.

In an embodiment, the method further includes:

receiving a message of providing a permission to join live streamer game sent by the live streamer terminal, where the message of providing permission to join live streamer game includes a virtual interactive room identifier and a game identifier; and

sending the message of providing a permission to join live streamer game to the first user terminal and parsing the message to obtain the virtual interactive room identifier and the game identifier.

According to a second aspect, embodiments of the present application provide a live stream interactive method, applied to a live stream server, the method including:

a first user terminal receiving a message of providing a permission to join live streamer game sent by a live stream server, and parsing the message to obtain a virtual interactive room identifier and a game identifier;

the first user terminal sending a request to join the live streamer game to the live stream server, where the request includes a user terminal identifier, the virtual interactive room identifier, and the game identifier; and

the first user terminal receiving real-time images and/or sounds of a user terminal game corresponding to a live streamer terminal game instructed to run by the live stream server.

In an embodiment, the method further includes:

the first user terminal sending game control commands to interact with the user terminal game corresponding to the live streamer terminal game.

In an embodiment, the method further includes:

when the live streamer terminal agrees to the first user terminal joining the live streamer game, and the live streamer game is a cloud game,

receiving the real-time images and/or sounds of the user terminal game sent by the cloud game server.

In an embodiment, the method further includes:

when the first user terminal does not match the live streamer game, receiving reference data for generating operation simulators corresponding to the live streamer game sent from the cloud game server instructed by the live stream server; and

In an embodiment, the method further includes:

when the live streamer terminal agrees to the user terminal joining the live streamer game, and the live streamer game is a non-cloud game, connecting the first user terminal to the live streamer terminal as a sub-input device; and

receiving the real-time images and/or sounds of the live streamer terminal game sent by the live stream server.

In an embodiment, the method further includes:

when the first user terminal does not match the live streamer game, receiving reference data for generating operation simulators corresponding to the live streamer game instructed to send by the live stream server to generate the operation simulator; and

the first user terminal sending game control commands to interact with the non-cloud game.

According to a third aspect, embodiments of the present application further provide a live stream interactive device, including:

an acquirer, configured to acquire a request to join a live streamer game sent by a first user terminal, where the request includes a user terminal identifier, avirtual interactive room identifier and a game identifier; and acquire feedback from a live streamer terminal on the request to join the live streamer game.

a parser, configured to parse the request to join the live streamer game sent by the first user terminal.

a sender, configured to send the request to join the live streamer game to the live streamer terminal; and

a generator, configured to instruct to run a user terminal game corresponding to a live streamer terminal game and sending real-time game images and/or sounds of the user terminal game to the first user terminal based on the feedback on the request and a live streamer game type.

In an embodiment, when the live streamer terminal agrees to the first user terminal joining the live streamer game and the live streamer game is a cloud game,

the generator is further configured to generate cloud game information based on the request to join live streamer game sent by the first user terminal;

the sender is further configured to send the cloud game information to the cloud game server, instruct the cloud game server to run the user terminal game corresponding to the live streamer terminal game based on the cloud game information, and instruct the cloud game server to send the real-time game images and/or sounds of the user terminal game to the first user terminal.

In an embodiment, the acquirer is configured to:

acquire images and/or sounds under current live stream status from the cloud game server;

acquire images and/or sounds under the current live stream status from the live streamer terminal;

the generator is further configured to mix and process the images and/or sounds of the cloud game server and/or live streamer terminal to generate live stream distribution images and/or sounds; and

the sender is further configured to distribute the live stream distribution images and/or sounds to a second user terminal.

In an embodiment, the images of the cloud game server acquired by the acquirer under the current live stream status are low-frame-rate images processed by frame extraction and compression.

In an embodiment, the generator is further configured to optimize the sounds from the cloud game server and the live streamer terminal under the current live stream status acquired by the live stream server using a mixing and noise reduction algorithm.

In an embodiment, when the first user terminal does not match the live streamer game, the sender is further configured to instruct the cloud game server to send reference data for generating operation simulator corresponding to the live streamer game to the first user terminal, to allow the first user terminal to generate the operation simulator based on the reference data for converting game operation signals between the first user terminal and the cloud game server.

In an embodiment, when the live streamer terminal agrees to the user terminal joining the live streamer game, and the live streamer game is a non-cloud game,

the sender is further configured to connect the first user terminal to the live streamer terminal as a sub-input device;

the acquirer is further configured to acquire the real-time images and/or sounds of the live streamer terminal game; and

the sender is further configured to send the real-time images and/or sounds of the live streamer terminal game to the first user terminal.

In an embodiment, when the first user terminal does not match the live streamer game, the sender is further configured to instruct the cloud game server to send the reference data for generating operation simulator corresponding to the live streamer game to the first user terminal, to allow the first user terminal to generate the operation simulator based on the reference data for converting game operation signals between the first user terminal and the live streamer terminal.

In an embodiment, the acquirer is further configured to acquire a message of providing a permission to join live streamer game sent by the live streamer terminal, where the message of providing permission to join live streamer game includes a virtual interactive room identifier and a game identifier;

the sender is further configured to send the message of providing permission to join live streamer game to the first user terminal; and

the parser is further configured to parse the message of providing permission to join live streamer game to obtain the virtual interactive room identifier and the game identifier.

According to a fourth aspect, embodiments of the present application further provide a live stream interactive device, including:

a receiver, configured to receive a message of providing a permission to join live streamer game sent by a live stream server and parse the message to obtain a virtual interactive room identifier and a game identifier.

a sender, configured to send a request to join live streamer game to the live server, where the request includes a user terminal identifier, the virtual interactive room identifier, and the game identifier.

the receiver is further configured to receive real-time images and/or sounds of a user terminal game corresponding to a live streamer terminal game instructed to run by the live stream server.

According to a fifth aspect, embodiments of the present application provide a live stream interactive device, including:

a receiver, configured to receive cloud game information;

a parser, configured to parse the cloud game information, and acquire the virtual interactive room identifier, user terminal identifier, and game identifier;

a starter, configured to start the game based on the game identifier, and add the user terminal to the live streamer game; and

a sender, configured to send content of live streamer game.

In an embodiment, the receiver is further configured to receive game operation instructions.

According to a sixth aspect, embodiments of the present application further provide an electronic apparatus, including:

one or more processors;

a storage device configured to store one or more programs; and

where when one or more programs are executed by the one or more processors, the one or more processors implement the method described above.

According to an eighth aspect, embodiments of the present application further provide a computer-readable storage medium, with a computer program stored thereon; when the program is executed by a processor, the method described above is implemented.

The embodiments of the present application provide, by means of the above technical solution, a new mode of game live stream interaction, which enables users to play games with a live streamer without special environmental restrictions using a simpler process and fewer devices, and enhances the experience of interaction between the user and the live streamer. Through the live stream server, according to the type of live streamer game and the type of user terminal, the command processing and distribution of live content is unified, which enhances the interactivity between the live audience and the live streamer and improves the user experience.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a flow diagram of a live stream interactive according to an embodiment of the present application;

FIG. 2 is a flow diagram of a live stream interactive method according to another embodiment of the present application;

FIG. 3 is a flow diagram of a live stream interactive method according to yet another embodiment of the present application;

FIG. 4 is a flow diagram of a live stream interactive method according to yet another embodiment of the present application;

FIG. 5 is a flow diagram of a live stream interactive method according to yet another embodiment of the present application;

FIG. 6 is a schematic structural diagram of a live stream interactive system according to an embodiment of the present application;

FIG. 6 (a) is a schematic structural diagram of a live stream server and a cloud game server of a live stream interactive system according to an embodiment of the present application;

FIG. 7 is a schematic diagram of a live stream interactive device according to an embodiment of the present application;

FIG. 8 is a schematic diagram of a live stream interactive device according to another embodiment of the present application;

FIG. 9 is a schematic diagram of a live stream interactive device according to yet another embodiment of the present application; and

FIG. 10 is a schematic diagram of an electronic apparatus according to an embodiment of the present application;

DETAILED DESCRIPTION OF EMBODIMENTS

Further details of the present application are given below in conjunction with the drawings and embodiments. It is understood that the embodiments are intended only to interpret and not limit the present application. It should also be noted that, for the purposes of description, only part of the structure relevant to the present application is shown in the drawings, not all of it.

Before discussing exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow diagrams. Although the flow diagram describes the steps as sequential processes, many of them can be implemented in parallel, concurrently, or simultaneously. In addition, the sequence of steps can be rearranged. The process may be terminated when the operation is completed, but there may also be additional steps that are not included in the drawings. The process may correspond to a method, function, procedure, subroutine, subprogram, and so on.

Game operation instruction mentioned herein refers to the operation instruction in the game. For example, the user sends the instruction to the game through the keyboard and mouse to realize the actions such as walking and attacking characters in the game, so as to affect the game process.

Live stream mentioned herein refers to the live stream service run through the live stream server, including acquisition of live streamer images through signal acquisition apparatus, import to the platform, upload to the server through the network, and release to the user terminal for users to watch.

Virtual interactive room mentioned herein refers to a virtual space (such as live stream room, real-time live stream, live streamer room, etc. ) that provides users with real-time synchronous update information exchange on the network, which can realize different functions such as live stream, online chat, and so on.

Game live streamer or live streamer mentioned herein refers to a “game streamer” who establishes, operates or manages a game, and accordingly, live streamer terminal, live streamer room and the like refer to a terminal, a room and the like in which the game streamer is virtually located.

User mentioned herein refers to “viewer” , “listener” and other users who can send and receive virtual interactive content online, rather than “live streamer” or “room host” .

Cloud game mentioned herein refers to a game mode that relies on a cloud game server.

Non-cloud game mentioned herein refers to a game mode that does not rely on the cloud game server. Typically, the non-cloud games can be run directly on a “live streamer terminal” or “host terminal” .

FIG. 6 is a schematic structural diagram of a live stream interactive system according to an embodiment of the present application. The system may include a user terminal (including a first user terminal, a second user terminal, and a third user terminal) , a live streamer terminal, a live stream server, and a cloud game server.

FIG. 1 is a flow diagram of a live stream interactive method according to an embodiment of the present application. The embodiment of the present application may be applied in the case of game live stream interaction. The method may be performed by a live stream interactive device (system) , implemented by software and/or hardware, and specifically configured in an electronic apparatus, which may be a mobile terminal or a fixed terminal.

As shown in FIG. 1, the live stream interactive method according to the embodiment of the present application is applied to a live stream server, the method including:

S110, acquiring a request to join live streamer game sent by a first user terminal, wherein the request comprises a user terminal identifier, a virtual interactive room identifier, and a game identifier.

It can be understood that in the process of target game live, the game live streamer in the virtual interactive room may collect audio and video data through the live streamer terminal such as mobile phone and computer, and then send the live stream data to the live stream server. At this time, users who want to watch the game live may obtain live stream data of the virtual interactive room from the live stream server through mobile phones, computers and other user terminals, where user terminal refers to the terminal held by the user watching the game live, and live streamer terminal refers to the terminal held by the game live streamer in the virtual interactive room.

In the process of live streamer live, the live streamer may choose to invite a user to play a game together. The live streamer may operate on the live streamer terminal to grant the “permission to join the live streamer game” that allows users to join the game. After receiving the information, the live stream server will notify users that the live streamer has granted the “permission to join the live streamer game” . After users make the “request to join the live streamer game” on the user terminal, they start to queue up, and the live streamer chooses a user to play the game together. The live streamer may set the way of making the “request to join the live streamer game” by itself. In one way, users may apply after giving the required gift; in the other way, users may apply without any conditions. In addition to these two ways, there are also other ways that may be used. Depending on the type of game, the live streamer may also set the number of users to play the game together, and choose the desired users in the queue to play the game together.

The above request to join the live streamer game includes the virtual interactive room identifier and the user terminal identifier. The virtual interactive room identifier is used to distinguish the live streamer room, the user terminal identifier is used to distinguish the user terminal, and the game identifier is used to distinguish the game played by the live streamer, so as to ensure that the same game is running in the user terminal and the live streamer terminal.

In accordance with some embodiments of the present application, the target game may be installed locally on the live streamer terminal, and the game live streamer may open the target game locally for live stream, so as to avoid the game restrictions of the live stream game library in the live stream server and the corresponding network restrictions. Of course, the target game may also be a game in the live stream game library.

Where the state of the target game on the live streamer terminal may include game not started, game paused, in-game, single player game and multiplayer game. In the process of target game live, the live streamer terminal may visually display the state of the corresponding target game on the user terminal through the live stream server. For example, different display icons may be set for different game states, and different display effects may be set accordingly.

When a user on the user terminal wants to join the virtual interactive room game and interact with the live streamer, it may initiate a request to join the live streamer game according to the state of the target game. For example, if the game live streamer has not set any co-play conditions, and the state of the target game is multiplayer game, the user terminal may send a request to join the live streamer game to the live streamer terminal; if the state of the target game is not multiplayer game, the user terminal cannot send a request to join the live streamer game to the live streamer terminal.

The first user terminal in this embodiment refers to a player user terminal that joins the live streamer game and interacts with the live streamer in the game.

In this embodiment, the live stream server may act as a hub for data transmission between the user terminal and the live streamer terminal, storing and forwarding the request to join the live streamer game initiated by the user terminal to the live streamer terminal; at the same time, when the live streamer terminal gives feedback on the results of the application for joining the game, it may store and forward the results of the application for joining the game sent by the live streamer terminal to the user terminal.

It can be understood that the live stream server may be used as a specialized data transmission hub to reduce the number of interactions between the user terminal and the live streamer terminal as well as the number of protocol headers contained when receiving data. For example, in some embodiments, multiple user terminals may send the request to join the game to the live streamer terminal at the same time. In this case, the live stream server may first receive these requests to join the game, and then send the requests to join the game to the live streamer terminal after unified sorting of them.

S120, parsing the request to join the live streamer game sent by the first user terminal and forwarding the request to a live streamer terminal.

Where the request to join the live streamer game may contain user information (i.e., the information carried by the user terminal identifier) and game information, etc., or only contain user information, including user account and user avatar, etc. The user information may be that provided to the live streamer, and the game information may be the specific content of a multiplayer game that the user suggests to the live streamer. If no game information is contained, by default, the user wants to play the current game with the live streamer.

S130, acquiring feedback on the request to join the live streamer game from the live streamer terminal, and based on the feedback on the request and a live streamer game type, instructing to run a user terminal game corresponding to a live streamer terminal game and sending real-time game images and/or sounds of the user terminal game to the first user terminal.

The live streamer may choose, according to the current situation, to agree on or refuse the “request to join the live streamer game” . When it is agreed, it indicates that the live streamer wants the user to join the target game; when it is refused, it indicates that the live streamer does not want the user to join the target game. Meanwhile, the live streamer terminal may observe the information of user application, which is displayed on the live streamer terminal in the form of user queue. The live streamer may freely choose the users to play the game together. When the number of users is full, other users who are not directly refused by the live streamer will be regarded as refused by default. The live streamer terminal sends feedback to the live stream server on whether it agrees with the user terminal’s request to join the live streamer game. After receiving the feedback information, the live stream server, based on the feedback results and the type of the live streamer game, commands to run the user terminal game corresponding to the live streamer terminal game and sends real-time game images and/or sounds of the user terminal game to the first user terminal.

Here, the first user terminal may directly receive the content of the live streamer game sent by the cloud game server.

FIG. 2 is a flow diagram of a live stream interactive method according to another embodiment of the present application. As shown in FIG. 2, when the live streamer terminal agrees to the first user terminal joining the live streamer game and the live streamer game is a cloud game:

S210, generating cloud game information based on the request to join live streamer game sent by the first user terminal and sending the cloud game information to a cloud game server;

S220, instructing the cloud game server to run the user terminal game corresponding to the live streamer terminal game based on the cloud game information; and

S230, instructing the cloud game server to send the real-time game images and/or sounds of the user terminal game to the first user terminal.

After receiving the confirmation that the live streamer agrees on users’ joining, the live stream server will parse the virtual interactive room identifier, the user terminal identifier and the game identifier. When the live streamer terminal agrees to the user terminal joining the live streamer game, and the live streamer game is a cloud game, cloud game information may be generated, and sent to the cloud game server, or directly send the game identifier alone to the cloud game server. In some embodiments, cloud game information is generated based on the virtual interactive room identifier, user terminal identifier, and game identifier, and depending on the system settings, it is also possible to generate cloud game information based on the game identifier only and the cloud game information is sent to the cloud game server. The application here depends on the way to set up the server, so there are several ways. It can be understood that in the following embodiments, although one of the ways described in the paragraph is used, other different ways may be applied, i.e., a certain logo may be sent or the information combined with the logo may be packaged and sent to the cloud game server.

Where it is considered that the corresponding target game is usually not installed on the user terminal during the game live, because the game live initiated by the live streamer is random to some extent. Therefore, the cloud game server may realize game live stream interaction by starting the same game as on the live streamer terminal.

The cloud game server may communicate with the live stream server and they trust each other, so as to realize the live stream interaction between the users and the live streamer through the cloud game server. The cloud game server may parse the information sent by the live stream server and run the corresponding game.

Where the cloud game server will start the same game as the live streamer game according to the cloud game information or the game selected by the live streamer according to users’ wishes, that is, the user terminal runs the cloud game on the server. In one way, the cloud game server allows the establishment of virtual computer to run the game, and the user directly controls the game running on the virtual computer by carrying out corresponding operations on the user terminal. Of course, the cloud game server may directly simulate the game environment and run the game without establishment of virtual computer, and the user may also operate on the user terminal for corresponding control. The game running in the virtual computer or simulated game environment is connected with the live streamer game.

It should be noted that the live streamer may not need to actively confirm objects of connection, for example, when a user is trusted by the live streamer and granted permission to join the game as a VIP in advance. The user may be a “number one fan” recognized by the live streamer. In this case, the live streamer may set the user as a special user. As long as the user submits the application for connection, the live streamer does not need to manually choose agreeing on the request, and the user may join. The specific implementation mode may be selected and set by the live streamer according to the needs of the scene.

In an embodiment, the cloud game server may consist of at least one server host. Typically, the number of server hosts on cloud game server is multiple. In the present embodiment, various types of games are pre-installed on each server. The games installed on different servers may be the same or different, and the choice may be made according to the actual demand and usage.

In some embodiments, before sending the cloud game start instruction to the cloud game server, it further includes: obtaining information about the local application program installed in the user terminal; according to the information about the local application program, determining whether the user terminal has a local game corresponding to the target game installed; if so, regarding the local game as the game application to be accessed, and executing the sending operation of stopping sending the cloud game start instruction to the cloud game server. Before obtaining the information about the local application program, the live stream server will send a request to the user terminal, including a request for scanning and a request for live stream of the user terminal image. If the user does not agree, the cloud game will be run.

In an embodiment, before sending the cloud game start instruction to the cloud game server, the user terminal application program may be scanned to obtain the information about the local application program installed on the user terminal. By analyzing and judging the information about the local application program, whether the target game is installed locally on the user terminal may be determined. If the target game is installed locally on the user terminal, the user may access the room of the target game through the local game, avoiding the waste of resources and traffic, and optimizing the process of the user terminal accessing the room of the target game.

FIG. 3 is a flow diagram of a live stream interactive method according to yet another embodiment of the present application. As shown in FIG. 3, the method includes:

S310, the live stream server acquiring images and/or sounds under current live stream status from the cloud game server;

S320, the live stream server acquiring images and/or sounds under the current live stream status from the live streamer terminal;

S330, the live stream server mixing and processing the images and/or sounds from the cloud game server

and/or

the images and/or sounds from the live streamer terminal to generate live stream distribution images and/or sounds;

S340, the live stream server distributing the live stream distribution images and/or sounds to a second user terminal.

It can be understood that when multiple players join the live streamer game for interaction, each player presents the game’s images and/or sounds from their own first-person perspective. However, in a live stream scenario, the live streamer and the audience often need to simultaneously view images and/or sounds from multiple player perspectives, so as to have a more comprehensive understanding of the game status and details. Therefore, the present application aims to real-time acquire images and/or sounds from the cloud game server and the live streamer terminal under the current live streaming status through the live stream server. Subsequently, these images and/or sounds are mixed and processed so that live stream images and/or sounds are generated and then distributed to the second user terminal.

The second user terminal in this embodiment refers to the audience’s terminal for viewing the live stream game.

In an embodiment, the distributed images and/or sounds in the live stream can be images and/or sounds from all player perspectives or from a single player perspective. The audience can switch images and/or sounds between all player perspectives and a single player perspective according to their preferences. They may also choose to synchronize with a specific player’s perspective.

In an embodiment, the images from the cloud game server acquired by the live stream server are low-frame-rate images that have been processed by frame extraction and compression.

Specifically, to better suit the live streaming scenario, the cloud game server renders high frame rate images (e.g., 60 frames per second) during running the game and simultaneously compresses and encodes before sending to the first user terminals of participating players. Meanwhile, the cloud game server evenly extracts frames from the high frame rate images at a lower frame rate (e.g., 30 frames per second) to acquire low-frame-rate images, which are then transmitted to the live stream server.

Directly sending high-frame-rate images to players, i.e., the first user terminals, ensures that users participating in the game have a good experience. Transmitting low-frame-rate images to the live stream server and distributing them to the second user terminals improves the network transmission efficiency between the live stream server and the large number of viewers, i.e., the second user terminals. This not only saves costs but also better ensures synchronization between viewers and players.

In an embodiment, the cloud server directly transmits low-frame-rate images to the live stream server using technologies such as RDMA (high-speed communication within a local network) and GPU shared memory (communication technology for multiple cards on the same physical machine) .

FIG. 6 (a) is a schematic structural diagram of a live stream server and a cloud game server of a live stream interactive system according to an embodiment of the present application. As shown in FIG. 6 (a) , the live stream server and the cloud game server can be disposed on the same physical machine. Through virtualization technology, multiple virtual machines can be created on a single physical machine. For example, an 8-GPU physical machine can virtualize one live stream server and seven cloud game servers. These seven cloud game servers can be on the same physical machine with the live stream server, allowing for high-speed interconnection.

A single GPU can also be split into multiple GPUs based on GPU virtualization technology. In this case, an 8-GPU physical machine can virtualize 16 or even more cloud game servers. Using network technologies based on RDMA, such as infiniband and RoCE, the transfer speed between machines can exceed 100 Gb/s, which is sufficient for transmitting multiple original video streams, thus avoiding wastage of computational resources and loss of image quality caused by repeated encoding and decoding.

Additionally, compressing low-frame-rate images through audio and video encoding and transmitting them to the live stream server via traditional network technologies such as Ethernet further enhances network transmission efficiency.

In an embodiment, the method also includes optimizing the audio from the cloud game server and from the live streamer terminal under the current live streaming status using a mixing and noise reduction algorithm.

It can be understood that when multiple players join the live streamer game for interaction, there may be voice communication between players, or through other voice tools. When the audio from all players is aggregated together, echoes or complex combinations of multiple sounds may occur. Therefore, by applying mixing and noise reduction processing to the audio from the cloud game server and the audio from the live streamer terminal, the viewing experience for the audience can be improved. Specifically, various methods, including but not limited to AEC algorithms and AI-based algorithms, can be used for mixing and noise reduction processing.

In an embodiment, the images from the cloud game server acquired by the live stream server are high-frame-rate images originally rendered.

The live stream server distributes the original high-frame-rate images to the third user terminals.

Specifically, during the live streaming process, the live stream server can also acquire the originally rendered high-frame-rate images from the cloud server based on the audience’s levels and permissions. These images are then distributed to special audience terminals, i.e., third user terminals, with corresponding viewing permissions, thereby enhancing the first-person experience and game interaction for special audiences.

In an embodiment, the method further includes: when the first user terminal does not match the live streamer game, instructing the cloud game server to send reference data for generating the operation simulator corresponding to the live streamer game, to allow the first user terminal to generate the operation simulator based on the reference data for converting game operation signals between the first user terminal and the cloud game server.

In this embodiment, it is understood that different computing capabilities of terminals imply differences in hardware, operating systems, and input systems. Therefore, when the first user terminal and the live streamer terminal join the same room to play a game together, It is necessary to convert the operation signals of the first user terminal and/or the live streamer terminal, and convert the operation signals of each terminal into real operations of the target game, so as to achieve the goal of jointly operating the game. In this embodiment, the operation signals of the first user terminal can be converted to match the operation signals of the live streamer terminal.

Specifically, the user terminal can be a mobile phone, computer, or other types of terminals, while the game being played by the live streamer may not match the user terminal. For example, if the live streamer is playing a mobile game and the user is watching the live stream on a computer, the cloud game server will automatically detect the user terminal. If the terminal does not match the game, it will send reference data for generating an operation simulator corresponding to the live streamer game to the user’s computer, meaning that simulate the mobile interface on the user’s computer, allowing the user to input commands through the simulator, thus achieving the goal of playing the game together with the live streamer. Of course, if the user terminal is a mobile phone and the live streamer terminal is a computer, or if there are other terminals, the similar process can be applied.

For another instance, if the user terminal is a mobile phone and the live streamer terminal is a computer, and the target game is a board game, the operation signals of both the user terminal and the live streamer terminal can be converted to the operation signals of a computer.

The operation signals include but are not limited to signals for moving up, down, left, and right, and specific operation signals can be set according to the specific content of the target game. The embodiment in the present application does not impose specific limitations on this.

In some embodiments, the operation signals of both the user terminal and the live streamer terminal can be simultaneously converted to meet the operational requirements of the target game. For example, if the target game is a board game and both the user terminal and the live streamer terminal are mobile phones, the operation signals of both the terminals can be converted to the operation signals of computer.

FIG. 4 is a flow diagram of a live stream interactive method according to yet another embodiment of the present application. As shown in FIG. 4, when the live streamer terminal agrees to the user terminal joining the live streamer game, and the live streamer game is a non-cloud game:

S410, connecting the first user terminal to the live streamer terminal as a sub-input device; and

S420, acquiring and sending the real-time images and/or sounds of the live streamer terminal game to the first user terminal.

It can be understood that when the game type is a non-cloud game type (single game type) , the user terminal to-be-accessed game application is the target game in the live streamer terminal, at this time, the user and the live streamer can interact with the game on the same screen, that is, the real-time game screen and music of the first user terminal and the live streamer terminal are consistent. When the live streamer agrees to the user’s joining, the live stream server will access the input device of the first user terminal as a sub-input device of the live streamer’s computer, and the input device may a joystick, a keyboard, a mouse, and other devices. The live stream server will ask the live streamer for permission before accessing, in order to achieve the purpose of accessing the input device of the first user terminal through the live stream server. This process is similar to remote control desktop (the technical difficulty is higher than that of remote control desktop, and it can be set to disallow the input device of the user terminal to carry out other operations other than the current page, etc. ) , and it can be set that, for example, the sub-input device has a lower priority than the live streamer’s original input device, so as to achieve the effect of playing the same game and will not interfere with the normal operation of the live streamer (such as the scenario of switching desktop to check the live situation, etc. ) .

In an embodiment, the method further includes: when the first user terminal does not match the live streamer game, sending the reference data for generating operation simulator corresponding to the live streamer game to the first user terminal, to allow the first user terminal to generate the operation simulator based on the reference data for converting game operation signals between the first user terminal and the live streamer terminal.

It can be understood that in this embodiment, terminal devices with different arithmetic power differ in hardware, operating system and input system, etc. Therefore, when the user terminal and the live streamer terminal are accessed to the same room to play the game together, it is necessary to convert the operation signals of the user terminal and/or the live streamer terminal, and convert the operation signals of the respective terminals into real operations of the target game, so as to achieve the purpose of operating the game together. In this embodiment, the operation signals of the user terminal can be converted to be consistent with the operation signals of the live streamer terminal.

The user terminal may be a mobile phone, a computer, and one of other types of terminals, and the game played by the live streamer may be a game that does not match the terminal, for example, the live streamer is playing a mobile phone game, and the user is using a computer to watch a live stream, at which time the live stream server automatically detects the user terminal, and sends the generation reference data of an operation simulator that corresponds to the live streamer game if the terminal is not matched to the game. That is, the mobile phone operation interface is simulated on the user terminal computer, and the user is allowed to input through this simulator for the purpose of playing with the live streamer. Of course, if the user terminal is a mobile phone and needs to be used to simulate a computer or other terminal, it can also be performed.

Another example is that the user terminal is a mobile phone, the live streamer terminal is a computer, and the target game is a board game, in which case the operation signal of the mobile phone terminal can be converted into the operation signal of the computer terminal. That is, reference data for generating the operation simulator corresponding to the target game is sent to the user terminal via the live stream server, so as to enable the user terminal to generate the operation simulator according to the reference data for converting the operation signals of the game of the user terminal.

The operation signals include, but are not limited to, operation signals such as upward shift, downward shift, left shift and right shift, and the specific operation signals may be set according to the specific contents of the target game, and the embodiments of the present application do not make specific limitations thereon.

In some embodiments, the operation signals of the user terminal and the live streamer terminal may also be converted at the same time so that they meet the operation requirements of the target game. For example, if the target game is a board game, and both the user terminal and the live streamer terminal are mobile phones, the operation signals of the user terminal and the live streamer terminal can be simultaneously converted to the operation signals of the computer terminal.

In an embodiment, the method further includes:

As mentioned above, during the live streaming of the target game, the live streamer can choose to invite users to play a certain game together, the live streamer performs an operation at the live streamer terminal to enable the “permission to join the live streamer game” that grants the user to join the game and sends a message thereof to the live stream server.

After a user puts forward the “request to join live streamer game” at the user terminal, the queue starts, and the live streamer makes its own selection and chooses the user to play the game together.

In this embodiment, RTC (Real time communication) technology can be used to realize the real-time transmission of real-time game images and music.

FIG. 5 is a flow diagram of a live stream interactive method according to yet another embodiment of the present application, and the flow diagram can be applied to a user terminal.

As shown in FIG. 5, the live stream interactive method includes:

S510, a first user terminal receiving a message of providing a permission to join live streamer game sent by a live stream server, and parsing the message to obtain a virtual interactive room identifier and a game identifier;

S520, the first user terminal sending a request to join the live streamer game to the live stream server, wherein the request comprises a user terminal identifier, the virtual interactive room identifier, and the game identifier.

S530, the first user terminal receiving real-time images and/or sounds of a user terminal game corresponding to a live streamer terminal game instructed to run by the live stream server.

During the game process, he first user terminal sends a game control command to realize interaction with the user terminal game corresponding to the live streamer terminal game.

After receiving the confirmation that the live streamer agrees on users’ joining, the live stream server will parse the virtual interactive room identifier, the user terminal identifier and the game identifier. When the live streamer terminal agrees to the user terminal joining the live streamer game, and the live streamer game is a cloud game, cloud game information may be generated, and sent to the cloud game server, or directly send the game identifier alone to the cloud game server. In some embodiments, cloud game information is generated based on the virtual interactive room identifier, user terminal identifier, and game identifier, and depending on the system settings, it is also possible to generate cloud game information based on the game identifier only and the cloud game information is sent to the cloud game server. The application here depends on the way to set up the server, so there are several ways.

In an embodiment, the second user terminal receives live distribution images and/or sounds distributed by the live server.

As shown in FIG. 6 (a) , the live stream server and the cloud game server can be disposed on the same physical machine. Through virtualization technology, multiple virtual machines can be created on a single physical machine. For example, an 8-GPU physical machine can virtualize one live stream server and seven cloud game servers. These seven cloud game servers can be on the same physical machine with the live stream server, allowing for high-speed interconnection.

In an embodiment, the live stream distribution sound distributed by the live stream server received by the second user terminal is the sound optimized after the live stream server obtains the cloud game server sound and the live streamer terminal sound in the current live streaming state by the live stream server by means of a mixing and noise reduction algorithm.

In an embodiment, the third user terminal receives the originally rendered high-frame-rate images.

In an embodiment, the method further includes: when the first user terminal does not match the live streamer game, instructing the cloud game server to send reference data for generating the operation simulator corresponding to the live streamer game, to allow the first user terminal to generate the operation simulator based on the reference data.

The first user terminal sends game control commands to realize interaction with the cloud game.

In an embodiment, when the first user terminal does not match the live streamer game, receiving reference data for generating operation simulators corresponding to the live streamer game sent from the cloud game server instructed by the live stream server; and

The user terminal may be a mobile phone, a computer, and one of other types of terminals, and the game played by the live streamer may be a game that does not match the terminal, for example, the live streamer is playing a mobile phone game, and the user is using a computer to watch a live stream, at which time the live stream server automatically detects the user terminal, and sends the generation reference data of an operation simulator that corresponds to the live streamer game if the terminal is not matched to the game. That is, the mobile phone operation interface is simulated on the user terminal computer, and the user is allowed to input through this simulator for the purpose of playing with the live streamer. The live stream server may also instruct the live streamer terminal to directly send the generated reference data of the operation simulator corresponding to the live streamer game. Of course, if the user terminal is a mobile phone and needs to be used to simulate a computer or other terminal, it can also be performed.

In an embodiment, the method further includes:

receiving a message of providing a permission to join live streamer game sent by a live stream server, and parsing the message to obtain a virtual interactive room identifier and a game identifier.

As mentioned above, in the process of live streamer live, the live streamer may choose to invite a user to play a game together. The live streamer may operate on the live streamer terminal to grant the “permission to join the live streamer game” that allows users to join the game. After receiving the information, the live stream server will notify users that the live streamer has granted the “permission to join the live streamer game” .

After users make the “request to join the live streamer game” on the user terminal, they start to queue up, and the live streamer chooses a user to play the game together.

Corresponding to the method provided in embodiments of the present application, an embodiment of the present application further provides a live stream interactive device, applied to a live stream server. As shown in FIG. 7, the device includes:

an acquirer 710, configured to acquire a request to join a live streamer game sent by a first user terminal, wherein the request comprises a user terminal identifier, a virtual interactive room identifier and a game identifier; and acquire feedback from a live streamer terminal on the request to join the live streamer game.

a parser 720, configured to parse the request to join the live streamer game sent by the first user terminal;

a sender 730, configured to send the request to join the live streamer game to the live streamer terminal; and

a generator 740, configured to instruct to run a user terminal game corresponding to a live streamer terminal game and sending real-time game images and/or sounds of the user terminal game to the first user terminal based on the feedback on the request and a live streamer game type.

the generator 740 is further configured to generate cloud game information based on the request to join live streamer game sent by the first user terminal;

the sender 730 is further configured to send the cloud game information to the cloud game server, instruct the cloud game server to run the user terminal game corresponding to the live streamer terminal game based on the cloud game information, and instruct the cloud game server to send the real-time game images and/or sounds of the user terminal game to the first user terminal.

In an embodiment, the acquirer 710 is further configured to:

the generator 740 is further configured to mix and process the images and/or sounds of the cloud game server and/or live streamer terminal to generate live stream distribution images and/or sounds; and

the sender 730 is further configured to distribute the live stream distribution images and/or sounds to a second user terminal.

The second user terminal in this embodiment refers to a viewer user terminal that watches a live stream game.

In an embodiment, the images of the cloud game server acquired by the acquirer 710 under the current live stream status are low-frame-rate images processed by frame extraction and compression.

In an embodiment, the generator 740 is further configured to optimize the sounds from the cloud game server and the live streamer terminal under the current live stream status acquired by the live stream server using a mixing and noise reduction algorithm.

In an embodiment, when the first user terminal does not match the live streamer game, the sender 730 is further configured to,

instruct the cloud game server to send reference data for generating operation simulator corresponding to the live streamer game to the first user terminal, to allow the first user terminal to generate the operation simulator based on the reference data for converting game operation signals between the first user terminal and the cloud game server.

the sender 730 is further configured to connect the first user terminal to the live streamer terminal as a sub-input device;

the acquirer 710 is further configured to acquire the real-time images and/or sounds of the live streamer terminal game; and

the sender 730 is further configured to send the real-time images and/or sounds of the live streamer terminal game to the first user terminal.

In an embodiment, when the first user terminal does not match the live streamer game, the sender 730 is further configured to instruct the cloud game server to send the reference data for generating operation simulator corresponding to the live streamer game to the first user terminal, to allow the first user terminal to generate the operation simulator based on the reference data for converting game operation signals between the first user terminal and the live streamer terminal.

In an embodiment, the acquirer 710 is further configured to acquire a message of providing a permission to join live streamer game sent by the live streamer terminal, wherein the message of providing permission to join live streamer game comprises a virtual interactive room identifier and a game identifier;

the sender 730 is further configured to send the message of providing permission to join live streamer game to the first user terminal; and

the parser 720 is further configured to parse the message of providing permission to join live streamer game to obtain the virtual interactive room identifier and the game identifier.

The embodiments of the present application are applied to a live stream server, in which a request to join a live streamer game initiated by the user terminal is acquired; the request is forwarded to a live streamer terminal, and the game content sent by the live streamer terminal is received. By means of the above technological solution, a new mode of game live stream interaction is provided, whereby the live streamer terminal can broadcast a game live broadcast based on the local target game without being restricted to the live broadcasting game library; at the same time, in the event that the target game has not been installed on the user terminal, the target game is initiated through the cloud game server, which reduces the threshold of the interaction between the live streamer and the user, and improves the user experience of both the live streamer and the user.

Corresponding to the method provided in embodiments of the present application, an embodiment of the present application further provides a live stream interactive device, applied to a first user terminal. As shown in FIG. 8, the device includes:

a receiver 810, configured to receive a message of providing a permission to join live streamer game sent by a live stream server and parse the message to obtain a virtual interactive room identifier and a game identifier.

a sender 820, configured to send a request to join live streamer game to the live server, wherein the request comprises a user terminal identifier, the virtual interactive room identifier, and the game identifier.

the receiver 810 is further configured to receive real-time images and/or sounds of a user terminal game corresponding to a live streamer terminal game instructed to run by the live stream server.

In an embodiment, the sender 820 is further configured to, send a game manipulation instruction to realize interaction with the user terminal game corresponding to the game of the live streamer terminal.

In an embodiment, when the live streamer terminal agrees that the first user terminal joins the live streamer game, and the live streamer game is a cloud game:

the receiver 810 is further configured to, receive the real-time images and/or sounds of the user terminal game sent by the cloud game server.

In an embodiment, when the first user terminal does not match the live streamer game.

The receiver 810 is further configured to, receive the generation reference data of the corresponding operation simulator of the live streamer game sent by the cloud game server at the instruction of the live stream server to generate the operation simulator;

The sender 820 is further configured to, send game manipulation instructions to realize interaction with the cloud game.

In an embodiment, when the live streamer terminal agrees to the first user terminal joining the live streamer game and the live streamer game is a non-cloud game:

the sender 820 is further configured to, access the live streamer terminal as a sub-input device of the live streamer terminal;

the receiver 810 is further configured to, receive the live images and/or sounds of the game of the live streamer terminal sent by the live stream server.

In a preferred embodiment provided in the present application, when said first user terminal does not match said live stream game, said receiving module 810 is further used for, receiving the generation reference data of the corresponding operation simulator of the live stream game sent by the live broadcast server at the command of said live broadcast server;

Said sending module 820 is further used to, send game manipulation instructions to realize interaction with said single player game.

The present embodiment is applied to the first user terminal to realize the live game interaction between the first user terminal and the live streamer terminal.

As shown in FIG. 9, embodiments of the present application provide a live stream interactive device, including: a receiver 910, a parser 920, a starter 930 and a sender 940.

The receiver 910 is configured to receive cloud game information;

the parser 920 is configured to parse the cloud game information to obtain a virtual interactive room identifier, a user terminal identifier and a game identifier;

the starter 930 is configured to start a game according to the game identifier and add a user terminal to live streamer game; and

the sender 940 is configured to send live streamer game content.

The embodiment of the present application is applied to the cloud game server, which starts the cloud game of the target game in response to the cloud game start instruction, and takes the cloud game as the game application to be accessed of the user terminal, so that the live stream server controls the access of the game application to be accessed to the room of the target game. Through the above technical solutions, a new interactive mode of game live is provided, so that when the user terminal does not have the target game installed, the cloud game server may start a cloud game corresponding to the target game according to the cloud game start instruction sent by the live stream server, and the user terminal may access the room of the target game through the cloud game for live interaction in multiplayer game, which reduces the threshold of interaction between the live streamer and the users and improves the experience of interaction between the live streamer and the users.

The live stream interactive device according to the embodiment of the present application may execute the live stream interactive method according to any embodiment of the present application, and has the corresponding function modules and beneficial effects of the implementation method.

FIG. 10 is a schematic diagram of an electronic apparatus according to an embodiment of the present application. FIG. 10 shows a block diagram of an exemplary electronic apparatus 1012 suitable for implementing the embodiment of the present application. The electronic apparatus 1012 shown in FIG. 10 is merely an example and should not impose any limitations on the functionality and scope of use of the embodiment of the present application.

As shown in FIG. 10, electronic apparatus 1012 takes the form of a general-purpose computing device. Components of electronic apparatus 1012 may include, but be not limited to, one or more processors or processing unit 1016, a system memory 1028, and a bus 1018 connecting different system components (including system memory 1028 and processing unit 1016) .

The bus 1018 represents one or more of several classes of bus structures, including memory bus or memory controller, peripheral bus, graphics acceleration port, processor, or local bus using any of several bus structures. For example, the architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MCA) bus, the enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

The electronic apparatus 1012 typically includes a variety of computer system readable media. These media may be any available media that may be accessed by the electronic apparatus 1012, including volatile and non-volatile media, mobile and non-mobile media.

The system memory 1028 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 1030 and/or cache memory 1032. electronic apparatus 1012 may further include other mobile/non-mobile, volatile/non-volatile computer system storage media. As an example only, storage system 1034 may be used to read and write non-removable and non-volatile magnetic media (not shown in FIG. 10, often referred to as “hard drive” ) . Although not shown in FIG. 10, disk drive for reading and writing to removable and non-volatile disks (such as “floppy disk” ) and optical drive for reading and writing to removable and non-volatile optical disks (such as CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 1018 through one or more data media interfaces. System memory 1028 may include at least one program product having a set of (e.g., at least one) program modules configured to perform the functions under the embodiment of the present application.

The program/utility 1040 with a set of (at least one) programmer 1042, including, but not limited to, an operating system, one or more application programs, other program modules, and program data, may be stored in, for example, system memory 1028. Each or some combination of these examples might include an implementation of a network environment. The programmer 1042 normally performs the functions and/or methods under the embodiments described in the present application.

The electronic apparatus 1012 may also communicate with one or more external devices 1014 (e.g., keyboard, pointing device, display 1024, etc. ) and with one or more devices enabling users to interact with the electronic apparatus 1012, and/or with any device (e.g., network card, modem, etc. ) that enables the electronic apparatus 1012 to communicate with one or more other computing devices. Such communication may take place through an input/output (I/O) interface 1022. Furthermore, the electronic apparatus 1012 may communicate with one or more networks (such as local area network (LAN) , wide area network (WAN) and/or public network (such as the Internet) through a network adapter 1020. As shown in the figure, the network adapter 1020 communicates with other modules of the electronic apparatus 1012 through the bus 1018. It should be understood that, although not shown in FIG. 10, other hardware and/or software modules may be used in conjunction with the electronic apparatus 1012, including but not limited to: microcode, device drive, redundant processing unit, external disk drive array, RAID system, tape drive, and data backup storage system, etc.

The processing unit 1016 executes various function applications and data processing by running at least one of the other programs among multiple programs stored in the system memory 1028, such as realizing any live stream interactive method according to the embodiment of the present application.

An embodiment of the present application further provides a computer-readable storage medium, with a computer program stored thereon; when the program is executed by the processor, the live stream interactive method according to any embodiment of the present application is implemented.

From the above description of embodiments, the technical personnel in the field may clearly understand that the present application may be implemented with the help of software and necessary general hardware, or of course by hardware, but in many cases the former is the preferred embodiment. Based on this understanding, the technical solutions of the present application may be represented in essence or contribute to the prior art in the form of a computer software product that may be stored in a computer-readable storage medium, such as a computer floppy disk, Read-Only Memory (ROM) , Random Access Memory (RAM) , FLASH, hard disk or optical disk, etc., including instructions to cause a computer device (which may be a personal computer, server, network device, etc. ) to perform the method described in any embodiment of the present application.

It is worth noting that in the above embodiments of live stream interactive device, the units and modules included are divided only according to the functional logic, and are not limited to the above division, as long as the corresponding functions may be realized. In addition, the specific name of each functional unit is only for the convenience of distinguishing, and is not used to limit the scope of protection in the present application.

Note that the above are only better embodiments of the present application and the technical principles used. The technical personnel in the field will understand that the present application is not limited to the specific embodiments described herein and that it is possible for the technical personnel in the field to make all kinds of obvious changes, realignments and substitutions without leaving the scope of protection of the present application. Therefore, although the present application is described in some detail by the above embodiments, the present application is not limited to the above embodiments but may include many other equivalent embodiments subject to the conception of the present application, the scope of which is determined by the scope of the claims.

Claims

A live stream interactive method, applied to a live stream server, the method comprising:

acquiring a request to join live streamer game sent by a first user terminal, wherein the request comprises a user terminal identifier, a virtual interactive room identifier, and a game identifier;

parsing the request to join the live streamer game sent by the first user terminal and forwarding the request to a live streamer terminal; and

acquiring feedback on the request to join the live streamer game from the live streamer terminal, and based on the feedback on the request and a live streamer game type, instructing to run a user terminal game corresponding to a live streamer terminal game and sending real-time game images and/or sounds of the user terminal game to the first user terminal.
The method according to claim 1, wherein the instructing to run a user terminal game corresponding to a live streamer terminal game and sending real-time game images and/or sounds of the first user terminal game to the first user terminal, further comprises:

generating cloud game information based on the request to join live streamer game sent by the first user terminal and sending the cloud game information to a cloud game server, and when the live streamer terminal agrees to the first user terminal joining the live streamer game and the live streamer game is a cloud game,

instructing the cloud game server to run the user terminal game corresponding to the live streamer terminal game based on the cloud game information; and

instructing the cloud game server to send the real-time game images and/or sounds of the user terminal game to the first user terminal.
The method according to claim 2, further comprising:

the live stream server acquiring images and/or sounds under current live stream status from the cloud game server;

the live stream server acquiring images and/or sounds under the current live stream status from the live streamer terminal;

the live stream server mixing and processing the images and/or sounds from the cloud game server

and/or

the images and/or sounds from the live streamer terminal to generate live stream distribution images and/or sounds; and

the live stream server distributing the live stream distribution images and/or sounds to a second user terminal.
The method according to claim 3, wherein:

the images from the cloud game server acquired by the live stream server under the current live stream status are low-frame-rate images processed by frame extraction and compression.
The method according to claim 3, further comprising:

optimizing the sounds from the cloud game server and the live streamer terminal under the current live stream status acquired by the live stream server using a mixing and noise reduction algorithm.
The method according to claim 3, wherein:

the live stream server distributing acquired images from the cloud game server under the current live stream status to a third user terminal, wherein the images from the cloud game server under the current live stream status are high-frame-rate images originally rendered.
The method according to claim 2, further comprising:

when the first user terminal does not match the live streamer game, instructing the cloud game server to send reference data for generating operation simulator corresponding to the live streamer game to the first user terminal, to allow the first user terminal to generate the operation simulator based on the reference data for converting game operation signals between the first user terminal and the cloud game server.
The method according to claim 1, wherein:

the instructing to run a user terminal game corresponding to a live streamer terminal game and sending real-time game images and/or sounds of the user terminal game to the first user terminal further comprises:

when the live streamer terminal agrees to the user terminal joining the live streamer game, and the live streamer game is a non-cloud game,

connecting the first user terminal to the live streamer terminal as a sub-input device; and

acquiring and sending the real-time images and/or sounds of the live streamer terminal game to the first user terminal.
The method according to claim 8, further comprising:

when the first user terminal does not match the live streamer game, sending the reference data for generating operation simulator corresponding to the live streamer game to the first user terminal, to allow the first user terminal to generate the operation simulator based on the reference data for converting game operation signals between the first user terminal and the live streamer terminal.
The method according to claim 1, further comprising:

receiving a message of providing a permission to join live streamer game sent by the live streamer terminal, wherein the message of providing permission to join live streamer game comprises a virtual interactive room identifier and a game identifier; and

sending the message of providing a permission to join live streamer game to the first user terminal and parsing the message to obtain the virtual interactive room identifier and the game identifier.
A live stream interactive method, applied to a first user terminal, the method comprising:

a first user terminal receiving a message of providing a permission to join live streamer game sent by a live stream server, and parsing the message to obtain a virtual interactive room identifier and a game identifier;

the first user terminal sending a request to join the live streamer game to the live stream server, wherein the request comprises a user terminal identifier, the virtual interactive room identifier, and the game identifier; and

the first user terminal receiving real-time images and/or sounds of a user terminal game corresponding to a live streamer terminal game instructed to run by the live stream server.
The method according to claim 11, further comprising:

the first user terminal sending game control commands to interact with the user terminal game corresponding to the live streamer terminal game.
The method according to claim 11, further comprising:

when the live streamer terminal agrees to the first user terminal joining the live streamer game, and the live streamer game is a cloud game,

receiving the real-time images and/or sounds of the user terminal game sent by the cloud game server.
The method according to claim 13, further comprising:

when the first user terminal does not match the live streamer game, receiving reference data for generating operation simulators corresponding to the live streamer game sent from the cloud game server instructed by the live stream server; and

the first user terminal sending game control commands to interact with the user terminal game corresponding to the live streamer terminal game.
The method according to claim 11, further comprising:

when the live streamer terminal agrees to the user terminal joining the live streamer game, and the live streamer game is a non-cloud game, connecting the first user terminal to the live streamer terminal as a sub-input device; and

receiving the real-time images and/or sounds of the live streamer terminal game sent by the live stream server.
The method according to claim 15, further comprising:

when the first user terminal does not match the live streamer game, receiving reference data for generating operation simulators corresponding to the live streamer game instructed to send by the live stream server to generate the operation simulator; and

the first user terminal sending game control commands to interact with the non-cloud game.
A live stream interactive device, comprising:

an acquirer, configured to acquire a request to join a live streamer game sent by a first user terminal, wherein the request comprises a user terminal identifier, a virtual interactive room identifier and a game identifier; and acquire feedback from a live streamer terminal on the request to join the live streamer game;

a parser, configured to parse the request to join the live streamer game sent by the first user terminal;

a sender, configured to send the request to join the live streamer game to the live streamer terminal; and

a generator, configured to instruct to run a user terminal game corresponding to a live streamer terminal game and sending real-time game images and/or sounds of the user terminal game to the first user terminal based on the feedback on the request and a live streamer game type.
The device according to claim 17, wherein:

when the live streamer terminal agrees to the first user terminal joining the live streamer game and the live streamer game is a cloud game,

the generator is further configured to generate cloud game information based on the request to join live streamer game sent by the first user terminal; and

the sender is further configured to send the cloud game information to the cloud game server, instruct the cloud game server to run the user terminal game corresponding to the live streamer terminal game based on the cloud game information, and instruct the cloud game server to send the real-time game images and/or sounds of the user terminal game to the first user terminal.
The device according to claim 18, wherein:

the acquirer is configured to:

acquire images and/or sounds under current live stream status from the cloud game server;

acquire images and/or sounds under the current live stream status from the live streamer terminal;

the generator is further configured to mix and process the images and/or sounds of the cloud game server and/or live streamer terminal to generate live stream distribution images and/or sounds; and

the sender is further configured to distribute the live stream distribution images and/or sounds to a second user terminal.
The device according to claim 19, wherein the images of the cloud game server acquired by the acquirer under the current live stream status are low-frame-rate images processed by frame extraction and compression.
The device according to claim 19, wherein:

the generator is further configured to optimize the sounds from the cloud game server and the live streamer terminal under the current live stream status acquired by the live stream server using a mixing and noise reduction algorithm.
The device according to claim 18, wherein when the first user terminal does not match the live streamer game,

the sender is further configured to instruct the cloud game server to send reference data for generating operation simulator corresponding to the live streamer game to the first user terminal, to allow the first user terminal to generate the operation simulator based on the reference data for converting game operation signals between the first user terminal and the cloud game server.
The device according to claim 17, wherein when the live streamer terminal agrees to the user terminal joining the live streamer game, and the live streamer game is a non-cloud game,

the sender is further configured to connect the first user terminal to the live streamer terminal as a sub-input device;

the acquirer is further configured to acquire the real-time images and/or sounds of the live streamer terminal game; and

the sender is further configured to send the real-time images and/or sounds of the live streamer terminal game to the first user terminal.
The device according to claim 23, wherein when the first user terminal does not match the live streamer game,

the sender is further configured to instruct the cloud game server to send the reference data for generating operation simulator corresponding to the live streamer game to the first user terminal, to allow the first user terminal to generate the operation simulator based on the reference data for converting game operation signals between the first user terminal and the live streamer terminal.
The device according to claim 17, wherein the acquirer is further configured to acquire a message of providing a permission to join live streamer game sent by the live streamer terminal, wherein the message of providing permission to join live streamer game comprises a virtual interactive room identifier and a game identifier;

the sender is further configured to send the message of providing permission to join live streamer game to the first user terminal; and

the parser is further configured to parse the message of providing permission to join live streamer game to obtain the virtual interactive room identifier and the game identifier.
A live stream interactive device, comprising:

a receiver, configured to receive a message of providing a permission to join live streamer game sent by a live stream server and parse the message to obtain a virtual interactive room identifier and a game identifier;

a sender, configured to send a request to join live streamer game to the live server, wherein the request comprises a user terminal identifier, the virtual interactive room identifier, and the game identifier;

wherein the receiver is further configured to receive real-time images and/or sounds of a user terminal game corresponding to a live streamer terminal game instructed to run by the live stream server.
A live interactive device, comprising:

a receiver, configured to receive cloud game information;

a parser, configured to parse the cloud game information, and acquire the virtual interactive room identifier, user terminal identifier, and game identifier;

a starter, configured to start the game based on the game identifier, and add the user terminal to the live streamer game; and

a sender, configured to send content of live streamer game.
The device according to claim 27, wherein the receiver is further configured to receive game operation instructions.
An electronic apparatus, comprising:

one or more processors;

a storage device configured to store one or more programs; and

where when one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1-16.
A computer-readable storage medium, with computer programs stored thereon, where when the programs are executed by a processor, the method of any one of claims 1-16 is implemented.