CN112135158B - Live broadcasting method based on mixed reality and related equipment - Google Patents
Live broadcasting method based on mixed reality and related equipment Download PDFInfo
- Publication number
- CN112135158B CN112135158B CN202010981861.8A CN202010981861A CN112135158B CN 112135158 B CN112135158 B CN 112135158B CN 202010981861 A CN202010981861 A CN 202010981861A CN 112135158 B CN112135158 B CN 112135158B
- Authority
- CN
- China
- Prior art keywords
- virtual
- information
- client
- target
- mixed reality
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 52
- 230000002452 interceptive effect Effects 0.000 claims abstract description 83
- 230000003993 interaction Effects 0.000 claims abstract description 55
- 238000009877 rendering Methods 0.000 claims abstract description 46
- 230000009471 action Effects 0.000 claims abstract description 26
- 238000007781 pre-processing Methods 0.000 claims description 20
- 238000013507 mapping Methods 0.000 claims description 5
- 230000000007 visual effect Effects 0.000 abstract description 10
- 230000008569 process Effects 0.000 description 11
- 230000006870 function Effects 0.000 description 8
- 238000010586 diagram Methods 0.000 description 6
- 238000005516 engineering process Methods 0.000 description 5
- 230000005540 biological transmission Effects 0.000 description 4
- 238000003062 neural network model Methods 0.000 description 4
- 238000012545 processing Methods 0.000 description 4
- 230000004927 fusion Effects 0.000 description 3
- 230000002194 synthesizing effect Effects 0.000 description 3
- 238000013528 artificial neural network Methods 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 230000001902 propagating effect Effects 0.000 description 2
- 241001310793 Podium Species 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 239000011521 glass Substances 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/21—Server components or server architectures
- H04N21/218—Source of audio or video content, e.g. local disk arrays
- H04N21/2187—Live feed
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/006—Mixed reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/20—Scenes; Scene-specific elements in augmented reality scenes
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/233—Processing of audio elementary streams
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/47—End-user applications
- H04N21/478—Supplemental services, e.g. displaying phone caller identification, shopping application
- H04N21/4788—Supplemental services, e.g. displaying phone caller identification, shopping application communicating with other users, e.g. chatting
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/698—Control of cameras or camera modules for achieving an enlarged field of view, e.g. panoramic image capture
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Databases & Information Systems (AREA)
- Computer Graphics (AREA)
- Computer Hardware Design (AREA)
- Software Systems (AREA)
- Processing Or Creating Images (AREA)
Abstract
The invention provides a live broadcasting method based on mixed reality and related equipment. The live broadcasting method based on mixed reality configures a virtual background, wherein the virtual background comprises a virtual object; acquiring first sound information and real first image information of a first object from a first client; acquiring second sound information and interaction information of a second object from a second client; rendering the first object into the virtual background according to the first image information, and controlling the virtual object to execute an action matched with the interaction information according to the interaction information; constructing a virtual scene based on the virtual background, the first object and the virtual object, newly building a first virtual camera in the virtual scene, and acquiring a virtual interactive video through the first virtual camera; and outputting the virtual interactive video to the third client to realize live broadcast of the virtual interactive video. The invention improves the visual experience of watching the live video of the user.
Description
Technical Field
The invention relates to the technical field of image processing, in particular to a live broadcast method and device based on mixed reality, computer equipment and a computer readable storage medium.
Background
When a plurality of objects carry VR/MR glasses to communicate and interact in a virtual scene, a third party object outside the virtual scene cannot see the communication and interaction process in the virtual scene. Is not beneficial to the popularization of VR/MR technology and the supervision and appreciation of the communication and interaction process by the third party object.
How to make a third party object see panoramic images and an interaction process of a plurality of objects improves the visual experience of watching live videos of the third party object, and becomes a problem to be solved.
Disclosure of Invention
In view of the foregoing, there is a need for a live broadcast method, apparatus, computer device and computer readable storage medium based on mixed reality, which can live broadcast a virtual scene, so that a user can see panoramic images of multiple objects and an interaction process, thereby improving the visual experience of the user watching live video.
The first aspect of the present application provides a mixed reality-based live broadcasting method, which includes:
configuring a virtual background, the virtual background comprising virtual objects;
acquiring first sound information and real first image information of a first object from a first client;
acquiring second sound information and interaction information of a second object from a second client;
rendering the first object into the virtual background according to the first image information, and controlling the virtual object to execute an action matched with the interaction information according to the interaction information;
constructing a virtual scene based on the virtual background, the first object and the virtual object, newly building a first virtual camera in the virtual scene, and acquiring a virtual interactive video through the first virtual camera;
and outputting the virtual interactive video to a third client to realize live broadcast of the virtual interactive video.
In another possible implementation manner, after the obtaining, from the first client, the first sound information and the real first image information of the first object, the mixed reality-based live broadcasting method further includes:
and transmitting the first sound information and the first image information to the second client, so that the second object obtains the first sound information and the first image information through the second client and interacts with the first object through the second client.
In another possible implementation manner, the controlling, according to the interaction information, the virtual object to execute the action matched with the interaction information includes:
acquiring head information in the interactive information through VR head equipment or MR head equipment, and controlling the head of the virtual object to execute head action matched with the head information in the interactive information;
and acquiring hand information in the interactive information through a VR handle, and controlling the hand of the virtual object to execute hand actions matched with the hand information in the interactive information.
In another possible implementation manner, the mixed reality-based live broadcasting method further includes:
newly building a second virtual camera at the position of the first object in the virtual scene;
second image information is collected through the second virtual camera, and the second sound information and the second image information are transmitted to the first client side, so that the first object obtains the second sound information and the second image information through the first client side and interacts with the second object through the first client side.
In another possible implementation manner, the capturing a virtual interactive video by the first virtual camera includes:
binding the first virtual camera with a rendering target, and updating each frame of image acquired by the first virtual camera to the rendering target;
generating a target texture according to the rendering target;
and mapping the target texture to a preset processor video memory, and generating a virtual interactive video.
In another possible implementation manner, the generating a target texture according to the rendering target includes:
newly building a target window, wherein the target window comprises an image in the rendering target;
binding the image in the target window to a first intermediate target texture of an independent context through a window handle of the target window;
and carrying out format preprocessing and size preprocessing on the first intermediate target texture to obtain the target texture.
In another possible implementation manner, the generating a target texture according to the rendering target includes:
creating a target independent context according to the rendering target, and creating a shared second intermediate target texture in the target independent context;
copying a render target of the target independent context BGRA channel to the second intermediate target texture;
and after the RB channel of the second intermediate target texture is converted into the BGRA channel, format preprocessing and size preprocessing are carried out on the second intermediate target texture of the BGRA channel to obtain the target texture.
The second aspect of this application provides a live device based on mixed reality, live device based on mixed reality includes:
a configuration module to configure a virtual context, the virtual context comprising a virtual object;
the first acquisition module is used for acquiring first sound information and real first image information of a first object from a first client;
the second acquisition module is used for acquiring second sound information and interaction information of the second object from the second client;
the control module is used for rendering the first object into the virtual background according to the first image information and controlling the virtual object to execute an action matched with the interaction information according to the interaction information;
the acquisition module is used for constructing a virtual scene based on the virtual background, the first object and the virtual object, newly building a first virtual camera in the virtual scene, and acquiring a virtual interactive video through the first virtual camera;
and the output module is used for outputting the virtual interactive video to a third client so as to realize live broadcast of the virtual interactive video.
A third aspect of the application provides a computer device comprising a processor for implementing the mixed reality based live method when executing computer readable instructions stored in a memory.
A fourth aspect of the present application provides a computer-readable storage medium having computer-readable instructions stored thereon which, when executed by a processor, implement the mixed reality based live method.
The method and the device for live broadcasting the virtual scene enable a user to see panoramic images of a plurality of objects and an interaction process, and improve the visual experience of watching live video of the user.
Drawings
Fig. 1 is a flowchart of a mixed reality-based live broadcasting method according to an embodiment of the present invention.
Fig. 2 is a schematic view of a virtual interactive video provided by an embodiment of the present invention.
Fig. 3 is a structural diagram of a mixed reality-based live broadcasting device according to an embodiment of the present invention.
Fig. 4 is a schematic diagram of a computer device provided by an embodiment of the present invention.
Detailed Description
In order that the above objects, features and advantages of the present invention can be more clearly understood, a detailed description of the present invention will be given below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments and features of the embodiments of the present application may be combined with each other without conflict.
In the following description, numerous specific details are set forth to provide a thorough understanding of the present invention, and the described embodiments are merely a subset of the embodiments of the present invention, rather than a complete embodiment.
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. The terminology used in the description of the invention herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.
Preferably, the mixed reality-based live broadcasting method of the present invention is applied to one or more computer devices. The computer device is a device capable of automatically performing numerical calculation and/or information processing according to a preset or stored instruction, and the hardware includes, but is not limited to, a microprocessor, an Application Specific Integrated Circuit (ASIC), a Programmable Gate Array (FPGA), a Digital Signal Processor (DSP), an embedded device, and the like.
The computer device can be a desktop computer, a notebook, a palm computer, a cloud server and other computing devices. The computer equipment can carry out man-machine interaction with a user in a keyboard mode, a mouse mode, a remote controller mode, a touch panel mode or a voice control equipment mode.
Example one
Fig. 1 is a flowchart of a mixed reality-based live broadcasting method according to an embodiment of the present invention. The live broadcast method based on mixed reality is applied to computer equipment and used for live broadcast of virtual scenes, so that a user can see panoramic images and interactive processes of a plurality of objects, and the visual experience of watching live video of the user is improved.
As shown in fig. 1, the mixed reality-based live broadcasting method includes:
101, configuring a virtual background, wherein the virtual background comprises virtual objects.
The virtual background can comprise a plurality of teaching scenes and interactive scenes with different styles and forms, such as classrooms, conference rooms, laboratories and the like. Different virtual experiences can be obtained by configuring different virtual backgrounds.
When the virtual background is configured as a classroom, the classroom can include the location of the teacher's podium and the seats of the students, and each of the student's seats can correspond to a virtual object (the virtual object in this example can be an virtual student figure).
In another embodiment, where the virtual background is configured as a laboratory, the virtual objects may include virtual characters, virtual laboratory items, and the like.
In another embodiment, the virtual background may be acquired by taking a panoramic image by a panoramic camera.
102, first sound information and real first image information of a first object are obtained from a first client.
In a specific embodiment, the first sound information of the first object may be acquired by a sound recording apparatus of the first client; and acquiring the image information of the first object through a photographic device of the first client, and extracting a real image of the first object from the image information of the first object to obtain the first image information. The first image information is a shot real image.
The extracting of the real image of the first object from the image information of the first object may include:
extracting a real image of the first object from the image information of the first object based on a color key keying technology;
and extracting an image part where the first object is located from the image information of the first object based on a biological body identification technology.
When the virtual background is configured as a classroom, the first object may comprise a teacher. When the virtual background is configured as a laboratory, the first object may comprise a teacher and/or a student.
And 103, acquiring second sound information and interaction information of the second object from the second client.
In a specific embodiment, the second sound information of the second object may be obtained by the sound recording apparatus of the second client; and acquiring the head information in the interactive information through VR (Virtual Reality) or MR (Mixed Reality) head equipment of the second client, and acquiring the hand information in the interactive information through a VR handle of the second client.
In another embodiment, obtaining the interaction information from the second client may include:
acquiring image information of the second object through the second client; acquiring a trained neural network model based on a picture skeleton convolution neural network algorithm; and calculating skeleton information of the second object through the neural network model by taking the image information of the second object as input, and identifying and tracking the action interaction information of the second object based on the skeleton information of the second object. The skeleton information comprises information of the head, the hands, the feet, the trunk and the like of the human body.
The second object may comprise a student when the virtual background is configured as a classroom. When the virtual background is configured as a laboratory, the first object may comprise a student and/or a real laboratory item.
When the second object includes a plurality of students, the virtual object includes a plurality of virtual characters. The plurality of students correspond to the plurality of virtual characters one by one. The character figures of the plurality of virtual characters may be the same. The character image of each virtual character may also be the same as the student image of the corresponding student.
In another embodiment, after the obtaining the first sound information and the real first image information of the first object from the first client, the mixed reality based live broadcasting method further includes:
and transmitting the first sound information and the first image information to the second client, so that the second object obtains the first sound information and the first image information through the second client and interacts with the first object through the second client.
In another embodiment, the mixed reality-based live broadcasting method further includes:
fusing the real image of the first object with the virtual background to obtain virtual fusion image information;
synthesizing the first sound information into the virtual fusion image information to obtain a first video;
and transmitting the first video to the second client, so that the second object can play the first video through a virtual display device of the second client. The second object can see the first object through the second client, so that the first object can interact with the second object.
104, rendering the first object into the virtual background according to the first image information, and controlling the virtual object to execute an action matched with the interaction information according to the interaction information.
In a specific embodiment, the rendering the first object to the virtual background according to the first image information includes:
extracting a real image of the first object from the image information of the first object;
rendering a real image of the first object to the virtual background; or
And fusing the real image of the first object with the virtual background.
In a specific embodiment, the controlling, according to the interaction information, the virtual object to perform the action matching with the interaction information includes:
acquiring head information in the interactive information through VR head equipment or MR head equipment, and controlling the head of the virtual object to execute head action matched with the head information in the interactive information;
acquiring hand information in the interactive information through a VR handle, and controlling the hand of the virtual object to execute hand actions matched with the hand information in the interactive information.
In another embodiment, the mixed reality-based live broadcasting method further includes:
creating a second virtual camera at the location of the first object in the virtual scene;
second image information is collected through the second virtual camera, and the second sound information and the second image information are transmitted to the first client side, so that the first object obtains the second sound information and the second image information through the first client side and interacts with the second object through the first client side.
In another embodiment, the mixed reality-based live broadcasting method further includes:
synthesizing the second sound information into the second image information to obtain a second video;
and transmitting the second video to the first client, so that the first object can play the second video through a virtual display device of the first client. The first object may view the second object through the first client, enabling the second object to interact with the first object.
And 105, constructing a virtual scene based on the virtual background, the first object and the virtual object, newly building a first virtual camera in the virtual scene, and acquiring a virtual interactive video through the first virtual camera.
In a specific embodiment, the newly creating a first virtual camera in the virtual scene includes:
newly building a first virtual camera at a target position in the virtual scene, and enabling the first virtual camera to shoot the panorama of the virtual scene including the first object and the virtual object.
In particular, the virtual object may be located intermediate the first object and the first virtual camera.
In a specific embodiment, the capturing a virtual interaction video by the first virtual camera includes:
binding the first virtual camera with a rendering target, and updating each frame of image acquired by the first virtual camera to the rendering target;
generating a target texture according to the rendering target;
and mapping the target texture to a preset processor video memory, and generating a virtual interactive video.
In a specific embodiment, the generating the target texture according to the rendering target includes:
newly building a target window, wherein the target window comprises an image in the rendering target;
binding the image in the target window to a first intermediate target texture of an independent context through a window handle of the target window;
and carrying out format preprocessing and size preprocessing on the first intermediate target texture to obtain the target texture.
Specifically, the format preprocessing and the size preprocessing of the first intermediate target texture include:
and converting the first intermediate target texture into a preset size and converting the first intermediate target texture into a YUV420P format.
In a specific embodiment, the generating the target texture according to the rendering target includes:
creating a target independent context according to the rendering target, and creating a shared second intermediate target texture in the target independent context;
copying the rendering target of the BGRA channel of the target independent context to the second intermediate target texture, converting the RB channel of the second intermediate target texture into the BGRA channel, and then performing format preprocessing and size preprocessing on the second intermediate target texture of the BGRA channel to obtain the target texture.
And 106, outputting the virtual interactive video to a third client to realize live broadcast of the virtual interactive video.
In a specific embodiment, the outputting the virtual interactive video to the third client includes:
and pushing the virtual interactive video to a content distribution network, and distributing the mixed information to the third client through the content distribution network.
The virtual interactive video comprises third image information, the first sound information and the second sound information of the virtual scene, which are acquired by the first virtual camera.
The third client can play the virtual interactive video, and a user can see and hear the panoramic images of the first object and the second object and the interactive process through the third client, so that the visual experience of watching the live video of the user is improved.
The first client, the second client, and the third client may be in different geographic locations, and the first client, the second client, and the third client may be computer devices. The first client and the second client at least comprise a camera device (such as an RGB high-definition lens, an IR (infrared) lens, a TOF (time of flight) depth lens and the like), a recording device, a virtual display device and the like. The third client at least comprises a virtual display device (such as a computer display device, a mobile terminal display device, a virtual head display device (such as a VR machine, a MR machine and the like)) and the like.
In another embodiment, the mixed reality-based live broadcasting method further comprises;
recording a live video of the virtual scene;
and propagating the live video through a social tool.
The live broadcasting method based on mixed reality in the first embodiment carries out live broadcasting on a virtual scene, so that a user can see panoramic images and an interactive process of a plurality of objects, and the visual experience of watching video live broadcast by the user is improved.
In the foregoing embodiments, a virtual camera (e.g., the first virtual camera) may be used to capture an interactive picture of a real teacher's image (the first object) and a virtual student (a virtual object corresponding to the second object) in a virtual scene, obtain first sound information including a teacher's sound and a courseware sound and second sound information including a student's sound, mix the teacher's sound, the courseware sound and the student's sound and then push the mixed information to a content distribution network, and meanwhile push the interactive picture to the content distribution network. And integrating the mixed sound and the interactive picture through a content distribution network to form a complete audio and video stream, and distributing the complete audio and video stream to a third client. The third client displays the interactive picture through broadcasting and plays the mixed sound at the same time, so that parents at the third client can see the interactive picture and hear the mixed sound. Live video of a virtual scene can be recorded; the parents can transmit the live video through social tools such as WeChat and the like. The first virtual camera is newly built in a virtual scene of a teacher end (the first client), and the virtual scene does not include a teacher end UI. The GPU is used for processing the images, and the CPU occupation is reduced. The virtual interactive video diagram shown in fig. 2 presents a combination of real teacher's image and virtual students in a virtual scene. The method comprises the steps of configuring a virtual scene comprising three virtual students, obtaining a real teacher image from a teacher end, and rendering the teacher image to the virtual scene. And acquiring the interactive information from the student terminal, and controlling the virtual student to execute the action matched with the interactive information according to the interactive information. The sound identification above the head of the virtual student shows the sound playing state of the corresponding student.
Example two
Fig. 3 is a block diagram of a mixed reality-based live device according to a second embodiment of the present invention. The mixed reality-based live broadcasting device 20 is applied to computer equipment. The live broadcast device 20 based on mixed reality is used for live broadcast of virtual scenes, so that a user can see panoramic images and an interactive process of a plurality of objects, and the visual experience of watching live video of the user is improved.
As shown in fig. 3, the mixed reality based live broadcasting device 20 may include a configuration module 201, a first obtaining module 202, a second obtaining module 203, a control module 204, an acquisition module 205, and an output module 206.
A configuration module 201 configured to configure a virtual background, wherein the virtual background includes a virtual object.
The virtual background may include a plurality of teaching scenes, interactive scenes such as classrooms, conference rooms, laboratories and the like in different styles and forms. Different virtual feelings can be obtained by configuring different virtual backgrounds.
When the virtual background is configured as a classroom, the classroom may include a teacher's lectern location and students ' seats, and each student's seat may correspond to a virtual object (the virtual object in this example may be an virtual student figure).
In another embodiment, where the virtual background is configured as a laboratory, the virtual objects may include virtual characters, virtual laboratory items, and the like.
In another embodiment, the virtual background may be obtained by taking a panoramic image by a panoramic camera.
A first obtaining module 202, configured to obtain first sound information and real first image information of a first object from a first client.
In a specific embodiment, the first sound information of the first object may be obtained by a sound recording apparatus of the first client; and acquiring the image information of the first object through a photographic device of the first client, and extracting a real image of the first object from the image information of the first object to obtain the first image information. The first image information is a shot real image.
The extracting of the real image of the first object from the image information of the first object may include:
extracting a real image of the first object from the image information of the first object based on a color key keying technology;
and extracting an image part where the first object is located from the image information of the first object based on a biological body identification technology.
When the virtual background is configured as a classroom, the first object may comprise a teacher. When the virtual background is configured as a laboratory, the first object may comprise a teacher and/or a student.
The second obtaining module 203 is configured to obtain second sound information and interaction information of the second object from the second client.
In a specific embodiment, the second sound information of the second object may be obtained by the sound recording apparatus of the second client; acquiring the head information in the interactive information through VR (Virtual Reality) or MR (Mixed Reality) head equipment of the second client, and acquiring the hand information in the interactive information through a VR handle of the second client.
In another embodiment, obtaining the interaction information from the second client may include:
acquiring image information of the second object through the second client; acquiring a trained neural network model based on a picture skeleton convolution neural network algorithm; and calculating skeleton information of the second object through the neural network model by taking the image information of the second object as input, and identifying and tracking the action interaction information of the second object based on the skeleton information of the second object. The skeleton information comprises information of the head, the hands, the feet, the trunk and the like of the human body.
The second object may comprise a student when the virtual background is configured as a classroom. When the virtual background is configured as a laboratory, the first object may comprise a student and/or a real laboratory item.
When the second object includes a plurality of students, the virtual object includes a plurality of virtual characters. The plurality of students correspond to the plurality of virtual characters one by one. The character figures of the plurality of virtual characters may be the same. The character image of each virtual character may also be the same as the student image of the corresponding student.
In another embodiment, after the first sound information and the real first image information of the first object are obtained from the first client, the mixed reality-based live broadcasting device further includes a transmission module configured to transmit the first sound information and the first image information to the second client, so that the second object obtains the first sound information and the first image information through the second client and interacts with the first object through the second client.
In another embodiment, the transmission module is further configured to fuse the real image of the first object and the virtual background to obtain virtual fused image information;
synthesizing the first sound information into the virtual fusion image information to obtain a first video;
and transmitting the first video to the second client, so that the second object can play the first video through a virtual display device of the second client. The second object may view the first object through the second client, enabling the first object to interact with the second object.
A control module 204, configured to render the first object into the virtual background according to the first image information, and control the virtual object to execute an action matching the interaction information according to the interaction information.
In a specific embodiment, the rendering the first object to the virtual background according to the first image information includes:
extracting a real image of the first object from the image information of the first object;
rendering a real image of the first object to the virtual background; or
And fusing the real image of the first object with the virtual background.
In a specific embodiment, the controlling, according to the interaction information, the virtual object to perform the action matched with the interaction information includes:
acquiring head information in the interaction information through VR head equipment or MR head equipment, and controlling the head of the virtual object to execute a head action matched with the head information in the interaction information;
acquiring hand information in the interactive information through a VR handle, and controlling the hand of the virtual object to execute hand actions matched with the hand information in the interactive information.
In another embodiment, the transmission module is further configured to create a second virtual camera at the location of the first object in the virtual scene;
second image information is collected through the second virtual camera, and the second sound information and the second image information are transmitted to the first client side, so that the first object obtains the second sound information and the second image information through the first client side and interacts with the second object through the first client side.
In another embodiment, the transmission module is further configured to synthesize the second sound information into the second image information to obtain a second video;
and transmitting the second video to the first client, so that the first object can play the second video through a virtual display device of the first client. The first object may view the second object through the first client, enabling the second object to interact with the first object.
The acquisition module 205 is configured to construct a virtual scene based on the virtual background, the first object, and the virtual object, create a new first virtual camera in the virtual scene, and acquire a virtual interactive video through the first virtual camera.
In a specific embodiment, the newly creating a first virtual camera in the virtual scene includes:
newly building a first virtual camera at a target position in the virtual scene, and enabling the first virtual camera to shoot the panorama of the virtual scene including the first object and the virtual object.
In particular, the virtual object may be located intermediate the first object and the first virtual camera.
In a specific embodiment, the capturing a virtual interaction video by the first virtual camera includes:
binding the first virtual camera with a rendering target, and updating each frame of image acquired by the first virtual camera to the rendering target;
generating a target texture according to the rendering target;
and mapping the target texture to a preset processor video memory, and generating a virtual interactive video.
In a specific embodiment, the generating the target texture according to the rendering target includes:
newly building a target window, wherein the target window comprises an image in the rendering target;
binding the image in the target window to a first intermediate target texture of an independent context through a window handle of the target window;
and carrying out format preprocessing and size preprocessing on the first intermediate target texture to obtain the target texture.
Specifically, the format preprocessing and the size preprocessing of the first intermediate target texture include:
and converting the first intermediate target texture into a preset size and converting the first intermediate target texture into a YUV420P format.
In a specific embodiment, the generating the target texture according to the rendering target includes:
creating a target independent context according to the rendering target, and creating a shared second intermediate target texture in the target independent context;
copying the rendering target of the BGRA channel of the target independent context to the second intermediate target texture, converting the RB channel of the second intermediate target texture into the BGRA channel, and performing format preprocessing and size preprocessing on the second intermediate target texture of the BGRA channel to obtain the target texture.
An output module 206, configured to output the virtual interactive video to a third client, so as to implement live broadcast of the virtual interactive video.
In a specific embodiment, the outputting the virtual interactive video to the third client includes:
and pushing the virtual interactive video to a content distribution network, and distributing the mixed information to the third client through the content distribution network.
The virtual interactive video comprises third image information, the first sound information and the second sound information of the virtual scene, which are acquired by the first virtual camera.
The third client can play the virtual interactive video, and a user can see and hear the panoramic images of the first object and the second object and the interactive process through the third client, so that the visual experience of watching the live video of the user is improved.
The first client, the second client, and the third client may be in different geographic locations, and the first client, the second client, and the third client may be computer devices. The first client and the second client at least comprise a camera device (such as an RGB high-definition lens, an IR (infrared) lens, a TOF (time of flight) depth lens and the like), a sound recording device, a virtual display device and the like. The third client at least comprises a virtual display device (such as a computer display device, a mobile terminal display device, a virtual head display device (such as a VR all-in-one machine, a MR all-in-one machine and the like)) and the like.
In another embodiment, the mixed reality-based live broadcasting device further comprises a propagation module, configured to record a live video of the virtual scene;
and propagating the live video through a social tool.
The live broadcasting device 20 based on mixed reality of the second embodiment live broadcasts the virtual scene, so that the user can see panoramic images and interactive processes of a plurality of objects, and the visual experience of watching live video of the user is improved.
EXAMPLE III
The present embodiment provides a computer-readable storage medium, which stores computer-readable instructions thereon, and when executed by a processor, the computer-readable instructions implement the steps in the embodiment of the mixed reality based live broadcasting method described above, such as steps 101 to 106 shown in fig. 1:
101, configuring a virtual background, wherein the virtual background comprises virtual objects;
102, acquiring first sound information and real first image information of a first object from a first client;
103, acquiring second sound information and interaction information of a second object from a second client;
104, rendering the first object into the virtual background according to the first image information, and controlling the virtual object to execute an action matched with the interaction information according to the interaction information;
105, constructing a virtual scene based on the virtual background, the first object and the virtual object, newly building a first virtual camera in the virtual scene, and acquiring a virtual interactive video through the first virtual camera;
and 106, outputting the virtual interactive video to a third client to realize live broadcast of the virtual interactive video.
Alternatively, the computer readable instructions, when executed by the processor, implement the functions of the modules in the above-described apparatus embodiments, such as modules 201-206 in fig. 3:
a configuration module 201, configured to configure a virtual background, where the virtual background includes virtual objects;
a first obtaining module 202, configured to obtain first sound information and real first image information of a first object from a first client;
a second obtaining module 203, configured to obtain second sound information and interaction information of the second object from the second client;
the control module 204 is configured to render the first object into the virtual background according to the first image information, and control the virtual object to execute an action matched with the interaction information according to the interaction information;
the acquisition module 205 is configured to construct a virtual scene based on the virtual background, the first object and the virtual object, create a new first virtual camera in the virtual scene, and acquire a virtual interactive video through the first virtual camera;
an output module 206, configured to output the virtual interactive video to a third client, so as to implement live broadcast of the virtual interactive video.
Example four
Fig. 4 is a schematic diagram of a computer device according to a third embodiment of the present invention. The computer device 30 comprises a memory 301, a processor 302 and computer readable instructions 303, e.g. a mixed reality based live program, stored in the memory 301 and executable on the processor 302. The processor 302, when executing the computer readable instructions 303, implements the steps in the mixed reality based live method embodiment described above, such as 101-106 shown in fig. 1:
101, configuring a virtual background, wherein the virtual background comprises a virtual object;
102, acquiring first sound information and real first image information of a first object from a first client;
103, acquiring second sound information and interaction information of a second object from a second client;
104, rendering the first object into the virtual background according to the first image information, and controlling the virtual object to execute an action matched with the interaction information according to the interaction information;
105, constructing a virtual scene based on the virtual background, the first object and the virtual object, newly building a first virtual camera in the virtual scene, and acquiring a virtual interactive video through the first virtual camera;
and 106, outputting the virtual interactive video to a third client to realize live broadcast of the virtual interactive video.
Or, the computer readable instructions, when executed by a processor, implement the functions of the modules in the above-described apparatus embodiments, such as modules 201-206 in fig. 3:
a configuration module 201 configured to configure a virtual background, the virtual background including a virtual object;
a first obtaining module 202, configured to obtain first sound information and real first image information of a first object from a first client;
a second obtaining module 203, configured to obtain second sound information and interaction information of the second object from the second client;
the control module 204 is configured to render the first object into the virtual background according to the first image information, and control the virtual object to execute an action matching the interaction information according to the interaction information;
an acquisition module 205, configured to construct a virtual scene based on the virtual background, the first object, and the virtual object, create a new first virtual camera in the virtual scene, and acquire a virtual interactive video through the first virtual camera;
an output module 206, configured to output the virtual interactive video to a third client, so as to implement live broadcast of the virtual interactive video.
Illustratively, the computer readable instructions 303 may be partitioned into one or more modules that are stored in the memory 301 and executed by the processor 302 to perform the present method. The one or more modules may be a series of computer-readable instructions capable of performing certain functions and describing the execution of the computer-readable instructions 303 in the computer device 30. For example, the computer readable instructions 303 may be divided into the configuration module 201, the first obtaining module 202, the second obtaining module 203, the control module 204, the collecting module 205, and the output module 206 in fig. 3, and specific functions of each module are described in embodiment two.
Those skilled in the art will appreciate that the schematic diagram 4 is merely an example of the computer device 30 and does not constitute a limitation of the computer device 30, and may include more or less components than those shown, or combine certain components, or different components, for example, the computer device 30 may also include input and output devices, network access devices, buses, etc.
The Processor 302 may be a Central Processing Unit (CPU), other general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic, discrete hardware components, etc. A general purpose processor may be a microprocessor or the processor 302 may be any conventional processor or the like, the processor 302 being the control center for the computer device 30 and connecting the various parts of the overall computer device 30 using various interfaces and lines.
The memory 301 may be used to store the computer-readable instructions 303, and the processor 302 may implement the various functions of the computer device 30 by executing or executing the computer-readable instructions or modules stored in the memory 301 and invoking data stored in the memory 301. The memory 301 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required by at least one function (such as a sound playing function, an image playing function, etc.), and the like; the storage data area may store data created according to use of the computer device 30, and the like. In addition, the Memory 301 may include a hard disk, a Memory, a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash Memory Card (Flash Card), at least one disk storage device, a Flash Memory device, a Read-Only Memory (ROM), a Random Access Memory (RAM), or other non-volatile/volatile storage devices.
The modules integrated by the computer device 30 may be stored in a computer-readable storage medium if they are implemented in the form of software functional modules and sold or used as separate products. Based on such understanding, all or part of the flow of the method according to the above embodiments may be implemented by hardware that is configured to be instructed by computer readable instructions, which may be stored in a computer readable storage medium, and when the computer readable instructions are executed by a processor, the steps of the method embodiments may be implemented. Wherein the computer readable instructions comprise computer readable instruction code which may be in source code form, object code form, an executable file or some intermediate form, and the like. The computer-readable medium may include: any entity or device capable of carrying said computer readable instruction code, a recording medium, a U disk, a removable hard disk, a magnetic diskette, an optical disk, a computer memory, a read-only memory (ROM), a Random Access Memory (RAM).
In the embodiments provided in the present invention, it should be understood that the disclosed system, apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the modules is only one logical functional division, and other divisions may be realized in practice.
The modules described as separate parts may or may not be physically separate, and parts displayed as modules may or may not be physical modules, may be located in one position, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment.
In addition, functional modules in the embodiments of the present invention may be integrated into one processing module, or each module may exist alone physically, or two or more modules are integrated into one module. The integrated module can be realized in a hardware form, and can also be realized in a form of hardware and a software functional module.
The integrated module implemented in the form of a software functional module may be stored in a computer-readable storage medium. The software functional module is stored in a storage medium and includes several instructions to enable a computer device (which may be a personal computer, a server, or a network device) or a processor (processor) to execute some steps of the mixed reality based live broadcast method according to various embodiments of the present invention.
It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential attributes thereof. The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference signs in the claims shall not be construed as limiting the claim concerned. Furthermore, it is to be understood that the word "comprising" does not exclude other modules or steps, and the singular does not exclude the plural. A plurality of modules or means recited in the system claims may also be implemented by one module or means in software or hardware. The terms first, second, etc. are used to denote names, but not to denote any particular order.
Finally, it should be noted that the above embodiments are only intended to illustrate the technical solutions of the present invention and not to limit the same, and although the present invention is described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.
Claims (8)
1. A live broadcasting method based on mixed reality is characterized in that the live broadcasting method based on mixed reality comprises the following steps:
configuring a virtual background, the virtual background comprising virtual objects;
acquiring first sound information and real first image information of a first object from a first client;
acquiring second sound information and interaction information of a second object from a second client;
rendering the first object into the virtual background according to the first image information, and controlling the virtual object to execute an action matched with the interaction information according to the interaction information;
constructing a virtual scene based on the virtual background, the first object and the virtual object, newly establishing a first virtual camera in the virtual scene, and acquiring a virtual interactive video through the first virtual camera, wherein the method comprises the following steps: binding the first virtual camera with a rendering target, and updating each frame of image acquired by the first virtual camera to the rendering target; generating a target texture according to the rendering target; mapping the target texture to a preset processor video memory, and generating a virtual interactive video;
creating a second virtual camera at the location of the first object in the virtual scene; acquiring second image information through the second virtual camera, and transmitting the second sound information and the second image information to the first client, so that the first object acquires the second sound information and the second image information through the first client, and interacts with the second object through the first client;
and outputting the virtual interactive video to a third client to realize live broadcast of the virtual interactive video.
2. The mixed reality-based live broadcasting method of claim 1, wherein after the acquiring the first sound information and the real first image information of the first object from the first client, the mixed reality-based live broadcasting method further comprises:
and transmitting the first sound information and the first image information to the second client, so that the second object acquires the first sound information and the first image information through the second client, and interacts with the first object through the second client.
3. The mixed reality-based live broadcasting method according to claim 1, wherein the controlling the virtual object to perform the action matched with the interaction information according to the interaction information comprises:
acquiring head information in the interactive information through VR head equipment or MR head equipment, and controlling the head of the virtual object to execute head action matched with the head information in the interactive information;
and acquiring hand information in the interactive information through a VR handle, and controlling the hand of the virtual object to execute hand actions matched with the hand information in the interactive information.
4. The mixed reality-based live broadcasting method of claim 1, wherein the generating a target texture according to the rendering target comprises:
newly building a target window, wherein the target window comprises an image in the rendering target;
binding the image in the target window to a first intermediate target texture of an independent context through a window handle of the target window;
and carrying out format preprocessing and size preprocessing on the first intermediate target texture to obtain the target texture.
5. The mixed reality-based live broadcasting method of claim 1, wherein the generating a target texture according to the rendering target comprises:
creating a target independent context according to the rendering target, and creating a shared second intermediate target texture in the target independent context;
copying a render target of the target independent context BGRA channel to the second intermediate target texture;
and after the RB channel of the second intermediate target texture is converted into the BGRA channel, format preprocessing and size preprocessing are carried out on the second intermediate target texture of the BGRA channel to obtain the target texture.
6. The utility model provides a live device based on mixed reality which characterized in that, live device based on mixed reality includes:
a configuration module to configure a virtual background, the virtual background including virtual objects;
the first acquisition module is used for acquiring first sound information and real first image information of a first object from a first client;
the second acquisition module is used for acquiring second sound information and interaction information of the second object from the second client;
the control module is used for rendering the first object into the virtual background according to the first image information and controlling the virtual object to execute an action matched with the interaction information according to the interaction information;
the acquisition module is configured to construct a virtual scene based on the virtual background, the first object and the virtual object, create a new first virtual camera in the virtual scene, and acquire a virtual interactive video through the first virtual camera, and includes: binding the first virtual camera with a rendering target, and updating each frame of image acquired by the first virtual camera to the rendering target; generating a target texture according to the rendering target; mapping the target texture to a preset processor video memory, and generating a virtual interactive video;
the acquisition module is further configured to newly build a second virtual camera at the position of the first object in the virtual scene; acquiring second image information through the second virtual camera, and transmitting the second sound information and the second image information to the first client, so that the first object acquires the second sound information and the second image information through the first client, and interacts with the second object through the first client;
and the output module is used for outputting the virtual interactive video to a third client so as to realize live broadcast of the virtual interactive video.
7. A computer device comprising a processor to execute computer readable instructions stored in a memory to implement the mixed reality based live method of any one of claims 1 to 5.
8. A computer readable storage medium having computer readable instructions stored thereon, which when executed by a processor implement the mixed reality based live method of any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010981861.8A CN112135158B (en) | 2020-09-17 | 2020-09-17 | Live broadcasting method based on mixed reality and related equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010981861.8A CN112135158B (en) | 2020-09-17 | 2020-09-17 | Live broadcasting method based on mixed reality and related equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112135158A CN112135158A (en) | 2020-12-25 |
| CN112135158B true CN112135158B (en) | 2022-10-25 |
Family
ID=73846039
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010981861.8A Active CN112135158B (en) | 2020-09-17 | 2020-09-17 | Live broadcasting method based on mixed reality and related equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112135158B (en) |
Families Citing this family (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116107472A (en) * | 2021-11-10 | 2023-05-12 | 北京字节跳动网络技术有限公司 | Display method, display device, electronic equipment and storage medium |
| CN114401415B (en) * | 2022-01-14 | 2024-04-12 | 北京字跳网络技术有限公司 | Live broadcast control method, live broadcast control device, computer equipment and storage medium |
| CN114630173A (en) * | 2022-03-03 | 2022-06-14 | 北京字跳网络技术有限公司 | Virtual object driving method and device, electronic equipment and readable storage medium |
| CN115209172B (en) * | 2022-07-13 | 2023-07-07 | 成都索贝数码科技股份有限公司 | XR-based remote interactive performance method |
| CN115278364B (en) * | 2022-07-29 | 2024-05-17 | 苏州创意云网络科技有限公司 | Video stream synthesis method and device |
| CN116110080B (en) * | 2023-04-04 | 2023-07-04 | 成都新希望金融信息有限公司 | Switching method of real facial mask and virtual facial mask |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8072470B2 (en) * | 2003-05-29 | 2011-12-06 | Sony Computer Entertainment Inc. | System and method for providing a real-time three-dimensional interactive environment |
| CN103164227B (en) * | 2013-03-28 | 2016-06-22 | 北京视博云科技有限公司 | The acquisition method of a kind of video in window and device |
| CN108459717A (en) * | 2018-03-13 | 2018-08-28 | 重庆虚拟实境科技有限公司 | Virtual education method, apparatus, computer installation and storage medium |
| CN111476903A (en) * | 2019-01-23 | 2020-07-31 | 北京奇虎科技有限公司 | Virtual interaction implementation control method and device, computer equipment and storage medium |
| CN110427107A (en) * | 2019-07-23 | 2019-11-08 | 德普信(天津)软件技术有限责任公司 | Virtually with real interactive teaching method and system, server, storage medium |
| CN111417014B (en) * | 2020-03-20 | 2022-12-13 | 深圳市企鹅网络科技有限公司 | Video generation method, system, device and storage medium based on online education |
-
2020
- 2020-09-17 CN CN202010981861.8A patent/CN112135158B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN112135158A (en) | 2020-12-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112135158B (en) | Live broadcasting method based on mixed reality and related equipment | |
| CN110570698B (en) | Online teaching control method and device, storage medium and terminal | |
| CN113240782B (en) | Streaming media generation method and device based on virtual roles | |
| CN108389249B (en) | Multi-compatibility VR/AR space classroom and construction method thereof | |
| CN107798932A (en) | A kind of early education training system based on AR technologies | |
| WO2019019403A1 (en) | Interactive situational teaching system for use in k12 stage | |
| Abu-Arqoub et al. | Interactive multimedia-based educational system for children using interactive book with augmented reality | |
| JP6683864B1 (en) | Content control system, content control method, and content control program | |
| CN108268139B (en) | Virtual scene interaction method and device, computer device and readable storage medium | |
| Pope et al. | The latest in immersive telepresence to support shared engineering education | |
| CN114463470A (en) | Virtual space browsing method and device, electronic equipment and readable storage medium | |
| Serubugo et al. | Comparison of wearable optical see-through and handheld devices as platform for an augmented reality museum guide | |
| KR20160136833A (en) | medical education system using video contents | |
| CN114846808A (en) | Content distribution system, content distribution method, and content distribution program | |
| CN116668605A (en) | Meta-universe studio system, playing method, storage medium and electronic device | |
| CN115297224A (en) | Campus virtual idol live broadcasting method, system, device and storage medium | |
| JP7465736B2 (en) | Content control system, content control method, and content control program | |
| CN210072615U (en) | Immersive training system and wearable equipment | |
| CN114245193A (en) | Display control method and device and electronic equipment | |
| CN108805951B (en) | Projection image processing method, device, terminal and storage medium | |
| KR20110107707A (en) | Online learning apparatus for augmented reality and method thereof | |
| JP2021009351A (en) | Content control system, content control method, and content control program | |
| Ariffin et al. | Enhancing tourism experiences via mobile augmented reality by superimposing virtual information on artefacts | |
| US20220343783A1 (en) | Content control system, content control method, and content control program | |
| CN112135157B (en) | Video content acquisition method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |