CN112653913A - Screen projection method and device, client device and readable storage medium - Google Patents
Screen projection method and device, client device and readable storage medium Download PDFInfo
- Publication number
- CN112653913A CN112653913A CN202011573439.5A CN202011573439A CN112653913A CN 112653913 A CN112653913 A CN 112653913A CN 202011573439 A CN202011573439 A CN 202011573439A CN 112653913 A CN112653913 A CN 112653913A
- Authority
- CN
- China
- Prior art keywords
- screen
- data
- client device
- screen projection
- audio data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 66
- 238000012545 processing Methods 0.000 claims abstract description 46
- 238000009877 rendering Methods 0.000 claims description 11
- 238000004806 packaging method and process Methods 0.000 claims description 10
- 238000004891 communication Methods 0.000 claims description 6
- 238000004458 analytical method Methods 0.000 claims description 3
- 239000000126 substance Substances 0.000 claims description 2
- 230000008569 process Effects 0.000 description 11
- 238000005516 engineering process Methods 0.000 description 10
- 238000005266 casting Methods 0.000 description 9
- 238000010586 diagram Methods 0.000 description 8
- 230000006870 function Effects 0.000 description 6
- 238000013475 authorization Methods 0.000 description 4
- 230000006835 compression Effects 0.000 description 4
- 238000007906 compression Methods 0.000 description 4
- 238000011084 recovery Methods 0.000 description 4
- 238000004590 computer program Methods 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 238000004422 calculation algorithm Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 239000003999 initiator Substances 0.000 description 2
- 239000004973 liquid crystal related substance Substances 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000012856 packing Methods 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 1
- 230000008094 contradictory effect Effects 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/41—Structure of client; Structure of client peripherals
- H04N21/4104—Peripherals receiving signals from specially adapted client devices
- H04N21/4122—Peripherals receiving signals from specially adapted client devices additional display device, e.g. video projector
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
- G06F3/1454—Digital output to display device ; Cooperation and interconnection of the display device with other functional units involving copying of the display data of a local workstation or window to a remote workstation or window so that an actual copy of the data is displayed simultaneously on two or more displays, e.g. teledisplay
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/439—Processing of audio elementary streams
- H04N21/4398—Processing of audio elementary streams involving reformatting operations of audio signals
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
- H04N21/4402—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display
- H04N21/440218—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display by transcoding between formats or standards, e.g. from MPEG-2 to MPEG-4
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/443—OS processes, e.g. booting an STB, implementing a Java virtual machine in an STB or power management in an STB
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
Abstract
The invention relates to data processing, and discloses a screen projection method, which comprises the following steps: acquiring a second client device identifier corresponding to a screen projection receiver, and judging the type of an operating system of the first client device; when the type of the operating system of the first client device is judged to be the first type, calling a first component to obtain screen picture data and audio data of the first client device, and performing coding processing on the screen picture data and the audio data to obtain coded screen picture data and coded audio data; and performing packet processing on the encoded screen picture data and the encoded audio data to obtain a plurality of small data packets, and sending the plurality of small data packets to the second client equipment based on the second client equipment identification so that the second client equipment can perform screen projection operation. The invention also provides a screen projection device, client equipment and a readable storage medium. The invention realizes that the small-screen equipment projects to the small-screen equipment and one-to-many projection screens.
Description
Technical Field
The present invention relates to the field of data processing, and in particular, to a screen projection method and apparatus, a client device, and a readable storage medium.
Background
With the popularization of intelligent mobile devices, the application of screen mutual projection between the intelligent mobile devices in the life of people is more and more extensive. For example, in a multi-person conference, each speaker projects its own computer screen onto a large-screen device.
The existing screen projection scheme usually only supports small-screen equipment to project to large-screen equipment (for example, a mobile phone projects a screen to a computer, a mobile phone projects a screen to a television, and a computer projects a screen to a television), but cannot realize small-screen equipment to project to small-screen equipment (for example, a mobile phone projects a screen to a mobile phone, and a mobile phone projects a screen to a panel); in addition, the existing screen projection schemes are all one-to-one screen projection, and one-to-many screen projection cannot be realized (for example, for a scene lacking large-screen equipment in a multi-person collaboration meeting). Therefore, a screen projection method is needed to realize the projection of the small-screen device to the small-screen device and the one-to-many screen projection.
Disclosure of Invention
In view of the above, it is necessary to provide a screen projection method aiming at realizing projection of a small-screen device to the small-screen device and one-to-many screen projection.
The screen projection method provided by the invention is applied to first client equipment, and comprises the following steps:
analyzing a screen projection request sent by a user, acquiring a second client equipment identifier corresponding to a screen projection receiver carried by the request, and judging the type of an operating system of the first client equipment;
when the type of the operating system of the first client device is judged to be a first type, calling a first component to obtain screen picture data and audio data of the first client device, and performing coding processing on the screen picture data and the audio data to obtain coded screen picture data and coded audio data;
and performing packet processing on the encoded screen picture data and the encoded audio data to obtain a plurality of small data packets, and sending the plurality of small data packets to the second client equipment based on the second client equipment identifier so that the second client equipment can perform screen projection operation.
Optionally, the first component includes a start unit, a pause unit, a resume unit, an end unit, and a callback unit, and the invoking the first component to obtain the screen data and the audio data of the first client device includes:
recording screen picture data and audio data of the first client equipment based on the starting unit, the suspending unit, the recovering unit and the ending unit;
and acquiring the screen picture data and the audio data based on the callback unit.
Optionally, the performing encoding processing on the screen picture data and the audio data to obtain encoded screen picture data and encoded audio data includes:
executing first coding on the screen picture data to obtain coded screen picture data in a first format;
and performing second coding on the audio data to obtain coded audio data in a second format.
Optionally, when it is determined that the operating system type of the first client device is not the first type, the method further includes:
and if the operating system type of the first client device is a second type, calling a second component and a third component to acquire the coded screen picture data and the coded audio data of the first client device.
Optionally, the second component includes a creating unit, the third component includes a setting unit and an obtaining unit, and the invoking the second component and the third component to obtain the encoded screen data and the encoded audio data of the first client device includes:
setting coding parameters based on the setting unit, and calling the creating unit to generate coded screen picture data and coded audio data of the first client device based on the coding parameters;
acquiring the encoded screen picture data and the encoded audio data based on the acquisition unit.
The invention also provides a screen projection method, which is applied to second client equipment, wherein the second client equipment is in communication connection with the first client equipment, and the method comprises the following steps:
receiving a plurality of small data packets sent by first client equipment, and performing packet packaging processing on the small data packets to obtain packaged screen projection data;
decoding the packaged screen projection data to obtain decoded screen projection data;
rendering and displaying the decoded screen projection data, and realizing screen projection of the screen picture data and the audio data of the first client device to the second client device.
In order to solve the above problems, the present invention also provides a screen projection apparatus, comprising:
the analysis module is used for analyzing a screen projection request sent by a user, acquiring a second client equipment identifier carried by the request and corresponding to a screen projection receiver, and judging the type of an operating system of the first client equipment;
the calling module is used for calling a first component to acquire screen picture data and audio data of the first client device and executing coding processing on the screen picture data and the audio data to obtain coded screen picture data and coded audio data when the type of the operating system of the first client device is judged to be a first type;
and the sending module is used for performing packet processing on the coded screen picture data and the coded audio data to obtain a plurality of small data packets, and sending the plurality of small data packets to the second client equipment based on the second client equipment identification so that the second client equipment can perform screen projection operation.
The present invention also provides a screen projection apparatus, the apparatus comprising:
the group packaging module is used for receiving a plurality of small data packets sent by first client equipment, and performing group packaging processing on the small data packets to obtain packaged screen projection data;
the decoding module is used for executing decoding processing on the packaged screen projection data to obtain decoded screen projection data;
and the screen projection module is used for rendering and displaying the decoded screen projection data to realize screen projection of the screen picture data and the audio data of the first client equipment to the second client equipment.
In order to solve the above problem, the present invention also provides a client device, including:
at least one processor; and the number of the first and second groups,
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores a screen projection program executable by the at least one processor, the screen projection program being executed by the at least one processor to enable the at least one processor to perform the above-described screen projection method.
In order to solve the above problem, the present invention also provides a computer-readable storage medium having a screen projection program stored thereon, where the screen projection program is executable by one or more processors to implement the screen projection method.
Compared with the prior art, the method comprises the steps of firstly obtaining the identification of the second client device corresponding to the screen projection receiver, and judging the type of the operating system of the first client device; then, when the type of the operating system of the first client device is judged to be the first type, calling the first component to obtain screen picture data and audio data of the first client device, and performing coding processing on the screen picture data and the audio data to obtain coded screen picture data and coded audio data; and finally, performing sub-packet processing on the coded screen picture data and the coded audio data to obtain a plurality of small data packets, and sending the small data packets to a second client device for the second client device to perform screen projection operation. Therefore, the invention realizes the projection of the small-screen equipment to the small-screen equipment and the one-to-many projection of the screen.
Drawings
Fig. 1 is a schematic flow chart of a screen projection method according to an embodiment of the present invention;
fig. 2 is a schematic flow chart of a screen projection method according to another embodiment of the present invention;
fig. 3 is a schematic block diagram of a screen projection apparatus according to an embodiment of the present invention;
fig. 4 is a schematic block diagram of a screen projection apparatus according to another embodiment of the present invention;
fig. 5 is a screen projection system including a first client device and a second client device according to an embodiment of the present invention;
fig. 6 is a schematic structural diagram of a first client device according to an embodiment of the present invention;
fig. 7 is a schematic structural diagram of a second client device according to an embodiment of the present invention;
the implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that the description relating to "first", "second", etc. in the present invention is for descriptive purposes only and is not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In addition, technical solutions between various embodiments may be combined with each other, but must be realized by a person skilled in the art, and when the technical solutions are contradictory or cannot be realized, such a combination should not be considered to exist, and is not within the protection scope of the present invention.
The invention provides a screen projection method. Fig. 1 is a schematic flow chart of a screen projection method according to an embodiment of the present invention. The method is performed by a first client device, which may be implemented by software and/or hardware.
In this embodiment, the screen projection method includes:
s11, a screen projection request sent by a user is analyzed, a second client device identification corresponding to a screen projection receiver carried by the request is obtained, and the type of the operating system of the first client device is judged.
In this embodiment, a scene lacking a large-screen device in a multi-user collaborative screen-casting conference is taken as an example to illustrate an implementation process of the present scheme, where a user sending a screen-casting request is a current speaker of the conference, a first client device is an intelligent mobile device corresponding to a screen-casting initiator (i.e., the current speaker), a second client device identifier is an identifier (e.g., an IP address or an MAC address of the second client device) of an intelligent mobile device corresponding to a screen-casting receiver (other attendees), the first client device and the second client device may be mobile phones or tablets, the types of the operating systems include a first type and a second type, the first type is an iOS operating system, the second type is an Android operating system, and meanwhile, the first client device and the second client device are located in the same local area network.
And S12, when the operating system type of the first client device is judged to be the first type, calling a first component to acquire the screen picture data and the audio data of the first client device, and performing coding processing on the screen picture data and the audio data to obtain coded screen picture data and coded audio data.
The first component comprises a start unit, a pause unit, a recovery unit, an end unit and a callback unit, and the calling the first component to acquire the screen image data and the audio data of the first client device comprises:
a11, recording the screen image data and the audio data of the first client device based on the starting unit, the pause unit, the recovery unit and the ending unit;
and A12, acquiring the screen picture data and the audio data based on the callback unit.
In this embodiment, this step is implemented by a Broadcast Upload Extension development framework of the iOS operating system, the first component is a samplehandle class (a samplehandle class may be automatically generated by creating a Broadcast Upload Extension in an APP supporting Broadcast), the start unit is a start method broadcaststartwithsetinfo of the samplehandle class, the pause unit is a pause method broadcastclosed, the resume unit is a resume method broadcastrated, the end unit is an end method broadcastFinished, and the callback unit is a callback method processSampleBuffer.
The encoding processing is performed on the screen picture data and the audio data to obtain encoded screen picture data and encoded audio data, and the encoding processing includes:
b11, executing the first coding to the screen picture data to obtain the coded screen picture data in the first format;
and B12, performing second coding on the audio data to obtain coded audio data in a second format.
The encoding is to compress data to improve subsequent transmission efficiency. In the embodiment, a h264 coding technology (a core algorithm adopted by the h264 coding technology is intraframe compression and interframe compression; the h264 coding technology can also flexibly set the resolution, code rate and frame rate of coding) is adopted to execute a first coding operation, so that coded screen picture data in a h264 coding format (namely the first format) is obtained; and performing a second encoding operation by using an aac (advanced Audio Coding) technology to obtain encoded Audio data in an aac encoding format (i.e., the second format) (compared with MP3, the aac encoding process is more efficient, and the encoded file is smaller).
In this embodiment, when it is determined that the operating system type of the first client device is not the first type, the method further includes:
and if the operating system type of the first client device is a second type, calling a second component and a third component to acquire the coded screen picture data and the coded audio data of the first client device.
The second component comprises a creation unit, the third component comprises a setting unit and an acquisition unit, and the calling the second component and the third component to acquire the encoded screen picture data and the encoded audio data of the first client device comprises:
c11, setting coding parameters based on the setting unit, and calling the creating unit to generate coded screen picture data and coded audio data of the first client device based on the coding parameters;
c12, acquiring the encoded screen picture data and the encoded audio data based on the acquiring unit.
The method is realized by a system screen recording service MediaProjectionmanager management class based on an Android operating system, and screen data can be acquired without root on a mobile phone or a tablet.
The second component is a screen recording operation object mediaproject (after acquiring a system screen recording service mediaproject manager management class, a user authorization window is called, and after a user clicks and agrees to authorization, the screen recording operation object mediaproject can be acquired).
The third component is an encoder object, Mediacodec, and in this embodiment, the setting unit is configured to set encoding parameters (e.g., resolution, code rate, and frame rate) related to h264, and the output format of the screen picture data can be determined by the encoding parameters.
The creation unit is a createVirtualDisplay method of mediaproject object. And calling a creating unit, and taking a Surface object of the Mediacodec object as a parameter to be transmitted in, namely, creating a system virtual screen object VirtualDisplay, wherein the object is responsible for generating screen data.
The obtaining unit is a getOutputBuffer method in a Mediacodec object.
Since the output format of the screen projection data is set by the setting unit, the encoded video data is obtained, and therefore, for the Android operating system, encoding operation does not need to be executed.
And S13, performing subpackage processing on the coded screen picture data and the coded audio data to obtain a plurality of small data packets, and sending the plurality of small data packets to the second client equipment based on the second client equipment identification so that the second client equipment can perform screen projection operation.
In this embodiment, the single image frame data may be divided into a plurality of small packets starting with 0000 data and one small packet starting with 0002 data, and the small packets may be sent to the second client device based on the tcp/udp network protocol.
As can be seen from the foregoing embodiment, the screen projection method provided by the present invention includes, first, obtaining an identifier of a second client device corresponding to a screen projection receiver, and determining an operating system type of a first client device; then, when the type of the operating system of the first client device is judged to be the first type, calling the first component to obtain screen picture data and audio data of the first client device, and performing coding processing on the screen picture data and the audio data to obtain coded screen picture data and coded audio data; and finally, performing sub-packet processing on the coded screen picture data and the coded audio data to obtain a plurality of small data packets, and sending the small data packets to a second client device for the second client device to perform screen projection operation. Therefore, the invention realizes the projection of the small-screen equipment to the small-screen equipment and the one-to-many projection of the screen.
Fig. 2 is a schematic flow chart of a screen projection method according to another embodiment of the present invention.
In this embodiment, the screen projection method is applied to a second client device, where the second client device is in communication connection with a first client device, and the method includes:
s21, receiving a plurality of small data packets sent by the first client device, and performing packaging processing on the small data packets to obtain packaged screen projection data;
s22, decoding the packaged screen projection data to obtain decoded screen projection data;
and S23, rendering and displaying the decoded screen projection data, and realizing screen projection of the screen picture data and the audio data of the first client device to the second client device.
In this embodiment, the packing process is the reverse of the packetizing process, and data beginning with 0000 data and data beginning with 0002 data that are continuously received are spliced to obtain a complete image frame data.
In the embodiment, decoding processing is performed by using an h264 decoding technology, and the decoded screen projection data is in a yuv data format.
In this embodiment, a general rendering scheme is used to render the decoded screen projection data, and the rendering process is not described again.
Through the technical scheme shown in fig. 1 and fig. 2, any intelligent mobile device can be projected to multiple intelligent mobile devices in the same local area network.
Fig. 3 is a schematic block diagram of a screen projection apparatus according to an embodiment of the present invention.
The screen projection device 100 is installed in a first client device. Depending on the implemented functions, the screen projection device 100 may include a parsing module 110, a calling module 120, and a sending module 130. The modules of the present invention, which may also be referred to as units, refer to a series of computer program segments that can be executed by a processor of a first client device and that can perform a fixed function, and that are stored in a memory of the first client device.
In the present embodiment, the functions regarding the respective modules/units are as follows:
the analysis module 110 is configured to analyze a screen projection request sent by a user, acquire a second client device identifier corresponding to a screen projection receiver carried in the request, and determine an operating system type of the first client device.
In this embodiment, a scene lacking a large-screen device in a multi-user collaborative screen-casting conference is taken as an example to illustrate an implementation process of the present scheme, where a user sending a screen-casting request is a current speaker of the conference, a first client device is an intelligent mobile device corresponding to a screen-casting initiator (i.e., the current speaker), a second client device identifier is an identifier (e.g., an IP address or an MAC address of the second client device) of an intelligent mobile device corresponding to a screen-casting receiver (other attendees), the first client device and the second client device may be mobile phones or tablets, the types of the operating systems include a first type and a second type, the first type is an iOS operating system, the second type is an Android operating system, and meanwhile, the first client device and the second client device are located in the same local area network.
The invoking module 120 is configured to, when the operating system type of the first client device is determined to be the first type, invoke a first component to obtain screen image data and audio data of the first client device, and perform encoding processing on the screen image data and the audio data to obtain encoded screen image data and encoded audio data.
The first component comprises a start unit, a pause unit, a recovery unit, an end unit and a callback unit, and the calling the first component to acquire the screen image data and the audio data of the first client device comprises:
a21, recording the screen image data and the audio data of the first client device based on the starting unit, the pause unit, the recovery unit and the ending unit;
and A22, acquiring the screen picture data and the audio data based on the callback unit.
In this embodiment, this step is implemented by a Broadcast Upload Extension development framework of the iOS operating system, the first component is a samplehandle class (a samplehandle class may be automatically generated by creating a Broadcast Upload Extension in an APP supporting Broadcast), the start unit is a start method broadcaststartwithsetinfo of the samplehandle class, the pause unit is a pause method broadcastclosed, the resume unit is a resume method broadcastrated, the end unit is an end method broadcastFinished, and the callback unit is a callback method processSampleBuffer.
The encoding processing is performed on the screen picture data and the audio data to obtain encoded screen picture data and encoded audio data, and the encoding processing includes:
b21, executing the first coding to the screen picture data to obtain the coded screen picture data in the first format;
and B22, performing second coding on the audio data to obtain coded audio data in a second format.
The encoding is to compress data to improve subsequent transmission efficiency. In the embodiment, a h264 coding technology (a core algorithm adopted by the h264 coding technology is intraframe compression and interframe compression; the h264 coding technology can also flexibly set the resolution, code rate and frame rate of coding) is adopted to execute a first coding operation, so that coded screen picture data in a h264 coding format (namely the first format) is obtained; and performing a second encoding operation by using an aac (advanced Audio Coding) technology to obtain encoded Audio data in an aac encoding format (i.e., the second format) (compared with MP3, the aac encoding process is more efficient, and the encoded file is smaller).
In this embodiment, when it is determined that the operating system type of the first client device is not the first type, the invoking module 120 is further configured to:
and if the operating system type of the first client device is a second type, calling a second component and a third component to acquire the coded screen picture data and the coded audio data of the first client device.
The second component comprises a creation unit, the third component comprises a setting unit and an acquisition unit, and the calling the second component and the third component to acquire the encoded screen picture data and the encoded audio data of the first client device comprises:
c21, setting coding parameters based on the setting unit, and calling the creating unit to generate coded screen picture data and coded audio data of the first client device based on the coding parameters;
c22, acquiring the encoded screen picture data and the encoded audio data based on the acquiring unit.
The method is realized by a system screen recording service MediaProjectionmanager management class based on an Android operating system, and screen data can be acquired without root on a mobile phone or a tablet.
The second component is a screen recording operation object mediaproject (after acquiring a system screen recording service mediaproject manager management class, a user authorization window is called, and after a user clicks and agrees to authorization, the screen recording operation object mediaproject can be acquired).
The third component is an encoder object, Mediacodec, and in this embodiment, the setting unit is configured to set encoding parameters (e.g., resolution, code rate, and frame rate) related to h264, and the output format of the screen picture data can be determined by the encoding parameters.
The creation unit is a createVirtualDisplay method of mediaproject object. And calling a creating unit, and taking a Surface object of the Mediacodec object as a parameter to be transmitted in, namely, creating a system virtual screen object VirtualDisplay, wherein the object is responsible for generating screen data.
The obtaining unit is a getOutputBuffer method in a Mediacodec object.
Since the output format of the screen projection data is set by the setting unit, the encoded video data is obtained, and therefore, for the Android operating system, encoding operation does not need to be executed.
A sending module 130, configured to perform packet-division processing on the encoded screen picture data and the encoded audio data to obtain multiple small data packets, and send the multiple small data packets to the second client device based on the identifier of the second client device, so that the second client device performs a screen-casting operation.
In this embodiment, the single image frame data may be divided into a plurality of small packets starting with 0000 data and one small packet starting with 0002 data, and the small packets may be sent to the second client device based on the tcp/udp network protocol.
Fig. 4 is a schematic block diagram of a screen projection apparatus according to another embodiment of the present invention.
The screen projection device 200 is installed in the second client device. Depending on the implemented functions, the screen projection device 200 may include a group package module 210, a decoding module 220, and a screen projection module 230. The module of the present invention, which may also be referred to as a unit, refers to a series of computer program segments that can be executed by a processor of a second client device and that can perform a fixed function, and that are stored in a memory of the second client device.
In the present embodiment, the functions regarding the respective modules/units are as follows:
the group packaging module 210 is configured to receive a plurality of small data packets sent by a first client device, and perform group packaging processing on the small data packets to obtain packaged screen projection data;
a decoding module 220, configured to perform decoding processing on the packaged screen projection data to obtain decoded screen projection data;
and a screen projection module 230, configured to render and display the decoded screen projection data, so as to implement screen projection of the screen image data and the audio data of the first client device to the second client device.
In this embodiment, the packing process is the reverse of the packetizing process, and data beginning with 0000 data and data beginning with 0002 data that are continuously received are spliced to obtain a complete image frame data.
In the embodiment, decoding processing is performed by using an h264 decoding technology, and the decoded screen projection data is in a yuv data format.
In this embodiment, a general rendering scheme is used to render the decoded screen projection data, and the rendering process is not described again.
Through the technical scheme shown in fig. 3 and 4, any intelligent mobile device can be projected to multiple intelligent mobile devices in the same local area network.
As shown in fig. 5, a screen projection system including a first client device and a second client device is provided for an embodiment of the present invention. In the present embodiment, the first client apparatus 1 is connected to a plurality of second client apparatuses 2 in a communication manner.
The first client device 1 and the second client device 2 are intelligent mobile devices.
In one embodiment of the invention, the first client device 1 is configured to:
analyzing a screen projection request sent by a user, acquiring a second client device 2 identifier corresponding to a screen projection receiver carried by the request, and judging the type of an operating system of the first client device 1;
when the operating system type of the first client device 1 is judged to be a first type, calling a first component to obtain screen picture data and audio data of the first client device 1, and performing coding processing on the screen picture data and the audio data to obtain coded screen picture data and coded audio data;
and performing packet processing on the encoded screen picture data and the encoded audio data to obtain a plurality of small data packets, and sending the plurality of small data packets to the second client device 2 based on the identifier of the second client device 2 so that the second client device 2 can perform screen projection operation.
The second client device 2 is configured to:
receiving a plurality of small data packets sent by the first client device 1, and performing packet processing on the small data packets to obtain packaged screen projection data;
decoding the packaged screen projection data to obtain decoded screen projection data;
rendering and displaying the decoded screen projection data, so as to realize screen projection of the screen picture data and the audio data of the first client device 1 to the second client device 2.
Fig. 6 is a schematic structural diagram of a first client device for implementing a screen projection method according to an embodiment of the present invention.
The first client apparatus 1 is an apparatus capable of automatically performing numerical calculation and/or information processing in accordance with a command set or stored in advance.
In the present embodiment, the first client device 1 includes, but is not limited to, a memory 11, a processor 12, and a network interface 13, which are communicatively connected to each other through a system bus, wherein the memory 11 stores a screen projection program 10, and the screen projection program 10 can be executed by the processor 12. Fig. 3 shows only the first client device 1 with the components 11-13 and the screen projection program 10, it being understood by a person skilled in the art that the structure shown in fig. 3 does not constitute a limitation of the first client device 1, and may comprise fewer or more components than shown, or combine certain components, or a different arrangement of components.
The storage 11 includes a memory and at least one type of readable storage medium. The memory provides cache for the operation of the first client device 1; the readable storage medium may be a non-volatile storage medium such as flash memory, a hard disk, a multimedia card, a card type memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a Read Only Memory (ROM), an Electrically Erasable Programmable Read Only Memory (EEPROM), a Programmable Read Only Memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the readable storage medium may be an internal storage unit of the first client device 1, such as a hard disk of the first client device 1; in other embodiments, the non-volatile storage medium may also be an external storage device of the first client device 1, such as a plug-in hard disk provided on the first client device 1, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like. In this embodiment, the readable storage medium of the memory 11 is generally used for storing an operating system and various application software installed in the first client device 1, for example, code of the screen projection program 10 in an embodiment of the present invention. Further, the memory 11 may also be used to temporarily store various types of data that have been output or are to be output.
Processor 12 may be a Central Processing Unit (CPU), controller, microcontroller, microprocessor, or other data Processing chip in some embodiments. The processor 12 is generally configured to control the overall operation of the first client device 1, for example, to perform control and processing related to data interaction or communication with other devices. In this embodiment, the processor 12 is configured to execute the program code stored in the memory 11 or process data, such as executing the screen projection program 10.
The network interface 13 may comprise a wireless network interface or a wired network interface, and the network interface 13 is used for establishing a communication connection between the first client device 1 and the second client device 2.
Optionally, the first client device 1 may further include a user interface, the user interface may include a Display (Display), an input unit such as a Keyboard (Keyboard), and the optional user interface may further include a standard wired interface and a wireless interface. Alternatively, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, or the like. The display, which may also be referred to as a display screen or display unit, is suitable for displaying information processed in the first client device 1 and for displaying a visualized user interface.
It is to be understood that the described embodiments are for purposes of illustration only and that the scope of the appended claims is not limited to such structures.
The screen projection program 10 stored in the memory 11 of the first client device 1 is a combination of instructions that, when executed in the processor 12, may implement:
analyzing a screen projection request sent by a user, acquiring a second client device 2 identifier corresponding to a screen projection receiver carried by the request, and judging the type of an operating system of the first client device 1;
when the operating system type of the first client device 1 is judged to be a first type, calling a first component to obtain screen picture data and audio data of the first client device 1, and performing coding processing on the screen picture data and the audio data to obtain coded screen picture data and coded audio data;
and performing packet processing on the encoded screen picture data and the encoded audio data to obtain a plurality of small data packets, and sending the plurality of small data packets to the second client device 2 based on the identifier of the second client device 2 so that the second client device 2 can perform screen projection operation.
Specifically, the processor 12 may refer to the description of the relevant steps in the embodiment corresponding to fig. 1 for a specific implementation method of the screen projection program 10, which is not described herein again.
Fig. 7 is a schematic structural diagram of a second client device 2 of the screen projection method according to an embodiment of the present invention.
In the present embodiment, the second client device 2 includes, but is not limited to, a memory 21, a processor 22, and a network interface 23, which are communicatively connected to each other through a system bus, wherein the memory 21 stores a screen projection program 20, and the screen projection program 20 can be executed by the processor 22.
The screen projection program 20 stored in the memory 21 of the second client device 2 is a combination of instructions that, when executed in the processor 22, may implement:
receiving a plurality of small data packets sent by a first client device 1, and performing packet packaging processing on the small data packets to obtain packaged screen projection data;
decoding the packaged screen projection data to obtain decoded screen projection data;
rendering and displaying the decoded screen projection data, so as to realize screen projection of the screen picture data and the audio data of the first client device 1 to the second client device 2.
Further, the modules/units integrated by the first client device 1 and/or the second client device 2 may be stored in a computer readable storage medium if they are implemented in the form of software functional units and sold or used as independent products. The computer readable medium may be non-volatile or non-volatile. The computer-readable medium may include: any entity or device capable of carrying said computer program code, recording medium, U-disk, removable hard disk, magnetic disk, optical disk, computer Memory, Read-Only Memory (ROM).
The computer-readable storage medium has a screen projection program 10 stored thereon, and the screen projection program 10 can be executed by one or more processors, and the embodiment of the computer-readable storage medium of the present invention is substantially the same as that of each embodiment of the screen projection method shown in fig. 1.
Further, the computer-readable storage medium may also store a screen projection program 20, and the screen projection program 20 may be executed by one or more processors, and the specific implementation of the computer-readable storage medium of the present invention is substantially the same as that of each embodiment of the screen projection method shown in fig. 2.
In the embodiments provided in the present invention, it should be understood that the disclosed apparatus, device and method can be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the modules is only one logical functional division, and other divisions may be realized in practice.
The modules described as separate parts may or may not be physically separate, and parts displayed as modules may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment.
In addition, functional modules in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, or in a form of hardware plus a software functional module.
It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential attributes thereof.
The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference signs in the claims shall not be construed as limiting the claim concerned.
Furthermore, it is obvious that the word "comprising" does not exclude other elements or steps, and the singular does not exclude the plural. A plurality of units or means recited in the system claims may also be implemented by one unit or means in software or hardware. The terms second, etc. are used to denote names, but not any particular order.
Finally, it should be noted that the above embodiments are only for illustrating the technical solutions of the present invention and not for limiting, and although the present invention is described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that modifications or equivalent substitutions may be made on the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.
Claims (10)
1. A screen projection method is applied to a first client device, and is characterized by comprising the following steps:
analyzing a screen projection request sent by a user, acquiring a second client equipment identifier corresponding to a screen projection receiver carried by the request, and judging the type of an operating system of the first client equipment;
when the type of the operating system of the first client device is judged to be a first type, calling a first component to obtain screen picture data and audio data of the first client device, and performing coding processing on the screen picture data and the audio data to obtain coded screen picture data and coded audio data;
and performing packet processing on the encoded screen picture data and the encoded audio data to obtain a plurality of small data packets, and sending the plurality of small data packets to the second client equipment based on the second client equipment identifier so that the second client equipment can perform screen projection operation.
2. The screen projection method of claim 1, wherein the first component comprises a start unit, a pause unit, a resume unit, an end unit, and a callback unit, and the invoking the first component to obtain the screen data and the audio data of the first client device comprises:
recording screen picture data and audio data of the first client equipment based on the starting unit, the suspending unit, the recovering unit and the ending unit;
and acquiring the screen picture data and the audio data based on the callback unit.
3. The screen projection method of claim 1, wherein the encoding the screen picture data and the audio data to obtain encoded screen picture data and encoded audio data comprises:
executing first coding on the screen picture data to obtain coded screen picture data in a first format;
and performing second coding on the audio data to obtain coded audio data in a second format.
4. The screen-projection method of claim 1, wherein when it is determined that the operating system type of the first client device is not a first type, the method further comprises:
and if the operating system type of the first client device is a second type, calling a second component and a third component to acquire the coded screen picture data and the coded audio data of the first client device.
5. The screen projection method of claim 4, wherein the second component comprises a creation unit, the third component comprises a setting unit and an acquisition unit, and the invoking the second component and the third component to acquire the encoded screen picture data and the encoded audio data of the first client device comprises:
setting coding parameters based on the setting unit, and calling the creating unit to generate coded screen picture data and coded audio data of the first client device based on the coding parameters;
acquiring the encoded screen picture data and the encoded audio data based on the acquisition unit.
6. A screen projection method is applied to a second client device which is in communication connection with a first client device, and is characterized by comprising the following steps:
receiving a plurality of small data packets sent by the first client device, and performing packet packaging processing on the small data packets to obtain packaged screen projection data;
decoding the packaged screen projection data to obtain decoded screen projection data;
rendering and displaying the decoded screen projection data, and realizing screen projection of the screen picture data and the audio data of the first client device to the second client device.
7. A screen projection apparatus, comprising:
the analysis module is used for analyzing a screen projection request sent by a user, acquiring a second client equipment identifier carried by the request and corresponding to a screen projection receiver, and judging the type of an operating system of the first client equipment;
the calling module is used for calling a first component to acquire screen picture data and audio data of the first client device and executing coding processing on the screen picture data and the audio data to obtain coded screen picture data and coded audio data when the type of the operating system of the first client device is judged to be a first type;
and the sending module is used for performing packet processing on the coded screen picture data and the coded audio data to obtain a plurality of small data packets, and sending the plurality of small data packets to the second client equipment based on the second client equipment identification so that the second client equipment can perform screen projection operation.
8. A screen projection apparatus, comprising:
the group packaging module is used for receiving a plurality of small data packets sent by first client equipment, and performing group packaging processing on the small data packets to obtain packaged screen projection data;
the decoding module is used for executing decoding processing on the packaged screen projection data to obtain decoded screen projection data;
and the screen projection module is used for rendering and displaying the decoded screen projection data to realize screen projection of the screen picture data and the audio data of the first client equipment to the second client equipment.
9. A client device, the client device comprising:
at least one processor; and the number of the first and second groups,
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores a screen projection program executable by the at least one processor to enable the at least one processor to perform the screen projection method of any one of claims 1 to 5 or claim 6.
10. A computer-readable storage medium having stored thereon a screen projection program executable by one or more processors to implement the screen projection method of any one of claims 1 to 5 or claim 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011573439.5A CN112653913A (en) | 2020-12-25 | 2020-12-25 | Screen projection method and device, client device and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011573439.5A CN112653913A (en) | 2020-12-25 | 2020-12-25 | Screen projection method and device, client device and readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112653913A true CN112653913A (en) | 2021-04-13 |
Family
ID=75363183
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011573439.5A Pending CN112653913A (en) | 2020-12-25 | 2020-12-25 | Screen projection method and device, client device and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112653913A (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113179307A (en) * | 2021-04-23 | 2021-07-27 | 努比亚技术有限公司 | Audio time delay control method and device and computer readable storage medium |
| CN114040242A (en) * | 2021-09-30 | 2022-02-11 | 荣耀终端有限公司 | Screen projection method and electronic equipment |
| CN114157903A (en) * | 2021-12-02 | 2022-03-08 | Oppo广东移动通信有限公司 | Redirection method, redirection device, redirection equipment, storage medium and program product |
| CN114691440A (en) * | 2022-06-01 | 2022-07-01 | 广州市高奈特网络科技有限公司 | Data extraction method and device of equipment, computer equipment and storage medium |
| CN115495307A (en) * | 2022-08-26 | 2022-12-20 | 深圳市旭联信息技术有限公司 | Screen transmission data processing method and device, computer readable storage medium and system |
| CN115515001A (en) * | 2021-06-22 | 2022-12-23 | 荣耀终端有限公司 | Screen mirroring method, device, equipment and storage medium |
| WO2024114427A1 (en) * | 2022-11-29 | 2024-06-06 | 蔚来移动科技有限公司 | Screen projection method and apparatus based on interconnection between onboard terminal and mobile device, and medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107770584A (en) * | 2017-11-01 | 2018-03-06 | 深圳市网旭科技有限公司 | A kind of method of the projection Android device screen picture to other platforms in real time |
| US20190037219A1 (en) * | 2017-07-28 | 2019-01-31 | Microsoft Technology Licensing, Llc | Effective intra encoding for screen data |
| US20190089760A1 (en) * | 2017-09-20 | 2019-03-21 | Junshan Zhang | Systems and methods for real-time content creation and sharing in a decentralized network |
| CN112035081A (en) * | 2020-09-01 | 2020-12-04 | 平安付科技服务有限公司 | Screen projection method and device, computer equipment and storage medium |
-
2020
- 2020-12-25 CN CN202011573439.5A patent/CN112653913A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20190037219A1 (en) * | 2017-07-28 | 2019-01-31 | Microsoft Technology Licensing, Llc | Effective intra encoding for screen data |
| US20190089760A1 (en) * | 2017-09-20 | 2019-03-21 | Junshan Zhang | Systems and methods for real-time content creation and sharing in a decentralized network |
| CN107770584A (en) * | 2017-11-01 | 2018-03-06 | 深圳市网旭科技有限公司 | A kind of method of the projection Android device screen picture to other platforms in real time |
| CN112035081A (en) * | 2020-09-01 | 2020-12-04 | 平安付科技服务有限公司 | Screen projection method and device, computer equipment and storage medium |
Non-Patent Citations (1)
| Title |
|---|
| MORRIS: "《CSDN,https://blog.csdn.net/Morris_/article/details/93465457》", 24 June 2019 * |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113179307A (en) * | 2021-04-23 | 2021-07-27 | 努比亚技术有限公司 | Audio time delay control method and device and computer readable storage medium |
| CN113179307B (en) * | 2021-04-23 | 2023-10-20 | 努比亚技术有限公司 | Audio delay control method, equipment and computer readable storage medium |
| CN115515001A (en) * | 2021-06-22 | 2022-12-23 | 荣耀终端有限公司 | Screen mirroring method, device, equipment and storage medium |
| CN115515001B (en) * | 2021-06-22 | 2023-10-24 | 荣耀终端有限公司 | Screen mirroring method, device, equipment and storage medium |
| CN114040242A (en) * | 2021-09-30 | 2022-02-11 | 荣耀终端有限公司 | Screen projection method and electronic equipment |
| CN114040242B (en) * | 2021-09-30 | 2023-07-07 | 荣耀终端有限公司 | Screen projection method, electronic equipment and storage medium |
| CN114157903A (en) * | 2021-12-02 | 2022-03-08 | Oppo广东移动通信有限公司 | Redirection method, redirection device, redirection equipment, storage medium and program product |
| CN114691440A (en) * | 2022-06-01 | 2022-07-01 | 广州市高奈特网络科技有限公司 | Data extraction method and device of equipment, computer equipment and storage medium |
| CN115495307A (en) * | 2022-08-26 | 2022-12-20 | 深圳市旭联信息技术有限公司 | Screen transmission data processing method and device, computer readable storage medium and system |
| WO2024114427A1 (en) * | 2022-11-29 | 2024-06-06 | 蔚来移动科技有限公司 | Screen projection method and apparatus based on interconnection between onboard terminal and mobile device, and medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112653913A (en) | Screen projection method and device, client device and readable storage medium | |
| US8443398B2 (en) | Architecture for delivery of video content responsive to remote interaction | |
| CN109085950B (en) | Multi-screen interaction method and device based on electronic whiteboard and electronic whiteboard | |
| CN110996160B (en) | Video processing method and device, electronic equipment and computer readable storage medium | |
| CN112055072A (en) | Cloud audio input method and device, cloud system, electronic equipment and storage medium | |
| CN102298491A (en) | Embedded graphical interface system and image generation method thereof | |
| CN113837939A (en) | Image processing method and device | |
| CN112422978A (en) | Intelligent transcoding method and device, electronic equipment and storage medium | |
| WO2021147556A1 (en) | Video data presentation method and apparatus, terminal device, and storage medium | |
| CN109788251B (en) | Video processing method, device and storage medium | |
| CN113395523B (en) | Image decoding method, device, equipment and storage medium based on parallel threads | |
| CN114938408A (en) | Data transmission method, system, equipment and medium of cloud mobile phone | |
| CN104883542A (en) | Image multi-screen display system | |
| CN110659080B (en) | Page display method and device, electronic equipment and storage medium | |
| CN108132767B (en) | Application window preview method and system | |
| CN112153412B (en) | Control method and device for switching video images, computer equipment and storage medium | |
| CN115599326A (en) | Information co-screen method, sending end, receiving end and computer readable storage medium | |
| CN103139757B (en) | A kind of data transmission system of application program for mobile terminal is unified method | |
| CN110798700B (en) | Video processing method, video processing device, storage medium and electronic equipment | |
| CN103974087A (en) | Video image file compressing system, client and method | |
| CN111355767B (en) | Remote control method, server, client and electronic equipment | |
| CN113961162A (en) | Screen projection control method, screen projection sending end, screen projection receiving end and screen projection system | |
| CN113066139A (en) | Picture processing method and device, storage medium and electronic equipment | |
| CN111475240A (en) | Data processing method and system | |
| CN113194277B (en) | Method, device and storage medium for multi-screen video of conference terminal |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210413 |