WO2024082947A1 - 多屏多设备交互方法、电子设备及*** - Google Patents
多屏多设备交互方法、电子设备及*** Download PDFInfo
- Publication number
- WO2024082947A1 WO2024082947A1 PCT/CN2023/122054 CN2023122054W WO2024082947A1 WO 2024082947 A1 WO2024082947 A1 WO 2024082947A1 CN 2023122054 W CN2023122054 W CN 2023122054W WO 2024082947 A1 WO2024082947 A1 WO 2024082947A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- screen
- target
- projection
- audio
- content
- Prior art date
Links
- 238000000034 method Methods 0.000 title claims abstract description 173
- 230000008569 process Effects 0.000 claims abstract description 76
- 230000005540 biological transmission Effects 0.000 claims description 56
- 238000009877 rendering Methods 0.000 claims description 35
- 238000007781 pre-processing Methods 0.000 claims description 27
- 238000003860 storage Methods 0.000 claims description 17
- 230000004044 response Effects 0.000 claims description 14
- 238000012545 processing Methods 0.000 description 77
- 238000007726 management method Methods 0.000 description 66
- 238000004891 communication Methods 0.000 description 58
- 238000013461 design Methods 0.000 description 33
- 238000010586 diagram Methods 0.000 description 23
- 230000006870 function Effects 0.000 description 12
- 238000004590 computer program Methods 0.000 description 10
- 230000015572 biosynthetic process Effects 0.000 description 9
- 238000003786 synthesis reaction Methods 0.000 description 9
- 230000001360 synchronised effect Effects 0.000 description 7
- 238000005266 casting Methods 0.000 description 6
- 230000003287 optical effect Effects 0.000 description 5
- 238000000926 separation method Methods 0.000 description 5
- 101100233916 Saccharomyces cerevisiae (strain ATCC 204508 / S288c) KAR5 gene Proteins 0.000 description 4
- 230000002452 interceptive effect Effects 0.000 description 4
- 229920001621 AMOLED Polymers 0.000 description 3
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 230000005236 sound signal Effects 0.000 description 3
- 230000010365 information processing Effects 0.000 description 2
- 239000004973 liquid crystal related substance Substances 0.000 description 2
- 230000007774 longterm Effects 0.000 description 2
- 239000002096 quantum dot Substances 0.000 description 2
- 230000011218 segmentation Effects 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 230000003190 augmentative effect Effects 0.000 description 1
- 230000010267 cellular communication Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000005520 cutting process Methods 0.000 description 1
- 230000001934 delay Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 235000012054 meals Nutrition 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000004148 unit process Methods 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/436—Interfacing a local distribution network, e.g. communicating with another STB or one or more peripheral devices inside the home
- H04N21/4363—Adapting the video stream to a specific local network, e.g. a Bluetooth® network
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0481—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
- G06F3/04817—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance using icons
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0484—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
- G06F3/0486—Drag-and-drop
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0487—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
- G06F3/0488—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures
- G06F3/04883—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures for inputting data by handwriting, e.g. gesture or text
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
- G06F3/1423—Digital output to display device ; Cooperation and interconnection of the display device with other functional units controlling a plurality of local displays, e.g. CRT and flat panel display
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/41—Structure of client; Structure of client peripherals
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/41—Structure of client; Structure of client peripherals
- H04N21/4104—Peripherals receiving signals from specially adapted client devices
- H04N21/4122—Peripherals receiving signals from specially adapted client devices additional display device, e.g. video projector
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/41—Structure of client; Structure of client peripherals
- H04N21/414—Specialised client platforms, e.g. receiver in car or embedded in a mobile appliance
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/41—Structure of client; Structure of client peripherals
- H04N21/414—Specialised client platforms, e.g. receiver in car or embedded in a mobile appliance
- H04N21/41422—Specialised client platforms, e.g. receiver in car or embedded in a mobile appliance located in transportation means, e.g. personal vehicle
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/436—Interfacing a local distribution network, e.g. communicating with another STB or one or more peripheral devices inside the home
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/436—Interfacing a local distribution network, e.g. communicating with another STB or one or more peripheral devices inside the home
- H04N21/4363—Adapting the video stream to a specific local network, e.g. a Bluetooth® network
- H04N21/43637—Adapting the video stream to a specific local network, e.g. a Bluetooth® network involving a wireless protocol, e.g. Bluetooth, RF or wireless LAN [IEEE 802.11]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/439—Processing of audio elementary streams
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
- H04N21/44012—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving rendering scenes according to scene graphs, e.g. MPEG-4 scene graphs
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
- H04N21/44016—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving splicing one content stream with another content stream, e.g. for substituting a video clip
Definitions
- the present application relates to the field of terminal technology, and in particular to a multi-screen multi-device interaction method, electronic device and system.
- the current screen projection scenarios have certain limitations. For example, initially, the user projects the video on the mobile phone on the TV for display. Later, when the user needs to project the document on the mobile phone on the laptop for office browsing, he needs to pause the screen projection on the TV first, and then project the document on the laptop by controlling the controls on the mobile phone. It can be seen that the current screen projection efficiency is low, and users need to perform complicated operations to switch the projection device.
- the present application provides a multi-screen multi-device interaction method, electronic device and system, which can improve the screen projection efficiency during the screen projection process.
- a multi-screen multi-device interaction method and interface are provided, which are applied to a first device or a component (such as a chip system) capable of implementing a function of the first device, and the method includes:
- the first interface including identifiers of a plurality of available screens on the second device
- the multiple available screens include the target screen
- projection contents and multiple target screens There are one or more projection contents and multiple target screens; or there are multiple projection contents and multiple target screens.
- a mobile phone displays an interface 30 (an example of a first interface), and the interface 30 includes logos of multiple available screens on a car (an example of a second device) (such as a dashboard screen, a central control screen, a co-pilot screen, etc.).
- the mobile phone After detecting that the user moves the virtual icon (or cursor) of the video application along the arrow direction to the area where the logo of the central control screen (an example of a target screen) is located, the mobile phone determines that the user wants to project the screen content of the video application on the mobile phone to the central control screen, and the mobile phone can send the projection image corresponding to the projection content to the central control screen, so that the central control screen displays the projection image corresponding to the video application.
- projection contents and multiple target screens There are one or more projection contents and multiple target screens; or there are multiple projection contents and multiple target screens.
- the first device can project the screen content of one or more applications to multiple target screens of one or more second devices.
- the first device projects the content of video application 1 (such as the played video) to screen 1 of the second device, and projects the content of game application 2 (such as the game interface, etc.) to screen 2 and screen 3.
- video application 1 such as the played video
- game application 2 such as the game interface, etc.
- the first device may project the content in one or more windows onto multiple target screens of the second device, for example, projecting video 1 played in window 1 onto screen 1 and screen 3 of the second device, and projecting the navigation interface displayed in window 2 onto screen 2.
- the first device may project the display content on the desktop onto multiple target screens of the second device.
- the user can operate through the first interface (an example of the multi-screen management interface 111 shown in Figure 2B) to control the first device to project the corresponding projection content to any one or more target screens of the second device.
- the user's audio-visual needs for displaying the projection content through any available screen in a multi-screen scenario can be met.
- the first device can be directly controlled to cast the content on device 2 and device 3 by operating the first interface, and the user does not need to pause the casting process on device 2 first and then The content is projected on device 3. It can be seen that the above technical solution has a high projection efficiency.
- the mobile phone desktop is projected to the co-pilot screen for display, and the co-pilot passenger sets driving information (ticket booking, meal ordering) on the co-pilot screen.
- the co-pilot passenger chooses to terminate the projection process of the second row left screen, the third row left screen, the central control screen, and the co-pilot screen on the multi-screen management interface of the co-pilot screen, and this projection process ends.
- the second device is one or more.
- the target screen is connected to multiple devices; the method further includes:
- Sending a projection image corresponding to the projection content to the target screen includes: sending the projection image to one or more target devices so that the one or more target devices process the projection image and send the processed projection image to the target screen.
- a mobile phone projects content to a three-row left screen (an example of a target screen) for display.
- the three-row left screen is connected to chips A and B.
- the mobile phone can determine chip A as the target device for processing the projected image from chips A and B, and send the projected image to chip A, so that chip A processes the projected image and sends the processed projected image to the three-row left screen, and the three-row left screen displays the projected content according to the projected image.
- the mobile phone can select the target processing component (such as chip A of the car computer) for processing the projected image according to the target screen selected by the user for displaying the projected content (such as the three-row left screen), so as to improve the efficiency of processing the projected image and thereby improve the efficiency of the entire projection operation.
- the target processing component such as chip A of the car computer
- determining one or more target devices from a plurality of devices includes:
- One or more target devices are determined from the multiple devices according to the computing power information of the multiple devices.
- the mobile phone can use chip A as the target device.
- using a device with better performance to process the projection image helps shorten the processing time and thus improve the efficiency of the projection process.
- chip A is connected to three rows of left screens and three rows of right screens
- chip B is only connected to three rows of left screens.
- the three rows of right screens are the same viewing screens as the three rows of left screens
- chip A consider using chip A to process the projection content corresponding to the three rows of left screens and the projection content corresponding to the three rows of right screens.
- the method further includes:
- the audio selection operation determining an audio module for playing audio corresponding to the projected image, and determining an audio transmission path between the first device and the audio module;
- the audio data corresponding to the projected image is sent to the audio module through the audio transmission path.
- the mobile phone displays card 305.
- the mobile phone can determine that the audio module for playing the audio corresponding to the projection image (i.e., the navigation audio) is the entire vehicle speaker, and that the audio transmission path between the mobile phone and the entire vehicle speaker is the vehicle computer-entire vehicle speaker. Afterwards, the mobile phone sends the audio data corresponding to the navigation to the entire vehicle speaker through the vehicle computer.
- the audio module for playing the audio corresponding to the projection image i.e., the navigation audio
- the mobile phone sends the audio data corresponding to the navigation to the entire vehicle speaker through the vehicle computer.
- the first device can determine the audio module for playing audio according to the user's audio selection operation, which can meet the user's audio listening needs and enhance the user's auditory experience.
- the target screen includes a first screen and a second screen; the projected content on the first screen corresponds to a first audio module and a first audio transmission path; the projected content on the second screen corresponds to a second audio module and a second audio transmission path.
- the mobile phone projects the content of the navigation application to the central control screen (an example of the first screen), and projects the video to the co-pilot screen (the second screen).
- the projected content on the central control screen corresponds to earphone 1 (first audio module) and the first audio transmission path, that is, the user listens to the audio of the navigation through the speakers of the entire vehicle.
- the projected content on the co-pilot screen corresponds to earphone 1 (second audio module) and the second audio transmission path, that is, the user listens to the audio corresponding to the video through earphone 1.
- the audio and video separation can be achieved during the screen projection process, so that the projection content on different screens has different audio modules and audio transmission paths.
- the projection content of the video application is displayed on the co-pilot screen of the car computer, and the audio corresponding to the projection content is played through earphone 1 (not the speaker fixed on the car computer) to meet the user's audio-visual needs.
- users can project the content they want to watch on a large-screen device (such as a smart screen) and listen to the projection content through mobile phone headphones. The audio should be played back to avoid disturbing other users.
- the target screen includes a first screen; the projected content includes the projected content of a first application and the projected content of a second application; the projected content of the first application on the first screen corresponds to a first audio module and a first audio transmission path; the projected content of the second application on the first screen corresponds to a second audio module and a second audio transmission path.
- the first device projects the content of the navigation application and the video to the co-pilot screen of the vehicle computer.
- the user can listen to the audio corresponding to the projected navigation interface through earphone 1 and listen to the audio of the video through the speakers throughout the vehicle.
- the projected audio of applications involving privacy can be listened to using personal headphones, and the projected audio of applications not involving privacy can be listened to using speakers throughout the car.
- the target screen includes a first screen and a second screen; the projected content includes the projected content of a first application and the projected content of a second application; the projected content of the first application on the first screen corresponds to a first audio module and a first audio transmission path; the projected content of the second application on the second screen corresponds to a second audio module and a second audio transmission path.
- the first device projects the content of the navigation application to the central control screen of the car computer and projects the video to the co-pilot screen of the car computer.
- the user can listen to the audio corresponding to the projected navigation interface through headphones 1 and listen to the audio of the video through the speakers throughout the car.
- the projection content of different applications on different screens has different audio modules and audio transmission paths, which can meet the user's listening needs for audio of different applications.
- the method before sending the projection image to the target screen, the method further includes: preprocessing the projection image;
- Sending the projection image to the target screen includes: sending the pre-processed projection image to the target screen.
- the first device can assist the selected second device with computing power.
- the first device can perform some processing operations on the projection content (such as video stream) in advance. In this way, the processing workload of the second device on the projection content can be reduced, the computing power of the second device can be saved, and the efficiency of the entire projection process can be improved.
- the projection image is preprocessed, including:
- the first condition includes: the load of the second device is insufficient to process the projected image.
- the first device can provide computing power assistance to the second device selected above to reduce the computing burden of the second device, thereby improving the efficiency of the projection process.
- the target screen includes a first screen and a second screen.
- Preprocess the projection image including:
- splicing and rendering the projection image of the first screen size and the projection image of the second screen size can reduce the rendering workload and thus improve the efficiency of the projection process.
- the target screen includes a first screen and a second screen.
- Preprocess the projection image including:
- Sending a projection image corresponding to the projection content to a target screen so that the target screen displays the projection image including: sending a rendered projection image to a first screen so that the first screen displays the projection image with a size of the first screen size; sending a rendered projection image to a second screen so that the second screen displays the projection image with a size of the second screen size after stretching or scaling.
- rendering according to the first screen size can reduce the rendering workload of the first device, thereby improving the efficiency of the projection process.
- the method further includes:
- the co-pilot screen and the left screen of the third row are the same screen.
- the mobile phone can adjust the image transmission path between the three-row left screen (an example of the second screen) and the audio transmission path between the audio module to achieve a better screen projection experience.
- the first interface further includes an identifier of a target application; and receiving an operation for instructing to project the screen projection content of the target application on the first device to the target screen includes:
- An indication that a user drags a virtual cursor from an indication of a target application to an indication of a target screen is received.
- the mobile phone when it is detected that the user drags the virtual cursor 304 from the logo of the navigation application to the logo of the central control screen, the mobile phone can send the corresponding projection image of the navigation application to the central control screen so as to project the projection content of the navigation application to the central control screen for display.
- the user can drag the icon of the application to be projected to the icon of any available screen through the first interface (an example of the multi-screen management interface 111 shown in Figure 2B) to control the mobile phone to project the content of the mobile phone application to the corresponding screen, thereby meeting the user's audio-visual needs to display the target content through any available screen in a multi-screen scenario.
- the first interface an example of the multi-screen management interface 111 shown in Figure 2B
- the method before displaying the first interface, the method further includes:
- Displaying the first interface includes: displaying the first interface in response to the first operation;
- Receiving an operation for instructing to project the projection content on the first device to the target screen includes: receiving an operation for selecting the target screen.
- the user can select video 1 as the projection object.
- the mobile phone can call out control 702.
- the mobile phone can display interface 10 (an example of the first interface) shown in (b) of Figure 25.
- an operation for selecting three rows of right screens and three rows of left screens is detected, and the mobile phone can send a projection image of video 1 to the three rows of left screens and three rows of right screens, so that the three rows of left screens and three rows of right screens display video 1.
- preprocessing includes decoding and rendering.
- the target screen includes a projected image and operation controls.
- the first interface can be displayed as a negative screen, or by sliding the desktop to cut into the next screen, or by other designs.
- a multi-screen multi-device interaction apparatus and interface are provided, which are applied to a first device or a component capable of implementing the first device, and the apparatus includes:
- a display module configured to display a first interface, wherein the first interface includes identifiers of multiple available screens on the second device;
- An input module configured to receive an operation for instructing to project the screen projection content on the first device to a target screen; the multiple available screens include the target screen;
- a communication module is used to send a projection image corresponding to the projection content to the target screen, so that the target screen displays the projection image; the projection image is used for the target screen to display the projection content;
- projection contents and multiple target screens There are one or more projection contents and multiple target screens; or there are multiple projection contents and multiple target screens.
- the second device is one or more.
- the target screen is connected to a plurality of devices; the apparatus further includes a processing module;
- a processing module configured to determine one or more target devices from a plurality of devices, the plurality of devices including the one or more target devices;
- Sending a projection image corresponding to the projection content to the target screen includes: sending the projection image to one or more target devices so that the one or more target devices process the projection image and send the processed projection image to the target screen.
- determining one or more target devices from a plurality of devices includes:
- One or more target devices are determined from the multiple devices according to the computing power information of the multiple devices.
- the input module is further used to receive an audio selection operation input by the user on the first interface
- the processing module is further used to determine, according to the audio selection operation, an audio module for playing the audio corresponding to the projected image, and to determine an audio transmission path between the first device and the audio module;
- the communication module is also used to send audio data corresponding to the projection image to the audio module through the audio transmission path.
- the target screen includes a first screen and a second screen; the projection content on the first screen corresponds to the projection content on the first screen. Audio module and a first audio transmission path; the projection content on the second screen corresponds to the second audio module and the second audio transmission path.
- the target screen includes a first screen; the projected content includes the projected content of a first application and the projected content of a second application; the projected content of the first application on the first screen corresponds to a first audio module and a first audio transmission path; the projected content of the second application on the first screen corresponds to a second audio module and a second audio transmission path.
- the target screen includes a first screen and a second screen; the projected content includes the projected content of a first application and the projected content of a second application; the projected content of the first application on the first screen corresponds to a first audio module and a first audio transmission path; the projected content of the second application on the second screen corresponds to a second audio module and a second audio transmission path.
- the processing module is further used to pre-process the projection image before sending the projection image to the target screen;
- Sending the projection image to the target screen includes: sending the pre-processed projection image to the target screen.
- the projection image is preprocessed, including:
- the first condition includes: the load of the second device is insufficient to process the projected image.
- the target screen includes a first screen and a second screen
- Preprocess the projection image including:
- the target screen includes a first screen and a second screen.
- Preprocess the projection image including:
- Sending a projection image corresponding to the projection content to a target screen so that the target screen displays the projection image including: sending a rendered projection image to a first screen so that the first screen displays the projection image with a size of the first screen size; sending a rendered projection image to a second screen so that the second screen displays the projection image with a size of the second screen size after stretching or scaling.
- the communication module is further used to receive a notification message of stopping screen projection from the first screen;
- the processing module is also used to adjust the image transmission path between the second screen and the audio transmission path between the audio module.
- the first interface further includes an identifier of the target application
- Receiving an operation for instructing to project screen content of a target application on a first device to a target screen including:
- An indication that a user drags a virtual cursor from an indication of a target application to an indication of a target screen is received.
- the input module is further used to receive an operation of selecting screen projection content from a user before displaying the first interface; receive a first operation input by the user;
- Displaying the first interface includes: displaying the first interface in response to the first operation;
- Receiving an operation for instructing to project the projection content on the first device to the target screen includes: receiving an operation for selecting the target screen.
- preprocessing includes decoding and rendering.
- an embodiment of the present application provides a computer program product.
- the computer program product When the computer program product is run on an electronic device, the electronic device executes a method of any aspect or any implementation method of any aspect.
- an embodiment of the present application provides a circuit system, the circuit system includes a processing circuit, and the processing circuit is configured to execute a method of any aspect or any implementation method of any aspect.
- an embodiment of the present application provides a chip system, comprising at least one processor and at least one interface circuit, wherein the at least one interface circuit is used to perform transceiver functions and send instructions to at least one processor, and when the at least one processor executes the instructions, the at least one processor executes a method of any aspect or any implementation method of any aspect.
- a computer-readable storage medium stores a computer program (also referred to as an instruction or code), and when the computer program is executed by an electronic device, the electronic device executes a method of any aspect or any implementation of any aspect.
- a computer program also referred to as an instruction or code
- a multi-screen multi-device interaction method, electronic device, and system are provided.
- the system includes a first device and a second device in any one of the above aspects or any implementation of any one of the aspects.
- FIG1A is a schematic diagram of a system architecture provided in an embodiment of the present application.
- FIG1B is a schematic diagram of a screen on a car provided in an embodiment of the present application.
- FIG2A is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application.
- FIG2B is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application.
- 3 to 5 are schematic diagrams of a process of establishing a connection between a mobile phone and a car provided in an embodiment of the present application;
- FIG6 is a schematic diagram of an interface provided in an embodiment of the present application.
- FIG7 is a schematic diagram of a flow chart of a multi-screen multi-device interaction method provided in an embodiment of the present application.
- FIG8 is a schematic diagram of a scenario provided in an embodiment of the present application.
- FIGS 9 and 10 are schematic diagrams of interfaces provided in embodiments of the present application.
- FIGS. 11A-11B are schematic diagrams of interfaces provided in embodiments of the present application.
- FIGS. 12 and 13 are schematic flow diagrams of a multi-screen multi-device interaction method provided in an embodiment of the present application.
- FIG19 is a schematic diagram of an interface provided in an embodiment of the present application.
- FIG20 is a schematic diagram of a flow chart of a multi-screen multi-device interaction method provided in an embodiment of the present application.
- FIG21 is a schematic diagram of an interface provided in an embodiment of the present application.
- FIG22 is a schematic diagram of a scenario provided in an embodiment of the present application.
- FIG23 is a schematic diagram of an interface provided in an embodiment of the present application.
- FIG24 is a schematic diagram of a scenario provided in an embodiment of the present application.
- FIG25 is a schematic diagram of an interface provided in an embodiment of the present application.
- FIG26 is a schematic diagram of the structure of a device provided in an embodiment of the present application.
- FIG. 27 is a schematic diagram of the structure of the chip system provided in an embodiment of the present application.
- first and second and the like in the specification and drawings of this application are used to distinguish different objects, or to distinguish different processing of the same object, rather than to describe a specific order of objects.
- a and/or B can mean: A exists alone, A and B exist at the same time, and B exists alone.
- the technical solution of the embodiment of this application is applicable to distributed systems across multiple devices and screens, such as in-vehicle, whole-house, and office scenarios.
- in-vehicle scenario is mainly used as an example for explanation, but this does not constitute a limitation on the applicable scenarios.
- FIG1A is a schematic diagram of a distributed system architecture of a vehicle-mounted scenario provided in an embodiment of the present application.
- the vehicle-mounted system includes a vehicle computer, which can be equipped with multiple screens.
- the center console of the car is equipped with a central control display or a central control screen.
- some cars have gradually digitized displays in the cockpit, and multiple display screens are provided in the cockpit to display digital instrument panels, in-car entertainment systems and other contents.
- multiple display screens are provided in the cockpit, such as a central control screen 11 on the center console and an instrument screen 12 on the instrument panel.
- display screens may also be included in the cockpit, such as a display screen in front of the passenger in the co-pilot seat (also called the front passenger) (which may be referred to as the co-pilot screen 13 for short), a display screen in front of the left rear passenger, and a display screen in front of the right rear passenger, etc.
- the central control screen is generally used to display the display content corresponding to the audio and video entertainment system.
- the central control screen can be used to display the application programs installed in the vehicle computer, and the user can use the application programs by operating on the central control screen.
- the machine detects that the user clicks on the map icon 1011 displayed on the central control screen, starts the map application, and displays the map interface on the central control screen to provide navigation services for the user.
- the instrument screen is generally used to display the display content corresponding to the digital instrument panel.
- the instrument screen can be used to display vehicle information such as vehicle speed 1021, cruising range 1022, and vehicle icon 1023. The user can quickly learn the vehicle status through the display content on the instrument screen.
- the system shown in Figure 1A may also include other devices.
- devices such as mobile phones and tablets with screens.
- a cross-device multi-screen management module can be configured in the mobile phone, and through the cross-device multi-screen management module, the mobile phone can display a multi-screen management interface (also referred to as a virtual cross-multi-screen interface). Users can manage multiple screens of the mobile phone and the vehicle-mounted system through the multi-screen management interface (an example of the first interface), so that the mobile phone and the vehicle-mounted system can share the screen to meet the user's audio-visual needs.
- the user can enter the corresponding operation in the multi-screen management interface of the mobile phone to control the mobile phone to project the video being played to the display screen in front of the left rear passenger and the co-pilot screen to meet the video viewing needs of the co-pilot passenger and the left rear passenger.
- FIG2A shows a schematic diagram of the structure of the electronic device 100.
- the electronic device 100 includes at least one processor 201 , a memory 202 and at least one transceiver 203 .
- Processor 201 can be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the present application.
- CPU central processing unit
- ASIC application-specific integrated circuit
- the above components may include communication lines for transmitting information between the above components.
- the transceiver 203 is used to communicate with other devices.
- the transceiver can be a module, a circuit, a bus, an interface or other device capable of realizing a communication function, and is used to communicate with other devices.
- the transceiver can be an independently arranged transmitter, which can be used to send information to other devices, and the transceiver can also be an independently arranged receiver, which is used to receive information from other devices.
- the transceiver can also be a component that integrates the functions of sending and receiving information, and the embodiment of the present application does not limit the specific implementation of the transceiver.
- the memory 202 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto.
- the memory may be independent and connected to the processor through a communication line.
- the memory may also be integrated with the processor.
- the memory 202 is used to store computer-executable instructions for implementing the solution of the present application, and the execution is controlled by the processor 201.
- the processor 201 is used to execute the computer-executable instructions stored in the memory 202, thereby implementing the various methods provided in the following embodiments of the present application.
- the computer-executable instructions in the embodiments of the present application may also be referred to as application code, instructions, computer program or other names, which are not specifically limited in the embodiments of the present application.
- the processor 201 may include one or more CPUs, such as CPU0 and CPU1.
- the electronic device 100 may include multiple processors. Each of these processors may be a single-CPU processor or a multi-CPU processor.
- the processor here may refer to one or more devices, circuits, and/or processing cores for processing data (e.g., computer program instructions).
- the electronic device 100 further includes a display screen 204 and an audio module 205 .
- Display screen 204 is used to display images, videos, etc.
- the display screen 204 includes a display panel.
- the display panel can be a liquid crystal display (LCD), for example, an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Mini-LED, Micro-LED, Micro-oled, a quantum dot light-emitting diode (quantum dot light-emitting diode), or a micro-OLED.
- the electronic device 100 may include 1 or N display screens 204, where N is a positive integer greater than 1.
- the display screen 204 includes a center console display screen (ie, a center control screen) and an instrument panel display screen (ie, an instrument screen).
- the audio module 205 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signal.
- the audio module 205 can also be used to encode and decode audio signals.
- the audio module 205 can be arranged in the processor 201, or some functional modules of the audio module 205 can be arranged in the processor 201.
- the electronic device 100 can implement audio functions through the audio module 205, etc. For example, music playing, recording, etc.
- the audio module 205 can include a speaker, a receiver, a microphone, a headphone interface, and an application processor, etc. to implement audio functions.
- the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100.
- the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently.
- the components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.
- the vehicle computer can establish a wired connection with the mobile terminal through a USB interface.
- the vehicle computer can establish a wireless connection with the mobile terminal through a wireless communication module, and the wireless connection method used can be, for example, Bluetooth (including low-power Bluetooth, classic Bluetooth, etc.), WLAN, NFC, IR, etc.
- the wireless connection method used can be, for example, Bluetooth (including low-power Bluetooth, classic Bluetooth, etc.), WLAN, NFC, IR, etc.
- the embodiment of the present application does not limit the wireless connection method.
- FIG. 2B another structural example of an electronic device in a distributed system is shown.
- the first device may also be referred to as a source device or a master device for casting
- the second device may also be referred to as a target device
- the first screen, the second screen, and the second subsidiary screen of the second device may be referred to as target screens.
- the first device can project the content of different applications onto different devices or different screens of the same device.
- the first device projects the content of application 1 onto screen 1 of device 1, and projects the content of application 2 onto screen 2 of device 2.
- the first device projects the content of application 1 onto screen 1 of device 1, and projects the content of application 2 onto screen 2 of device 1.
- the first device can project the content of different applications onto the same screen of a certain device.
- the first device projects the content of application 1 onto screen 1 of device 1, and projects the content of application 2 onto screen 1 of device 1.
- Device 1 can display the content of application 1 and the content of application 2 in small windows.
- the first device may include the following functional modules:
- the multi-screen management interface 111 can be presented on the third screen of the first device.
- the multi-screen management interface can present multiple screens on the second device.
- the user can use the multi-screen management interface 111 to project the application or desktop on the first device to the screen of the second device.
- the mobile phone screen can display the multi-screen management interface 111 or the multi-screen management interface 101.
- the multi-screen management interface 111 can be managed at the granularity of the screen, and the multi-screen management interface 101 can be managed at the granularity of the application.
- the multi-screen management interface 111 can be an interface 20 (an example of the first interface) as shown in (b) of Figure 5.
- the interface 20 may include icons of one or more applications, such as icons of applications such as video, settings, and a recorder.
- the interface 20 may also include identification of one or more available screens, such as icons of virtual available screens such as the instrument screen, central control screen, and co-pilot screen of a car (an example of the second device).
- the user can drag the application that he wants to project to the corresponding virtual screen.
- the mobile phone can generate a virtual icon 302' (or virtual cursor) corresponding to the video application, and the user can drag the virtual icon 302' to the area where the icon 303 of the central control screen is located.
- the mobile phone can project the content of the video application to the central control screen of the car.
- the user can project the multi-screen management interface 111 onto the screen of the second device, and the user can interact with the multi-screen management interface 111 through the second device, so that the user can perform the screen projection operation conveniently and quickly.
- the first application on the first device can run in the background.
- the first device is equivalent to a master control device, which displays the interface of the first application on the second device, and the first device can run the first application in the background.
- the multi-window management module 120 is used to manage multiple windows across multiple screens of the device.
- the first device can control the second device to display the interface of the second application and run the second application in the background of the first device.
- the interface of a navigation application is displayed in a small window on the central control screen.
- the driver switches the navigation application interface in the window from the central control screen to the co-pilot screen.
- the co-pilot screen can display the navigation application interface in the window, and the passenger in front of the co-pilot screen sets the navigation address.
- the passenger in front of the co-pilot screen switches the interface of the navigation application back to the central control screen. In this way, the driver can view the navigation interface through the central control screen.
- the multi-screen management module 130 is used to manage multiple screens across devices and is also used to select a target device from multiple devices connected to the target screen and determine a computing power scheduling strategy for the target device.
- the multi-screen management module 130 of the mobile phone will select device 1 or device 2 to process the projected content (video stream) according to the selection strategy. Assuming that device 1 processes the video stream, device 1 can send the processing result of the video stream to display screen 1, and display screen 1 plays the video of application 1.
- the strategy of the source device (such as a mobile phone) selecting the target device (such as device 1) will be described in detail below.
- the mobile phone can also assist the computing power of the above-selected device 1.
- the mobile phone can perform some processing operations on the projection content (such as video stream) in advance. In this way, the processing workload of device 1 on the projection content can be reduced, the computing power of device 1 can be saved, and the efficiency of the entire projection process can be improved.
- the multi-screen management module 130 can also be used to determine the audio module used to play the corresponding audio of the projected content (such as the sound of the video), and the communication path between the source device and the audio module (which can be simply referred to as the audio transmission path).
- the multi-screen management module 130 can confirm whether the audio transmission path selected by the audio smart selection module 152 is normal through the communication module 160, and determine the audio transmission path.
- the multi-screen management module 130 may determine that the audio and video share the same path, that is, the projected video and audio are transmitted through the same communication path.
- the audio of the screen (such as the left screen of the third row of the car computer) can come from the target screen device (such as chip A connected to the co-pilot screen of the car computer), or directly from the source device (such as a mobile phone).
- the target screen device such as chip A connected to the co-pilot screen of the car computer
- the source device such as a mobile phone
- Co-viewing screen refers to multiple screens with the same projected content that do not require additional processing.
- the co-pilot screen and the three-row left screen both display the projected video 1, so the co-pilot screen and the three-row left screen are co-viewing screens.
- the video 1 displayed on the co-pilot screen and the three-row left screen can be displayed in full screen or in a window.
- the source device can separate the audio and video (audio, screen projection video) and no longer send audio data to the target device, but directly send the audio data to the target headset so that the target headset can play the audio.
- the source device sends the audio data to the target headset through an intermediate device (such as a third device).
- the multi-screen management module 130 may also be used to determine a communication path (referred to as a video path) between a source device and a target screen.
- a communication path referred to as a video path
- the source device confirms the target video path between the source device and the target device through the communication module, such as USB, HDMI, WiFi, Sparklink, vehicle Ethernet, video cable, whole house bus, etc.
- the source device sends the video stream to the target device through the target video path, and then the target device sends the video stream to the target screen.
- the source device can intelligently determine the audio transmission path and video path for the screen projection, which helps to ensure the screen projection effect.
- the device account authentication module 140 is used to authenticate and authorize other devices. Devices that pass the authentication can establish a connection with the first device.
- the cross-device screen projection module 150 is used to control the screen projection process.
- the pre-decoding and pre-rendering module 151 is used to pre-process the projection content.
- the pre-processing may include but is not limited to: pre-decoding and pre-rendering, so as to reduce the computing power burden of the second device, reduce the time required for projection, and improve the projection efficiency.
- Decoding may be a process of pulling a compressed media stream from a server and converting the compressed media stream into a playable format.
- the audio smart selection module 152 is used to select the audio module for playing the screen projection audio and the corresponding audio transmission path.
- the communication module 160 is used to communicate with other devices.
- the second device includes the following modules: a second screen, a first screen, a second subsidiary screen, a multi-window management module 120, a multi-screen management module 130, a device account authentication module 140, an audio processing module 153, a cross-device screen projection quick display module 154, a communication module 160 and a designated screen information processing module 155.
- the first screen and the second screen may be screens equipped by the second device itself, and the second auxiliary screen may be an external screen of the second device.
- the second auxiliary screen may be an external tablet on the rear of the car computer.
- modules such as the second screen, the first screen, the second subsidiary screen, the multi-window management module 120, the multi-screen management module 130, the device account authentication module 140, and the communication module 160 can be found in the introduction of the first device related modules.
- the designated screen information processing module 155 is used to process the projection screen (such as a video frame).
- the processing includes but is not limited to segmentation and super-resolution processing.
- segmentation and super-resolution please refer to the following embodiments.
- the cross-device screen projection quick display module 154 is used to further process the pre-processing result of the projection screen from the first device when the first device has pre-processed the projection screen. For example, the first device pre-decodes and pre-renders the projection screen, and the cross-device screen projection quick display module 154 of the second device synthesizes the projection screen. Afterwards, the second device can send the synthesis result of the projection screen to, for example, the second screen, which plays the projection screen.
- FIG2B illustrates a distributed system including a first device and a second device as an example.
- the technical solution of the embodiment of the present application can also be applied to a distributed system including more devices, more screens and other components.
- the second device can include more screens
- the distributed system can also include a third device and a screen and an audio module of the third device, and for another example, the distributed system includes more screens and audio modules of more devices.
- the multiple devices in the embodiments of the present application may be multiple terminal devices.
- the first device may be a mobile phone, and the second device may be a car computer.
- the first device is a tablet computer, and the second device may be a dual-screen personal computer (PC) or a dual-screen conference room display device.
- the first device may be a mobile phone, and the second device may be multiple smart large screens throughout the house.
- the first device may be the central control module of the whole house, and the second device may include a first screen controlled by a server throughout the house, etc.
- the multiple devices may include cloud-side devices and end-side devices.
- the first device may be the master control module of a cloud server, and the second device may be a car computer.
- the server may provide a multi-screen management interface for managing multiple screens across devices in the system.
- the first device may be a car computer, which may be installed with a car-home interactive interface
- the second device may include a first screen, a second screen, and a second subsidiary screen controlled by a whole-house server. Users can control the first screen, the second screen, and the like in their home through the car-home interactive interface installed on the car computer.
- the first device may be a PC with a cloud-based car control application installed on it, and the second device may be a car computer.
- the whole-house server may provide a multi-screen management interface for controlling multiple screens throughout the house. Multiple devices may also be other combinations of devices to achieve cross-device interaction between end-cloud and end-cloud-end.
- the first device and the second device may also be different logical modules in the same device, which is equipped with multiple screens.
- one of the multiple devices is a Dongle (screenless decoding device), for example, the first device is a Dongle, and the second device is a car computer or a large screen; or the first device is a mobile phone or a remote control device, the second device is a Dongle, and the third device is a multi-screen car computer, etc.
- a Dongle screenless decoding device
- the first device is a Dongle
- the second device is a car computer or a large screen
- the first device is a mobile phone or a remote control device
- the second device is a Dongle
- the third device is a multi-screen car computer, etc.
- the embodiments of the present application do not limit the device forms of the multiple devices.
- the device forms and structures of the multiple devices may be different, or the multiple devices may have the same form.
- the modules in the above-mentioned first device and second device can also have other deployment methods.
- the first device of the embodiment of the present application can also control the third device to project the screen to the second device.
- the multi-screen management interface is deployed in the first device, and other modules such as the multi-window management module 120, the multi-screen management module 130, the device account authentication module 140, the cross-device screen projection module 150, the audio smart selection module 152, the pre-decoding and pre-rendering module 151 and the communication module 160 are deployed in the third device.
- the mobile terminal is described by taking a mobile phone as an example.
- the vehicle computer and the mobile phone can establish a communication connection through a first application (such as a HiCar application).
- a first application such as a HiCar application
- the main menu 701 is displayed on the vehicle computer.
- the Hicar application icon 702 and other application icons are displayed in the main menu.
- the vehicle computer can send a connection request.
- the mobile phone receives the connection request sent by the vehicle computer, it can display an interface 704 as shown in (2) of FIG3 .
- a prompt box 703 is displayed in the interface 704 to prompt the user to choose whether to connect to the vehicle computer.
- the mobile phone replies to the vehicle computer with a response agreeing to the connection.
- the vehicle computer After receiving the response agreeing to the connection sent by the mobile phone, the vehicle computer can display a connection code as shown in (3) of FIG3 .
- the mobile phone displays an interface 706 as shown in (4) of FIG3 , prompting the user to enter the connection code displayed by the vehicle computer.
- the vehicle computer After the user enters the connection code, the mobile phone sends the connection code entered by the user to the vehicle computer, and the vehicle computer verifies it. After the verification is successful, the communication is established.
- the vehicle computer may display an interface 801 as shown in FIG. 4 (1), and the mobile phone may display an interface 802 as shown in FIG. 4 (2), respectively to prompt the user that a connection is being established.
- the mobile phone may display a prompt interface 804 as shown in FIG. 4 (3), prompting that the connection has been established to the vehicle computer, and then the user may close it manually, or the mobile phone may automatically close the prompt interface 804 after a preset time period.
- the car computer and the mobile phone can also be paired directly through Bluetooth without the user entering a connection code.
- the user can open the Hicar application on the mobile phone side and choose to connect to the car computer that has been connected before, so the mobile phone and the car computer can directly establish a communication connection.
- the mobile phone and the vehicle computer can automatically establish a communication connection without user operation.
- the mobile phone can obtain information about multiple display screens available in the vehicle from the vehicle computer, and display a multi-screen management interface based on the information of the multiple display screens. Through the multi-screen management interface, the user can project the display content on the mobile phone to multiple display screens of one or more target devices in the vehicle to achieve screen sharing across devices.
- the technical solution of the embodiment of the present application is introduced in multiple scenarios as follows:
- Scenario 1 Single target device scenario
- the mobile phone may display an interface 10, which may include multiple tags (e.g., tag 102, tag 103, tag 103).
- tags e.g., tag 102, tag 103, tag 103.
- the mobile phone may display a corresponding interface.
- the mobile phone in response to detecting that the user clicks on the desktop tag 103, the mobile phone may display icons of one or more applications on the desktop (e.g., icons of video, settings, and recorder applications).
- the interface 10 may also display a switching control 105. The user may slide left or right to cause the mobile phone to display more application icons on other interfaces.
- the mobile phone in response to detecting that the user clicks the "All" label 102, can obtain information about multiple screens available on the vehicle from the vehicle computer and display an interface 20 (an example of the multi-screen management interface 111 shown in FIG2B).
- the interface 20 may include icons of multiple screens available on the vehicle (such as icons of multiple screens such as the instrument screen and the central control screen) and icons of one or more applications on the mobile phone.
- the icons of multiple screens available on the vehicle may be displayed in area 201 of the interface 20.
- the mobile phone may display icons of multiple available screens in the vehicle.
- the user in order to more clearly view the icons of multiple screens on the vehicle on the mobile phone interface, as shown in FIG5(c), the user can place the mobile phone in landscape orientation.
- the mobile phone posture such as adjusting the mobile phone from portrait display to landscape display
- the mobile phone can adaptively adjust the display content of the interface. For example, as shown in FIG5(c), the mobile phone can display interface 30, and display icons of multiple available screens on the vehicle in area 301 of interface 30.
- the user can drag the icon 302 of the video application in the direction of the central control screen icon 303.
- the icon 302 of the video application does not move with the change of the dragging position.
- the mobile phone can generate and display a virtual icon 302' (or a virtual cursor) of the video application, and the virtual icon 302' can move with the user's drag.
- the user can lift the finger off the screen, and the mobile phone will no longer display the virtual icon 302'.
- the mobile phone After the mobile phone detects that the user moves the virtual icon 302' to the area where the central control screen icon 303 is located and releases it, it is determined that the user wants to cast the content of the video application on the central control screen of the vehicle, and the mobile phone can cast the video application of the mobile phone to the central control screen of the vehicle according to the user's casting intention.
- the mobile phone can display the logo of the video application on the icon 303 of the central control screen, such as displaying the text "video application of the mobile phone" on the icon 303 of the central control screen.
- the mobile phone may display “APP Video from Jane’s phone” (video application on Jane’s phone) on the central control screen icon 303 .
- a corner mark such as “APP sync-Video from Jane’s phone” may be displayed on the central control screen icon 303.
- Sync indicates that the central control screen is a co-viewing screen.
- the user can determine the content of the mobile phone's video application being projected and displayed on the vehicle's central control screen based on the video application identifier on the central control screen icon 303 .
- the mobile phone in response to the user dragging the virtual icon 302' of the video application to the area where the central control screen icon 303 is located, as shown in (d) of Figure 5, can also display a corner mark on the icon 302 of the video application, and the corner mark is used to indicate that the content of the video application is being projected and displayed on the screen of another device.
- the corner mark can also be in other forms, and the embodiments of the present application do not limit the specific presentation method of the corner mark.
- the corner mark can be the text "central control screen" to indicate that the content of the video application is being projected and displayed on the central control screen of the vehicle.
- FIG. 5 is illustrated by taking the video application logo displayed on the central control screen icon 303 in text form as an example.
- the application logo can also be displayed on the central control screen icon 303 in other forms, such as in the form of an application icon, etc.
- the embodiment of the present application does not limit the display method of the application logo in the interface.
- FIG6 shows the display interface of the central control screen. It can be understood that after the vehicle computer receives the projection content corresponding to the video application from the mobile phone, the projection content can be displayed. For example, the vehicle computer receives the code stream of the video to be played from the mobile phone, and plays the video 50 shown in FIG6 according to the code stream of the video.
- the logo of the application is no longer displayed in the corresponding screen icon.
- the central control screen icon 303 shown in (d) of FIG. 5 no longer displays "mobile phone video application”.
- the user can drag the icon of the application to be projected to the icon of any available screen through the multi-screen management interface 30 (an example of the multi-screen management interface 111 shown in Figure 2B) to control the mobile phone to project the content of the mobile phone application to the corresponding screen, thereby meeting the user's audio-visual needs to display the target content through any available screen in a multi-screen scenario.
- the multi-screen management interface 30 an example of the multi-screen management interface 111 shown in Figure 2B
- the above mainly takes the example of the user dragging a virtual icon, taking the example of dragging the icon of the corresponding application (such as a video application) to the corresponding screen (such as the central control screen).
- the user can also instruct the mobile phone to project the content of the mobile application to the screen of other devices through other operations.
- the cursor is displayed, and the user can drag the cursor from the icon position of application 1 (such as a video application) to the icon position of the car's central control screen.
- the user can drag the cursor in an area outside the application icon, or drag the cursor in a specified area of the interface 20.
- FIG7 shows an example of a process of a cross-device device control method according to an embodiment of the present application. As shown in FIG7 , the method includes the following steps:
- the mobile phone displays a multi-screen management interface.
- the mobile phone can display a multi-screen management interface 30 as shown in (a) of Figure 9, where the interface 30 includes icons of one or more applications (such as icons of applications such as video, settings, and recorder), and icons of multiple available screens on the vehicle (such as icons of screens such as the instrument screen and the central control screen).
- applications such as icons of applications such as video, settings, and recorder
- icons of multiple available screens on the vehicle such as icons of screens such as the instrument screen and the central control screen.
- the mobile phone receives a screen projection instruction input by the user.
- the user can use the above-mentioned multi-screen management interface to drag the icon of the application to be projected to the icon of the target screen to control the mobile phone to perform the projection operation.
- the mobile phone displays the multi-screen management interface 30.
- the mobile phone can generate a virtual cursor 304.
- the user's finger can slide along the trajectory shown by the dotted line, and the virtual cursor 304 follows the user's finger to move in the direction of the central control screen icon 303.
- the virtual cursor 304 is moved to the area where the central control screen icon 303 is located (screen projection instruction 1), the user's finger can be lifted off the screen, and the mobile phone no longer displays the virtual cursor 304.
- the mobile phone After detecting that the user moves the virtual cursor 304 to the area where the central control screen icon 303 is located and releases it, as shown in FIG9(b), the mobile phone can display the name of the navigation application on the central control screen icon 303.
- the mobile phone can also display a corner mark on the icon of the navigation application to indicate that the content of the navigation application is being projected and displayed on other devices.
- the mobile phone can generate a virtual cursor, and the user can drag the virtual cursor so that the virtual cursor is moved from the area where the video application icon 302 is located to the area where the co-pilot screen icon is located (screen projection instruction 2).
- the mobile phone can display the name of the video application on the icon on the co-pilot screen.
- the mobile phone can also display a corner mark on the video application icon 302 to indicate that the content of the video application is being projected and displayed on other devices.
- entertainment applications such as videos
- AR augmented reality
- HUD head-up display
- S103 The mobile phone receives an audio selection instruction input by the user.
- the user can long press the icon 303 on the central control screen to trigger the mobile phone to pop up a card 305.
- the card 305 may include an audio module that can be used by the central control screen. For example, it includes the car head-whole car speaker, car head Bluetooth-headset 1, car head Bluetooth-headset 2.
- the mobile phone determines that the user wants to select all car speakers to play the audio of the navigation application.
- the mobile phone can display the "All Car Speakers" logo in the icon 303 of the central control screen, indicating that when the navigation application screen is displayed on the central control screen, the audio matched by the navigation application is played by all car speakers.
- the user can also select an audio device connected to the vehicle to play audio.
- the user can choose to play the audio of a video application through earphone 1 such as a mobile phone.
- earphone 1 such as a mobile phone.
- the mobile phone can display the logo of "mobile phone earphone 1" in the icon on the co-pilot screen, which is used to indicate that when the screen of the video application is displayed on the co-pilot screen, the earphone 1 connected to the mobile phone plays the audio matching the video screen.
- the mobile phone can cooperate with other terminals in the car to determine whether the audio module selected by the user for the screen projection content is valid. For example, assuming that the audio module selected by the user for the video application includes the car Bluetooth-mobile phone headset 1, the mobile phone can communicate with the car to determine whether the audio module related to the car Bluetooth-mobile phone headset 1 can work normally. After judgment, if the car has a Bluetooth channel, the Bluetooth module of the car can work normally, and the mobile phone also has a Bluetooth channel, and the Bluetooth module of the mobile phone can also work normally, then the mobile phone can preliminarily confirm that the audio module selected by the user for playing the audio of the video application is valid.
- the mobile phone can prompt the user to reselect the audio module used for this screen projection.
- the mobile phone can communicate with the car computer to determine whether the relevant audio module can work normally. After judgment, the path between the Bluetooth module of the car computer and the Bluetooth module of the headset 2 is faulty, and the car computer cannot transmit the audio data to the headset 2 for playback through the Bluetooth module, that is, the audio module selected by the user is invalid.
- the mobile phone can pop up a prompt window 306 to prompt the user to reselect the audio module.
- the mobile phone before the mobile phone pops up the card 305, it can determine in advance whether each audio module is available. Afterwards, the mobile phone can display only the available audio modules in the pop-up card 305 to reduce the probability of the user reselecting the audio module. For example, after detection, the mobile phone determines that the path between the car Bluetooth module and the headset 2 is faulty, then the audio option "Car Bluetooth-Headphone 2" may not be displayed in the card 305 shown in (a) of Figure 10 to prevent the user from mistakenly selecting the faulty audio module.
- the mobile phone can display the audio option "Car Bluetooth-Headphone 2" in the card 305, but the audio option is set to an unselectable or disabled state, and the user cannot select the audio module corresponding to the audio option.
- S104 The mobile phone determines whether the conditions for preprocessing the projection content are met. If the preprocessing conditions are met, the following S105 is executed; if the preprocessing conditions are not met, S106-S113 are executed.
- the preprocessing conditions include any one or more of the following conditions: the load of the target device is greater than or equal to a certain threshold, the load of the source device is less than or equal to a certain threshold, and the communication parameters between the target device and the source device meet the conditions.
- the communication parameters between the target device and the source device include but are not limited to bandwidth, bit error rate, etc.
- the communication parameters meet the conditions, which may mean that the bandwidth is higher than a certain threshold (for example, above 500Mbps), and/or the bit error rate is lower than a certain threshold.
- the source device refers to the device from which the projection content comes
- the target device refers to the device that receives and displays the projection content. For example, if you want to project the content of the navigation app on your phone to the central control screen of your car, the source device is the phone and the target device is the car.
- Meeting the preprocessing conditions means that the current processing capacity of the target device is low, while the current processing capacity of the source device is high. Therefore, you can consider letting the source device preprocess the projection content in advance. This can reduce the workload of the target device, and the target device can avoid or reduce the processing of the projection content, thus improving the efficiency of the entire projection process.
- the mobile phone interacts with the vehicle computer through the communication module 160 to obtain the performance parameters of the vehicle computer (such as load status).
- the multi-screen management module 130 can determine whether the vehicle computer needs computing power mutual assistance based on the load status of the vehicle computer. If the vehicle computer needs computing power mutual assistance, the mobile phone can pre-process the projection content. Conversely, if the vehicle computer does not need computing power mutual assistance, the mobile phone does not need to pre-process the projection content.
- the mobile phone pre-processes the projection content of the navigation application and the video application.
- the mobile phone determines that it can provide computing power assistance to the car computer, and the mobile phone can pre-process the projection content such as navigation applications and video applications.
- the CPU threshold for example, 60%
- the load of the mobile phone is smaller, and the bandwidth between the mobile phone and the car computer is higher and the bit error rate is lower, then the mobile phone determines that it can provide computing power assistance to the car computer, and the mobile phone can pre-process the projection content such as navigation applications and video applications.
- preprocessing of the projected content may include early decoding and/or early rendering of the projected content.
- the mobile phone sends the projection content of the video application to the co-pilot screen of the vehicle computer.
- the mobile phone sends the projection content of the video application to the car computer, and the car computer sends the projection content of the video application to the co-pilot screen.
- the projection content of the video application is the projection content that has been pre-decoded and pre-rendered by the mobile phone.
- the car computer can further process the decoded and rendered projection content (such as synthesis), and send the processing results of the projection content of the video application to the co-pilot screen. It can be seen that since the mobile phone has pre-processed the projection content, the processing task of the car computer can be reduced. When the car computer load is high, it can effectively reduce the probability of processing delays or processing failures caused by high load.
- the projection content is projection content that has not been pre-decoded or pre-rendered by the mobile phone.
- the car computer after the car computer receives the projection content of the video application from the mobile phone, it can decode, render, synthesize, etc. the projection content and send the processing results of the projection content to the co-pilot screen.
- the mobile phone can obtain a remote video file from the server, and send the remote video file (i.e., the projection content) to the communication module of the vehicle computer through the communication module.
- the vehicle computer After the vehicle computer receives the video file, it decodes the video file according to the encoding format of the video file, and renders and synthesizes the video frames included in the video file according to the size of the target screen (co-pilot screen). Afterwards, the vehicle computer can send the processing results of the video file to the co-pilot screen, and the co-pilot screen will display the corresponding video frames.
- the mobile phone can also project local video files to the co-pilot screen of the vehicle computer for display.
- the mobile phone obtains a video file, and pre-decodes and pre-renders the video file. Afterwards, the mobile phone can send the preprocessed video file to the communication module of the vehicle computer through the communication module. After the vehicle computer receives the video file, it performs synthesis and other processing, and sends the processing result of the video file to the co-pilot screen, which displays the corresponding video frame of the video file. Optionally, the vehicle computer can also perform frame cutting or super-resolution processing on the video file.
- the mobile phone when it sends the projection content of the video application, it also sends the timestamp corresponding to the projection content, so that the projection picture displayed on the co-pilot screen (such as video frame) and the audio played by the audio module (such as earphones 1 selected by the user for playing the audio of the video application) can be synchronized.
- the mobile phone sends the audio of the video application to earphone 1 .
- the mobile phone sends audio to the vehicle computer, and the vehicle computer sends the audio of the video application to the headset 1 through the Bluetooth module.
- the mobile phone sends the timestamp corresponding to the audio while sending the audio, so that the audio played by the headset 1 can be synchronized with the projection screen displayed by the display module (such as the co-pilot screen selected by the user).
- the projection content and audio of the video application can be carried in the same message or in different messages.
- the embodiment of the present application does not limit the execution order of S106 and S107.
- the co-pilot screen of the vehicle computer plays the video frames of the video application.
- the co-driver screen obtains the projection content of the video application, it can play the video frames of the video application according to the projection content.
- the co-pilot screen plays the corresponding video frame at the corresponding time according to the timestamp corresponding to the projected content, so that the video frame displayed on the co-pilot screen can be synchronized with the audio played by the audio module (such as the earphone 1 selected by the user).
- the audio module such as the earphone 1 selected by the user.
- the co-pilot screen can play the projected video frame in the target area of the co-pilot screen.
- the target area can be a window of the co-pilot screen, or the co-pilot screen can display the video frame on the entire screen.
- Earphone 1 plays the sound of the video.
- the headset After the headset acquires the audio of the video application, it can play the audio of the video application.
- earphone 1 plays the corresponding audio at the corresponding time according to the timestamp corresponding to the audio, so that the audio played by earphone 1 can be synchronized with the video frame displayed on the co-pilot screen.
- the embodiment of the present application does not limit the execution order of S108 and S109.
- the mobile phone sends the projection content of the navigation application to the central control screen.
- the mobile phone sends the projection content of the navigation application to the vehicle computer, and the vehicle computer sends the projection content of the navigation application to the central control screen.
- the projection content is the projection content that has been pre-decoded and pre-rendered by the mobile phone.
- the car computer after the car computer receives the projection content of the navigation application from the mobile phone, it can further process the decoded and rendered projection content (such as synthesis) and send the processing result of the projection content of the navigation application to the central control screen.
- the projection content is projection content that has not been pre-decoded or pre-rendered by the mobile phone.
- the car computer after the car computer receives the projection content of the navigation application from the mobile phone, it can decode, render, synthesize, etc. the projection content and send the processing results of the projection content to the central control screen.
- the mobile phone when it sends the projection content to the car computer, it also sends the timestamp corresponding to the projection content, so that the projection picture displayed on the central control screen and the audio played by the audio module (such as the car speakers selected by the user) can be synchronized.
- the embodiment of the present application does not limit the execution order of S106 and S110.
- the mobile phone sends the audio of the navigation application to the speakers of the entire vehicle.
- the mobile phone sends the audio of the navigation application to the vehicle computer, and the vehicle computer sends the audio of the navigation application to the speakers of the entire vehicle.
- the mobile phone when it sends the audio of the navigation application, it also sends the timestamp corresponding to the audio, so that the navigation audio played by the speakers of the entire vehicle and the projected screen (such as the navigation interface) displayed by the display module (such as the central control screen selected by the user) can be synchronized.
- the embodiment of the present application does not limit the execution order of S110 and S111.
- the central control screen plays the navigation image.
- the central control screen plays the corresponding navigation interface at the corresponding time according to the timestamp corresponding to the projection content of the navigation application, so that the navigation interface displayed on the central control screen can be synchronized with the audio played by the audio module (such as the vehicle speakers selected by the user).
- the audio and video separation can be achieved during the screen projection process.
- the projection content of the video application is displayed on the co-pilot screen of the car computer, and the audio corresponding to the projection content is played through the earphone 1 (not the speaker fixed on the car computer) to meet the user's audio-visual needs.
- users can project the content they want to watch on a large-screen device (such as a smart screen) and listen to the audio corresponding to the projection content through the mobile phone earphones to avoid disturbing other users.
- the target screen (central control screen, co-pilot screen) being connected to a device (car computer).
- the target screen may be connected to multiple processing units (such as chips or devices).
- the mobile phone can select a target processing unit from the multiple processing units connected to the target screen, and the target processing unit processes the projection content to be displayed on the target screen.
- the method may include the following steps:
- the mobile phone displays a multi-screen management interface.
- the mobile phone displays a multi-screen management interface 30 as shown in (a) of Figure 9.
- the interface 30 may include icons of one or more applications (such as icons of applications such as video, settings, and recorder), and icons of multiple available screens on the vehicle (such as icons of instrument screens, central control screens, etc.).
- the mobile phone receives an operation instruction from the user to project the content of the video application onto the three-row left screen.
- the user can drag the icon of the video application in the multi-screen management interface until it collides with the icon of the three-row left screen.
- the mobile phone can determine that the user wants to project the content of the video application to the three-row left screen display of the car computer.
- S203 The mobile phone receives an operation instruction from the user to play the audio of the video application through earphone 2 .
- the mobile phone obtains the computing power information of chip A from the vehicle computer.
- the mobile phone After the mobile phone determines that the user wants to project the content of the video application to the three-row left screen of the car computer, it can query the device/chip connected to the three-row left screen. After querying, the mobile phone determines that the three-row left screen is connected to the chips A and B of the car computer, and then the mobile phone can obtain the computing power information of chips A and B, so as to select a chip with better performance to process the projection content of the video application according to the computing power information of chips A and B.
- the mobile phone obtains the computing power information of chip A from chip A through the communication module.
- the mobile phone obtains the computing power information of chip B from the vehicle computer.
- the mobile phone determines chip A as the target chip based on the computing power information of chips A and B.
- the computing power information includes, but is not limited to, any one or more of the following information: CPU performance information (such as load), GPU performance information (such as load), I/O performance information, and whether there is a shared projection application between the connected screen and other screens.
- CPU performance information such as load
- GPU performance information such as load
- I/O performance information whether there is a shared projection application between the connected screen and other screens.
- the existence of a shared projection application between screens means that the projection content currently played on the two screens comes from the same application.
- the mobile phone can determine the performance of chips A and B based on one or more of the above computing power information, and use the chip with better performance as the target chip for processing the projection content.
- the multi-screen management module of the mobile phone can select a chip with better performance as the target chip based on the computing power information of chips A and B.
- the mobile phone can use chip A as the target chip.
- chip A is connected to three rows of left screens and three rows of right screens
- chip B is only connected to three rows of left screens.
- the three rows of right screens are the same viewing screens as the three rows of left screens
- chip A consider using chip A to process the projection content corresponding to the three rows of left screens and the projection content corresponding to the three rows of right screens.
- the mobile phone sends the projection content of the video application to chip A.
- the mobile phone can determine whether to pre-process the projection content of the video application. In some examples, if the computing power of chip A is poor, the mobile phone can pre-process the projection content of the video application in advance. For example, assuming that the screen size of the three-row left screen is 1920*2048, decoding and rendering an image of size 1920*2048 requires 25% of the CPU computing power of chip A and 1/3 of the GPU computing power, but the CPU load of chip A has exceeded 60% and the GPU has supported a certain amount of rendering tasks, which is not enough to complete the decoding and rendering tasks. The mobile phone can pre-process the projection content and send the pre-processed projection content to chip A. In this way, the number of tasks that chip A needs to process can be reduced.
- the mobile phone does not need to pre-process the projection content of the video application.
- S208 The mobile phone sends the audio of the video application to earphone 2 .
- the embodiment of the present application does not limit the execution order between S207 and S208.
- Chip A processes the screen projection content of the video application.
- the projection content of the video application may be data that has not been pre-processed.
- chip A needs to decode, render, synthesize, and process the projection content.
- the projection content of the video application can be pre-processed data.
- the projection content is the projection content obtained through decoding and rendering.
- chip A can process on this basis without performing decoding and rendering separately.
- chip A can synthesize the decoded and rendered projection content. It can be seen that when preprocessing is performed, the number of task processing of chip A can be reduced, and the processing delay caused by the high load of chip A can be avoided, thereby improving the efficiency of the entire projection process.
- Chip A sends the projection content of the video application to the left screen of the third row.
- chip A After chip A processes the projection content of the video application, it sends the processed projection content to the left screen of the three rows.
- the left screen of the third row receives the projection content of the video application, it can display the projection content, such as playing video frames.
- Earphone 2 plays the sound of the video.
- the embodiment of the present application does not limit the execution order between S211 and S212.
- the mobile phone can select the target processing component (such as chip A of the car computer) for processing the projected content based on the target screen selected by the user for displaying the projected content, so as to improve the efficiency of processing the projected content and thereby improve the efficiency of the entire projection operation.
- the target processing component such as chip A of the car computer
- the audio module selected by the user through the multi-screen management interface may be invalid.
- the mobile phone can prompt the user to reselect the audio module for playing audio.
- the user chooses to play the audio of the video application through earphone 2, but the path between the target chip A and the earphone 2 may be faulty.
- the mobile phone can pop up a prompt window 306 such as shown in (b) of Figure 11A to prompt the user to reselect an available audio module.
- the audio modules and audio transmission paths can also be selected based on the application as the granularity. For example, the user can choose to play audio that does not involve personal privacy (such as navigation audio) through the speakers of the entire car, and play audio that involves personal privacy or is not convenient to play externally. Play through personal headphones.
- Figure 12 mainly uses the target screen connected to different processing components (chips) as an example for explanation.
- the target screen can also be connected to different devices.
- the mobile phone can obtain the computing power information of multiple devices connected to the target screen, and select the device with the best performance as the target device based on the computing power information of multiple devices. In this way, by selecting the target device with the best performance to process the projection content, the processing efficiency can be improved, which helps to improve the efficiency of the entire projection operation.
- users can project the content of the desktop or application to multiple screens through the multi-screen management interface.
- the user can project Video 1 in the mobile phone to the co-pilot screen and the three-row left screen of the car computer for display.
- the multi-screens viewed together can be divided into target screens and co-viewing screens.
- the user can first project Video 1 to the target screen for display, and then project Video 1 to the co-viewing screen for display.
- the projection timing of the target screen and the co-viewing screen may not be distinguished.
- the technical solution of the embodiment of the present application may include the following steps:
- the mobile phone displays a multi-screen management interface.
- the user operates the multi-screen management interface to choose to project the content of the video application to the co-pilot screen and the left screen of the third row.
- the user selects the audio transmission path corresponding to the co-pilot screen and the third-row left screen by operating the multi-screen management interface.
- the user selects the corresponding audio transmission path through the interface 30 shown in (a) of FIG10.
- the car machine Bluetooth-earphone 1 is selected to play the audio corresponding to the screen projection content on the co-pilot screen
- the car machine Bluetooth-earphone 2 is selected to play the audio corresponding to the screen projection content on the left screen of the third row.
- the mobile phone pre-decodes the projected image.
- the mobile phone can pre-decode the projected image.
- the mobile phone splices the projection image of the co-pilot screen and the projection image of the three-row left screen, and renders the spliced image.
- the mobile phone can splice the screen of the co-pilot screen (video frame of video 1) and the screen of the three rows of left screen (video frame of video 1) into a whole screen according to the size of the co-pilot screen and the three rows of left screen, and fill the gap between the two screens.
- the mobile phone can render the spliced screen and send the rendering result to the vehicle computer.
- the process in which the mobile phone splices the image of the co-pilot screen and the image of the three-row left screen into one image and renders the spliced image can be called splicing rendering or merge rendering.
- the source device such as a mobile phone
- the above-mentioned splicing rendering (S304) and pre-decoding (S305) can also be performed by the target device (such as a car computer).
- the mobile phone sends the processing result of the spliced image to the vehicle computer.
- the processing result of the spliced image is the projection image information that has been pre-decoded and pre-rendered by the mobile phone.
- the processing result of the spliced image is the projection image information that has not been pre-decoded and pre-rendered by the mobile phone.
- the vehicle computer splits the spliced image into the projection image of the co-pilot screen and the projection image of the three-row left screen.
- the vehicle computer can divide the spliced screen into the image of the co-pilot screen and the image of the three-row left screen according to the size of the co-pilot screen and the three-row left screen.
- the vehicle computer sends the projection image of the co-pilot screen to the co-pilot screen.
- the co-pilot screen displays the projected image.
- the vehicle computer sends the projection image of the three-row left screen to the three-row left screen.
- the mobile phone controls the playback of the audio corresponding to the co-pilot screen and the third-row left screen.
- the mobile phone can send the audio to earphone 1 and earphone 2 respectively through the corresponding audio transmission path.
- user A can use earphone 1 to listen to the corresponding audio while watching the screen projection (such as a video) on the co-pilot screen.
- User B can use earphone 2 to listen to the corresponding audio while watching the screen projection on the three-row left screen.
- earphone 1 and earphone 2 are earphones connected to a mobile phone, as shown in Figure 16, chip A is connected to the co-pilot screen, and chip B is connected to the third-row left screen.
- the mobile phone can send audio to earphone 1 through the mobile phone Bluetooth module (path 1).
- the mobile phone can send audio to earphone 2 through the mobile phone Bluetooth module (path 2).
- the mobile phone can send audio to the Bluetooth module of the car computer chip A through the mobile phone Bluetooth module, and then the Bluetooth module of the car computer chip A sends audio to earphones 1 and 2.
- the mobile phone can also send audio to earphones 1 and 2 through other audio transmission paths.
- the embodiments of the present application do not limit the specific audio transmission paths used for audio transmission.
- the mobile phone sends audio to chip B, and chip B sends audio to earphone 2.
- the mobile phone sends audio to chip A, chip A sends audio to chip B, and chip B sends audio to earphone 2.
- the device splices the projection images of multiple screens and performs splicing rendering as an example.
- other rendering methods can also be used.
- the source device mobile phone can render according to the size of the target screen (such as the co-pilot screen) and send the rendering result to the car computer.
- the car computer After the car computer receives the rendering result, it can execute the super-resolution algorithm according to the size of the screen (such as the three-row left screen) to stretch or scale the picture size to the size of the three-row left screen.
- the car computer can instruct the three-row left screen and the co-pilot screen to display the same projection screen.
- the projection screen displayed on the three-row left screen is a super-resolution processed picture.
- the super-resolution algorithm may be a linear interpolation super-resolution algorithm, or a super-resolution interpolation algorithm after learning and optimization.
- the embodiment of the present application does not limit the specific implementation of the super-resolution algorithm.
- the target device (such as a car computer) for cross-device rendering may have an external Dongle in some scenarios, and the Dongle can be connected to the target screen via an HDMI interface.
- the basic process of cross-device screen projection can be similar to the above-mentioned embodiment.
- the external Dongle can be used to receive the projection content from the source device (such as a mobile phone), process the projection content, and send the processing results to the target device.
- the target device instructs the target screen to display the projection content based on the processing results of the projection content.
- the Dongle performs some processing operations on the projection content in advance, which can reduce the processing task load of the target device and improve the efficiency of the entire projection process.
- the source device pre-renders the projection content and sends the rendering result of the projection content to the Dongle plug-in of the target device, which decodes the projection content.
- the Dongle decodes the projection content
- it sends the decoding result of the projection content to the target device (such as a car computer), and the target device divides or super-divides the projection content into display content of the target screen and performs a synthesis operation.
- the target device performs the synthesis operation, it sends the synthesis result to the target screen for display.
- the target screen can project the projection content and touch interactive controls (such as a progress bar and pause button for users to control the progress of video viewing). Only the projection content can be projected on the same viewing screen, without projecting controls for touch interaction.
- the projection interface displayed on the target screen (co-pilot screen) is the playback interface 60 of video 1, which includes a progress bar and pause controls for controlling the progress of the video.
- the projection interface displayed on the same viewing screen (three-row left screen) is also the playback interface 70 of video 1, but the interface 70 does not include controls such as a video progress bar and a video pause control.
- the user can control the target screen and the projection content displayed on the same viewing screen by manipulating the target screen.
- the co-pilot screen and the left screen of the third row are both projecting and playing video 1 from the mobile phone.
- the passenger on the co-pilot screen can click on the pause control such as in (a) of Figure 19.
- the operation information can be returned to the mobile phone through the car computer, and the mobile phone controls the co-pilot screen and the left screen of the third row to stop playing video 1 at the same time.
- the projection content and touch-interactive controls can be projected onto both the target screen and the co-viewing screen.
- the mobile phone can control the co-viewing screen to switch to a new target screen and adjust the communication path of the new target screen (including the transmission path of the projection screen and the transmission path of the corresponding audio of the projection screen).
- the method includes the following steps:
- the user can exit the playback interface of Video 1 on the co-pilot screen, or stop projecting on the co-pilot screen through other operations.
- the co-pilot screen sends a notification to the mobile phone to stop screen projection.
- the co-pilot screen will send a notification to stop screen projection to the vehicle computer, and the vehicle computer will send a notification to stop screen projection from the co-pilot screen to the mobile phone.
- the mobile phone adjusts the communication path of the audio stream corresponding to the projection image between the three rows of left screens.
- the target screen switching process can be started through the multi-screen management module to switch the current co-viewing screen to the new target screen.
- the multi-screen management module of the mobile phone can cooperate with the communication module to adjust the communication path of the projected screen (such as video stream) between the mobile phone and the new target screen and the audio stream corresponding to the projected screen to achieve a better screen projection experience.
- the user chooses to play the audio corresponding to the projection screen on the three-row left screen through earphone 2.
- mobile phone 2 is connected to chip A of the car computer, and the mobile phone sends an audio stream to chip A of the car computer through the Bluetooth module, and then chip A forwards the audio stream to earphone 2, so that user A can listen to the audio corresponding to the projection screen on the co-pilot screen through earphone 2.
- the co-pilot screen stops projecting, the mobile phone can establish a connection with earphone 2, and audio can be sent directly from the mobile phone to earphone 2. In this way, the length of the communication path can be shortened and the audio playback delay can be reduced.
- the mobile phone adjusts the communication path between the projection image and the three-row left screen.
- the mobile phone sends the projection screen to the chip B of the car computer, which processes it and sends the processing result to the three-row left screen for display.
- the mobile phone can send the projection screen to the chip A with better computing power, which processes it and sends the processing result to the three-row left screen for display.
- the mobile phone updates the multi-screen management interface.
- the multi-screen management interface 30a of the mobile phone is as shown in FIG. 21 (a)
- the co-pilot screen is the target screen
- the three-row left screen is the same-viewing screen.
- the mobile phone switches the previous same-viewing screen (three-row left screen) to the target screen, as shown in the interface 30b of FIG. 21 (b), the three-row left screen has been marked as the target screen.
- the mobile phone transmits the projection image information and the corresponding audio stream according to the new communication channel.
- the mobile phone can directly send the audio stream corresponding to the projection screen to the headset 2 through the Bluetooth module.
- the mobile phone can send the projection screen information to the chip A of the vehicle computer, and the chip A processes the projection screen information and sends the processing result to the three-row left screen for display.
- the multi-screen management interface displays multiple screens across devices by screen.
- the multi-screen management interface can also display multiple screens across devices by application granularity.
- the multi-screen management interface can present one or more applications and the shared data and operation status of each application.
- the operation status of the application can be divided into active (Active) or inactive.
- the interface of the active application is presented on at least one screen of at least one device.
- the shared data of the application includes but is not limited to: the identifier of the device/screen presenting the Active application, the account corresponding to the device/screen, the display status of the Active application is desktop or full screen, and the audio transmission path corresponding to the Active application.
- FIG23 shows a multi-screen management interface 30c with application granularity (an example of the multi-screen management interface 101 in FIG2B ).
- the navigation application is projected on the central control screen for display, and the account logged into the central control screen is the main driving account.
- the navigation interface displayed on the central control screen is displayed in full screen, and the audio corresponding to the navigation interface is sent from the mobile phone to the car speakers of the vehicle computer, and the audio is played by the car speakers.
- the shared data and operating status of other applications in the interface 30c in FIG23 will not be described in detail.
- the shared data and operating status of each application in the interface 30c in FIG23 are shown in Table 1.
- the vehicle computer can prioritize the operation instructions of high-authority users based on the permissions of different accounts. Assume that the permission of the co-pilot account is higher than that of the third-row left screen account. In some scenarios, if the co-pilot screen and the third-row left screen play the projection content at the same time, the user in front of the co-pilot screen enters operation instruction A on the co-pilot screen, and the user in front of the third-row left screen enters operation instruction B on the third-row left screen, the car computer can give priority to the high-authority operation instruction A.
- the driver can use the multi-screen management interface on the phone screen to project the desktop + navigation to the central control screen, project video 2 to the co-driver screen and the left screen of the third row, and project video 3 to the left screen of the third row.
- the chip A of the mobile phone pre-processes (such as decoding and rendering) the projection screen of the desktop and navigation applications (such as navigation), and sends the processing results to the chip B of the car computer, which synthesizes them and sends the synthesized results to the central control screen for display.
- the navigation sound can be directly sent to the Bluetooth headset 2 by the mobile phone Bluetooth module for playback.
- chip A of the mobile phone pre-processes video 2 (such as decoding and rendering), and sends the processing results to chip B of the car computer, which is synthesized by chip B.
- Chip B sends the synthesis results of video 2 to the co-pilot screen and the three-row left screen respectively, and the co-pilot screen and the three-row left screen display video 2 respectively.
- each screen can generate a corresponding window and play video 2 in the corresponding window.
- the co-pilot screen plays video 2 in local window 1
- the three-row left screen plays video 2 in local window 1.
- the sound of Video 2 is transmitted to the chip B of the vehicle computer through the mobile phone and played through the player of the entire vehicle.
- Video 3 reaches the chip C of the car computer through the mobile phone, and the chip C decodes, renders, and synthesizes Video 3, and sends the synthesis result to the three rows of left screens for display.
- the sound of Video 3 can be transmitted from the mobile phone to the chip C through the Wi-Fi connection, and then transmitted to the Bluetooth headset 1 through the chip C for playback.
- the embodiment of the present application also provides a multi-screen multi-device interaction method, and the user can start the screen projection process during video viewing or in other scenarios.
- the user swipes left to call out the multi-screen management card 702 while watching video 1.
- the mobile phone can display the pop-up window 106 shown in Figure 25 (b).
- the mobile phone can determine that the user wants to project video 1 to the three rows of left screens and the three rows of right screens for display.
- the mobile phone can then send the projection screen of video 1 to the three rows of left screens and the three rows of right screens according to the scheme of the above embodiment.
- the user can also select multiple contents that he wants to project (such as videos 1, 2, etc.) in applications such as gallery, and call out the pop-up window 106 as shown in (b) of Figure 25 through a certain operation to select the target screen for playing the projected content.
- contents such as videos 1, 2, etc.
- applications such as gallery
- the first device can also automatically project some content on the first device to the second device for display based on the user's usage habits. For example, when it is detected that the user has entered the car, the mobile phone can automatically project the content of the navigation application to the central control screen based on the user's usage habits if the user is sitting in the main driver's seat. If the user is sitting in the co-pilot seat and the user is accustomed to using the co-pilot screen to watch videos, the mobile phone can project video applications and other audio and video entertainment content to the co-pilot screen to meet the user's screen projection needs.
- steps in the method embodiment may be equivalently replaced by other possible steps.
- some steps in the method embodiment may be optional and may be deleted in certain usage scenarios.
- other possible steps may be added in the method embodiment.
- the execution subject (such as a functional module) of some steps in the method embodiment may be replaced by other execution subjects.
- the latest instruction is selected for execution.
- the first device master controller plays or stops according to the latest instruction (regardless of which screen it comes from); to simplify the entire system.
- the operation priority can be restricted and managed in combination with the account authority. Considering the complexity and logical unity of the embodiment of the present application, the description will not be expanded, but the foreseeable expansion module and multi-screen priority operation control processing are also included in the protection scope of the embodiment of the present application.
- Some other embodiments of the present application provide a device, which may be the above-mentioned first device, second device, etc.
- the device may include: a display screen, a memory, and one or more processors.
- the display screen, the memory, and the processor are coupled.
- the memory is used to store computer program code, and the computer program code includes computer instructions.
- the processor executes the computer instructions, the device may execute the various functions or steps performed by the mobile phone in the above-mentioned method embodiment.
- the structure of the device can refer to the electronic device (device) shown in Figure 26.
- the core structure of the device can be represented as the structure shown in Figure 26, and the device includes: a processing module 1301, an input module 1302, a storage module 1303 and a display module 1304.
- the processing module 1301 may include at least one of a central processing unit (CPU), an application processor (AP) or a communication processor (CP).
- the processing module 1301 may perform operations or data processing related to the control and/or communication of at least one of other elements of the user electronic device. Specifically, the processing module 1301 may be used to control the content displayed on the main screen according to certain trigger conditions.
- the processing module 1301 is also used to process input instructions or data and determine the display style according to the processed data.
- the input module 1302 is used to obtain the instructions or data input by the user and transmit the obtained instructions or data to other modules of the electronic device.
- the input mode of the input module 1302 may include touch, gesture, approaching the screen, etc., or voice input.
- the input module may be the screen of the electronic device, obtain the user's input operation and generate an input signal according to the obtained input operation, and transmit the input signal to the processing module 1301.
- the storage module 1303 may include a volatile memory and/or a non-volatile memory.
- the storage module is used to store at least one instruction or data related to other modules of the user device. Specifically, the storage module may record the location of the interface where the device interface element UI is located.
- the display module 1304 may include, for example, a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a micro-electromechanical system (MEMS) display, or an electronic paper display, and is used to display content (e.g., text, images, videos, icons, symbols, etc.) that can be viewed by a user.
- LCD liquid crystal display
- LED light emitting diode
- OLED organic light emitting diode
- MEMS micro-electromechanical system
- a communication module 1305 is also included to support personal devices (via a communication network) to communicate with other personal devices.
- the communication module can be connected to a network via wireless communication or wired communication to communicate with other personal devices or network servers.
- Wireless communication can use at least one of cellular communication protocols, such as long-term evolution (LTE), advanced long-term evolution (LTE-A), code division multiple access (CDMA), wideband code division multiple access (WCDMA), universal mobile telecommunications system (UMTS), wireless broadband (WiBro) or global mobile communication system (GSM).
- Wireless communication may include, for example, short-range communication.
- Short-range communication may include at least one of wireless fidelity (Wi-Fi), Bluetooth, near field communication (NFC), magnetic stripe transmission (MST) or GNSS.
- each functional module of the device can execute one or more steps in the above method embodiment.
- the chip system includes at least one processor 1401 and at least one interface circuit 1402.
- the processor 1401 and the interface circuit 1402 can be interconnected by lines.
- the interface circuit 1402 can be used to receive signals from other devices (such as the memory of the electronic device).
- the interface circuit 1402 can be used to send signals to other devices (such as the processor 1401).
- the interface circuit 1402 can read the instructions stored in the memory and send the instructions to the processor 1401.
- the electronic device can perform the various steps in the above embodiments.
- the chip system can also include other discrete devices, which are not specifically limited in the embodiment of the present application.
- An embodiment of the present application also provides a computer storage medium, which includes computer instructions.
- the computer instructions When the computer instructions are executed on the above-mentioned electronic device, the electronic device executes each function or step executed by the mobile phone in the above-mentioned method embodiment.
- the embodiment of the present application also provides a computer program product.
- the computer program product When the computer program product is run on a computer, the computer is enabled to execute each function or step executed by the first device or the second device in the above method embodiment.
- the disclosed devices and methods can be implemented in other ways.
- the device embodiments described above are only schematic.
- the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed.
- Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
- the units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
- each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
- the above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
- the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium.
- the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the method described in each embodiment of the present application.
- the aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- General Engineering & Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Networks & Wireless Communication (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
一种多屏多设备交互方法、电子设备及***,涉及终端技术领域,可以提升投屏过程中的效率,该方法包括:显示第一界面,第一界面包括第二设备上多个可用屏幕的标识;接收用于指示将第一设备上的投屏内容投屏到目标屏幕的操作;多个可用屏幕包括目标屏幕;向目标屏幕发送投屏内容对应的投屏图像,以使得目标屏幕显示投屏图像;投屏图像用于目标屏幕显示投屏内容;其中,投屏内容为一个或多个,且目标屏幕为多个;或者,投屏内容为多个,且目标屏幕为一个或多个。
Description
本申请要求于2022年10月18日提交国家知识产权局、申请号为202211275625.X、申请名称为“多屏多设备交互方法、电子设备及***”的中国专利申请的优先权,其全部内容通过引用结合在本申请中。
本申请涉及终端技术领域,尤其涉及多屏多设备交互方法、电子设备及***。
目前,用户可以通过投屏将手机上的数字显示内容投射到电视等大屏设备上,虽然在一定程度上能够满足用户在大屏设备上观看投屏内容的需求,但目前的投屏场景具有一定局限性。比如,初始时,用户将手机上的视频投屏在电视上进行显示,之后,用户需要将手机上的文档投屏在笔记本电脑上进行办公浏览时,需要先暂停电视上的投屏,再通过操控手机上的控件,将文档投屏在笔记本电脑上。可以看出,目前的投屏效率较低,用户需进行繁杂的操作才能切换投屏设备。
发明内容
本申请提供多屏多设备交互方法、电子设备及***,可以提升投屏过程中的投屏效率。
为了实现上述目的,本申请实施例提供了以下技术方案:
第一方面,提供一种多屏多设备交互方法和界面,应用于第一设备或能够实现第一设备功能的组件(比如芯片***),方法包括:
显示第一界面,第一界面包括第二设备上多个可用屏幕的标识;
接收用于指示将第一设备上的投屏内容投屏到目标屏幕的操作;多个可用屏幕包括目标屏幕;
向目标屏幕发送投屏内容对应的投屏图像,以使得目标屏幕显示投屏图像;投屏图像用于目标屏幕显示投屏内容;
其中,投屏内容为一个或多个,且目标屏幕为多个;或者,投屏内容为多个,且目标屏幕为一个或多个。
示例性的,如图5的(c),手机(第一设备的一个示例)显示界面30(第一界面的一个示例),界面30包括汽车(第二设备的一个示例)上多个可用屏幕的标识(比如仪表屏、中控屏、副驾屏等标识)。在检测到用户将视频应用的虚拟图标(或光标)沿着箭头方向移动至中控屏(目标屏幕的一个示例)的标识所在区域之后,手机确定用户想要将手机上视频应用的投屏内容投屏到中控屏,则手机可以向中控屏发送该投屏内容对应的投屏图像,以使得中控屏显示视频应用对应的投屏图像。
其中,投屏内容为一个或多个,且目标屏幕为多个;或者,投屏内容为多个,且目标屏幕为一个或多个。
示例性的,第一设备可以将一个或多个应用的投屏内容投射到一个或多个第二设备的多个目标屏幕上,比如,第一设备将视频应用1的内容(比如播放的视频)投射到第二设备的屏幕1,将游戏应用2的内容(比如游戏界面等)投射到屏幕2和屏幕3。
再示例性的,第一设备可以将一个或多个窗口中的内容投射到第二设备的多个目标屏幕上。比如,将窗口1中播放的视频1投射到第二设备的屏幕1和屏幕3,将窗口2中显示的导航界面投射到屏幕2。
再示例性的,第一设备可以将桌面上的显示内容投射到第二设备的多个目标屏幕上。
上述技术方案中,用户可以通过第一界面(图2B所示多屏管理界面111的一个示例),进行操作,以控制第一设备将相应的投屏内容投屏到第二设备的任意一个或多个目标屏幕,如此,能够满足在多屏场景中用户通过任意可用屏幕展示投屏内容的视听需求。
以第一设备将内容投屏到设备2和设备3为例,上述技术方案中,可以通过操作第一界面直接控制第一设备将内容投屏在设备2和设备3上,用户无需先暂停在设备2上的投屏过程,再将
内容投屏在设备3上。可见,上述技术方案的投屏效率较高。
在一个示例中,以驾车场景为例,手机桌面被投屏到副驾屏进行显示,由副驾乘客在副驾屏上设置行车(订票、订餐)等信息。目的地到达之后,副驾乘客在副驾屏的多屏管理界面上选择中止二排左、三排左屏、中控屏、副驾屏的投屏过程,本次投屏流程结束。
在一些可能的设计中,第二设备为一个或多个。
在一些可能的设计中,目标屏幕连接有多个设备;方法还包括:
从多个设备中确定一个或多个目标设备,多个设备包括一个或多个目标设备;
向目标屏幕发送投屏内容对应的投屏图像,包括:向一个或多个目标设备发送投屏图像,以使得一个或多个目标设备对投屏图像进行处理,并向目标屏幕发送处理后的投屏图像。
示例性的,如图12,以手机(第一设备的一个示例)将内容投屏到三排左屏(目标屏幕的一个示例)进行显示为例,三排左屏连接有芯片A、B。手机可以芯片A、B中确定芯片A作为用于处理投屏图像的目标设备,并向芯片A发送投屏图像,以使得芯片A对投屏图像进行处理,并向三排左屏发送处理后的投屏图像,由三排左屏根据投屏图像显示投屏内容。
通过该方案,手机可以根据用户选择的用于显示投屏内容的目标屏幕(比如三排左屏),选择用于处理投屏图像的目标处理组件(比如车机的芯片A),以提升处理投屏图像的效率,进而提升整个投屏操作的效率。
在一些可能的设计中,从多个设备中确定一个或多个目标设备,包括:
获取多个设备的算力信息,算力信息包括设备的负载信息;
根据多个设备的算力信息,从多个设备中确定一个或多个目标设备。
示例性的,若芯片A的CPU性能、GPU性能、I/O性能均比芯片B高,则手机可以将芯片A作为目标设备。如此,采用性能更优的设备处理投屏图像,有助于缩短处理时长,进而提升投屏过程的效率。
再示例性的,假设芯片A连接三排左屏以及三排右屏,芯片B仅连接三排左屏。假设三排右屏为三排左屏的同看屏,则考虑使用芯片A处理三排左屏对应的投屏内容以及三排右屏对应的投屏内容。
在一些可能的设计中,方法还包括:
接收用户在第一界面上输入的音频选择操作;
根据音频选择操作,确定用于播放投屏图像对应音频的音频模块,以及确定第一设备与音频模块之间的音频传输通路;
通过音频传输通路,向音频模块发送投屏图像对应的音频数据。
示例性的,如图10的(a),手机显示卡片305。检测到用户在卡片305上选择车机-全车扬声器的操作,手机可以根据用户的音频选择操作,确定用于播放投屏图像对应音频(即导航音频)的音频模块为全车扬声器,以及确定手机与全车扬声器之间的音频传输通路为车机-全车扬声器。之后,手机通过车机向全车扬声器发送导航对应的音频数据。
该方案中,第一设备可以根据用户的音频选择操作,确定用于播放音频的音频模块,能够满足用户的音频收听需求,提升用户的听觉体验。
在一种可能的设计中,目标屏幕包括第一屏幕和第二屏幕;第一屏幕上的投屏内容对应第一音频模块以及第一音频传输通路;第二屏幕上的投屏内容对应第二音频模块以及第二音频传输通路。
示例性的,如图7,手机(第一设备)将导航应用的内容投屏到中控屏(第一屏幕的一个示例),将视频投屏到副驾屏(第二屏幕),中控屏幕上的投屏内容对应耳机1(第一音频模块)以及第一音频传输通路,即用户通过全车扬声器收听导航的音频。副驾屏幕上的投屏内容对应耳机1(第二音频模块)以及第二音频传输通路,即用户通过耳机1收听视频对应的音频。
通过该方案,可实现投屏过程中的音画分离,使得不同屏幕上的投屏内容有不同的音频模块以及音频传输通路。比如通过车机的副驾屏显示视频应用的投屏内容,通过耳机1(并非车机上固定安装的扬声器)播放投屏内容对应的音频,以满足用户的视听需求。比如,在一些场景中,用户可以将想要观看的内容投屏在大屏设备(比如智慧屏)上,并通过手机耳机收听投屏内容对
应的音频,以免音频打扰到其他用户。
在一种可能的设计中,目标屏幕包括第一屏幕;投屏内容包括第一应用的投屏内容和第二应用的投屏内容;第一屏幕上第一应用的投屏内容对应第一音频模块以及第一音频传输通路;第一屏幕上第二应用的投屏内容对应第二音频模块以及第二音频传输通路。
示例性的,第一设备将导航应用的内容,以及视频均投屏到车机的副驾屏,用户可以通过耳机1收听所投射的导航界面对应的音频,通过全车扬声器收听视频的音频。
再示例性的,涉及隐私的应用的投屏音频,可使用个人耳机收听,不涉及隐私的应用的投屏音频,可使用全车扬声器收听。
该音画分离的方案中,同一屏幕上不同应用的投屏内容有不同的音频模块以及音频传输通路。如此,能够满足用户对不同应用的音频的收听需求。
在一种可能的设计中,目标屏幕包括第一屏幕和第二屏幕;投屏内容包括第一应用的投屏内容和第二应用的投屏内容;第一屏幕上第一应用的投屏内容对应第一音频模块以及第一音频传输通路;第二屏幕上第二应用的投屏内容对应第二音频模块以及第二音频传输通路。
示例性的,第一设备将导航应用的内容投屏到车机的中控屏,将视频投屏到车机的副驾屏,用户可以通过耳机1收听所投射的导航界面对应的音频,通过全车扬声器收听视频的音频。
该音画分离的方案中,不同屏幕上不同应用的投屏内容有不同的音频模块以及音频传输通路,能够满足用户对不同应用的音频的收听需求。
在一种可能的设计中,向目标屏幕发送投屏图像之前,方法还包括:对投屏图像进行预处理;
向目标屏幕发送投屏图像,包括:向目标屏幕发送预处理后的投屏图像。
如此,第一设备可以对上述选择的第二设备进行算力辅助。比如,第一设备可以提前对投屏内容(比如视频流)进行部分处理操作。如此,可以降低第二设备对投屏内容的处理工作量,节约第二设备的算力,提升整个投屏流程的效率。
在一种可能的设计中,对投屏图像进行预处理,包括:
满足第一条件时,对投屏图像进行预处理;
第一条件包括:第二设备的负载不足以处理投屏图像。
如此,在第二设备的负载足以处理投屏图像时,可以由第二设备处理投屏图像。第二设备的负载不足以处理投屏图像时,第一设备可以对上述选择的第二设备进行算力辅助,以降低第二设备的计算负担,进而提升投屏过程的效率。
在一种可能的设计中,目标屏幕包括第一屏幕和第二屏幕,
对投屏图像进行预处理,包括:
将第一屏幕的投屏图像和第二屏幕的投屏图像进行拼接,得到拼接图像;
对拼接图像进行渲染。
相比于对第一屏幕尺寸的投屏图像和第二屏幕尺寸的投屏图像分别进行渲染,将第一屏幕尺寸的投屏图像和第二屏幕尺寸的投屏图像进行拼接渲染,能够降低渲染工作量,进而提升投屏过程的效率。
在一种可能的设计中,目标屏幕包括第一屏幕和第二屏幕,
对投屏图像进行预处理,包括:
按照第一屏幕的尺寸,对投屏图像进行渲染;
向目标屏幕发送投屏内容对应的投屏图像,以使得目标屏幕显示投屏图像,包括:向第一屏幕发送渲染后的投屏图像,以使得第一屏幕显示尺寸为第一屏幕尺寸的投屏图像;向第二屏幕发送渲染后的投屏图像,以使得第二屏幕显示经拉伸或缩放后尺寸为第二屏幕尺寸的投屏图像。
相比于对第一屏幕尺寸的投屏图像和第二屏幕尺寸的投屏图像分别进行渲染,按照第一屏幕尺寸进行渲染,能够降低第一设备的渲染工作量,进而提升投屏过程的效率。
在一种可能的设计中,方法还包括:
从第一屏幕接收停止投屏的通知消息;
调整与第二屏幕之间的图像传输通路以及与音频模块之间的音频传输通路。
示例性的,如图20,副驾屏和三排左屏为同看屏。副驾屏(第一屏幕的一个示例)停止投屏
之后,手机可以调整与三排左屏(第二屏幕的一个示例)之间的图像传输通路以及与音频模块之间的音频传输通路,以实现更优的投屏体验。
在一种可能的设计中,第一界面还包括目标应用的标识;接收用于指示将第一设备上目标应用的投屏内容投屏到目标屏幕的操作,包括:
接收用户将虚拟光标由目标应用的标识拖动到目标屏幕的标识。
示例性的,如图9的(a),检测到用户将虚拟光标304由导航应用的标识拖拽到中控屏的标识,则手机可以向中控屏发送导航应用的相应投屏图像,以便将导航应用的投屏内容投射到中控屏进行显示。
上述技术方案中,用户可以通过第一界面(图2B所示多屏管理界面111的一个示例),将需要投屏的应用的图标拖拽到任意可用屏幕的图标上,以控制手机将手机应用的内容投屏到相应屏幕,满足在多屏场景中用户通过任意可用屏幕展示目标内容的视听需求。
在一种可能的设计中,显示第一界面之前,方法还包括:
接收用户选择投屏内容的操作;
接收用户输入的第一操作;
显示第一界面,包括:响应于第一操作,显示第一界面;
接收用于指示将第一设备上的投屏内容投屏到目标屏幕的操作,包括:接收用于选择目标屏幕的操作。
示例性的,如图25的(a),用户可以选择视频1作为投屏对象。检测到用户输入的左滑操作,手机可以唤出控件702。检测到用户点击控件702中的多屏管理选项的操作(第一操作的一个示例)之后,手机可显示图25的(b)所示的界面10(第一界面的一个示例)。如图25的(b),检测到用于选择三排右屏、三排左屏的操作(比如点击三排左屏、三排右屏),手机可以向三排左屏、三排右屏发送视频1的投屏图像,以使得三排左屏、三排右屏显示视频1。
在一种可能的设计中,预处理包括解码、渲染。
可选的,目标屏幕包括投屏图像以及操控控件。
可选的,第一界面可以作为负一屏,或者滑动桌面切入下一屏来体现,或者其他设计体现。
第二方面,提供一种多屏多设备交互装置和界面,应用于第一设备或能够实现第一设备的组件,装置包括:
显示模块,用于显示第一界面,第一界面包括第二设备上多个可用屏幕的标识;
输入模块,用于接收用于指示将第一设备上的投屏内容投屏到目标屏幕的操作;多个可用屏幕包括目标屏幕;
通信模块,用于向目标屏幕发送投屏内容对应的投屏图像,以使得目标屏幕显示投屏图像;投屏图像用于目标屏幕显示投屏内容;
其中,投屏内容为一个或多个,且目标屏幕为多个;或者,投屏内容为多个,且目标屏幕为一个或多个。
在一种可能的设计中,第二设备为一个或多个。
在一种可能的设计中,目标屏幕连接有多个设备;装置还包括处理模块;
处理模块,用于从多个设备中确定一个或多个目标设备,多个设备包括一个或多个目标设备;
向目标屏幕发送投屏内容对应的投屏图像,包括:向一个或多个目标设备发送投屏图像,以使得一个或多个目标设备对投屏图像进行处理,并向目标屏幕发送处理后的投屏图像。
在一种可能的设计中,从多个设备中确定一个或多个目标设备,包括:
获取多个设备的算力信息,算力信息包括设备的负载信息;
根据多个设备的算力信息,从多个设备中确定一个或多个目标设备。
在一种可能的设计中,输入模块,还用于接收用户在第一界面上输入的音频选择操作;
处理模块,还用于根据音频选择操作,确定用于播放投屏图像对应音频的音频模块,以及确定第一设备与音频模块之间的音频传输通路;
通信模块,还用于通过音频传输通路,向音频模块发送投屏图像对应的音频数据。
在一种可能的设计中,目标屏幕包括第一屏幕和第二屏幕;第一屏幕上的投屏内容对应第一
音频模块以及第一音频传输通路;第二屏幕上的投屏内容对应第二音频模块以及第二音频传输通路。
在一种可能的设计中,目标屏幕包括第一屏幕;投屏内容包括第一应用的投屏内容和第二应用的投屏内容;第一屏幕上第一应用的投屏内容对应第一音频模块以及第一音频传输通路;第一屏幕上第二应用的投屏内容对应第二音频模块以及第二音频传输通路。
在一种可能的设计中,目标屏幕包括第一屏幕和第二屏幕;投屏内容包括第一应用的投屏内容和第二应用的投屏内容;第一屏幕上第一应用的投屏内容对应第一音频模块以及第一音频传输通路;第二屏幕上第二应用的投屏内容对应第二音频模块以及第二音频传输通路。
在一种可能的设计中,处理模块,还用于在向目标屏幕发送投屏图像之前,对投屏图像进行预处理;
向目标屏幕发送投屏图像,包括:向目标屏幕发送预处理后的投屏图像。
在一种可能的设计中,对投屏图像进行预处理,包括:
满足第一条件时,对投屏图像进行预处理;
第一条件包括:第二设备的负载不足以处理投屏图像。
在一种可能的设计中,目标屏幕包括第一屏幕和第二屏幕;
对投屏图像进行预处理,包括:
将第一屏幕的投屏图像和第二屏幕的投屏图像进行拼接,得到拼接图像;
对拼接图像进行渲染。
在一种可能的设计中,目标屏幕包括第一屏幕和第二屏幕,
对投屏图像进行预处理,包括:
按照第一屏幕的尺寸,对投屏图像进行渲染;
向目标屏幕发送投屏内容对应的投屏图像,以使得目标屏幕显示投屏图像,包括:向第一屏幕发送渲染后的投屏图像,以使得第一屏幕显示尺寸为第一屏幕尺寸的投屏图像;向第二屏幕发送渲染后的投屏图像,以使得第二屏幕显示经拉伸或缩放后尺寸为第二屏幕尺寸的投屏图像。
在一种可能的设计中,通信模块,还用于从第一屏幕接收停止投屏的通知消息;
处理模块,还用于调整与第二屏幕之间的图像传输通路以及与音频模块之间的音频传输通路。
在一种可能的设计中,第一界面还包括目标应用的标识;
接收用于指示将第一设备上目标应用的投屏内容投屏到目标屏幕的操作,包括:
接收用户将虚拟光标由目标应用的标识拖动到目标屏幕的标识。
在一种可能的设计中,输入模块,还用于在显示第一界面之前,接收用户选择投屏内容的操作;接收用户输入的第一操作;
显示第一界面,包括:响应于第一操作,显示第一界面;
接收用于指示将第一设备上的投屏内容投屏到目标屏幕的操作,包括:接收用于选择目标屏幕的操作。
在一种可能的设计中,预处理包括解码、渲染。
第三方面,本申请实施例提供一种计算机程序产品,当计算机程序产品在电子设备上运行时,使得电子设备执行任一方面或任一方面中任意一种实施方式的方法。
第四方面,本申请实施例提供一种电路***,电路***包括处理电路,处理电路被配置为执行任一方面或任一方面中任意一种实施方式的方法。
第五方面,本申请实施例提供一种芯片***,包括至少一个处理器和至少一个接口电路,至少一个接口电路用于执行收发功能,并将指令发送给至少一个处理器,当至少一个处理器执行指令时,至少一个处理器执行任一方面或任一方面中任意一种实施方式的方法。
第六方面,提供一种计算机可读存储介质。计算机可读存储介质存储有计算机程序(也可称为指令或代码),当该计算机程序被电子设备执行时,使得电子设备执行任一方面或任一方面中任意一种实施方式的方法。
第七方面,提供一种多屏多设备交互方法、电子设备及***。该***包括上述任一方面或任一方面中任意一种实施方式的第一设备和第二设备。
图1A为本申请实施例提供的***架构示意图;
图1B为本申请实施例提供的汽车上的屏幕的示意图;
图2A为本申请实施例提供的电子设备的结构示意图;
图2B为本申请实施例提供的电子设备的结构示意图;
图3-图5为本申请实施例提供的手机与汽车建立连接的过程的示意图;
图6为本申请实施例提供的界面示意图;
图7为本申请实施例提供的多屏多设备交互方法的流程示意图;
图8为本申请实施例提供的场景示意图;
图9-图10为本申请实施例提供的界面示意图;
图11A-图11B为本申请实施例提供的界面示意图;
图12、图13为本申请实施例提供的多屏多设备交互方法的流程示意图;
图14-图18为本申请实施例提供的场景示意图;
图19为本申请实施例提供的界面示意图;
图20为本申请实施例提供的多屏多设备交互方法的流程示意图;
图21为本申请实施例提供的界面示意图;
图22为本申请实施例提供的场景示意图;
图23为本申请实施例提供的界面示意图;
图24为本申请实施例提供的场景示意图;
图25为本申请实施例提供的界面示意图;
图26为本申请实施例提供的装置的结构示意图;
图27为本申请实施例提供的芯片***的结构示意图。
下面结合附图对本申请实施例提供的一种显示用户界面的方法及车机进行详细地描述。
本申请的说明书以及附图中的术语“第一”和“第二”等是用于区别不同的对象,或者用于区别对同一对象的不同处理,而不是用于描述对象的特定顺序。
此外,本申请的描述中所提到的术语“包括”和“具有”以及它们的任何变形,意图在于覆盖不排他的包含。例如包含了一系列步骤或单元的过程、方法、***、产品或设备没有限定于已列出的步骤或单元,而是可选地还包括其他没有列出的步骤或单元,或可选地还包括对于这些过程、方法、产品或设备固有的其它步骤或单元。
需要说明的是,本申请实施例中,“示例性的”或者“例如”等词用于表示作例子、例证或说明。本申请实施例中被描述为“示例性的”或者“例如”的任何实施例或设计方案不应被解释为比其它实施例或设计方案更优选或更具优势。确切而言,使用“示例性的”或者“例如”等词旨在以具体方式呈现相关概念。
在本申请的描述中,除非另有说明,“多个”的含义是指两个或两个以上。
本文中的“和/或”仅仅是一种描述关联对象的关联关系,表示可以存在三种关系,例如,A和/或B,可以表示:单独存在A,同时存在A和B,单独存在B这三种情况。
本本申请实施例的技术方案适用于车载、全屋、办公场景等跨设备多屏幕的分布式***中。如下,主要以车载场景为例进行说明,但这并不构成对适用场景的限制。
图1A为本申请实施例提供的一种车载场景的分布式***架构示意图。车载***包括车机,车机可以配备多个屏幕。其中,汽车的中控台配备中控显示屏或中控屏。另外,有一些汽车的座舱内逐步数字化显示,座舱内设置有多块显示屏,用于显示数字仪表盘、车载娱乐***等内容。如图1A所示,座舱内设置有多块显示屏,如中控台上的中控屏11,仪表盘上的仪表屏12。可以理解的是,座舱内还可以包括其他显示屏,如副驾驶位上的乘客(也称为前排乘客)面前的显示屏(可简称为副驾屏13),左侧后排乘客面前的显示屏以及右侧后排乘客面前的显示屏等。
可选的,中控屏一般用于显示影音娱乐***对应的显示内容,如图1B中(a)所示,中控屏可以用于显示车机中安装的应用程序,用户能够通过在中控屏上的操作实现使用应用程序。如车
机检测到用户点击中控屏上显示的地图图标1011的操作,启动地图应用,并在中控屏上显示地图界面,实现为用户提供导航服务。
可选的,仪表屏一般用于显示显示数字仪表盘对应的显示内容。如图1B中(b)所示,仪表屏可以用于显示车速1021、续航里程1022、车辆图标1023等车辆信息。用户能够通过仪表屏上的显示内容快速获知车辆状态。
可选的,图1A所示***还可以包括其他设备。比如,具有屏幕的手机、平板等设备。以上述***包括手机为例,在一些实施例中,手机中可以配置跨设备多屏管理模块,通过跨设备多屏管理模块,手机可以显示多屏管理界面(也可以称为虚拟跨多屏界面)。用户可以通过多屏管理界面(第一界面的一个示例)管理手机以及车载***的多个屏幕,使得手机与车载***可以实现共享屏幕,满足用户的视听需求。比如,用户可以在手机的多屏管理界面输入相应操作,以控制手机将正在播放的视频投屏到后排左侧乘客面前的显示屏以及副驾屏,以满足副驾乘客以及后排左侧乘客的视频观看需求。
本申请实施例提供的跨设备的控制方法,可以应用于电子设备100中。示例性的,图2A示出了电子设备100的结构示意图。
该电子设备100包括至少一个处理器201,存储器202以及至少一个收发器203。
处理器201可以是一个通用中央处理器(central processing unit,CPU),微处理器,特定应用集成电路(application-specific integrated circuit,ASIC),或一个或多个用于控制本申请方案程序执行的集成电路。
上述各组件之间可包括通信线路,用于在上述组件之间传送信息。
收发器203,用于与其他设备通信。在本申请实施例中,收发器可以是模块、电路、总线、接口或者其它能实现通信功能的装置,用于与其他设备通信。可选的,该收发器可以为独立设置的发送器,该发送器可用于向其他设备发送信息,该收发器也可以为独立设置的接收器,用于从其他设备接收信息。该收发器也可以是将发送、接收信息功能集成在一起的部件,本申请实施例对收发器的具体实现不做限制。
存储器202可以是只读存储器(read-only memory,ROM)或可存储静态信息和指令的其他类型的静态存储设备,随机存取存储器(random access memory,RAM)或者可存储信息和指令的其他类型的动态存储设备,也可以是电可擦可编程只读存储器(electrically erasable programmable read-only memory,EEPROM)、只读光盘(compact disc read-only memory,CD-ROM)或其他光盘存储、光碟存储(包括压缩光碟、激光碟、光碟、数字通用光碟、蓝光光碟等)、磁盘存储介质或者其他磁存储设备、或者能够用于携带或存储具有指令或数据结构形式的期望的程序代码并能够由计算机存取的任何其他介质,但不限于此。存储器可以是独立存在,通过通信线路与处理器相连接。存储器也可以和处理器集成在一起。
其中,存储器202用于存储用于实现本申请方案的计算机执行指令,并由处理器201来控制执行。处理器201用于执行存储器202中存储的计算机执行指令,从而实现本申请下述实施例提供的各个方法。
可选的,本申请实施例中的计算机执行指令也可以称之为应用程序代码、指令、计算机程序或者其它名称,本申请实施例对此不作具体限定。
在具体实现中,作为一种实施例,处理器201可以包括一个或多个CPU,例如CPU0和CPU1。
在具体实现中,作为一种实施例,电子设备100可以包括多个处理器。这些处理器中的每一个可以是一个单核(single-CPU)处理器,也可以是一个多核(multi-CPU)处理器。这里的处理器可以指一个或多个设备、电路、和/或用于处理数据(例如计算机程序指令)的处理核。
本申请实施例中,电子设备100还包括显示屏204以及音频模块205。
显示屏204,用于显示图像,视频等。
在一些示例中,显示屏204包括显示面板。显示面板可以采用液晶显示屏(liquid crystal display,LCD),例如采用有机发光二极管(organic light-emitting diode,OLED),有源矩阵有机发光二极体或主动矩阵有机发光二极体(active-matrix organic light emitting diode的,AMOLED),柔性发光二极管(flex light-emitting diode,FLED),Mini-led,Micro-led,Micro-oled,量子点发光二极管(quantum
dot light emitting diodes,QLED)等生产制造。在一些实施例中,电子设备100可以包括1个或N个显示屏204,N为大于1的正整数。
在一些实施例中,若电子设备100为车机,则显示屏204包括中控台显示屏(即中控屏)、仪表盘显示屏(即仪表屏)。
音频模块205,用于将数字音频信息转换成模拟音频信号输出,也用于将模拟音频输入转换为数字音频信号。音频模块205还可以用于对音频信号编码和解码。在一些实施例中,音频模块205可以设置于处理器201中,或将音频模块205的部分功能模块设置于处理器201中。电子设备100可以通过音频模块205等实现音频功能。例如音乐播放,录音等。音频模块205可以包括扬声器,受话器,麦克风,耳机接口,以及应用处理器等实现音频功能。
可以理解的是,本申请实施例示意的结构并不构成对电子设备100的具体限定。在本申请另一些实施例中,电子设备100可以包括比图示更多或更少的部件,或者组合某些部件,或者拆分某些部件,或者不同的部件布置。图示的部件可以以硬件,软件或软件和硬件的组合实现。
在一些实施例中,车机可以通过USB接口与移动终端建立有线连接。或者,车机通过无线通信模块与移动终端建立无线连接,采用的无线连接方式例如可以是蓝牙(包括低功耗蓝牙、经典蓝牙等)、WLAN、NFC,IR等,本申请实施例对无线连接方式不做限定。
如图2B示出了分布式***中电子设备的另一种结构示例。以用户将第一设备的第三屏幕上的第一应用投屏到第二设备的第二屏幕,将第一设备的第三屏幕上的第二应用投屏到第二设备的屏幕(比如第一屏幕、第二屏幕、第二附属屏幕)为例,第一设备还可称为投屏的源设备或主控设备,第二设备还可称为目标设备,第二设备的第一屏幕、第二屏幕、第二附属屏幕可称为目标屏幕。
可选的,第一设备可以将不同应用的内容投屏到不同设备或相同设备的不同屏幕上。比如,第一设备将应用1的内容投屏到设备1的屏幕1上,将应用2的内容投屏到设备2的屏幕2上。再比如,第一设备将应用1的内容投屏到设备1的屏幕1上,将应用2的内容投屏到设备1的屏幕2上。或者,第一设备可以将不同应用的内容投屏到某个设备的同一屏幕上。比如,第一设备将应用1的内容投屏到设备1的屏幕1上,将应用2的内容也投屏到设备1的屏幕1上。设备1可以小窗方式显示应用1的内容和应用2的内容。
可选的,第一设备可包括如下功能模块:
多屏管理界面111,可以呈现在第一设备的第三屏幕上。多屏管理界面可呈现第二设备上的多个屏。用户可以通过多屏管理界面111,将第一设备上的应用程序或桌面等投屏到第二设备的屏幕上。
以第一设备为手机为例,手机屏幕可以显示多屏管理界面111或多屏管理界面101。示例性的,多屏管理界面111可以按照屏幕为粒度进行管理,多屏管理界面101可以按照应用为粒度进行管理。示例性的,多屏管理界面111可以是如图5的(b)所示的界面20(第一界面的一个示例)。界面20可包括一个或多个应用的图标,比如包括视频、设置、录音机等应用的图标。界面20还可以包括一个或多个可用屏幕的标识,比如汽车(第二设备的一个示例)的仪表屏、中控屏、副驾屏等虚拟可用屏幕的图标。
用户可以将想要投屏显示的应用拖拽到相应虚拟屏幕。以第一设备的第一应用为手机上的视频应用为例,比如,如图5的(c),检测到用户长按视频应用,手机可以生成视频应用对应的虚拟图标302’(或者虚拟光标),用户可以将虚拟图标302’拖拽至中控屏的图标303所在区域。响应于用户的拖拽释放操作,手机可以将视频应用的内容投屏到汽车的中控屏上。
在一些示例中,用户可以将多屏管理界面111投屏在第二设备的屏幕上,用户可通过第二设备与多屏管理界面111进行交互,使得用户可以方便快捷的执行投屏操作。
如图2B,第一设备的第一应用的内容(比如视频)投屏到第二设备的第二屏幕上显示之后,第一设备上的第一应用可以在后台运行。意味着,第一设备相当于总控设备,把第一应用的界面显示在第二设备上,而第一设备可以在后台运行第一应用。
多窗管理模块120,用于管理跨设备的多个屏幕的多个窗口。类似的,第一设备可以控制第二设备显示第二应用的界面,并在第一设备的后台运行第二应用。
示例性的,有时用户需要在跨设备的屏幕之间切换应用的界面,比如,中控屏上以小窗口显示导航应用的界面,司机将窗口中的导航应用界面从中控屏切换到副驾屏,副驾屏可以在窗口中显示导航应用界面,由副驾屏前的乘客设置导航地址,设置完成之后,副驾屏前的乘客将导航应用的界面切换回中控屏,如此,司机可通过中控屏观看导航界面。
多屏管理模块130,用于管理跨设备的多个屏幕。还用于从目标屏幕连接的多个设备中选择目标设备,以及确定对目标设备的算力调度策略。
示例性的,假设手机将应用1的界面(比如视频)投屏到全屋的显示屏1显示,显示屏1与设备1和设备2之间均存在连接,则手机的多屏管理模块130会根据选择策略,选择设备1或设备2对投屏内容(视频流)进行处理。假设由设备1对视频流进行处理,设备1可以将视频流的处理结果发送给显示屏1,由显示屏1播放应用1的视频。源设备(比如手机)选择目标设备(比如设备1)的策略将在下文给出具体介绍。
再示例性的,手机还可以对上述选择的设备1进行算力辅助。具体的,手机可以提前对投屏内容(比如视频流)进行部分处理操作。如此,可以降低设备1对投屏内容的处理工作量,节约设备1的算力,提升整个投屏流程的效率。
可选的,多屏管理模块130,还可用于确定用于播放投屏内容相应音频(比如视频的声音)的音频模块,以及源设备到音频模块之间的通信通路(可简称为音频传输通路)。
多屏管理模块130可以通过通信模块160,确认音频智选模块152选择的音频传输通路是否正常,并确定音频传输通路。
在一些示例中,多屏管理模块130可以确定音频和视频共享相同的通路,即通过相同通信通路传输投屏视频和音频。
在一些示例中,多屏同看场景中,同看屏(比如车机的三排左屏)的音频可以来自目标屏设备(比如车机副驾屏连接的芯片A),或直接来自源设备(比如手机)。
同看屏,指的是不需要额外处理,包括同样投屏内容的多个屏幕。示例性的,如图14,副驾屏和三排左屏均显示投屏的视频1,则副驾屏和三排左屏为同看屏。其中,副驾屏、三排左屏中显示的视频1可以是全屏显示,也可以是在窗口中显示。
在一些示例中,若目标设备音路不够,不支持接收或处理音频,则源设备可以将音画(音频、投屏视频)分离,且不再向目标设备发送音频数据,而是直接向目标耳机下发音频数据,以使得目标耳机能播放该音频。或者,源设备将音画进行分离之后,通过中间设备(比如第三设备)将音频数据发送给目标耳机。
可选的,多屏管理模块130,还可用于确定源设备到目标屏幕之间的通信通路(可简称为视频通路)。
源设备通过通信模块确认源设备与目标设备之间的目标视频通路,例如USB、HDMI、WiFi、Sparklink、车载以太网、视频线、全屋内总线等。源设备通过目标视频通路向目标设备发送视频流,再由目标设备向目标屏幕发送视频流。
如此,源设备可以智能的确定此次投屏的音频传输通路、视频通路,有助于确保投屏效果。
设备账户认证模块140,用于对其他设备进行认证授权,认证通过的设备可与第一设备建立连接。
跨设备投屏模块150,用于对投屏流程进行控制。
预解码和预渲染模块151,用于对投屏内容进行预处理。预处理可包括但不限于:预先解码和预先渲染,以便降低第二设备的算力负担,降低投屏所需的用时,提升投屏效率。
其中,解码,可以是从服务器拉取压缩的媒体流,并将压缩的媒体流转换成可以播放格式的过程。
音频智选模块152,用于选择用于播放投屏音频的音频模块以及相应的音频传输通路。
通信模块160,用于与其他设备进行通信。
可选的,如图2B,第二设备包括如下模块:第二屏幕、第一屏幕、第二附属屏幕、多窗管理模块120、多屏管理模块130、设备账户认证模块140、音频处理模块153、跨设备投屏快显模块154、通信模块160以及指定屏信息处理模块155。
在一些示例中,第一屏幕、第二屏幕可以是第二设备自身配备的屏幕,第二附属屏幕可以是第二设备外接的屏幕。例如,第二设备是车机时,第二附属屏幕可以是车机的后排外接平板。
第二屏幕、第一屏幕、第二附属屏幕、多窗管理模块120、多屏管理模块130、设备账户认证模块140、通信模块160等模块的具体实现可参见第一设备相关模块的介绍。
指定屏信息处理模块155,用于对投屏画面(比如视频帧)进行处理。处理包括但不限于切分、超分处理。关于切分、超分的具体实现可参见下述实施例。
跨设备投屏快显模块154,用于在第一设备已对投屏画面进行预处理的情况下,对来自第一设备的投屏画面的预处理结果进行进一步的处理。比如,由第一设备对投屏画面进行预解码、预渲染,由第二设备的跨设备投屏快显模块154对投屏画面进行合成。之后,第二设备可以将投屏画面的合成结果发送给诸如第二屏幕,由第二屏幕播放该投屏画面。
图2B以分布式***包括第一设备和第二设备为例进行说明,应理解,本申请实施例的技术方案还可以适用于包括更多设备、更多屏幕以及其他组件的分布式***。比如,第二设备可以包括更多屏幕,再比如,分布式***还可以包括第三设备以及第三设备的屏幕和音频模块,再比如,分布式***包括更多设备的更多屏幕和音频模块。
本申请实施例的多个设备可以是多个终端设备。例如,第一设备可以是手机,第二设备可以是车机。第一设备是平板电脑,第二设备可以是双屏个人电脑(personal computer,PC)或者双屏的会议室显示设备。第一设备可以是手机,第二设备可以是全屋的多个智慧大屏。第一设备可以是全屋的中控模块,第二设备可以包括受全屋服务器控制的第一屏幕等。
或者,多个设备可以包括云侧设备和端侧设备。例如,第一设备可以是云端服务器的总控模块,第二设备可以是车机。服务器可以提供多屏管理界面,用于管理***中跨设备的多屏。再比如,第一设备可以是车机,车机可以安装车家互动界面,第二设备可以包括受全屋服务器控制的第一屏幕、第二屏幕和第二附属屏幕。用户可以通过车机安装的车家互动界面控制家中的第一屏幕、第二屏幕等。再比如,第一设备可以是PC,PC上安装有云端车控应用,第二设备可以是车机。再比如,全屋服务器可以提供多屏管理界面,用于控制全屋中的多个屏幕。多个设备还可以是其他的设备组合方式,以实现端-云、端-云-端的跨设备交互。
在一些示例中,第一设备和第二设备还可以是同一设备中的不同逻辑模块,该设备配备有多个屏幕。
或者,多个设备中的一个设备是Dongle(无屏解码设备)等,例如第一设备是Dongle,第二设备是车机或大屏;或者第一设备是手机或遥控设备,第二设备是Dongle,第三设备是多屏车机等。
本申请实施例对多个设备的设备形态不做限制。多个设备的设备形态、结构可以各不相同,或者,多个设备中可以有相同形态的设备。
在一些场景中,比如云端控制、全屋总控等场景中,上述第一设备、第二设备中的模块还可以有其他部署方式。示例性的,本申请实施例的第一设备也可控制第三设备向第二设备投屏。此种情况下,多屏管理界面部署在第一设备中,多窗管理模块120、多屏管理模块130、设备账户认证模块140、跨设备投屏模块150、音频智选模块152、预解码和预渲染模块151以及通信模块160等其他模块部署在第三设备。
综上,本申请实施例可应用于更多不限于此的同类型场景应用。
如下,以移动终端为手机为例进行说明。
在本申请的一个具体实施例中,车机和手机可以通过第一应用(例如HiCar应用)建立通信连接。示例性的,如图3中(1)所示,为车机显示的主菜单701。在主菜单中显示有Hicar应用图标702,以及其他应用图标。响应于用户操作图标702,车机可以发送连接请求。手机接收到车机发送的连接请求后,可以显示如图3中(2)所示的界面704。界面704中显示提示框703,用于提示用户选择是否连接该车机。响应于用户在提示框703中点击“连接”按钮,手机向车机回复同意连接的响应。车机在接收手机发送的同意连接的响应之后,可以显示如图3中(3)所示的连接码。与此同时,手机显示如图3中(4)所示的界面706,提示用户输入车机显示的连接码。在用户输入连接码后,手机将用户输入的连接码发送给车机,车机进行验证,验证成功后建立通
信连接。可选的,在建立通信连接的过程中,车机可以显示如图4中(1)所示的界面801,手机可以显示如图4中(2)所示的界面802,分别用于提示用户正在建立连接。可选的,在车机和手机建立通信连接后,手机可以显示如图4中(3)所示的提示界面804,提示已连接到车机,而后用户可以手动关闭,或者手机也可以在预设时间段后自动关闭提示界面804。
另一些示例中,若车机与手机之前建立过通信连接,那么,车机和手机也可以直接进行蓝牙配对,而无需用户输入连接码。例如,用户可以在手机侧开启Hicar应用,并选择连接到曾经连接过的车机,那么手机和车机直接建立通信连接。
又一些示例中,若车机与手机之前建立过通信连接,那么,当手机和车机都开启蓝牙,且二者的距离位于预设范围内,则无需用户操作,手机和车机也可以自动建立通信连接。
本申请实施例中,在车机和手机建立通信连接后,手机可以从车机获取车辆中可用的多个显示屏的信息,并根据多个显示屏的信息,显示多屏管理界面。用户可以通过多屏管理界面,将手机上的显示内容投屏到车内的一个或多个目标设备的多个显示屏,以实现跨设备之间的屏幕共享。如下,分多个场景对本申请实施例的技术方案进行介绍:
场景一:单个目标设备的场景
示例性的,如图5的(a)所示,手机可以显示有界面10,该界面10可以包括多个标签,(例如,标签102,标签103,标签103)。当某一标签被选中时,手机可以显示相应界面。比如,如图5中的(a)所示,响应于检测到用户点击桌面标签103的操作,手机可以显示桌面上一个或多个应用的图标(比如视频、设置、录音机应用的图标)。当应用的图标较多时,界面10还可以显示切换控件105。用户可以向左或向右滑动,使得手机在其他界面上显示更多应用的图标。
再比如,如图5的(b),响应于检测到用户点击“所有”标签102的操作,手机可以从车机获取车辆上可用的多个屏幕的信息,并显示界面20(图2B所示多屏管理界面111的一个示例)。界面20可以包括车辆上多个可用屏幕的图标(比如仪表屏、中控屏等多个屏幕的图标)以及手机上一个或多个应用的图标。其中,车辆上多个可用屏幕的图标可以显示在界面20的区域201内。
再比如,响应于检测到用户点击虚拟多屏标签104的操作,手机可以显示车辆内多个可用屏幕的图标。
在一些场景,为了在手机界面上更加清晰的观看车辆上多个屏幕的图标,如图5的(c),用户可以将手机横屏放置。响应于用户调整手机姿态(比如将手机由竖屏显示调整为横屏显示)的操作,手机可以自适应调整界面的显示内容。比如,如图5的(c),手机可以显示界面30,并在界面30的区域301内显示车辆上多个可用屏幕的图标。
之后,如图5的(c),用户可以向中控屏图标303的方向拖拽视频应用的图标302。作为一种示例,拖拽过程中,视频应用的图标302不随拖拽位置的改变发生位移。手机可以生成并显示视频应用的虚拟图标302’(或者虚拟光标),虚拟图标302’可以随用户的拖拽发生位移。用户将虚拟图标302’移至中控屏图标303所在区域之后,可以将手指抬离屏幕,手机不再显示虚拟图标302’。手机检测到用户将虚拟图标302’移至中控屏图标303所在区域并释放后,确定用户想要在车辆的中控屏上投屏显示视频应用的内容,则手机可以根据用户的投屏意图,将手机的视频应用投屏到车辆的中控屏。并且,如图5的(d)所示,手机可以在中控屏的图标303上显示视频应用的标识,比如在中控屏的图标303上显示文本“手机的视频应用”。再比如,手机可以在中控屏图标303上显示“APP Video from Jane’s phone”(Jane的手机上的视频应用)。
可选的,若中控屏是同看屏,则在中控屏被选择为投射屏幕时,中控屏图标303上可以显示诸如“APP sync-Video from Jane’s phone”的角标。其中,sync表示中控屏为同看屏。
如此,用户可以根据中控屏图标303上的视频应用标识,确定车辆的中控屏上正在投屏显示手机的视频应用的内容。
可选的,响应于用户将视频应用的虚拟图标302’拖拽至中控屏图标303所在区域之后,如图5的(d)所示,手机还可以在视频应用的图标302上显示角标,该角标用于表示正在其他设备的屏幕上投屏显示视频应用的内容。可选的,角标还可以是其他形式,本申请实施例对角标的具体呈现方式不做限制。比如,角标可以是文字“中控屏”,用于表示正在车辆的中控屏上投屏显示视频应用的内容。
需要说明的是,图5的(d)以中控屏图标303上显示的视频应用的标识为文字形式为例进行说明,在另一些实施例中,应用标识还可以以其他形式展示在中控屏图标303上,比如以应用图标的形式展示等,本申请实施例对应用标识在界面中的展示方式不做限制。
图6示出了中控屏的显示界面。可以理解,车机从手机接收视频应用对应的投屏内容之后,可以对投屏内容进行展示。比如,车机从手机接收待播放视频的码流,并根据视频的码流,播放如图6所示的视频50。
在一些实施例中,当停止投屏时,相应屏幕图标中不再显示应用的标识。比如,当停止投屏时,如图5的(d)所示的中控屏图标303中不再显示“手机的视频应用”。
上述技术方案中,用户可以通过多屏管理界面30(图2B所示多屏管理界面111的一个示例),将需要投屏的应用的图标拖拽到任意可用屏幕的图标上,以控制手机将手机应用的内容投屏到相应屏幕,满足在多屏场景中用户通过任意可用屏幕展示目标内容的视听需求。
上述主要以用户拖拽虚拟图标,以将相应应用(比如视频应用)的图标拖拽到相应屏幕(比如中控屏)为例,在另一些实施例中,用户还可以通过其他操作指示手机将手机应用的内容投屏到其他设备的屏幕上。比如,在图5的(b)所示的界面20中显示光标,用户可以将光标从应用1(比如视频应用)的图标位置拖拽到车机中控屏的图标位置。可选的,若用户想要将整个桌面投屏在其他设备的屏幕上显示,用户可以在应用图标之外的区域拖拽光标,或在界面20的某个指定区域拖拽光标。
需要说明的是,上述主要以用户点击如图5的(b)所示标签102,调出车辆上的多屏的图标为例,在另一些实施例中,还可以采用其他方式调出车辆上多屏的图标,本申请实施例对此不做限制。
图7示出了本申请实施例的跨设备的设备控制方法的流程示例。如图7,该方法包括如下步骤:
S101、手机显示多屏管理界面。
示例性的,手机可以显示如图9的(a)所示多屏管理界面30,界面30包括一个或多个应用的图标(比如视频、设置、录音机等应用的图标),以及车辆上多个可用屏幕的图标(比如仪表屏、中控屏等屏幕的图标)。
S102、手机接收用户输入的投屏指令。
可以理解,用户可以通过上述多屏管理界面,将需要投屏的应用的图标拖拽至目标屏幕的图标上,以控制手机执行投屏操作。示例性的,如图9的(a),手机显示多屏管理界面30,检测到用户拖拽导航应用图标的意图时,手机可以生成虚拟光标304。用户手指可以沿着虚线所示轨迹滑动手指,虚拟光标304跟随用户手指向中控屏图标303的方向进行移动。当虚拟光标304被移至中控屏图标303所在区域(投屏指令1)时,用户手指可以抬离屏幕,手机不再显示虚拟光标304。
检测到用户将虚拟光标304移动至中控屏图标303的所在区域并释放之后,如图9的(b)所示,手机可以在中控屏图标303上显示导航应用的名称。可选的,手机还可以在导航应用的图标上显示角标,以表征正在其他设备上投屏显示导航应用的内容。
类似的,检测到用户拖拽视频应用图标302的意图时,手机可以生成虚拟光标,用户可以拖拽虚拟光标,使得虚拟光标由视频应用图标302的所在区域被移动到副驾屏图标的所在区域(投屏指令2)。检测到用户将虚拟光标移动至副驾屏图标的所在区域并释放之后,如图9的(b)所示,手机可以在副驾屏的图标上显示视频应用的名称。可选的,手机还可以在视频应用的图标302上显示角标,以表征正在其他设备上投屏显示视频应用的内容。
在一些实施例中,可以设置只有驻车时才允许从源设备向仪表屏、增强现实(augmented reality,AR)抬头显示(head up display,HUD)屏投影娱乐应用(例如视频),在行车时禁止从源设备向仪表屏、AR HUD屏投影娱乐应用,以提升行车安全。
S103、手机接收用户输入的音频选择指令。
示例性的,如图10的(a),用户可以长按中控屏的图标303,以触发手机弹出卡片305。卡片305可包括可供中控屏使用的音频模块。比如,包括车机-全车扬声器,车机蓝牙-耳机1,车机
蓝牙-耳机2。响应于用户勾选卡片305中的“车机-全车扬声器”选项,手机确定用户想要选择全车扬声器播放导航应用的音频。可选的,如图10的(b),手机可以在中控屏的图标303中显示“全车扬声器”的标识,用于表示当中控屏上显示导航应用的画面时,由全车扬声器播放导航应用匹配的音频。
一些实施例中,用户还可以选择与车机相连接的设备的音频设备播放音频。例如,用户可以选择通过诸如手机的耳机1播放视频应用的音频。如图10的(b),用户选择通过手机的耳机1播放视频应用的音频之后,手机可以在副驾屏的图标中显示“手机耳机1”的标识,用于表示当副驾屏上显示视频应用的画面时,由手机连接的耳机1播放与视频画面匹配的音频。
在一些实施例中,手机可以与车内的其他终端协同,以便判断用户为此次投屏内容选择的音频模块是否有效。例如,假设用户为视频应用选择的音频模块包括车机蓝牙-手机耳机1,则手机可以与车机通信,以便判断车机蓝牙-手机耳机1相关的音频模块是否能够正常工作。经判断,若车机具备蓝牙通路,车机的蓝牙模块能够正常工作,且手机也具备蓝牙通路,手机的蓝牙模块也能够正常工作,则手机可以初步确认用户选择的用于播放视频应用的音频的音频模块有效。
在一些示例中,如果用户选择的音频模块无效,比如车机的蓝牙通路出现故障,则手机可以提示用户重新选择此次投屏使用的音频模块。示例性的,如图11A的(a),检测到用户选择卡片305上的“车机蓝牙-耳机2”音频选项之后,手机可以与车机通信,判断相关音频模块是否能够正常工作。经判断,车机的蓝牙模块与耳机2的蓝牙模块之间的通路故障,车机无法通过蓝牙模块将音频数据传输给耳机2进行播放,即用户选择的音频模块无效。如图11A的(b),手机可以弹出提示窗306,用于提示用户重新选择音频模块。
在一些实施例中,手机在弹出卡片305之前,可以提前判断各音频模块是否可用。之后,手机可以在弹出的卡片305中仅显示可用的音频模块,以降低用户重新选择音频模块的概率。比如,经检测,手机确定车机蓝牙模块与耳机2之间的通路故障,则可以在图10的(a)所示卡片305中不显示“车机蓝牙-耳机2”这一音频选项,以免用户误选该故障音频模块。或者,如图11B所示,手机可以在卡片305中显示“车机蓝牙-耳机2”这一音频选项,但该音频选项被设置为不可选或禁用状态,用户不能选择该音频选项对应的音频模块。
S104、手机判断是否满足对投屏内容进行预处理的条件。若满足预处理条件,则执行如下S105,若不满足预处理条件,则执行S106-S113。
可选的,预处理条件包括如下任一项或多项条件:目标设备的负载大于或等于一定门限、源设备的负载小于或等于一定门限、目标设备和源设备之间的通信参数满足条件。可选的,目标设备与源设备之间的通信参数包括但不限于带宽、误码率等。通信参数满足条件,可以指带宽高于某个阈值(比如500Mbps以上),和/或误码率低于某个阈值。通过一系列通信参数以及相应阈值的设置,可使得目标设备与源设备之间满足分布式体验的基础保障要求。较高的通信带宽,可用于保证目标设备和源设备之间保证实时和稳定的通信体验。
源设备,指的是投屏内容的来源设备,目标设备,指的是接收并展示投屏内容的设备。以将手机中导航应用的内容投屏到车机的中控屏上为例,源设备即手机,目标设备即车机。
满足预处理条件,意味着,目标设备当前的处理能力较低,源设备当前的处理能力较高。因此,可以考虑让源设备提前对投屏内容进行预处理,如此,能够降低目标设备的工作量,目标设备可以无需或减少对投屏内容的处理,提升整个投屏流程的效率。
作为一种可能的实现方式,手机通过通信模块160与车机进行交互,获取车机的性能参数(比如负载状态)。多屏管理模块130可以根据车机的负载状态,判断车机是否需要算力互助。若车机需要算力互助,则手机可以对投屏内容进行预处理。反之,若车机不需要算力互助,则手机无需对投屏内容进行预处理。
可选的,在另一些实施例中,还可以以应用为粒度判断是否进行预处理。比如,考虑到导航应用的投屏内容的渲染工作量较小,对导航应用的投屏内容可以不进行预处理。反之,考虑到视频应用的投屏内容的渲染工作量较大,为了降低车机的渲染工作量,手机可以对视频应用的投屏内容进行预处理。
S105、手机对导航应用、视频应用的投屏内容进行预处理。
示例性的,假设车机的负载超出CPU门限值(例如60%),手机的负载较小,且手机和车机之间的带宽较高,误码率较低,则手机确定可以对车机进行算力辅助,手机可以对诸如导航应用、视频应用的投屏内容进行预处理。
可选的,对投屏内容的预处理可包括对投屏内容的提前解码和/或提前渲染。
S106、手机向车机的副驾屏发送视频应用的投屏内容。
作为一种可能的实现方式,手机向车机发送视频应用的投屏内容,由车机向副驾屏发送视频应用的投屏内容。
在一些示例中,在满足上述预处理条件时,视频应用的投屏内容是已被手机预解码、预渲染的投屏内容。此种情况下,车机从手机接收视频应用的投屏内容之后,可以对已解码、渲染的投屏内容进行进一步处理(比如合成),并将视频应用的投屏内容的处理结果发送到副驾屏。可以看出,由于手机已对投屏内容进行了预处理,因此,可以降低车机的处理任务量,在车机负载较高的情况下,能够有效降低车机因高负载导致的处理延迟或处理失败的概率。
或者,在一些示例中,在不满足上述预处理条件时,投屏内容是未被手机预解码、预渲染的投屏内容。此种情况下,车机从手机接收视频应用的投屏内容后,可以对投屏内容进行解码、渲染、合成等处理,并将投屏内容的处理结果发送到副驾屏。
示例性的,以手机不对投屏内容进行预处理为例,如图8的(a),手机可以从服务器获取远程视频文件,并通过通信模块向车机的通信模块发送该远程视频文件(即投屏内容)。车机接收视频文件之后,根据视频文件的编码格式,对视频文件进行解码,并按照目标屏幕(副驾屏)的尺寸,对视频文件包括的视频帧进行渲染、合成。之后,车机可以将视频文件的处理结果发送给副驾屏,由副驾屏显示相应视频帧。可选的,手机也可以将本地的视频文件投屏到车机的副驾屏进行显示。
再示例性的,以手机对投屏内容进行预处理为例,如图8的(b),手机获取视频文件,并对视频文件进行预解码、预渲染。之后,手机可以通过通信模块向车机的通信模块发送预处理后的视频文件。车机接收视频文件之后,执行合成等处理,并将视频文件的处理结果发送给副驾屏,由副驾屏显示视频文件的相应视频帧。可选的,车机还可以对视频文件进行切帧或超分处理。
作为一种可能的实现方式,手机发送视频应用的投屏内容的同时,还发送投屏内容对应的时间戳,以便于副驾屏显示的投屏画面(比如视频帧)与音频模块(比如用户选择的用于播放视频应用的音频的耳机1)播放的音频能够同步。
S107、手机向耳机1发送视频应用的音频。
作为一种可能的实现方式,手机向车机发送音频,车机通过蓝牙模块向耳机1发送视频应用的音频。
作为一种可能的实现方式,手机发送音频的同时,还发送音频对应的时间戳,以便于耳机1播放的音频与显示模块(比如用户选择的副驾屏)显示的投屏画面能够同步。
可选的,视频应用的投屏内容与音频可以携带在同一消息或不同消息中。
本申请实施例对S106和S107的执行顺序不做限制。
S108、车机的副驾屏播放视频应用的视频帧。
应理解,副驾屏获取视频应用的投屏内容之后,可以根据投屏内容,播放视频应用的视频帧。
作为一种可能的实现方式,副驾屏根据投屏内容对应的时间戳,在相应时间播放相应的视频帧,以便副驾屏显示的视频帧能够与音频模块(比如用户选择的耳机1)播放的音频同步。
作为一种可能的实现方式,副驾屏可以在副驾屏的目标区域播放投屏的视频帧。目标区域可以是副驾屏的一个窗口,或者,副驾屏可以在整个屏幕上显示视频帧。
S109、耳机1播放视频的声音。
耳机获取视频应用的音频之后,可以播放视频应用的音频。
作为一种可能的实现方式,耳机1根据音频对应的时间戳,在相应时间播放相应的音频,以便耳机1播放的音频能够与副驾屏显示的视频帧同步。
本申请实施例对S108和S109的执行顺序不做限制。
S110、手机向中控屏发送导航应用的投屏内容。
作为一种可能的实现方式,手机向车机发送导航应用的投屏内容,由车机向中控屏发送导航应用的投屏内容。
在一些示例中,在满足上述预处理条件时,投屏内容是已被手机预解码、预渲染的投屏内容。此种情况下,车机从手机接收导航应用的投屏内容之后,可以对已解码、渲染的投屏内容进行进一步处理(比如合成),并将导航应用的投屏内容的处理结果发送到中控屏。
或者,在一些示例中,在不满足上述预处理条件时,投屏内容是未被手机预解码、预渲染的投屏内容。此种情况下,车机从手机接收导航应用的投屏内容后,可以对投屏内容进行解码、渲染、合成等处理,并将投屏内容的处理结果发送到中控屏。
作为一种可能的实现方式,手机向车机发送投屏内容的同时,还发送投屏内容对应的时间戳,以便于中控屏显示的投屏画面与音频模块(比如用户选择的全车扬声器)播放的音频能够同步。
本申请实施例对S106和S110的执行顺序不做限制。
S111、手机向全车扬声器发送导航应用的音频。
作为一种可能的实现方式,手机向车机发送导航应用的音频,由车机向全车扬声器发送导航应用的音频。
作为一种可能的实现方式,手机发送导航应用的音频的同时,还发送音频对应的时间戳,以便于全车扬声器播放的导航音频与显示模块(比如用户选择的中控屏)显示的投屏画面(比如导航界面)能够同步。
本申请实施例对S110和S111的执行顺序不做限制。
S112、中控屏播放导航图像。
作为一种可能的实现方式,中控屏根据导航应用的投屏内容对应的时间戳,在相应时间播放相应的导航界面,以便中控屏显示的导航界面能够与音频模块(比如用户选择的全车扬声器)播放的音频同步。
S113、全车扬声器播放导航音频。
通过该方案,可实现投屏过程中的音画分离,比如通过车机的副驾屏显示视频应用的投屏内容,通过耳机1(并非车机上固定安装的扬声器)播放投屏内容对应的音频,以满足用户的视听需求。比如,在一些场景中,用户可以将想要观看的内容投屏在大屏设备(比如智慧屏)上,并通过手机耳机收听投屏内容对应的音频,以免音频打扰到其他用户。
场景二:多个目标设备的场景
上述主要以目标屏幕(中控屏、副驾屏)连接一个设备(车机)为例进行说明,在另一些实施例中,目标屏幕可能连接多个处理单元(比如芯片或设备)。此种情况下,手机可以从目标屏幕连接的多个处理单元中选择目标处理单元,由目标处理单元处理目标屏幕上待显示的投屏内容。示例性的,以目标屏幕包括车机上的三排左屏,三排左屏连接车机的芯片A、B为例,如图12,该方法可包括如下步骤:
S201、手机显示多屏管理界面。
示例性的,手机显示如图9的(a)所示的多屏管理界面30。界面30可包括一个或多个应用的图标(比如视频、设置、录音机等应用的图标),以及车辆上多个可用屏幕的图标(比如仪表屏、中控屏等屏幕的图标)。
S202、手机接收到用户将视频应用的内容投屏到三排左屏上的操作指令。
示例性的,用户可以将多屏管理界面中视频应用的图标拖拽到与三排左屏的图标碰撞,如此,手机可以确定用户想要将视频应用的内容投屏到车机的三排左屏显示。
S203、手机接收到用户选择通过耳机2播放视频应用的音频的操作指令。
S204、手机从车机获取芯片A的算力信息。
手机确定用户想要将视频应用的内容投屏到车机的三排左屏显示之后,可以查询三排左屏连接的设备/芯片。经查询,手机确定三排左屏连接了车机的芯片A、B,则手机可以获取芯片A、B的算力信息,以便根据芯片A、B的算力信息,选择性能更好的芯片来处理视频应用的投屏内容。
作为一种可能的实现方式,手机通过通信模块从芯片A获取芯片A的算力信息。
S205、手机从车机获取芯片B的算力信息。
S206、手机根据芯片A、B的算力信息,确定芯片A为目标芯片。
可选的,算力信息包括但不限于如下任一项或多项信息:CPU性能信息(比如负载)、GPU性能信息(比如负载)、I/O性能信息、连接的屏幕与其他屏幕是否存在共享的投屏应用。屏幕之间存在共享的投屏应用,指的是两个屏幕当前播放的投屏内容来自同一应用。手机可以根据上述一项或多项算力信息,确定芯片A、B的性能,并将性能更好的芯片作为用于处理投屏内容的目标芯片。
作为一种可能的实现方式,手机的多屏管理模块可以根据芯片A、B的算力信息,选择性能更好的芯片作为目标芯片。
示例性的,若芯片A的CPU性能、GPU性能、I/O性能均比芯片B高,则手机可以将芯片A作为目标芯片。
再示例性的,假设芯片A连接三排左屏以及三排右屏,芯片B仅连接三排左屏。假设三排右屏为三排左屏的同看屏,则考虑使用芯片A处理三排左屏对应的投屏内容以及三排右屏对应的投屏内容。
S207、手机向芯片A发送视频应用的投屏内容。
可选的,手机可以判断是否对视频应用的投屏内容进行预处理。在一些示例中,若芯片A的算力较差,则手机可以提前对视频应用的投屏内容进行预处理。比如,假设三排左屏的屏幕尺寸为1920*2048,解码、渲染1920*2048尺寸的图像需要占用芯片A25%的CPU算力和1/3的GPU算力,但芯片A的CPU负载已超过60%、GPU已支撑一定量渲染任务,不足以完成解码、渲染任务,则手机可以对投屏内容进行预处理,并将预处理之后的投屏内容发送给芯片A。如此,能够降低芯片A所需处理的任务数量。
在另一些示例中,若芯片的算力较强,则手机无需对视频应用的投屏内容进行预处理。
S208、手机向耳机2发送视频应用的音频。
本申请实施例对S207、S208之间的执行顺序不做限制。
S209、芯片A对视频应用的投屏内容进行处理。
在一些示例中,视频应用的投屏内容可以是未经过预处理的数据。此种情况下,芯片A需要对投屏内容进行解码、渲染、合成等处理。
在另一些示例中,视频应用的投屏内容可以是经过预处理的数据。比如,投屏内容是经过解码、渲染得到的投屏内容。此种情况下,芯片A可以在此基础上进行处理,无需另行执行解码、渲染。比如芯片A可以对已解码、渲染的投屏内容进行合成。可见,在执行预处理的情况下,可减少芯片A的任务处理数目,避免芯片A因负载较高导致的处理延迟,进而能够提升整个投屏流程的效率。
S210、芯片A向三排左屏发送视频应用的投屏内容。
芯片A对视频应用的投屏内容进行处理之后,向三排左屏发送处理后的投屏内容。
S211、三排左屏播放视频帧。
三排左屏接收视频应用的投屏内容之后,可以根据显示该投屏内容,比如播放视频帧。
S212、耳机2播放视频的声音。
本申请实施例对S211、S212之间的执行顺序不做限制。
通过该方案,手机可以根据用户选择的用于显示投屏内容的目标屏幕,选择用于处理投屏内容的目标处理组件(比如车机的芯片A),以提升处理投屏内容的效率,进而提升整个投屏操作的效率。
在一些实施例中,用户通过多屏管理界面选择的音频模块可能是无效的。此种情况下,手机可以提示用户重新选择用于播放音频的音频模块。示例性的,仍以图12为例,用户选择通过耳机2播放视频应用的音频,但目标芯片A与耳机2之间的通路可能存在故障。此种情况下,手机可以弹出诸如图11A的(b)所示提示窗306,用于提示用户重新选择可用的音频模块。
需要说明的是,上述以按照屏幕为粒度选择音频模块以及音频传输通路为例,在另一些实施例中,还可以按照应用为粒度选择音频模块以及音频传输通路。比如,用户可以选择将不涉及个人隐私的音频(比如导航音频)通过全车扬声器进行播放,将涉及个人隐私或不便于外放的音频
通过个人耳机进行播放。
此外,图12主要以目标屏幕连接不同的处理组件(芯片)为例进行说明,在另一些实施例中,目标屏幕还可以连接不同的设备。此种情况下,手机可以获取目标屏幕所连接多个设备的算力信息,并根据多个设备的算力信息,选择性能最好的设备作为目标设备。如此,通过选择性能最好的目标设备处理投屏内容,可提升处理效率,有助于提升整个投屏操作的效率。
场景三:多屏同看场景
一些场景中,用户可以通过多屏管理界面将桌面或应用的内容投射到多个屏。示例性的,用户可以将手机中的视频1投屏到车机的副驾屏和三排左屏上进行显示。在多屏同看场景中,同看的多屏可以分为目标屏和同看屏。在一些示例中,用户可以先将视频1投射到目标屏上进行显示,之后,将视频1投射到同看屏上进行显示。在另一些示例中,可以不区分目标屏和同看屏的投屏时机。
如图13,以手机为投屏的源设备,车机为目标设备,目标屏是车机的副驾屏,同看屏是车机的三排左屏为例,多屏同看场景中,本申请实施例的技术方案可包括如下步骤:
S301、手机显示多屏管理界面。
S302、用户通过操控多屏管理界面,选择将视频应用的内容投屏到副驾屏以及三排左屏上。
S303、用户通过操控多屏管理界面,选择副驾屏、三排左屏对应的音频传输通路。
示例性的,用户通过诸如图10的(a)所示的界面30,选择相应的音频传输通路。比如,选择车机蓝牙-耳机1用于播放副驾屏上投屏内容对应的音频,选择车机蓝牙-耳机2用于播放三排左屏上投屏内容对应的音频。
S304、手机对投屏画面进行预解码。
在一些实施例中,若需要手机对投屏画面进行预处理,则手机可以对投屏画面进行预解码。
S305、手机将副驾屏的投屏画面和三排左屏的投屏画面进行拼接,对拼接画面进行渲染。
在一些实施例中,若需要手机对投屏画面进行预处理,如图14所示,手机可以按照副驾屏以及三排左屏的尺寸,将副驾屏的画面(视频1的视频帧)和三排左屏的画面(视频1的视频帧)拼接为一个整体画面,对于拼接两个画面产生的缝隙,可以进行填充。手机可以对拼接画面进行渲染,并将渲染结果发送给车机。
手机将副驾屏的画面和三排左屏的画面拼接为一个画面,并对拼接画面进行渲染的过程,可称为拼接渲染或合并渲染。
在另一些实施例中,若无需源设备(比如手机)进行预处理,则上述拼接渲染(S304)、预解码(S305)也可以由目标设备(比如车机)执行。
S306、手机将拼接画面的处理结果发送给车机。
可选的,在手机对车机进行算力辅助的情况下,该拼接画面的处理结果是已经过手机预解码、预渲染的投屏画面信息。
或者,可选的,在手机未对车机进行算力辅助的情况下,该拼接画面的处理结果是未经过手机预解码、预渲染的投屏画面信息。
S307、车机将拼接画面拆分为副驾屏的投屏画面和三排左屏的投屏画面。
示例性的,车机从手机接收拼接画面的渲染结果之后,如图14,车机可以根据副驾屏和三排左屏的尺寸大小,将拼接画面切分为副驾屏的画面和三排左屏的画面。
S308、车机将副驾屏的投屏画面发送给副驾屏。
S309、副驾屏显示投屏画面。
S310、车机将三排左屏的投屏画面发送给三排左屏。
S311、三排左屏显示投屏画面。
S312、手机控制播放副驾屏、三排左屏对应的音频。
示例性的,假设用户选择通过耳机1播放副驾屏上投屏画面对应的音频,通过耳机2播放三排左屏上投屏画面对应的音频,则手机可以通过相应的音频传输通路,将音频分别发送给耳机1、耳机2。如此,如图15,用户A可以在观看副驾屏上的投屏画面(比如视频)的同时,使用耳机1收听相应音频。用户B可以在观看三排左屏上的投屏画面的同时,使用耳机2收听相应音频。
示例性的,假设耳机1、耳机2是手机连接的耳机,如图16,芯片A与副驾屏连接,芯片B与三排左屏连接,手机可以通过手机蓝牙模块(路径1)向耳机1发送音频。手机可以通过手机蓝牙模块(路径2)向耳机2发送音频。
再示例性的,假设耳机1、耳机2是车机连接的耳机,如图17,手机可以通过手机蓝牙模块向车机芯片A的蓝牙模块发送音频,再由车机芯片A的蓝牙模块向耳机1、耳机2发送音频。手机还可以通过其他音频传输通路向耳机1、2发送音频,本申请实施例并不限制用于音频传输的具体音频传输通路。比如,手机向芯片B发送音频,由芯片B向耳机2发送音频。再比如,手机向芯片A发送音频,芯片A向芯片B发送音频,由芯片B向耳机2发送音频。
通过该方案,可实现投屏过程中的音画分离,以满足不同用户的不同视听需求。
上述以多屏同看场景中,设备对多个屏的投屏画面进行拼接,进行拼接渲染为例,在另一些实施例中,还可以采用其他渲染方式。比如,如图18,源设备手机可以按照目标屏(比如副驾屏)的尺寸进行渲染,并将渲染结果发送给车机。车机接收渲染结果之后,可以按照同看屏(比如三排左屏)的尺寸,执行超分算法,以将画面尺寸拉伸或缩放到三排左屏的尺寸。之后,车机可以指示三排左屏、副驾屏显示相同的投屏画面。其中,三排左屏显示的投屏画面是经过超分处理的画面。
可选的,超分算法可以是线性插值超分算法,也可以是经过学习和优化之后的超分插值算法。本申请实施例对超分算法的具体实现不做限定。
可选的,跨设备进行渲染的目标设备(比如车机),在部分场景有可能外挂Dongle,Dongle可以通过HDMI接口连接到目标屏。此种场景中,跨设备投屏的基本流程可以和上述实施例类似,外挂Dongle可用于从源设备(比如手机)接收投屏内容,对投屏内容进行处理,并将处理结果发送给目标设备。目标设备根据投屏内容的处理结果,指示目标屏对投屏内容进行显示。如此,由Dongle提前对投屏内容执行部分处理操作,可降低目标设备的处理任务量,提升整个投屏流程的效率。
示例性的,源设备(比如手机)对投屏内容进行预渲染,并将投屏内容的渲染结果发送给目标设备外挂的Dongle,由Dongle对投屏内容进行解码。Dongle对投屏内容进行解码之后,将投屏内容的解码结果发送给目标设备(比如车机),由目标设备将投屏内容切分或超分为目标屏的显示内容,并执行合成操作。目标设备执行合成操作之后,将合成结果发送给目标屏进行显示。
在一些实施例中,目标屏上可以投射投屏内容以及触控交互的控件(比如用户控制视频观看进度的进度条、暂停按钮等)。同看屏上可以仅投射投屏内容,不投射用于触控交互的控件。示例性的,如图19的(a),目标屏(副驾屏)上显示的投屏界面为视频1的播放界面60,该界面60包括用于控制视频进度的进度条以及暂停控件等。如图19的(b),同看屏(三排左屏)上显示的投屏界面同样为视频1的播放界面70,但界面70不包括视频进度条、视频暂停控件等控件。
作为一种可能的实现方式,用户可以通过操控目标屏,实现对目标屏以及同看屏中显示的投屏内容的控制。示例性的,副驾屏与三排左屏上均在投屏播放来自手机的视频1。副驾屏的乘客可点击诸如图19的(a)中的暂停播放控件。该操作信息可通过车机返回给手机,由手机控制副驾屏、三排左屏同时停止播放视频1。
在另一些实施例中,目标屏、同看屏上均可以投射投屏内容以及触控交互的控件(比如用户控制视频观看进度的进度条、暂停按钮等)。此种情况下,如果用户停止通过在目标屏上进行投屏,则手机可以控制同看屏切换为新的目标屏,并调整新目标屏的通信通路(包括投屏画面的传输通路以及投屏画面相应音频的传输通路)。仍以手机作为投屏的源设备,车机作为目标设备,副驾屏最为最初的目标屏,三排左屏作为最初的同看屏为例,如图20,该方法包括如下步骤:
S401、副驾屏停止投屏。
示例性的,以副驾屏和三排左屏同时投射视频1的内容为例,用户可以在副驾屏上退出视频1的播放界面,或通过其他操作停止在副驾屏上进行投屏。
S402、副驾屏将停止投屏的通知发送给手机。
可选的,副驾屏将停止投屏的通知发送给车机,由车机将副驾屏停止投屏的通知发送给手机。
S403、手机调整与三排左屏之间投屏画面对应的音频流的通信通路。
应理解,手机接收目标屏停止投屏的通知之后,可以通过多屏管理模块启动目标屏切换流程,将当前的同看屏切换为新的目标屏。作为一种可能的实现方式,手机的多屏管理模块可以协同通信模块,对手机到新目标屏之间的投屏画面(比如视频流)、投屏画面相应的音频流的通信通路进行调整,以实现更优的投屏体验。
示例性的,如图22的(a),用户选择通过耳机2播放三排左屏上投屏画面相应的音频。假设初始时,手机2与车机的芯片A连接,手机通过蓝牙模块向车机的芯片A发送音频流,再由芯片A向耳机2转发音频流,以便用户A能够通过耳机2收听副驾屏上投屏画面相应的音频。之后,如图22的(b),副驾屏停止投屏,手机可以建立与耳机2之间的连接,并可以直接由手机向耳机2发送音频。如此,能够缩短通信通路的长度,降低音频播放时延。
S404、手机调整与三排左屏之间投屏画面的通信通路。
示例性的,如图22的(a),初始时,手机将投屏画面发往车机的芯片B,由芯片B进行处理之后,将处理结果发送给三排左屏进行显示。后续,如图22的(b),副驾屏停止投屏之后,手机可以将投屏画面发往算力更好的芯片A,由芯片A进行处理之后,将处理结果发送给三排左屏进行显示。
S405、手机更新多屏管理界面。
示例性的,初始时,手机的多屏管理界面30a如图21的(a)所示,副驾屏为目标屏,三排左屏为同看屏。之后,副驾屏停止显示之后,手机将之前的同看屏(三排左屏)切换为目标屏,如图21的(b)的界面30b所示,三排左屏已被标记为目标屏。
S406、手机根据新的通信通路,传输投屏画面的信息以及相应音频流。
示例性的,如图22的(b),手机可以通过蓝牙模块直接向耳机2发送投屏画面对应的音频流。手机可以向车机的芯片A发送投屏画面的信息,由芯片A对投屏画面的信息进行处理,并将处理结果发送给三排左屏进行显示。
上述主要以多屏管理界面是按屏显示跨设备的多屏为例,在另一些实施例中,多屏管理界面还可以按应用为粒度显示跨设备的多屏。可选的,多屏管理界面可呈现一个或多个应用以及每个应用的共享数据和操作状态。应用的操作状态可分为激活(Active)或非Active。Active应用的界面被呈现在至少一个设备的至少一个屏幕上。
应用的共享数据包括但不限于:呈现Active应用的设备/屏幕的标识,该设备/屏幕对应的账户,Active应用的显示状态是桌面或应用全屏,Active应用对应的音频传输通路。
示例性的,图23显示了以应用为粒度的多屏管理界面30c(图2B中的多屏管理界面101的一个示例)。其中,导航应用被投射在中控屏上显示,中控屏登录的账号为主驾账号。中控屏上显示的导航界面是全屏显示的,导航界面对应的音频由手机发往车机的全车扬声器,由全车扬声器播放该音频。类似的,对图23中界面30c中其他应用的共享数据、操作状态不再赘述。图23中界面30c中各应用的共享数据、操作状态如表1所示。
表1
在一些实施例中,车机可以根据不同账户的权限,优先处理高权限用户的操作指令。示例性
的,假设副驾账户的权限高于三排左屏账户的权限。在一些场景中,若副驾屏和三排左屏同时播放投屏内容,副驾屏前用户在副驾屏上输入了操作指令A,三排左屏前用户在三排左屏上输入了操作指令B,则车机可以优先处理高权限的操作指令A。
场景四:多账户场景
该场景中,使用不同账户的用户可以共享***中的多屏。示例性的,如图24,主驾用户通过手机屏幕上的多屏管理界面,将桌面+导航投屏到中控屏上显示,将视频2投屏到副驾屏以及三排左屏上进行显示,将视频3投屏到三排左屏上进行显示。
其中,对于桌面以及导航应用的投屏处理,如图24,手机的芯片A对桌面以及导航应用的投屏画面(比如导航)进行预处理(比如解码、渲染),并将处理结果发送给车机的芯片B,由芯片B进行合成,并将合成结果发送给中控屏进行显示。导航的声音可由手机蓝牙模块直接发送蓝牙耳机2进行播放。
对于视频2的投屏处理,如图24,手机的芯片A对视频2进行预处理(比如解码、渲染),并将处理结果发送给车机的芯片B,由芯片B进行合成。芯片B将视频2的合成结果分别发送给副驾屏和三排左屏,由副驾屏和三排左屏分别显示视频2。可选的,每个屏幕(副驾屏、后排屏)可以生成相应窗口,并在相应窗口中播放视频2。比如,副驾屏在本地窗口1中播放视频2,三排左屏在本地窗口1中播放视频2。
如图24,视频2的声音通过手机传输给车机的芯片B,并经由全车播放器播放。
对于视频3的投屏处理,如图24,视频3通过手机到达车机的芯片C,由芯片C对视频3进行解码、渲染、合成,并将合成结果发送给三排左屏,由三排左屏进行显示。视频3的声音可以由手机通过Wi-Fi连接传输给芯片C,再通过芯片C传输给蓝牙耳机1进行播放。
本申请实施例还提供一种多屏多设备交互方法,用户可以在视频观看过程中,或者在其他场景中,启动投屏流程。示例性的,如图25的(a),用户在观看视频1的过程中,左滑唤出多屏管理卡片702。检测到用户点击卡片702,手机可以显示图25的(b)所示弹窗106。检测到用户点击弹窗106中的三排左屏和三排右屏之后,手机可确定用户想要将视频1投屏到三排左屏和三排右屏进行显示,则手机可以按照上述实施例的方案,向三排左屏和三排右屏发送视频1的投屏画面。
再示例性的,用户还可以在图库等应用中选择想要进行投屏显示的多个内容(比如视频1、2等),并通过某个操作唤出如图25的(b)所示弹窗106,以便选择用于播放投屏内容的目标屏幕。
上述以用户通过多屏管理界面来选择目标屏幕(比如副驾屏等)、音频模块、音频传输通路为例,在另一些实施例中,第一设备还可以根据用户的使用习惯,自动将第一设备上的一些内容投屏到第二设备上显示。比如,检测到用户进入车中,手机根据用户的使用习惯,若用户坐在主驾位置,则手机可自动将导航应用的内容投屏到中控屏。若用户坐在副驾位置,且用户习惯使用副驾屏观看视频,则手机可以将视频应用等影音娱乐的内容投屏到副驾屏,以满足用户的投屏需求。
需要说明的是,可以对上述多个实施例进行组合,并实施组合后的方案。可选的,各方法实施例的流程中的一些操作任选地被组合,并且/或者一些操作的顺序任选地被改变。并且,各流程的步骤之间的执行顺序仅是示例性的,并不构成对步骤之间执行顺序的限制,各步骤之间还可以是其他执行顺序。并非旨在表明所述执行次序是可以执行这些操作的唯一次序。本领域的普通技术人员会想到多种方式来对本文所述的操作进行重新排序。另外,应当指出的是,本文某个实施例涉及的过程细节同样以类似的方式适用于其他实施例,或者,不同实施例之间可以组合使用。
此外,方法实施例中的某些步骤可等效替换成其他可能的步骤。或者,方法实施例中的某些步骤可以是可选的,在某些使用场景中可以删除。或者,可以在方法实施例中增加其他可能的步骤。或者,方法实施例中的某些步骤的执行主体(比如功能模块)可以替换为其他执行主体。
并且,上述各方法实施例之间可以单独实施,或结合起来实施。
示例性的,多屏同时有控制信息(用户交互的触屏指令之类操作命令)时,选取最新一条指令执行,例如第一设备总控方按最新指令(无论来自哪个屏幕)进行播放、停止;来简化整个系
统的复杂度。更进一步,可以结合账户权限,对操作进行优先权限制和管理,考虑本申请实施例的的复杂度和逻辑统一性,不再展开描述,但可以预见的扩展模块,多屏优先权操作控制处理,也包含在本申请实施例的保护范围之中。
本申请另一些实施例提供了一种装置,该装置可以是上述第一设备、第二设备等。该装置可以包括:显示屏、存储器和一个或多个处理器。该显示屏、存储器和处理器耦合。该存储器用于存储计算机程序代码,该计算机程序代码包括计算机指令。当处理器执行计算机指令时,装置可执行上述方法实施例中手机执行的各个功能或者步骤。该装置的结构可以参图26所示的电子设备(装置)。
其中,该装置的核心结构可以表示为图26所示的结构,装置包括:处理模块1301、输入模块1302、存储模块1303和显示模块1304。
处理模块1301,可包括中央处理器(CPU)、应用处理器(Application Processor,AP)或通信处理器(Communication Processor,CP)中的至少一个。处理模块1301可执行与用户电子设备的其他元件中的至少一个的控制和/或通信相关的操作或数据处理。具体地,处理模块1301可用于根据一定的触发条件,控制主屏上显示的内容。处理模块1301还用于将输入的指令或数据进行处理,并根据处理后的数据确定显示样式。
输入模块1302,用于获取用户输入的指令或数据,并将获取到的指令或数据传输到电子设备的其他模块。具体地说,输入模块1302的输入方式可以包括触摸、手势、接近屏幕等,也可以是语音输入。例如,输入模块可以是电子设备的屏幕,获取用户的输入操作并根据获取到的输入操作生成输入信号,将输入信号传输至处理模块1301。
存储模块1303,可包括易失性存储器和/或非易失性存储器。存储模块用于存储用户装置设备的其他模块中的至少一个相关的指令或数据,具体地说,存储模块可记录装置界面元素UI所在界面的位置。
显示模块1304,可包括例如液晶显示器(LCD)、发光二极管(LED)显示器、有机发光二极管(OLED)显示器、微机电***(MEMS)显示器或电子纸显示器。用于显示用户可观看的内容(例如,文本、图像、视频、图标、符号等)。
可选的,还包括通信模块1305,用于支持个人装置(通过通信网络)与其他个人装置通信。例如,通信模块可经由无线通信或有线通信连接到网络,以与其他个人装置或网络服务器进行通信。无线通信可采用蜂窝通信协议中的至少一个,诸如,长期演进(LTE)、高级长期演进(LTE-A)、码分多址(CDMA)、宽带码分多址(WCDMA)、通用移动通信***(UMTS)、无线宽带(WiBro)或全球移动通信***(GSM)。无线通信可包括例如短距通信。短距通信可包括无线保真(Wi-Fi)、蓝牙、近场通信(NFC)、磁条传输(MST)或GNSS中的至少一个。
需要说明的是,装置的各功能模块可执行上述方法实施例中的一个或多个步骤。
本申请实施例还提供一种芯片***,如图27所示,该芯片***包括至少一个处理器1401和至少一个接口电路1402。处理器1401和接口电路1402可通过线路互联。例如,接口电路1402可用于从其它装置(例如电子设备的存储器)接收信号。又例如,接口电路1402可用于向其它装置(例如处理器1401)发送信号。示例性的,接口电路1402可读取存储器中存储的指令,并将该指令发送给处理器1401。当所述指令被处理器1401执行时,可使得电子设备执行上述实施例中的各个步骤。当然,该芯片***还可以包含其他分立器件,本申请实施例对此不作具体限定。
本申请实施例还提供一种计算机存储介质,该计算机存储介质包括计算机指令,当所述计算机指令在上述电子设备上运行时,使得该电子设备执行上述方法实施例中手机执行的各个功能或者步骤。
本申请实施例还提供一种计算机程序产品,当所述计算机程序产品在计算机上运行时,使得所述计算机执行上述方法实施例中第一设备或第二设备等执行的各个功能或者步骤。
通过以上实施方式的描述,所属领域的技术人员可以清楚地了解到,为描述的方便和简洁,仅以上述各功能模块的划分进行举例说明,实际应用中,可以根据需要而将上述功能分配由不同的功能模块完成,即将装置的内部结构划分成不同的功能模块,以完成以上描述的全部或者部分功能。
在本申请所提供的几个实施例中,应该理解到,所揭露的装置和方法,可以通过其它的方式实现。例如,以上所描述的装置实施例仅是示意性的,例如,所述模块或单元的划分,仅仅为一种逻辑功能划分,实际实现时可以有另外的划分方式,例如多个单元或组件可以结合或者可以集成到另一个装置,或一些特征可以忽略,或不执行。另一点,所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口,装置或单元的间接耦合或通信连接,可以是电性,机械或其它的形式。
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的,作为单元显示的部件可以是一个物理单元或多个物理单元,即可以位于一个地方,或者也可以分布到多个不同地方。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
另外,在本申请各个实施例中的各功能单元可以集成在一个处理单元中,也可以是各个单元单独物理存在,也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现,也可以采用软件功能单元的形式实现。
所述集成的单元如果以软件功能单元的形式实现并作为独立的产品销售或使用时,可以存储在一个可读取存储介质中。基于这样的理解,本申请实施例的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的全部或部分可以以软件产品的形式体现出来,该软件产品存储在一个存储介质中,包括若干指令用以使得一个设备(可以是单片机,芯片等)或处理器(processor)执行本申请各个实施例所述方法的全部或部分步骤。而前述的存储介质包括:U盘、移动硬盘、只读存储器(read only memory,ROM)、随机存取存储器(random access memory,RAM)、磁碟或者光盘等各种可以存储程序代码的介质。
以上内容,仅为本申请的具体实施方式,但本申请的保护范围并不局限于此,任何在本申请揭露的技术范围内的变化或替换,都应涵盖在本申请的保护范围之内。因此,本申请的保护范围应以所述权利要求的保护范围为准。
Claims (18)
- 一种多屏多设备交互方法和界面,其特征在于,应用于第一设备,所述方法包括:显示第一界面,所述第一界面包括第二设备上多个可用屏幕的标识;接收用于指示将所述第一设备上的投屏内容投屏到目标屏幕的操作;所述多个可用屏幕包括所述目标屏幕;向所述目标屏幕发送所述投屏内容对应的投屏图像,以使得所述目标屏幕显示所述投屏图像;所述投屏图像用于所述目标屏幕显示所述投屏内容;其中,所述投屏内容为一个或多个,且所述目标屏幕为多个;或者,所述投屏内容为多个,且所述目标屏幕为一个或多个。
- 根据权利要求1所述的方法和界面,其特征在于,所述第二设备为一个或多个。
- 根据权利要求1或2所述的方法和界面,其特征在于,所述目标屏幕连接有多个设备;所述方法还包括:从所述多个设备中确定一个或多个目标设备,所述多个设备包括所述一个或多个目标设备;向所述目标屏幕发送所述投屏内容对应的投屏图像,包括:向所述一个或多个目标设备发送所述投屏图像,以使得所述一个或多个目标设备对所述投屏图像进行处理,并向所述目标屏幕发送处理后的投屏图像。
- 根据权利要求3所述的方法和界面,其特征在于,从所述多个设备中确定一个或多个目标设备,包括:获取所述多个设备的算力信息,所述算力信息包括设备的负载信息;根据所述多个设备的算力信息,从所述多个设备中确定一个或多个目标设备。
- 根据权利要求1-4任一项所述的方法和界面,其特征在于,所述方法还包括:接收用户在所述第一界面上输入的音频选择操作;根据所述音频选择操作,确定用于播放所述投屏图像对应音频的音频模块,以及确定所述第一设备与所述音频模块之间的音频传输通路;通过所述音频传输通路,向所述音频模块发送所述投屏图像对应的音频数据。
- 根据权利要求1-5任一项所述的方法和界面,其特征在于,所述目标屏幕包括第一屏幕和第二屏幕;所述第一屏幕上的投屏内容对应第一音频模块以及第一音频传输通路;所述第二屏幕上的投屏内容对应第二音频模块以及第二音频传输通路。
- 根据权利要求1-6任一项所述的方法和界面,其特征在于,所述目标屏幕包括第一屏幕;所述投屏内容包括第一应用的投屏内容和第二应用的投屏内容;所述第一屏幕上第一应用的投屏内容对应第一音频模块以及第一音频传输通路;所述第一屏幕上第二应用的投屏内容对应第二音频模块以及第二音频传输通路。
- 根据权利要求1-7任一项所述的方法和界面,其特征在于,所述目标屏幕包括第一屏幕和第二屏幕;所述投屏内容包括第一应用的投屏内容和第二应用的投屏内容;所述第一屏幕上第一应用的投屏内容对应第一音频模块以及第一音频传输通路;所述第二屏幕上第二应用的投屏内容对应第二音频模块以及第二音频传输通路。
- 根据权利要求1-8任一项所述的方法和界面,其特征在于,向所述目标屏幕发送所述投屏图像之前,所述方法还包括:对所述投屏图像进行预处理;向所述目标屏幕发送所述投屏图像,包括:向所述目标屏幕发送预处理后的投屏图像。
- 根据权利要求9所述的方法和界面,其特征在于,对所述投屏图像进行预处理,包括:满足第一条件时,对所述投屏图像进行预处理;所述第一条件包括:所述第二设备的负载不足以处理所述投屏图像。
- 根据权利要求9或10所述的方法和界面,其特征在于,所述目标屏幕包括第一屏幕和第二屏幕,对所述投屏图像进行预处理,包括:将所述第一屏幕的投屏图像和所述第二屏幕的投屏图像进行拼接,得到拼接图像;对所述拼接图像进行渲染。
- 根据权利要求9或10所述的方法和界面,其特征在于,所述目标屏幕包括第一屏幕和第二屏幕,对所述投屏图像进行预处理,包括:按照所述第一屏幕的尺寸,对所述投屏图像进行渲染;向所述目标屏幕发送所述投屏内容对应的投屏图像,以使得所述目标屏幕显示所述投屏图像,包括:向所述第一屏幕发送渲染后的投屏图像,以使得所述第一屏幕显示尺寸为第一屏幕尺寸的投屏图像;向所述第二屏幕发送渲染后的投屏图像,以使得所述第二屏幕显示经拉伸或缩放后尺寸为第二屏幕尺寸的投屏图像。
- 根据权利要求11或12所述的方法和界面,其特征在于,所述方法还包括:从所述第一屏幕接收停止投屏的通知消息;调整与所述第二屏幕之间的图像传输通路以及与音频模块之间的音频传输通路。
- 根据权利要求1-13任一项所述的方法和界面,其特征在于,所述第一界面还包括目标应用的标识;接收用于指示将所述第一设备上目标应用的投屏内容投屏到目标屏幕的操作,包括:接收用户将虚拟光标由所述目标应用的标识拖动到所述目标屏幕的标识。
- 根据权利要求1-13任一项所述的方法和界面,其特征在于,显示第一界面之前,所述方法还包括:接收用户选择所述投屏内容的操作;接收用户输入的第一操作;显示第一界面,包括:响应于所述第一操作,显示所述第一界面;接收用于指示将所述第一设备上的投屏内容投屏到目标屏幕的操作,包括:接收用于选择所述目标屏幕的操作。
- 根据权利要求9-13任一项所述的方法和界面,其特征在于,所述预处理包括解码、渲染。
- 一种计算机可读存储介质,其特征在于,包括程序或指令,当所述程序或指令被执行时,如权利要求1至16中任一项所述的方法被实现。
- 一种电子设备,其特征在于,所述电子设备包括处理器和存储器;所述存储器用于存储计算机执行指令,当所述电子设备运行时,所述处理器执行所述存储器存储的所述计算机执行指令,以使所述电子设备执行如权利要求1-16中任意一项所述的方法。
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202211275625.X | 2022-10-18 | ||
CN202211275625.XA CN117956219A (zh) | 2022-10-18 | 2022-10-18 | 多屏多设备交互方法、电子设备及*** |
Publications (2)
Publication Number | Publication Date |
---|---|
WO2024082947A1 true WO2024082947A1 (zh) | 2024-04-25 |
WO2024082947A9 WO2024082947A9 (zh) | 2024-05-30 |
Family
ID=90736897
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2023/122054 WO2024082947A1 (zh) | 2022-10-18 | 2023-09-27 | 多屏多设备交互方法、电子设备及*** |
Country Status (2)
Country | Link |
---|---|
CN (1) | CN117956219A (zh) |
WO (1) | WO2024082947A1 (zh) |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20150172766A1 (en) * | 2013-12-12 | 2015-06-18 | Samsung Electronics Co., Ltd. | Image display apparatus, method for driving image display apparatus, method for displaying an image, and computer readable recording medium therefor |
CN112286477A (zh) * | 2020-11-16 | 2021-01-29 | Oppo广东移动通信有限公司 | 投屏显示方法及相关产品 |
CN112988102A (zh) * | 2021-05-11 | 2021-06-18 | 荣耀终端有限公司 | 投屏方法和装置 |
CN113590059A (zh) * | 2020-04-30 | 2021-11-02 | 青岛海信移动通信技术股份有限公司 | 一种投屏方法及移动终端 |
CN114647395A (zh) * | 2022-03-28 | 2022-06-21 | 北京小米移动软件有限公司 | 投屏方法、装置、电子设备及存储介质 |
-
2022
- 2022-10-18 CN CN202211275625.XA patent/CN117956219A/zh active Pending
-
2023
- 2023-09-27 WO PCT/CN2023/122054 patent/WO2024082947A1/zh unknown
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20150172766A1 (en) * | 2013-12-12 | 2015-06-18 | Samsung Electronics Co., Ltd. | Image display apparatus, method for driving image display apparatus, method for displaying an image, and computer readable recording medium therefor |
CN113590059A (zh) * | 2020-04-30 | 2021-11-02 | 青岛海信移动通信技术股份有限公司 | 一种投屏方法及移动终端 |
CN112286477A (zh) * | 2020-11-16 | 2021-01-29 | Oppo广东移动通信有限公司 | 投屏显示方法及相关产品 |
CN112988102A (zh) * | 2021-05-11 | 2021-06-18 | 荣耀终端有限公司 | 投屏方法和装置 |
CN114647395A (zh) * | 2022-03-28 | 2022-06-21 | 北京小米移动软件有限公司 | 投屏方法、装置、电子设备及存储介质 |
Also Published As
Publication number | Publication date |
---|---|
CN117956219A (zh) | 2024-04-30 |
WO2024082947A9 (zh) | 2024-05-30 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11327711B2 (en) | External visual interactions for speech-based devices | |
KR102183413B1 (ko) | 콘텐트 표현 방법 및 시스템 | |
EP2892239A1 (en) | Living room computer with small form-factor pluggable port | |
CN112214186B (zh) | 信息共享方法及车载终端 | |
US10019224B2 (en) | Electronic device and method of operating the same | |
EP3029650A1 (en) | Electronic device, audio device, and methods of controlling electronic device and audio device power supply | |
CN102857805A (zh) | 一种移动终端信号的处理方法及装置 | |
KR102304365B1 (ko) | 디스플레이 장치 및 그의 명령 전송 방법 | |
US11051147B2 (en) | Electronic apparatus and method of outputting content by the electronic apparatus | |
JPWO2017068759A1 (ja) | 車載表示システムおよびこの車載表示システムの制御方法 | |
US11076121B2 (en) | Apparatus and associated methods for video presentation | |
JP2020197688A (ja) | 画像伝送システム、伝送制御装置及び伝送制御方法 | |
US9661375B2 (en) | Display apparatus and method of controlling content output of display apparatus | |
JP2020197687A (ja) | 画像表示システム、表示制御装置及び表示制御方法 | |
US10142576B2 (en) | Control device, sound switching method of control device, and program | |
JP2005199998A (ja) | 車載用画像表示システム | |
JP6922170B2 (ja) | 情報処理装置、情報処理方法、プログラム、及び情報処理システム | |
WO2024082947A1 (zh) | 多屏多设备交互方法、电子设备及*** | |
KR101638612B1 (ko) | 외부 기기와 연결되는 차량의 멀티미디어 시스템 및 그 제어방법 | |
KR20220139783A (ko) | 이동 단말기 | |
JP6325231B2 (ja) | 情報表示システム、情報表示システムの制御方法およびプログラム | |
WO2024140496A1 (zh) | 一种音频播放方法及电子设备 | |
KR20160149991A (ko) | 멀티미디어 디스플레이 장치 | |
KR20130101801A (ko) | 콘텐츠 처리 장치 및 그 방법 | |
CN118077209A (zh) | 显示设备、外接设备、音频播放及音效处理方法 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 23878940 Country of ref document: EP Kind code of ref document: A1 |