CN117097878A - Cloud control interaction system based on ultralow-delay video transmission technology - Google Patents
Cloud control interaction system based on ultralow-delay video transmission technology Download PDFInfo
- Publication number
- CN117097878A CN117097878A CN202311334197.8A CN202311334197A CN117097878A CN 117097878 A CN117097878 A CN 117097878A CN 202311334197 A CN202311334197 A CN 202311334197A CN 117097878 A CN117097878 A CN 117097878A
- Authority
- CN
- China
- Prior art keywords
- follow
- user
- interaction
- image
- equipment
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000003993 interaction Effects 0.000 title claims abstract description 113
- 230000005540 biological transmission Effects 0.000 title claims abstract description 37
- 238000005516 engineering process Methods 0.000 title claims abstract description 23
- 230000009471 action Effects 0.000 claims description 26
- 238000004088 simulation Methods 0.000 claims description 26
- 230000002452 interceptive effect Effects 0.000 claims description 19
- 238000012795 verification Methods 0.000 claims description 13
- 230000002159 abnormal effect Effects 0.000 claims description 9
- 238000003384 imaging method Methods 0.000 claims description 9
- 238000013507 mapping Methods 0.000 claims description 9
- 239000003550 marker Substances 0.000 claims description 9
- 238000012937 correction Methods 0.000 claims description 8
- 230000004044 response Effects 0.000 claims description 8
- 238000000034 method Methods 0.000 claims description 7
- 230000000007 visual effect Effects 0.000 claims description 5
- 238000012545 processing Methods 0.000 abstract description 7
- 230000000694 effects Effects 0.000 abstract description 6
- 230000033001 locomotion Effects 0.000 description 7
- 230000036544 posture Effects 0.000 description 4
- 238000013461 design Methods 0.000 description 3
- 238000001514 detection method Methods 0.000 description 3
- 238000003491 array Methods 0.000 description 2
- 230000006399 behavior Effects 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 230000000903 blocking effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000001010 compromised effect Effects 0.000 description 1
- 230000002950 deficient Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000007474 system interaction Effects 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/194—Transmission of image signals
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/106—Processing image signals
- H04N13/122—Improving the 3D impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/106—Processing image signals
- H04N13/167—Synchronising or controlling image signals
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/275—Image signal generators from 3D object models, e.g. computer-generated stereoscopic image signals
- H04N13/279—Image signal generators from 3D object models, e.g. computer-generated stereoscopic image signals the virtual viewpoint locations being selected by the viewers or determined by tracking
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/296—Synchronisation thereof; Control thereof
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/30—Image reproducers
- H04N13/332—Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
- H04N13/344—Displays for viewing with the aid of special glasses or head-mounted displays [HMD] with head-mounted left-right displays
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/30—Image reproducers
- H04N13/366—Image reproducers using viewer tracking
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Processing Or Creating Images (AREA)
Abstract
The invention relates to a cloud control interaction system based on an ultralow-delay video transmission technology, which comprises a user control end, follow-up shooting equipment, a control background and a model positioning component; the user control end can control corresponding follow-up equipment to move through a remote output interaction instruction, an image fed back by the follow-up equipment can be output through video output equipment of the user control end, other follow-up equipment and the position of the follow-up equipment are obtained through the positioning unit, the original mode of image processing in a control background is changed, image processing is directly carried out on an image caching module, even if the follow-up equipment appears in a picture, image rationalization can be carried out, the effect of abrupt display of the follow-up equipment is avoided, meanwhile, the time increase caused by image identification and image data transmission is avoided in a direct accurate replacement mode, and occupation of display efficiency is avoided as much as possible in an equipment display link.
Description
Technical Field
The invention relates to intelligent household equipment, in particular to a cloud control interaction system based on an ultralow-delay video transmission technology.
Background
Through the model cloud control platform, players in the south, the south and the north can quickly and conveniently face a professional-grade high-quality model site which is thousands of kilometers away through interaction equipment such as mobile phones, and the model site comprises an outdoor site with local characteristics (such as deserts, karst landforms, forests, grasslands and the like) so as to control the model at the field personnel visual angle, and the experience of the field players is achieved. The cloud control platform enables a business mode of time-sharing payment of the model to be achieved, and compared with a traditional product purchase mode, the experience threshold of the professional model is greatly reduced. The cloud experience mode makes the space distance in the real world no longer a problem, players in different cities can interact with each other in the same place at the same time, and the geographic position of the place is irrelevant to the players, so long as the network can be reached. Therefore, villages and towns far from cities and rural areas can provide specialized and characteristic model cloud services for whole-network users with extremely low site cost and labor cost.
While model scenes can present several problems in practical applications, firstly, because the follow-up devices on behalf of the user can be in the same scene, the presence of the follow-up devices can reduce the realism of the scene. 2. The imaging of the follow-up shooting equipment on the basis of the above is to reflect the real scene image of the real person, and ensure higher interaction and response efficiency, so that the implementation difficulty is higher. 3. Other follow-up devices move in the scene, errors can easily occur through picture replacement, the follow-up device identification errors lead to the follow-up device itself or parts of devices to directly appear in the scene, and fourth, transmission efficiency needs to be guaranteed if video feedback is to be achieved, and meanwhile, if the video is further processed, the time for interaction video appearance can be increased undoubtedly.
Disclosure of Invention
In view of the above, the present invention aims to provide a cloud control interaction system based on an ultra-low delay video transmission technology.
In order to solve the technical problems, the technical scheme of the invention is as follows: a cloud control interaction system based on an ultralow-delay video transmission technology comprises a user control end, a follow-up shooting device, a control background and a model positioning component;
the user control end comprises a dynamic collector, an image cache module and an equipment virtual image module, wherein the dynamic collector is used for collecting user actions to generate interaction instructions to the control background, the control background comprises an action control module and an interaction index module, and the action control module controls the actions of the follow-up shooting equipment corresponding to the user control end according to the interaction instructions;
the model positioning assembly comprises a plurality of positioning units arranged in a model simulation scene, wherein the positioning units are used for collecting the positions of the follow-up shooting equipment to form positioning position data corresponding to the follow-up shooting equipment;
the interaction index module determines the follow-up equipment related to the follow-up equipment as the associated follow-up equipment according to the positioning position data of each follow-up equipment, acquires an interaction instruction of the associated follow-up equipment to generate interaction index information and sends the interaction index information to the corresponding user control end;
the image caching module loads a user image model located in the same model simulation scene to an image caching area of a user control end through a control background in advance, the user image model takes associated user data as an index, and the user image model outputs a corresponding associated user image according to the input associated gesture data;
the equipment virtual image module is used for calling a corresponding associated user image from the image cache module according to interaction index information, and the associated user image is covered at a corresponding position of an imaging picture of the video output equipment according to corresponding positioning position data, wherein the interaction index information comprises the associated user data, the associated gesture data and the positioning position data.
Further: each user image model is provided with a reference attitude image, the device virtual image module calculates the field distance between the follow-up shooting devices according to the positioning position data, and the device virtual image module is provided withWherein, the method comprises the steps of, wherein,is->The field of view distance of the associated heel-beat apparatus from the heel-beat apparatus where the user is located, < >>Reflecting the influence of the position distance on the field distance for a preset position distance mapping function, +.>For the position coordinates corresponding to the heel-beat device where the user is located, < >>Is->Position coordinates corresponding to the associated beat-following device, < >>Reflecting the influence of the effective shielding area on the field distance for a preset effective shielding mapping function, +.>For the current shooting view angle coordinates of the follow-up shooting device where the user is located,/->Reflecting the influence of the focal distance on the field distance for a preset focal distance mapping function, +.>For a preset occlusion decay weight,the method comprises the steps of setting a preset defocus attenuation weight; dividing the associated follow-up shooting equipment into a short distance level, a medium distance level and a long distance level according to the field of view distance, outputting the associated user image from a user image model when the associated follow-up shooting equipment is the short distance level, and when the associated follow-up shooting equipment is the associated follow-up shooting equipmentAnd when the user image model is at a long distance level, a preset reference replacement image is used as an associated user image.
Further: the equipment virtual image module is configured with a prediction loading sub-module, the associated user images in the user image model are configured with continuous associated values according to the continuity of the gestures of the associated user images, the prediction loading sub-module calculates loading priority values of other associated user images according to the current associated gesture data of the associated follow-up equipment as a reference, and sequentially loads corresponding associated user images according to the magnitude sequence of the loading priority values, wherein the prediction loading sub-module comprisesWherein->For loading priority values ++>Is the firstThe +.f. of the corresponding associated beat-following device>Successive association values of the individual associated user images, +.>Is->The associated pair corresponds to the field of view distance of the photographing apparatus.
Further: the user image model is provided with a coverage maximum area, the coverage maximum area reflects the maximum area which can be covered by all relevant user images in the user image model, and the reference replacement image is larger than or equal to the corresponding coverage maximum area;
the equipment virtual image module is configured with an abnormal response sub-module, and the user of the abnormal response sub-module analyzes abnormal data in the interaction index information and replaces corresponding associated user images through the reference replacement images when the abnormal data are generated.
Further: the image caching module further comprises AR interaction information for preloading a corresponding model simulation scene, wherein the AR interaction information takes position conditions and view angle conditions as indexes;
the user control terminal also comprises an AR display module, wherein the AR display module is used for judging whether the position coordinates corresponding to the follow-up shooting equipment where the user is located meet the corresponding position conditions and whether the associated gesture data of the follow-up shooting equipment where the user is located meet the view angle conditions, and when the position conditions and the view angle conditions are met, the corresponding AR interaction information is called to be displayed at the corresponding position of the imaging picture of the video output equipment.
Further: the control background comprises an interaction configuration module, a plurality of interaction verification units are configured in the model simulation scene, the interaction verification units are used for capturing interaction rotation data of the follow-up shooting equipment to the interaction configuration module, the interaction configuration module is used for comparing the interaction rotation data with rotation instruction data in interaction instructions to generate rotation deviation data, and correction configuration information is obtained from a pre-built deviation correction database according to the rotation deviation data and sent to the corresponding follow-up shooting equipment to configure rotation parameters of the follow-up shooting equipment.
Further: the interactive checking unit is configured as an infrared receiving array, an infrared emitter is arranged on the follow-up shooting device, and the infrared receiving array responds to the infrared emitter to generate rotation interactive data.
Further: the interaction verification unit is configured to be a wireless transmission device and a marker arranged in a model simulation scene, the follow-up shooting device is configured to obtain corresponding interaction rotation data by analyzing images of the marker, and according to different shooting visual angles, the content of the interaction rotation data is different, and the follow-up shooting device sends the interaction rotation data to the interaction verification unit through the wireless transmission module.
Further: the positioning unit is arranged as a photosensitive array arranged on the ground, and the model simulation scene is provided with a fixed light source.
Further: the positioning unit is configured as an RFID.
The technical effects of the invention are mainly as follows: through such setting, the user control end can be through the removal of the corresponding follow-up equipment of remote output interaction instruction control, and the image of follow-up equipment feedback can be through the video output equipment of user control end export, the position acquisition to other follow-up equipment and this follow-up equipment through positioning unit, original mode of image processing is done at the control backstage has been changed, direct image processing is done at image buffer module, even the follow-up equipment appears in the assurance picture, can carry out the image rationalization, avoid appearing the effect of abrupt demonstration with the equipment, simultaneously through less data volume, the direct accurate alternative mode, the increase of the time that causes of image recognition and image data transmission has been avoided, the occupation to the display efficiency production is avoided as far as possible in the equipment display link.
Drawings
Fig. 1: the invention relates to a cloud control interaction system architecture schematic diagram based on an ultralow-delay video transmission technology;
fig. 2: the invention relates to a cloud control interaction system interaction control principle topological graph based on an ultralow-delay video transmission technology;
fig. 3: the invention relates to a virtual image imaging principle topological graph of a cloud control interaction system based on an ultralow-delay video transmission technology;
fig. 4: the invention discloses an interaction correction principle topological graph of a cloud control interaction system based on an ultralow-delay video transmission technology.
Reference numerals: 100. a user control end; 110. a dynamic collector; 120. an image caching module; 130. a virtual device module; 131. a predictive loading sub-module; 132. an anomaly response sub-module; 140. an AR display module; 200. a heel beat device; 300. controlling a background; 310. a motion control module; 320. an interactive index module; 330. an interaction configuration module; 401. a positioning unit; 501. and an interactive checking unit.
Detailed Description
The following detailed description of the invention is provided in connection with the accompanying drawings to facilitate understanding and grasping of the technical scheme of the invention.
Referring to fig. 1, embodiment 1 is a cloud control interaction system based on an ultra-low delay video transmission technology, which includes a user control end 100, a follow-up shooting device 200, a control background 300, and a model positioning component; firstly, a general remote interaction platform is provided with a user control end 100, a traditional user control end 100 collects user behaviors so as to generate control instructions corresponding to a follow-up shooting device 200 and control the follow-up shooting device 200 to move, so the user control end 100 sends the instructions through network data and receives video returned by the follow-up shooting device 200 so as to achieve the effect of browsing in the model simulation scene in the field, along with the popularization of the technology, the follow-up shooting device 200 can interact with people, things and objects in the simulation model scene through interaction, the interaction mode is not a design point of the invention, and the design point of the invention is not repeated. Firstly, the first requirement is analyzed, that is, the following photographing device 200 is "removed" from the acquired picture of the scene, that is, the picture of the following photographing device 200 is replaced by other pictures, so as to ensure the consistency of the simulated model scene, the traditional mode is to identify the image and process the image, that is, the identifier or the identification pattern is arranged on the following photographing device 200, when the pattern is identified, the image is replaced by the preset image, firstly, the identification of the pattern or the identifier cannot be ensured to be in error in real time, so that the following photographing device 200 is caused to appear at the image due to the identification error, or the situation that part of the content of the following photographing device 200 is drawn due to the factors such as the movement of the following photographing device 200, on the other hand, the situation that the view angle is blocked by the scene is also possible to appear due to the model simulation scene, so that the identification of the following photographing device 200 needs a large amount of algorithm support, and thus the efficiency is insufficient to meet the current remote interaction efficiency requirement. On the other hand, the data is processed in the control background 300, and there is a great problem in synchronization and real-time.
First, referring to fig. 2, the user control terminal 100 includes a dynamic collector 110, an image buffer module 120, and a device virtual image module 130, where the dynamic collector 110 is configured to collect user actions to generate interaction instructions to the control background 300, the dynamic collector 110 may be a head-mounted sensor or a key or other wearable sensor, and the user monitors user actions through the sensor to generate interaction instructions, for example, the head-mounted sensor may monitor steering actions of the head, so as to generate interaction instructions for controlling rotation of the clapping device 200, for example, the key may control interaction instructions corresponding to a moving direction of the clapping device 200. The control background 300 includes an action control module 310 and an interaction index module 320, where the action control module 310 controls the actions of the follow-up device 200 corresponding to the user control terminal 100 according to the interaction instruction; the control background 300 is disposed near the model simulation scene, and can communicate with the follow-up shooting device 200 and the sensing device in the model simulation scene, while ensuring data transmission efficiency.
Referring to fig. 3, the model positioning assembly includes a plurality of positioning units 401 disposed in a model simulation scene, where the positioning units 401 are configured to collect positions of the heel-beat apparatus 200 to form positioning position data corresponding to the heel-beat apparatus 200; the positioning unit 401 is arranged as a photosensitive array arranged on the ground, and the model simulation scene is provided with a fixed light source. The location unit 401 is configured as an RFID. Firstly, the invention is characterized in that the structural features of the model simulation scene are used for directly determining the image picture to be covered, and then the image picture is replaced according to different data postures, so that the image recognition is not needed, and the key point is that the accurate positioning position is needed to be obtained. The invention provides two means, one is that the feedback of the photosensitive array is not changed under normal conditions through the photosensitive array, but if the follow-up device 200 enters a scene, the photosensitive array is blocked, the position of the follow-up device 200 can be judged by blocking the output signals of the sensor, and the other is that a plurality of RFID receivers are buried in the ground, a plurality of RFID transmitters are arranged in different directions at the bottom of the follow-up device 200, and the position of the follow-up device 200 can be judged according to the positions of the receivers and which signal is preferentially received by the receivers, and meanwhile, the two means can be combined to improve the accuracy of position judgment. Since the structural model in the whole scene is known, the coordinates of the camera in the scene (the height of the follow-up device 200 is known) are also known by knowing the position information, so that the structure can be quickly pre-determined, and the corresponding region can be determined.
The interaction index module 320 determines, according to the positioning position data of each heel-beat device 200, the heel-beat device 200 related to the heel-beat device 200 as the associated heel-beat device 200, and obtains the interaction instruction of the associated heel-beat device 200 to generate interaction index information and send the interaction index information to the corresponding user control terminal 100; firstly, a coordinate system is established according to the whole model simulation scene, and the coordinates of each follow-up device 200 are positioned, so that the content to be displayed in the model scene is known in advance under each view angle, the picture can be divided into a plurality of layers according to the entity structure, so that the picture layer needing to be inserted with the image is quickly determined, quick retrieval of video information can be realized, the interaction index information reflects the orientation of a camera of other follow-up devices 200, the changing user side of the camera orientation of the device can be known at the first time, if the data of the model are loaded in advance, the image is very well added under the condition that the picture layer is known in advance, and interaction is realized, namely, the action of one follow-up device 200 can be captured by another follow-up device 200, for example, the rotation of the camera can judge whether the view angle of the follow-up device 200 is observing the user, so that a foundation can be provided for interaction, for example, elements such as a manipulator can be added to embody interactivity, the action of the follow-up device 200 in the scene and the image can be associated with the image, and the action of the follow-up device, instead of simply transmitting the data, the image can be adjusted, and error adjustment can be realized in real time, and error adjustment can be realized after the scene is realized.
The image caching module 120 loads a user image model located in the same model simulation scene to an image cache area of the user control terminal 100 through the control background 300 in advance, the user image model takes associated user data as an index, and the user image model outputs a corresponding associated user image according to the input associated gesture data; the other core point of the invention is that the user image model is stored in the image buffer area of the other user in advance, so that when the user makes action, the user directly calls the image buffer area, the efficiency is highest, and the local only needs to transmit the associated gesture data and the associated user data as index conditions, and the corresponding associated user image does not need to be transmitted, because the data volume of the associated user image is large, the interaction effect is easily influenced by distortion or time delay through the transmission mode.
The device virtual image module 130 retrieves a corresponding associated user image from the image buffer module 120 according to interaction index information, and the associated user image is overlaid at a corresponding position of an imaging frame of the video output device according to corresponding positioning position data, where the interaction index information includes the associated user data, the associated gesture data, and the positioning position data. And because the image layer is known, the associated user image to be inserted is known, and the position to be inserted is known, the generation of a new imaging picture can be completed only by inserting the information flow of the corresponding associated user image, and the interaction effect is ensured.
Example 2, based on example 1, further calculates the field distance in order to reduce the amount of analysis of the data: each user image model is provided with a reference attitude image, the device virtual image module calculates the field distance between the follow-up shooting devices according to the positioning position data, and the device virtual image module is provided withWherein->Is->The field of view distance of the associated heel-beat apparatus from the heel-beat apparatus where the user is located, < >>Reflecting the influence of the position distance on the field distance for a preset position distance mapping function, +.>For the position coordinates corresponding to the heel-beat device where the user is located, < >>Is->Position coordinates corresponding to the associated beat-following device, < >>Reflecting the influence of the effective shielding area on the field distance for a preset effective shielding mapping function, +.>For the current shooting view angle coordinates of the follow-up shooting device where the user is located,/->Reflecting the influence of the focal distance on the field distance for a preset focal distance mapping function, +.>For a preset occlusion decay weight, +.>The method comprises the steps of setting a preset defocus attenuation weight; dividing the associated follow-up shooting equipment into a short distance level, a middle distance level and a long distance level according to the field of view distance, outputting the associated user image from a user image model when the associated follow-up shooting equipment is at the short distance level, taking a reference posture image of the user image model as the associated user image when the associated follow-up shooting equipment is at the middle distance level, and taking a preset reference replacement image as the associated user image when the user image model is at the long distance level. For example, images with relatively large effective positions blocked and relatively far from the focal position of the device are subjected to different gradesFor example, if the distance is medium, only the reference posture image of the user is displayed, and specific directions and actions are not displayed, but the user can realize resolution, and if the distance is long, the user does not need to be identified, and the reference posture image is used for replacing, so that the operation amount can be reduced, and the visual angle reliability can be ensured.
Embodiment 3, based on embodiment 2, to further improve efficiency, by predicting possible actions of the user, invoking the corresponding possible changed user gestures, when the user actions are generated, such as the user moves or changes the viewing angle direction, and other user moves, all the changes of the positions of the images in the frame are caused, the device virtual image module 130 is configured with a prediction loading sub-module 131, the associated user images in the user image model are configured with continuous association values according to the continuity of the gestures, the values of the continuous association values consider the following factors, the use habit of the user, the moving direction of the other user, the viewing angle direction of the other user, so that the possibility of each associated user image can be determined, the continuous association values reflect, the prediction loading sub-module 131 calculates the loading priority values of the other associated user images according to the current associated gesture data of the associated follow-up device 200 as a reference, and sequentially loads the corresponding associated user images according to the magnitude of the loading priority values, and has the following habitsWherein->For loading priority values ++>Is->The +.f. of the corresponding associated beat-following device>Successive association values of the individual associated user images, +.>Is->The associated pair corresponds to the field of view distance of the photographing apparatus. And by loading the priority value, the associated user images corresponding to the follow-up shooting equipment with larger field distance are given higher priority, and the loading is performed in advance, so that the user can quickly receive the corresponding image information. The user image model is provided with a coverage maximum area, the coverage maximum area reflects the maximum area which can be covered by all relevant user images in the user image model, and the reference replacement image is larger than or equal to the corresponding coverage maximum area;
the purpose of this is to ensure that the reference replacement image can be overlaid if the viewing angle and the follow-up device 200 are moved without the device itself "drawing out". The device virtual image module 130 is configured with an anomaly response sub-module 132, and the anomaly response sub-module 132 analyzes anomaly data in the interaction index information and replaces a corresponding associated user image with the reference replacement image when the anomaly data is generated. For example, the position of the user is lost, or the steering data of the user is lost or reported wrong, and the like, the user is directly replaced by the reference replacement image, so that the abnormal image caused by information errors is avoided.
Embodiment 4, based on embodiment 1, the image buffer module 120 further includes pre-loading AR interaction information corresponding to the model simulation scene, where the AR interaction information is indexed by a position condition and a view angle condition;
the user control terminal 100 further includes an AR display module 140, where the AR display module 140 is configured to determine whether a position coordinate corresponding to the heel-beat device 200 where the user is located meets a corresponding position condition, and whether associated gesture data of the heel-beat device 200 where the user is located meets a viewing angle condition, and when both the position condition and the viewing angle condition are met, retrieve corresponding AR interaction information and display the information at a corresponding position of an imaging picture of the video output device. That is, the user can call the corresponding introduction information, such as the introduction information to the article, at the corresponding position and view angle, so that the reliability of the output of the information is ensured.
Referring to fig. 4, in embodiment 5, on the basis of embodiment 1, the control background 300 includes an interaction configuration module 330, in which a plurality of interaction verification units 501 are configured in the model simulation scene, the interaction verification units 501 are configured to capture interaction rotation data of the follow-up shooting device 200 to the interaction configuration module 330, the interaction configuration module 330 is configured to compare the interaction rotation data with rotation instruction data in an interaction instruction to generate rotation deviation data, and obtain correction configuration information from a pre-constructed deviation correction database according to the rotation deviation data, and send the correction configuration information to the corresponding follow-up shooting device 200 to configure rotation parameters of the follow-up shooting device 200. The interactive configuration module 330 is configured to configure initial parameters, such as a basic speed of rotation of the camera of the follow-up device 200, and adjust the initial parameters according to actual collected data of a user at the basic speed, so that the rotation speed of the follow-up device 200 accords with expectations, and deviation of a maximum rotation angle and deviation of other actions are the same, and the principle is that by setting a detection device at a preset position, corrected parameters, such as that when the camera rotates, the rotation angle of each time is deficient, the rotation angle of each time can be increased to compensate the data. The method comprises the following steps: in one embodiment, the interactive verification unit 501 is configured as an infrared receiving array, and an infrared emitter is disposed on the heel-strike device 200, and the infrared receiving array is responsive to the infrared emitter to generate the rotational interactive data. The change amount can be rapidly determined through the infrared transmitter and the infrared receiver, so that judgment is carried out, and the deviation condition is found in time. In another embodiment, the interactive verification unit 501 is configured as a wireless transmission device and a marker disposed in the model simulation scene, the follow-up shooting device 200 is configured to obtain corresponding interactive rotation data by analyzing an image of the marker, and according to different shooting angles, the content of the interactive rotation data is different, and the follow-up shooting device 200 sends the interactive rotation data to the interactive verification unit 501 through the wireless transmission module. The data acquisition is directly carried out on the marker, for example, the marker can be a coded sphere with a marker pattern, the code is acquired through image recognition and used as interactive rotation data, the theoretically acquired code and the actual difference value represent corresponding deviation, and two rotation data results can be combined in a simultaneous mode, so that the detection accuracy is improved.
The control support function required by the invention is as follows: 1. and receiving and decoding the control instruction through WiFi or 4G/5G, and checking the validity of the instruction. 2. And converting the received control instruction into driving signals of each hardware of the model, such as electric tuning driving, steering engine driving and the like. 3. Realizing a set of control and feedback instruction coding protocols: the coding format is realized based on the byte data frame, the receiving and transmitting of the instruction are realized by using as little data as possible, and the error rate in the transmission process is reduced. 4. The cooperative control end (instruction sending end) realizes a set of communication link transmission control protocol: packet loss control, delay processing, drop detection, reconnection and the like, and ensures reliable transmission of control instructions and received feedback data. A runaway self-protection mechanism: when the model starts to move after receiving the forward command, the network is suddenly disconnected, and if the model is not processed, the model loses control, so that safety problems can be caused. The cloud control system realizes a runaway protection mechanism of the model under the condition of unstable communication link or link disconnection by continuously sending instructions. For the motion control instruction, the sending end continuously sends the motion control instruction every 50ms, and if the receiving end does not receive the motion control instruction for more than a plurality of periods (such as 3 periods, namely 150 ms), the receiving end immediately and actively sends a stop signal to the driving module of the model.
For a clearer understanding of the present invention, for example, the technology is applied to a large-scale on-line deduction platform, firstly, the user side logs on the platform through a special terminal device, the device at least comprises a display device and an input device, the display device is used for matching with an AR display module 140 to complete the display of a situation virtual image, the input device is used for realizing the function of a dynamic collector 110, firstly, a user can control the motion of a follow-up shooting device 200 arranged in a virtual scene of the field through an interactive instruction, a camera of the follow-up shooting device feeds back corresponding image information in real time in the scene, that is, the camera simulates an actual shot image of the user, the content is realized through a motion control module 310, for example, vegetation, stones, buildings and the like Jing Waiguan in the scene are all present in the camera, and the logic of the content is realized in a traditional on-line simulation platform.
The main part of the present invention is that if this scene needs to support a plurality of users to perform interactive experiences, such as completing some specified actions, chatting in a virtual scene, etc., voice information and text information can be easily implemented, but in an actual scene, the user sees the following device 200, but in this way, the actual experience is greatly compromised, if the user needs to display a virtual reality image selected in advance by the user, such as selecting a child figure, a real-time display technique is required to enable the child figure to be displayed, and in a general design, when the following device captures an image of another following device 200, the captured features of the following device 200 are identified, then the image is overlaid, and then the overlaid image is transmitted to the user control terminal 100, in this way, the user control end can generate a clear video stream, the corresponding follow-up shooting device 200 is replaced by the child's figure, but the delay generated by processing the image data by the follow-up shooting device 200 can cause the synchronous display of the image simultaneous transmission to be reduced, the invention loads the figures corresponding to all users in the scene in advance through the image caching module, for example, the virtual scene logs in 10 users, the 10 figures are prestored in the image caching module when loaded, then the image data transmitted to the user control end 100 by the control background 300 is not processed, and is quickly fetched and processed through the image caching module 120, the processing efficiency is higher because the figures are cached, a new image can be obtained by directly overlaying the original image through the basic image, the device virtual image module 130 finds the corresponding user through the associated user data, and opening an image cache gallery of a corresponding user, determining specific image content through the associated gesture data, and determining the placement angle and the proportion of the image through the positioning position data. Specifically, in order to reduce the operand of image calling, the real situation is represented by fuzzy action, specifically, the field of view distance is calculated, if the distance is far, the image is directly replaced by a preset reference, for example, the user A is a girl, the reference image of the girl is directly replaced by a reference portrait, until the middle distance level is reached, the reference image of the girl is called, and if the distance is short, the image related to the specific behavior of the girl is called. Because the video stream is to be served, images at the next moment are to be retrieved preferentially, for example, possible actions of the user A are analyzed, images of girls corresponding to the actions are called and ordered by loading priority values, and therefore quick indexing of video stream display can be completed.
If text information or other image information needs to be displayed in the real scene, the corresponding display can be triggered by the acquisition of the visual angle and the acquisition of the position. For example, before a user walks to a panel, specific panel content can be displayed for convenient reading, if the above synchronous transmission is to be realized, there is a greater requirement for the accuracy degree of the position of the follow-up device, so that the position judgment of the follow-up device cannot be relied on, on one hand, a plurality of positioned photosensitive arrays are arranged in a scene through a positioning unit, whether the photosensitive arrays are blocked or not is judged, the corresponding position of the follow-up device 200 is judged, and the real-time position information is returned to the control background 300 instead of the follow-up device 200, so that the data efficiency is higher because the data can be transmitted in a wired mode, and on the other hand, the mechanical deviation of the follow-up device 200 is corrected through an interactive checking unit, for example, the follow-up device reaches the scene where the interactive checking unit is located, the follow-up device 200 rotates by 30 degrees through the rotation of a camera of the follow-up device 200, and the infrared receiving array detects that only 29.8 degrees, so that the generated rotation deviation data is 0.2 degrees, and the corresponding rotation deviation parameter is required to be adjusted, and the accuracy is improved.
Of course, the above is only a typical example of the invention, and other embodiments of the invention are also possible, and all technical solutions formed by equivalent substitution or equivalent transformation fall within the scope of the invention claimed.
Claims (10)
1. A cloud control interaction system based on an ultralow-delay video transmission technology is characterized in that: the system comprises a user control end, a follow-up shooting device, a control background and a model positioning component;
the user control end comprises a dynamic collector, an image cache module and an equipment virtual image module, wherein the dynamic collector is used for collecting user actions to generate interaction instructions to the control background, the control background comprises an action control module and an interaction index module, and the action control module controls the actions of the follow-up shooting equipment corresponding to the user control end according to the interaction instructions;
the model positioning assembly comprises a plurality of positioning units arranged in a model simulation scene, wherein the positioning units are used for collecting the positions of the follow-up shooting equipment to form positioning position data corresponding to the follow-up shooting equipment;
the interaction index module determines the follow-up equipment related to the follow-up equipment as the associated follow-up equipment according to the positioning position data of each follow-up equipment, acquires an interaction instruction of the associated follow-up equipment to generate interaction index information and sends the interaction index information to the corresponding user control end;
the image caching module loads a user image model located in the same model simulation scene to an image caching area of a user control end through a control background in advance, the user image model takes associated user data as an index, and the user image model outputs a corresponding associated user image according to the input associated gesture data;
the equipment virtual image module is used for calling a corresponding associated user image from the image cache module according to interaction index information, and the associated user image is covered at a corresponding position of an imaging picture of the video output equipment according to corresponding positioning position data, wherein the interaction index information comprises the associated user data, the associated gesture data and the positioning position data.
2. The cloud-controlled interaction system based on the ultra-low delay video transmission technology as claimed in claim 1, wherein: each user image model is provided with a reference attitude image, the device virtual image module calculates the field distance between the follow-up shooting devices according to the positioning position data, and the device virtual image module is provided withWherein->Is->The field of view distance of the associated heel-beat apparatus from the heel-beat apparatus where the user is located, < >>Reflecting the influence of the position distance on the field distance for a preset position distance mapping function, +.>For the position coordinates corresponding to the heel-beat device where the user is located, < >>Is->Position coordinates corresponding to the associated beat-following device, < >>Reflecting the influence of the effective shielding area on the field distance for a preset effective shielding mapping function, +.>For the current shooting view angle coordinates of the follow-up shooting device where the user is located,/->Reflecting the influence of the focal distance on the field distance for a preset focal distance mapping function, +.>For a preset occlusion decay weight, +.>The method comprises the steps of setting a preset defocus attenuation weight; dividing the associated follow-up shooting equipment into a short distance level, a middle distance level and a long distance level according to the field of view distance, outputting the associated user image from a user image model when the associated follow-up shooting equipment is at the short distance level, taking a reference posture image of the user image model as the associated user image when the associated follow-up shooting equipment is at the middle distance level, and taking a preset reference replacement image as the associated user image when the user image model is at the long distance level.
3. The cloud-controlled interaction system based on the ultra-low delay video transmission technology as claimed in claim 2, wherein: the equipment virtual image module is configured with a prediction loading sub-module, the associated user images in the user image model are configured with continuous associated values according to the continuity of the gestures of the associated user images, the prediction loading sub-module calculates loading priority values of other associated user images according to the current associated gesture data of the associated follow-up equipment as a reference, and sequentially loads corresponding associated user images according to the magnitude sequence of the loading priority values, wherein the prediction loading sub-module comprisesWherein->For loading priority values ++>Is->The +.f. of the corresponding associated beat-following device>Successive association values of the individual associated user images, +.>Is->The associated pair corresponds to the field of view distance of the photographing apparatus.
4. The cloud-controlled interaction system based on the ultra-low delay video transmission technology as claimed in claim 3, wherein: the user image model is provided with a coverage maximum area, the coverage maximum area reflects the maximum area which can be covered by all relevant user images in the user image model, and the reference replacement image is larger than or equal to the corresponding coverage maximum area;
the equipment virtual image module is configured with an abnormal response sub-module, and the user of the abnormal response sub-module analyzes abnormal data in the interaction index information and replaces corresponding associated user images through the reference replacement images when the abnormal data are generated.
5. The cloud-controlled interaction system based on the ultra-low delay video transmission technology as claimed in claim 1, wherein: the image caching module further comprises AR interaction information for preloading a corresponding model simulation scene, wherein the AR interaction information takes position conditions and view angle conditions as indexes;
the user control terminal also comprises an AR display module, wherein the AR display module is used for judging whether the position coordinates corresponding to the follow-up shooting equipment where the user is located meet the corresponding position conditions and whether the associated gesture data of the follow-up shooting equipment where the user is located meet the view angle conditions, and when the position conditions and the view angle conditions are met, the corresponding AR interaction information is called to be displayed at the corresponding position of the imaging picture of the video output equipment.
6. The cloud-controlled interaction system based on the ultra-low delay video transmission technology as claimed in claim 1, wherein: the control background comprises an interaction configuration module, a plurality of interaction verification units are configured in the model simulation scene, the interaction verification units are used for capturing interaction rotation data of the follow-up shooting equipment to the interaction configuration module, the interaction configuration module is used for comparing the interaction rotation data with rotation instruction data in interaction instructions to generate rotation deviation data, and correction configuration information is obtained from a pre-built deviation correction database according to the rotation deviation data and sent to the corresponding follow-up shooting equipment to configure rotation parameters of the follow-up shooting equipment.
7. The cloud-controlled interaction system based on the ultra-low delay video transmission technology as claimed in claim 6, wherein: the interactive checking unit is configured as an infrared receiving array, an infrared emitter is arranged on the follow-up shooting device, and the infrared receiving array responds to the infrared emitter to generate rotation interactive data.
8. The cloud-controlled interaction system based on the ultra-low delay video transmission technology as claimed in claim 6, wherein: the interaction verification unit is configured to be a wireless transmission device and a marker arranged in a model simulation scene, the follow-up shooting device is configured to obtain corresponding interaction rotation data by analyzing images of the marker, and according to different shooting visual angles, the content of the interaction rotation data is different, and the follow-up shooting device sends the interaction rotation data to the interaction verification unit through the wireless transmission module.
9. The cloud-controlled interaction system based on the ultra-low delay video transmission technology as claimed in claim 1, wherein: the positioning unit is arranged as a photosensitive array arranged on the ground, and the model simulation scene is provided with a fixed light source.
10. The cloud-controlled interaction system based on the ultra-low delay video transmission technology as claimed in claim 1, wherein: the positioning unit is configured as an RFID.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311334197.8A CN117097878A (en) | 2023-10-16 | 2023-10-16 | Cloud control interaction system based on ultralow-delay video transmission technology |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311334197.8A CN117097878A (en) | 2023-10-16 | 2023-10-16 | Cloud control interaction system based on ultralow-delay video transmission technology |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117097878A true CN117097878A (en) | 2023-11-21 |
Family
ID=88770237
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311334197.8A Pending CN117097878A (en) | 2023-10-16 | 2023-10-16 | Cloud control interaction system based on ultralow-delay video transmission technology |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117097878A (en) |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140192159A1 (en) * | 2011-06-14 | 2014-07-10 | Metrologic Instruments, Inc. | Camera registration and video integration in 3d geometry model |
| WO2015159289A1 (en) * | 2014-04-15 | 2015-10-22 | Navigate Surgical Technologies, Inc. | Marker-based pixel replacement |
| CN108289220A (en) * | 2018-01-15 | 2018-07-17 | 深圳市奥拓电子股份有限公司 | Virtual image processing method, image processing system and storage medium |
| CN109035185A (en) * | 2018-06-29 | 2018-12-18 | 努比亚技术有限公司 | A kind of image processing method and terminal |
| CN109389550A (en) * | 2018-09-17 | 2019-02-26 | 联想(北京)有限公司 | Data processing method, device and calculating equipment |
| CN112672057A (en) * | 2020-12-25 | 2021-04-16 | 维沃移动通信有限公司 | Shooting method and device |
| CN114913308A (en) * | 2022-05-20 | 2022-08-16 | 深圳市励得数码科技有限公司 | Camera tracking method, device, equipment and storage medium |
| CN116137655A (en) * | 2021-11-17 | 2023-05-19 | 通用汽车环球科技运作有限责任公司 | Intelligent vehicle system and control logic for surround view enhancement |
| CN116506700A (en) * | 2023-05-16 | 2023-07-28 | 深圳市南方网通网络技术开发有限公司 | Video release system based on multiple models |
-
2023
- 2023-10-16 CN CN202311334197.8A patent/CN117097878A/en active Pending
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140192159A1 (en) * | 2011-06-14 | 2014-07-10 | Metrologic Instruments, Inc. | Camera registration and video integration in 3d geometry model |
| WO2015159289A1 (en) * | 2014-04-15 | 2015-10-22 | Navigate Surgical Technologies, Inc. | Marker-based pixel replacement |
| CN108289220A (en) * | 2018-01-15 | 2018-07-17 | 深圳市奥拓电子股份有限公司 | Virtual image processing method, image processing system and storage medium |
| CN109035185A (en) * | 2018-06-29 | 2018-12-18 | 努比亚技术有限公司 | A kind of image processing method and terminal |
| CN109389550A (en) * | 2018-09-17 | 2019-02-26 | 联想(北京)有限公司 | Data processing method, device and calculating equipment |
| CN112672057A (en) * | 2020-12-25 | 2021-04-16 | 维沃移动通信有限公司 | Shooting method and device |
| CN116137655A (en) * | 2021-11-17 | 2023-05-19 | 通用汽车环球科技运作有限责任公司 | Intelligent vehicle system and control logic for surround view enhancement |
| CN114913308A (en) * | 2022-05-20 | 2022-08-16 | 深圳市励得数码科技有限公司 | Camera tracking method, device, equipment and storage medium |
| CN116506700A (en) * | 2023-05-16 | 2023-07-28 | 深圳市南方网通网络技术开发有限公司 | Video release system based on multiple models |
Non-Patent Citations (2)
| Title |
|---|
| TUULI KESKINEN: "The Effect of Camera Height, Actor Behavior, and Viewer Position on the User Experience of 360° Videos", 2019 IEEE CONFERENCE ON VIRTUAL REALITY AND 3D USER INTERFACES (VR) * |
| 王德鑫;张茂军;熊志辉;刘青宝;: "四路摄像头协同的多重触控***", 计算机辅助设计与图形学学报, no. 10 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11587292B2 (en) | Triggered virtual reality and augmented reality events in video streams | |
| CN110753218A (en) | Digital twinning system and method and computer equipment | |
| KR102069930B1 (en) | Immersion communication client and server, and method for obtaining content view | |
| CN105898346A (en) | Control method, electronic equipment and control system | |
| WO2019051115A1 (en) | Personalized live media content | |
| WO2002073955A1 (en) | Image processing apparatus, image processing method, studio apparatus, storage medium, and program | |
| CN104536579A (en) | Interactive three-dimensional scenery and digital image high-speed fusing processing system and method | |
| CN106327583A (en) | Virtual reality equipment for realizing panoramic image photographing and realization method thereof | |
| CN109561333A (en) | Video broadcasting method, device, storage medium and computer equipment | |
| CN112379815A (en) | Image capturing method and device, storage medium and electronic equipment | |
| CN108564613A (en) | A kind of depth data acquisition methods and mobile terminal | |
| CN106060523A (en) | Methods for collecting and displaying panoramic stereo images, and corresponding devices | |
| CN113485626A (en) | Intelligent display device, mobile terminal and display control method | |
| CN115379125A (en) | Interactive information sending method, device, server and medium | |
| CN112558761A (en) | Remote virtual reality interaction system and method for mobile terminal | |
| JP3122002B2 (en) | Interactive information provision device | |
| CN117097878A (en) | Cloud control interaction system based on ultralow-delay video transmission technology | |
| CN112288876A (en) | Long-distance AR identification server and system | |
| CN108712604B (en) | Panoramic shooting method and mobile terminal | |
| CN108550182B (en) | Three-dimensional modeling method and terminal | |
| CN109816795A (en) | A kind of system and method for the augmented reality AR of distributed scene target identification | |
| CN102457711A (en) | 3D (three-dimensional) digital image monitoring system and method | |
| CN114327072A (en) | Action triggering interaction method for real person and virtual object in MR virtual environment | |
| KR20120137607A (en) | Web service system and method using spherical image, web map, gis database | |
| CN214122755U (en) | Remote control system based on 5G and VR technology |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |