WO2021218683A1 - 图像处理的方法和装置 - Google Patents

图像处理的方法和装置 Download PDF

Info

Publication number
WO2021218683A1
WO2021218683A1 PCT/CN2021/088120 CN2021088120W WO2021218683A1 WO 2021218683 A1 WO2021218683 A1 WO 2021218683A1 CN 2021088120 W CN2021088120 W CN 2021088120W WO 2021218683 A1 WO2021218683 A1 WO 2021218683A1
Authority
WO
WIPO (PCT)
Prior art keywords
information
pose
server
historical
image
Prior art date
Application number
PCT/CN2021/088120
Other languages
English (en)
French (fr)
Inventor
黄茵
熊华平
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Publication of WO2021218683A1 publication Critical patent/WO2021218683A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G06T19/006—Mixed reality
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/131—Protocols for games, networked simulations or virtual reality
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/14—Picture signal circuitry for video frequency region

Definitions

  • This application relates to the field of network transmission technology, and in particular to an image processing method and device.
  • Cloud VR cloud virtual reality
  • Cloud VR is applied to human-computer interaction scenarios.
  • the second device first obtains the first image according to the game logic processing, the first device displays the first image, and the user can perform actions on the game object in the first image.
  • the first device generates an action instruction according to the detected user action, and transmits the action instruction to the second device through the network.
  • the second device obtains the second image according to the received action instruction and game logic processing, and returns the second image to the first device. After that, the first device displays the received second image again, thereby completing a game process of human-computer interaction.
  • the action command received by the second device is a time-lag command. Since the second device runs the game logic uninterruptedly, when the second device delays receiving the action instruction, the game object has already deviated, so that the user action and the game in the second image rendered by the second device according to the action instruction and the game logic The objects do not match each other, the accuracy of the image is low, and the display effect is poor. Therefore, it is urgent to provide an image processing method to improve the accuracy of the image and avoid affecting the user experience.
  • the embodiments of the present application provide an image processing method and device to solve the problem of low image accuracy provided by related technologies.
  • the technical solutions are as follows:
  • an image processing method is provided.
  • the method is applied to a network device.
  • the method includes: acquiring first pose information indicating the pose and position of the target object at the first moment, and indicating the target Historical pose information of the subject's posture and position at historical moments. Afterwards, the posture and position indicated by the first posture information are compensated according to the time delay information and the historical posture information, so as to obtain the second posture information. Then, the target instruction matching the second pose information can be obtained, and the target instruction is used to obtain the image to be displayed.
  • the embodiment of the application adjusts the first pose information through the time delay information and the historical pose information, improves the hysteresis of the target command obtained based on the first pose information, and calibrates the command error caused by the time delay, so that After the target instruction arrives at the second device with delay due to the time delay, the action described by the second pose information carried by the target instruction can still match the logic run by the second device, thereby improving the accuracy of the action response.
  • the solution is highly adaptable, can generate images or video streams with higher accuracy, and improves the user's operating experience.
  • the posture and position indicated by the first posture information are compensated according to the time delay information and the historical posture information to obtain the second posture information, including: determining according to the time delay information and the historical posture information
  • the pose change information, the pose change information includes the amount of the pose change and the direction of the pose change.
  • the pose and position indicated by the first pose information are compensated according to the amount of pose change, and the second pose information is obtained according to the compensated pose and position.
  • the posture and position compensation indicated by the first posture information is performed only when the time delay information is large, thereby avoiding the waste of processing resources.
  • determining the pose change information according to the time delay information and the historical pose information includes: determining the pose change direction and the pose change rate according to the historical pose information. Determine the amount of pose change according to the time delay information and the rate of pose change.
  • the method further includes: obtaining time delay information; in response to the time delay information being greater than the delay threshold, performing a comparison of the first pose information based on the time delay information and the historical pose information The step of compensating the posture and position indicated by the posture information to obtain the second posture information.
  • the network device is a terminal. After obtaining the target instruction matching the second pose information, the method further includes: sending the target instruction to the server, receiving the image to be displayed returned by the server according to the target instruction, and performing the image processing on the image. show.
  • the network device is a server, and after acquiring the target instruction matching the second pose information, the method further includes: acquiring the image to be displayed according to the target instruction, and sending the image to be displayed to the terminal for display.
  • an image processing device configured to acquire first pose information of a target object, where the first pose information is used to indicate the target object’s position at the first moment Posture and position;
  • the acquisition module is further configured to acquire historical pose information of the target object, where the historical pose information is used to indicate the pose and position of the target object at a historical moment;
  • the compensation module is configured to compensate the posture and position indicated by the first posture information according to the time delay information and the historical posture information to obtain second posture information, and the time delay information is based on the reference posture information
  • the time interval between the collection time and the display time of the reference image is obtained, the reference image is an image corresponding to the reference pose information, and the collection time of the reference pose information is earlier than the first time;
  • the acquiring module is further configured to acquire a target instruction matching the second pose information, and the target instruction is used to acquire an image to be displayed.
  • the compensation module is configured to determine pose change information according to the time delay information and the historical pose information, and the pose change information includes a pose change amount and a pose change direction; Based on the pose change direction, the pose and position indicated by the first pose information are compensated according to the pose change amount, and the second pose information is obtained according to the compensated pose and position.
  • the compensation module is configured to determine the pose change direction and the pose change rate according to the historical pose information; determine the pose change rate according to the time delay information and the pose change rate The amount of posture change.
  • the acquisition module is further configured to acquire the delay information; the compensation module is further configured to execute the delay information and the delay information in response to the delay information being greater than the delay threshold.
  • the historical pose information compensates for the pose and position indicated by the first pose information to obtain the second pose information.
  • the device further includes: a display module, configured to send the target instruction to the server, receive the image to be displayed returned by the server according to the target instruction, and display the image.
  • a display module configured to send the target instruction to the server, receive the image to be displayed returned by the server according to the target instruction, and display the image.
  • the device further includes a sending module, configured to obtain the image to be displayed according to the target instruction, and send the image to be displayed to the terminal for display.
  • a communication device which includes a transceiver, a memory, and a processor.
  • the transceiver, the memory, and the processor communicate with each other through an internal connection path, the memory is used to store instructions, and the processor is used to execute the instructions stored in the memory to control the transceiver to receive signals and control the transceiver to send signals And when the processor executes the instructions stored in the memory, the processor is caused to execute the method provided in the first aspect or any exemplary embodiment of the first aspect.
  • processors there are one or more processors, and one or more memories.
  • the memory may be integrated with the processor, or the memory and the processor may be provided separately.
  • the memory can be a non-transitory (non-transitory) memory, such as a read only memory (ROM), which can be integrated with the processor on the same chip, or can be set in different On the chip, the embodiment of the present application does not limit the type of the memory and the setting mode of the memory and the processor.
  • ROM read only memory
  • a computer program includes: computer program code, when the computer program code is run by a computer, the computer executes the methods in the above aspects.
  • a readable storage medium stores a program or instruction.
  • the program or instruction runs on a computer, the methods in the above aspects are executed.
  • a chip including a processor, configured to call and execute instructions stored in the memory from a memory, so that a communication device installed with the chip executes the methods in the above aspects.
  • a chip including: an input interface, an output interface, a processor, and a memory, the input interface, output interface, the processor, and the memory are connected by an internal connection path, and the processor uses To execute the code in the memory, when the code is executed, the processor is used to execute the methods in the foregoing aspects.
  • FIG. 1 is a flowchart of an image processing method provided by an embodiment of the application
  • FIG. 2 is a flowchart of an image processing method provided by an embodiment of the application
  • FIG. 3 is a schematic diagram of an image processing flow provided by an embodiment of the application.
  • FIG. 4 is a schematic diagram of a flow chart of image processing provided by an embodiment of the application.
  • FIG. 5 is a schematic diagram of pose information provided by an embodiment of the application.
  • FIG. 6 is a schematic diagram of compensating for the first pose information provided by an embodiment of the application.
  • FIG. 7 is a flowchart of an image processing method provided by an embodiment of the application.
  • FIG. 8 is a schematic diagram of an image processing flow provided by an embodiment of the application.
  • FIG. 9 is a structural diagram of an image processing apparatus provided by an embodiment of the application.
  • Virtual reality (virtual reality, VR) technology is a technology that combines computer-generated virtual content with a real environment. It has been applied to many fields such as video, games, education, and medical treatment, and has a huge market space.
  • Cloud VR introduces the concept and technology of cloud computing into VR technology, so that the calculation, rendering and other processes that originally need to be performed by the first device are transferred to the second device for execution, and the first device only needs to have decoding capabilities. Reduced the configuration requirements for the first device. Therefore, Cloud VR can reduce user consumption costs and help accelerate the popularization of VR technology. It has received widespread attention in recent years.
  • the first device often transmits instructions to the second device through the network, and the second device processes one or more frames of images according to the received instructions, and composes one or more frames of images.
  • the video stream is returned to the first device for display by the first device.
  • the instructions received by the second device are instructions that are delayed in time.
  • the images and videos generated by the second device according to the delayed instructions The accuracy of the flow is low. Therefore, the above application process needs to be adjusted in order to obtain images and video streams with higher accuracy.
  • the system includes: a terminal 101, a network node 102, and a server 103.
  • the terminal 101 is connected to the server 103 through the network node 102.
  • the terminal 101 includes, but is not limited to, a virtual reality (VR) helmet and a handle.
  • the VR helmet is used for browsing images and video streams of a target object, and the handle is used for operating the target object.
  • the VR helmet can be an all-in-one VR machine, a split VR including a mobile phone and a VR head display, or a personal computer (PC) VR.
  • One or more components such as buttons and triggers can be provided on the handle as required.
  • the network node 102 may be a device capable of connecting the terminal 101 to a communication network, such as an optical network terminal (ONT), a switch, a router, or a wireless access point (access point, AP).
  • a communication network such as an optical network terminal (ONT), a switch, a router, or a wireless access point (access point, AP).
  • ONT optical network terminal
  • AP wireless access point
  • the communication network may be the Internet, an internal network of an enterprise organization, or a local network in a certain area.
  • the server 103 may be a personal computer (PC) local to the terminal, an independent server, a server cluster composed of multiple servers, or a virtual machine or computer provided by a cloud service.
  • PC personal computer
  • the terminal 101 is used to collect the pose information of the target object, and the pose information describes the action performed by the target object.
  • the pose information is used to indicate the posture and position of the target object at a certain moment
  • the posture describes the rotation action performed by the target object
  • the position describes the stationary action or translational action performed by the target object.
  • the action performed by the target object includes the action generated by the movement of the body part of the target object and the operation performed by the target object on the handle. Actions generated by the movement of the body parts of the target object, such as the rotation of the head of the target object, the movement of the body, the waving of the hand holding the handle, and so on.
  • the operation performed by the target object on the handle for example, the target object presses a button set on the handle, pulls a trigger set on the handle, and so on.
  • the operation of the target object on the handle can be directly captured by the built-in sensor of the handle, and the action generated by the movement of the body part of the target object needs to adopt the inside-out method.
  • outside-in (outside-in) mode to capture, so as to achieve the collection of pose information.
  • the VR helmet and the built-in device of the handle will detect the change of the target object relative to the environment where the target object is located, and then calculate the body part of the target object based on the change of the target object relative to the environment where the target object is located. Perform the actions produced by the exercise.
  • built-in devices include but are not limited to sensors, cameras and other devices.
  • the VR helmet can detect the movement of the VR helmet through the built-in device, and the head movement of the target object can be calculated through the movement of the VR helmet.
  • the handle can detect the movement of the handle through the built-in device, and the movement of the handle can be used to calculate the swing of the hand of the target object holding the handle.
  • an external tracking device such as a tracking device such as a positioning base/base station, an optical camera, etc., needs to be used to track the movement of the body part of the target object.
  • the terminal 101 is also used to access the communication network through the network node 102, and send the collected pose information to the server 103.
  • the server 103 is configured to obtain an instruction matching the received first pose information, so as to perform calculation and rendering according to the instruction matching the pose information, and generate an image or video stream corresponding to the pose information.
  • the server 103 encodes the image or video stream, it returns the encoded image or video stream to the terminal 101 through the communication network.
  • the terminal 101 is also used to decode the encoded image or video stream, and project the image or video stream through the VR helmet to display the image or video stream, that is, the target object can be browsed to the image or video stream through the VR helmet .
  • any one of the terminal 101 or the network node 102 and the server 103 constitute a delay detection subsystem, and the delay detection subsystem is used to detect delay information. That is to say, in this embodiment, the terminal 101 and the server 103 may constitute the delay detection subsystem, and the network node 102 and the server 103 may also constitute the delay detection subsystem.
  • the time delay information is obtained based on the time interval between the collection time of the pose information and the display time of the image corresponding to the pose information. Since the terminal 101 can be used to collect the pose information, the collection moment of the pose information is the moment when the terminal 101 collects the pose information. Exemplarily, in this embodiment, the time when the terminal 101 collects the pose information and the time when the pose information is generated (that is, the object performs an action) can be regarded as the same time, in other words, the collection time of the pose information and the generation time of the pose information The time interval between can be ignored.
  • the terminal 101 After the terminal 101 collects the pose information, it will send the pose information to the server 103, and the server 103 will process the pose information based on the received pose information, thereby returning the image or video stream corresponding to the pose information.
  • the time when the terminal 101 displays the image or video stream returned by the server 103 is the display time of the image corresponding to the above-mentioned pose information. Since the collection time of the pose information and the display time of the image corresponding to the pose information involve the process of information transmission and processing, and the process of transmission and processing takes time, there is a time interval between the collection time and the display time , The time interval can be used as the above-mentioned time delay information.
  • the above-mentioned time delay information may be determined based on the pose information of the target object at the historical moment, or it may be based on the historical information of other objects other than the target object.
  • the pose information at the time determines the above-mentioned time delay information.
  • the time delay information in this embodiment can not only be determined based on the pose information based on historical moments, but also can be determined in real time based on the pose information of the target object or other objects at the current moment. The moment of the pose information is limited.
  • the delay detection subsystem includes a client and a server.
  • the client is used to trigger the detection of delay information
  • the server is used to cooperate with the client to complete the detection of delay information.
  • the terminal 101 may be used as a client and the server 103 may be used as a server, or the terminal 101 may be used as a server and the server 103 may be used as a client.
  • the network node 102 can be used as a client and the server 103 can be used as a server, or the network node 102 can be used as a server and the server 103 can be used as a client.
  • the delay information detected by the delay detection subsystem can be the network delay caused by the transmission of pose information between the terminal 101 and the server 103, or the network delay generated by the terminal 101 or the server 103 during operation.
  • other delays include, but are not limited to, when the terminal 101 sends the pose information to the server 103, the pose information is waiting to be sent in the sending queue of the terminal 101, and the terminal 101 sends the pose information from the sending queue. The delay in sending to the network.
  • the client in the delay detection subsystem sends a detection signal containing the transmission time to the server to trigger the detection of delay information.
  • the server can calculate the difference between the time when the detection signal is received and the time before the transmission time included in the detection signal. Since the difference is the time required for the detection signal to be transmitted between the terminal 101 and the server 103 in the communication network, the difference can be regarded as the network delay.
  • the network delay to be detected is the network delay between the terminal 101 and the server 103
  • the client or server in the delay detection subsystem is the network node 102
  • the network The position of the node 102 in the communication network needs to be close to the terminal.
  • the difference between the length of time the detection signal is transmitted between the network node 102 and the server 103 and the time length of the transmission between the terminal 101 and the server 103 can be close to zero, so that the difference between the network node 102 and the server 103 can be reduced.
  • the network delay between the terminal 101 and the server 103 replaces the network delay.
  • this embodiment may use software probes to detect other time delays.
  • software probes can add time stamps to the services performed by the device, and other time delays can be calculated by calculating the time stamp difference between different services.
  • the software probe can add the first time stamp to record the current time when the terminal 101 executes the service of "adding the collected pose information to the sending queue".
  • 101 executes the service of "is about to send the pose information in the sending queue” and adds a second time stamp to record the current time.
  • the time stamp difference between the first time stamp and the second time stamp is calculated to obtain the queuing delay of the pose information waiting to be sent in the sending queue. For different delays other than the queuing delay, we will not repeat them here.
  • the system provided in this embodiment also includes a computing node, which is used to obtain the pose and position indicated by the pose information of the target object based on the delay information and the historical pose information. Compensation is performed to obtain the second pose information, so as to further obtain a target instruction that matches the second pose information and is used to obtain the image to be displayed.
  • the historical pose information is used to indicate the pose and position of the target object at a historical moment.
  • the computing node may be the terminal 101 or the server 103. Next, the process of obtaining historical pose information is described first, and then the two cases where the computing nodes are the terminal 101 and the server 103 are respectively described:
  • the historical pose information is information collected by the terminal 101. After the terminal 101 collects the historical pose information, it can store the historical pose information locally in the terminal 101, or send the historical pose information to the server 103 for storage, or send the historical pose information to a server other than the server 103. Other storage devices for storage. Therefore, no matter the computing node is the terminal 101 or the server 103, the historical pose information can be obtained from the device that stores the historical pose information.
  • the pose information of the target object is used to indicate the pose and position of the target object at the current moment
  • the historical pose information is used to indicate the pose and position of the target object at the historical moment.
  • current time may refer to one instantaneous moment, or may refer to a period of time composed of multiple instantaneous moments.
  • the target object's hand When performing a wave motion, the target object's hand will move from one place to another. This movement process is often not completed in one instant, but needs to occupy multiple instantaneous moments. A period of time composed. Therefore, when the current moment and the historical moment represent an instantaneous moment, the action described by the posture information is the action at a certain point passed by the hand in the process of moving. In this case, the historical posture information and the posture information of the target object can both be actions at a certain point in the process of waving the hand, but the action described by the historical posture information occurs in the action described by the posture information of the target object Before.
  • the posture information of the target object describes the complete movement of the target object's hand from one place to another. Therefore, in this case, what the historical posture information describes is one or more other complete actions that occurred before the complete action of waving the hand of the target object.
  • This embodiment does not limit the time indicated by the time, and the above two cases can be selected according to needs.
  • the computing node is the terminal 101.
  • the terminal 101 After the terminal 101 locally collects the pose information of the target object, when the terminal 101 obtains the delay information, in response to the terminal 101 being the server in the delay detection subsystem, the terminal 101 can receive the transmission from the client. Time detection signal, so as to obtain time delay information through calculation. In response to the terminal 101 being the client in the delay detection subsystem, the terminal 101 can obtain the delay information from the server in the delay detection subsystem. In addition, for the case where the terminal 101 does not constitute a delay detection subsystem, the terminal 101 may also obtain delay information from any one of the network nodes or servers in the delay detection subsystem.
  • the terminal 101 may obtain it locally, the server 103, or other devices that store the historical pose information. Then, the terminal can compensate the pose and state indicated by the collected pose information according to the acquired time delay information and historical pose information to obtain new pose information, thereby determining to match the new pose information The target instruction.
  • the terminal 101 After the terminal 101 obtains the target instruction, the terminal 101 also needs to send the target instruction to the server 103, so that the server 103 performs calculation and rendering according to the target instruction, and encodes the rendered image or video stream back to the terminal 101. After that, the terminal 101 can display the received image or video stream.
  • the pose information of the target object is still collected by the terminal, so the server 103 needs to receive the pose information sent by the terminal 101, so as to realize the acquisition of the pose information.
  • the server 103 may receive the detection signal containing the transmission time sent by the client, and thereby obtain the delay information through calculation.
  • the server 103 can obtain the delay information from the server in the delay detection subsystem.
  • the server 103 may obtain the historical action information from the terminal 101, locally, or other devices that store historical pose information.
  • the server 103 compensates for the posture and position indicated by the posture information through the time delay information and the historical posture information. Perform calculation and rendering to obtain an image or video stream, and encode and push the image or video stream to the terminal 101. It should be noted that the server 103 can call one or more bypass devices to perform the above-mentioned delay information detection and pose information compensation, or it can directly perform detection and compensation without calling the bypass device. In this embodiment, the server 103 Whether to call the bypass device is not limited.
  • the terminal 101, the network node 102, and the server 103 shown in FIG. 1 can constitute the following four systems. It can be understood that the following four systems are only examples. In practical applications, the system provided in this embodiment is not limited to the four systems in the following examples, and may also include other different systems based on the combination of the terminal 101, the network node 102, and the server 103:
  • the first system the terminal 101 and the server 103 are used to form a time delay detection subsystem to detect the time delay, and the terminal 101 is used to perform the compensation calculation of the pose information.
  • the delay detection subsystem includes a client and a server
  • the first system can be further divided into the two combinations in Table 1:
  • Delay detection subsystem client Delay detection subsystem server Compensation calculation of pose information terminal server terminal server terminal terminal
  • the second system the terminal 101 and the server 103 are used to form a time delay detection subsystem, and the server 103 is used to perform the compensation calculation of the pose information.
  • the second system can be divided into two combinations as shown in Table 2:
  • Delay detection subsystem client Delay detection subsystem server Compensation calculation of pose information terminal server server server terminal server
  • the third system the network node 102 and the server 103 constitute a delay detection subsystem, and the terminal 101 is used to perform the compensation calculation of the pose information.
  • the third system can be divided into two combinations as shown in Table 3:
  • Delay detection subsystem client Delay detection subsystem server Compensation calculation of pose information Network node server terminal server Network node terminal
  • the fourth system the network node 102 and the server 103 constitute a delay detection subsystem, and the server 103 is used to perform the compensation calculation of the pose information.
  • the fourth system can be divided into two combinations shown in Table 4:
  • Delay detection subsystem client Delay detection subsystem server Compensation calculation of pose information Network node server server server Network node server
  • this embodiment also provides an image processing method, which can be applied to a network device, and the network device may be the terminal shown in FIG. 1, or the terminal shown in FIG. Server.
  • the method in response to the network device being a terminal, the method includes:
  • Step 201 Acquire first pose information of the target object, where the first pose information is used to indicate the pose and position of the target object at the first moment.
  • the first pose information describes the actions performed by the target object at the first moment by indicating the pose and position of the target object at the first moment.
  • the actions performed by the target object include actions generated by the movement of the body parts of the target object and operations performed by the target object on VR devices such as handles.
  • the actions generated by the movement of the body parts of the target object are, for example, the rotation of the head of the target object, the movement of the body, and the waving of the hand holding the handle, etc.
  • the operation performed by the target object on the VR device is, for example, the pressing of the target object. Buttons set on the device, etc.
  • the actions performed by the target object can be categorized into three different types including stationary actions, translation actions, and rotation actions.
  • the hand waving of the target object can be regarded as a translational movement
  • the head rotation of the target object can be regarded as a rotating movement.
  • the first posture information may include one or both of the first posture information and the first position information, the first posture information is used to describe the rotation action performed by the target object, and the first position information is used to describe Static motion and translational motion performed by the target object.
  • the space where the target object is located can be represented by three-dimensional coordinates, and the three-dimensional coordinate system includes X, Y, and Z axes that are perpendicular to each other.
  • the any body part of the target object is fixed at a position, and the first position information can be represented by a coordinate point in the three-dimensional coordinate system.
  • the first position information includes multiple coordinate points in the three-dimensional coordinate system.
  • the translational change can be regarded as a translational change along at least one of the X axis, the Y axis, and the Z axis in the three-dimensional coordinate system.
  • the first posture information includes the rotation angles in various directions at a coordinate point.
  • the rotation process can be regarded as a rotation around at least one of the X-axis, Y-axis, and Z-axis in the three-dimensional coordinate system.
  • Step 202 Obtain historical pose information of the target object.
  • the historical pose information is used to indicate the pose and position of the target object at a historical moment.
  • the historical moment is any moment earlier than the above-mentioned first moment. Therefore, the action described by the historical pose information is an action that has been completed by the target object before the action described by the first pose information.
  • the first moment and the historical moment may represent an instantaneous moment, or may represent a period of time composed of multiple instantaneous moments.
  • the target object's hand When performing a wave motion, the target object's hand will move from one place to another. This movement process is often not completed in one instant, but needs to occupy multiple instantaneous moments.
  • the composition takes a while to complete. Therefore, when the first moment and the historical moment represent an instantaneous moment, the actions described by the first posture information and the historical posture information are all actions at a certain point passed by the hand in the process of moving. . Assuming that the hand of the target object first passes the first point and then the second point during the movement process, since the action described by the historical pose information is completed earlier than the action described by the first pose information, the historical pose The information describes the action at the first point, and the first pose information describes the action at the second point.
  • the first posture information describes the complete movement of the target object's hand from one place to another. Therefore, what the historical posture information describes is one or more other complete actions that occurred before the complete action of waving the hand of the target object.
  • This embodiment does not limit the above two cases, and can be selected according to the action type of the target object and actual needs.
  • Step 203 Compensate the posture and position indicated by the first posture information according to the time delay information and the historical posture information to obtain the second posture information.
  • the time delay information is based on the collection time of the reference posture information and the display of the reference image. The time interval between the times is obtained, the reference image is the image corresponding to the reference pose information, and the collection time of the reference pose information is earlier than the first moment.
  • the reference pose information can be used to indicate the posture and position of the target object at a certain moment earlier than the first moment, or the reference pose information can also be used to indicate that other objects other than the target object are earlier than the first moment.
  • this embodiment does not limit the posture and position of which object the reference pose information is used to indicate.
  • the collection time of the reference pose information and the generation time of the reference pose information can be regarded as the same time, or the time interval between the collection time and the generation time can be ignored.
  • the terminal After the reference pose information is collected at a certain time earlier than the first moment, the terminal will obtain an instruction that matches the reference pose information (or the compensated reference pose information), and send the instruction to the server, which is based on the instruction Obtain the reference image (or video stream) corresponding to the reference pose information, and the terminal can receive and display the reference image returned by the server.
  • the time when the terminal displays the reference image is the display time of the above-mentioned reference image. Since the acquisition time and the display time involve at least the process of acquiring instructions, instruction transmission, reference image generation, and reference image transmission, and may also involve the process of compensating the reference pose information, there will be a gap between the acquisition time and the display time. Time interval, which can be used as time delay information for compensating the posture and position indicated by the first posture information.
  • the time delay information when compensating the posture and position indicated by the first posture information, can be determined according to the reference posture information before the first posture information.
  • the time delay information may be calculated after acquiring the collection time of the reference pose information and the display time of the reference image, and the time delay information may be stored. Afterwards, when the posture and position indicated by the first posture information need to be compensated, the stored time delay information is directly read to facilitate the completion of the compensation process.
  • this embodiment can also detect the time delay information in real time when the posture and position indicated by the first posture information need to be compensated.
  • the time delay information can be the network delay between the terminal and the server, or the equipment delay generated during the operation of the terminal and the server.
  • the delay information between the terminal and the server is determined by the delay detection subsystem. Referring to Figure 3, the delay detection subsystem can be composed of a terminal and a server. Or, referring to FIG. 4, the delay detection subsystem may also be composed of a network node and a server.
  • the terminal and server shown in Figure 3 are used as the delay detection subsystem
  • the terminal in response to the terminal being the delay detection client and the server being the server, the terminal sends a detection signal to the server, and the server calculates the sending time of the detection signal sent by the terminal
  • the difference between the reception time of the detection signal and the local reception time is used as the delay information between the terminal and the server.
  • the server may send the calculated delay information to the terminal, so that the terminal can obtain the delay information.
  • the terminal In response to the situation that the terminal is the delay detection server and the server is the client, the terminal will receive the detection signal sent by the server, and the terminal calculates the difference between the time when the server sends the detection signal and the reception time when the detection signal is received locally . After the terminal calculates the difference, the difference can be used as the delay information, thereby realizing the acquisition of the delay information.
  • the network node or server calculates the difference between the sending time and the receiving time of the detection signal to obtain the delay information. After that, the network node or server that has calculated the delay information can directly send the delay information to the terminal, so that the terminal can obtain the delay information. Alternatively, the network node or server that has calculated the delay information may also wait for the acquisition request sent by the terminal, and then return the delay information to the terminal according to the acquisition request, so that the terminal can also obtain the delay information.
  • the terminal After obtaining the time delay information, the terminal can compensate the posture and position indicated by the first posture information according to the time delay information and the historical posture information obtained in step 202, so as to avoid the posture and position indicated by the first posture information.
  • the generated instruction has a time lag, which affects the subsequent image processing process, and makes the generated image less accurate.
  • compensating the posture and position indicated by the first posture information according to the time delay information and the historical posture information to obtain the second posture information includes the following steps 2031 and 2032:
  • Step 2031 Determine pose change information according to the time delay information and the historical pose information, where the pose change information includes the amount of pose change and the direction of the pose change.
  • determining the pose change information according to the time delay information and the historical pose information includes: determining the pose change direction and the pose change rate according to the historical pose information, and determining the pose change rate according to the time delay information and the pose change rate The direction of posture change.
  • the movement trend of the target object can be determined according to the historical pose information of the target object, so as to determine the direction of the pose change according to the movement trend.
  • the direction of the pose and position indicated by the historical pose information may be used as the pose change direction.
  • the direction of the movement is constantly changing or even turned in a wave, zigzag, etc.
  • the direction of the pose change may be the same or opposite to the direction of the posture and position indicated by the historical pose information, or it may be There is a deviation from the direction of the posture and position indicated by the historical pose information.
  • the direction change law of the action trend can be summarized according to at least three historical pose information, such as the period of the direction change or the rotation, and the direction change law is combined to determine the direction of the pose change.
  • t1 and t2 are two adjacent historical moments, and t1 is a moment earlier than t2.
  • the coordinates of the position A of the target object at time t1 in the three-dimensional coordinate system are (x1, y1, z1)
  • the coordinates of the position B of the target object at time t2 in the three-dimensional coordinate system are (x2, y2, z2).
  • the direction from position A to position B can be directly used as the direction of pose change.
  • the direction of the pose change may be the direction where position A points to position B, the direction where position B points to position A, or the direction where position B points to any position other than position A.
  • Pose change rate [(x2, y2, z2)-(x1, y1, z1)]/(t2-t1).
  • the amount of change in the pose during the time delay can be obtained.
  • the amount of pose change can be calculated according to the following formula:
  • the amount of pose change [(x2, y2, z2)-(x1, y1, z1)] ⁇ (t3-t2)/(t2-t1), where (t3-t2) is the above-mentioned delay information.
  • Step 2032 Compensate the posture and position indicated by the first posture information according to the posture change amount based on the posture change direction, and obtain the second posture information according to the compensated posture and position.
  • this embodiment sums the first pose information and the pose change information in the pose change direction, and uses the sum of the first pose information and the pose change as the second pose information, In this way, the compensation of the first pose information is realized, and the second pose information is obtained.
  • the second pose information can be calculated according to the following formula:
  • (x3, y3, z3) is the second pose information.
  • the purpose of compensating the first pose information is to improve the lag of the subsequent generated instructions, and avoid the lag of the instructions from reducing the accuracy of the subsequent generated images or video streams, thereby affecting the user experience.
  • the lag of the command is weak, even if the command has a lag, it will not affect the accuracy of the image or video stream. Therefore, if the first pose information is still compensated when the instruction lag is weak and the accuracy of the image or video stream is not affected, the processing resources of the terminal, network node, and server will be wasted.
  • the method further includes: obtaining time delay information, and in response to the time delay information being greater than the time delay threshold, in response to the time delay information being greater than the time delay threshold, executing The step of compensating the posture and position indicated by the first posture information according to the time delay information and the historical posture information to obtain the second posture information.
  • the delay threshold can be set according to actual needs or experience.
  • the delay threshold can be 30 milliseconds, or other values other than 30 milliseconds.
  • Step 204 Obtain a target instruction matching the second pose information, where the target instruction is used to obtain an image to be displayed.
  • the terminal may encapsulate the second pose information and other information that needs to be used in the transmission process to obtain a target instruction that matches the second pose information.
  • Other information that needs to be used during transmission includes, but is not limited to, transmission protocol information, routing information, and so on.
  • the method further includes: sending the target instruction to the server, receiving the image to be displayed returned by the server according to the target instruction, and displaying the image.
  • the first pose information is the pose information at time t2. Since there is a time delay between the terminal and the server, and the logic on the server runs uninterrupted, when the server receives an instruction that matches the first pose information, the logic that the server runs is the logic at time t3, and the logic at time t2 The first posture information is lagging behind the logic run by the server.
  • the server generates an image or video stream based on the logic running at t3 and the first pose information at t2, which will cause a mismatch between the action described by the first pose information at t2 and the logic.
  • the accuracy is low. For example, in a human-computer interaction game, a mismatch between the action and the logic will cause the game object that the target object could have hit cannot be hit, which affects the use experience of the target object.
  • the target pose information included in the target instruction sent by the terminal to the server at time t2 is the second pose information corresponding to time t3.
  • the second pose information included in the target instruction is the pose information at time t3, it matches the logic running on the server, so that the target motion of the target object matches the logic, and the accuracy of the video stream is high.
  • the server After the server receives the target instruction sent by the terminal, it calculates and renders based on the target instruction to obtain an image or video stream, then encodes the image or video stream, and sends the encoded image or video stream to the terminal for convenience
  • the terminal displays. Since the server sends the encoded image or video stream, the terminal can receive the encoded image or video stream accordingly. After receiving, the terminal can decode the encoded image or video stream to obtain the original target video stream. After receiving the image or video stream sent by the server, the terminal also displays the target video stream. In other words, the terminal can display the target video stream through a VR helmet or other display device, so that the user can browse the target video stream with higher accuracy.
  • this embodiment can provide a terminal as the client of the delay detection subsystem, the server as the server of the delay detection subsystem, and the terminal is used to compensate for the posture and position indicated by the first pose information system.
  • the terminal may collect the first pose information.
  • the first device also needs to detect the delay information with the server through interaction with the server, so as to realize the acquisition of the delay information.
  • the terminal also needs to obtain historical pose information required for compensation calculation. After the terminal obtains the delay information and the historical pose information, the terminal may compensate and adjust the first pose information according to the delay information and the historical pose information in step 304 to obtain the second pose information.
  • the terminal obtains the target instruction matching the second pose information, encapsulates the target instruction into a data packet, and sends the data packet to the server.
  • the server may perform calculation and rendering based on the received target instruction, and generate an image or video stream for display.
  • the image or video stream needs to be encoded, and the encoded image or video stream is returned to the terminal.
  • the terminal can also decode and display the received encoded image or video stream.
  • this embodiment can also provide a network node as the client of the delay detection subsystem, the server as the server of the delay detection subsystem, and the terminal is used to compensate for the posture and position indicated by the first pose information. system.
  • the terminal may collect the first pose information. Since the network node and the server are respectively the client and the server in the delay detection subsystem, the delay information between the terminal and the server is obtained through the interaction between the network node and the server in step 402.
  • the terminal can obtain the delay information from any device in the network node or the server.
  • the terminal also needs to obtain historical pose information, so that in step 404, according to the delay information and the historical pose information, the first The posture and position indicated by the posture information are adjusted to obtain the second posture information.
  • the terminal obtains the target instruction matched by the second pose information, encapsulates the target instruction into a data packet, and sends the data packet to the server.
  • the server may perform calculation and rendering based on the received target instruction, and generate an image or video stream for display.
  • the server after the server generates the image or video stream, it also needs to encode the image or video stream, and return the encoded image or video stream to the terminal.
  • the terminal can also decode and display the received encoded image or video stream.
  • the embodiment of the present application compensates the first pose information through the time delay information and the historical pose information, improves the lag of the target command, and calibrates the command error caused by the time delay, so that the target command is due to After the delay in reaching the server due to the existence of time delay, the posture and position indicated by the second posture information matched by the target instruction can still match the logic of the server operation, thereby improving the accuracy of the action response.
  • the solution is highly adaptable, can generate images or video streams with higher accuracy, and improves the user's operating experience.
  • this embodiment provides an image processing method, which is applied to a network device, and the network device may be the server shown in FIG. 1.
  • the method includes:
  • Step 701 Acquire first pose information, where the first pose information is used to indicate the pose and position of the target object at the first moment.
  • the terminal since the terminal has the function of collecting the first pose information, the first pose information is acquired by the terminal side. Since the server needs to compensate for the first pose information, the server needs to first obtain the first pose information to be adjusted from the terminal.
  • the first pose information acquired by the server may be the first pose information pushed by the terminal, and also the first pose information returned by the terminal according to the acquisition request sent by the server.
  • Step 702 Obtain historical pose information, which is used to indicate the pose and position of the target object at a historical moment.
  • the historical pose information is also collected by the terminal.
  • the server needs to obtain the historical pose information from the terminal.
  • the server can obtain the historical pose information locally.
  • the server needs to obtain the historical pose information from the other device.
  • Step 703 Compensate the posture and position indicated by the first posture information according to the time delay information and the historical posture information to obtain second posture information.
  • the delay information can be determined by the delay detection subsystem.
  • the delay detection subsystem can be composed of a terminal and a server. Or, referring to FIG. 8, the delay detection subsystem may also be composed of a network node and a server.
  • the terminal and the server are used as the delay detection subsystem
  • the terminal in response to the terminal being the delay detection client and the server being the server, the terminal sends a detection signal to the server, and the server calculates the sending time of the terminal sending the detection signal and the local receiving the detection signal
  • the difference between the receiving time of, the difference is used as the delay information between the terminal and the server, so that the server can obtain the delay information.
  • the server In response to the situation that the terminal is the delay detection server and the server is the client, the server first sends a detection signal to the terminal, the terminal will receive the detection signal sent by the server, and the terminal calculates the sending time of the detection signal sent by the server and the detection signal received locally The difference between the receiving time. After the terminal calculates the difference, it can return the difference to the server, so that the server can obtain the delay information.
  • the network node and the server are used as the delay detection subsystem
  • the network node in response to the network node being the delay detection client and the server being the server, the network node sends a detection signal to the server, and the server computing network node sends The difference between the sending time of the detection signal and the receiving time of the local reception of the detection signal is used as the delay information between the network node and the server, so that the server can obtain the delay information.
  • the server In response to the situation that the network node is the delay detection server and the server is the client, the server first sends a detection signal to the network node, the network node will receive the detection signal sent by the server, and the network node calculates the time when the server sends the detection signal and the local The difference between the reception time when the detection signal is received. After the network node calculates the difference, the difference can be returned to the server, so that the server can obtain the delay information.
  • step 203 For the process of compensating the posture and position indicated by the first posture information according to the time delay information and the historical posture information, refer to the description in step 203, which will not be repeated here.
  • Step 704 Obtain a target instruction matching the second pose information, where the target instruction is used to obtain an image to be displayed.
  • the method further includes: obtaining the image to be displayed according to the target instruction, and sending the image to be displayed to the terminal for display.
  • the server can obtain the image or video stream to be displayed by performing calculation and rendering processes based on the target instruction.
  • the terminal After receiving the image or video stream sent by the server, the terminal can be used to display the image or video stream.
  • this embodiment can provide a network node as the client of the delay detection subsystem, the server as the server of the delay detection subsystem, and the server is used to compare the posture and posture indicated by the first pose information. Position compensation system.
  • the terminal since the terminal is a device that can perform human-computer interaction with the target object, the collection of the first pose information is executed by the terminal.
  • the terminal sends the first pose information to the server, and accordingly, the server can receive the first pose information and historical pose information to be sent by the terminal.
  • the server also needs to detect the delay information with the terminal through interaction with the network node, so as to realize the acquisition of the delay information.
  • step 804 in addition to acquiring time delay information, the server also needs to acquire historical pose information required to compensate for the first pose information.
  • step 805 after the server obtains the delay information and the historical pose information, the first pose information may be compensated according to the delay information and the historical pose information, so as to obtain the second pose information.
  • step 806 since the image or video stream can be generated on the server side, after the server obtains the target instruction matching the second pose information, it can continue to calculate and render based on the target instruction to generate an image for display Video streaming.
  • the server After the server generates the image and video stream, referring to step 807, the server also needs to encode the image or video stream, and return the encoded image or video stream to the terminal. In the subsequent process, the terminal can also decode and display the received encoded image or video stream.
  • the embodiment of the present application compensates the first pose information through the time delay information and the historical pose information, improves the lag of the target command, and calibrates the command error caused by the time delay, so that the target command is due to After the delay in reaching the server due to the existence of time delay, the posture and position indicated by the second posture information matched by the target instruction can still match the logic of the server operation, thereby improving the accuracy of the action response.
  • the solution is highly adaptable, can generate images or video streams with higher accuracy, and improves the user's operating experience.
  • an embodiment of the present application also provides an image processing device, which includes:
  • the obtaining module 901 is configured to obtain first pose information of the target object, and the first pose information is used to indicate the pose and position of the target object at the first moment;
  • the acquiring module 901 is also used to acquire historical pose information of the target object, and the historical pose information is used to indicate the pose and position of the target object at a historical moment;
  • the compensation module 902 is used to compensate the posture and position indicated by the first posture information according to the time delay information and the historical posture information to obtain the second posture information.
  • the time delay information is based on the collection time and reference of the reference posture information The time interval between the display moments of the image is obtained, the reference image is the image corresponding to the reference pose information, and the collection time of the reference pose information is earlier than the first moment;
  • the acquiring module 901 is used to acquire a target instruction matching the second pose information, and the target instruction is used to acquire an image to be displayed.
  • the compensation module 902 is used to determine the pose change information according to the time delay information and the historical pose information.
  • the pose change information includes the amount of pose change and the direction of the pose change; based on the direction of the pose change, according to The pose change amount compensates for the pose and position indicated by the first pose information, and the second pose information is obtained according to the compensated pose and position.
  • the compensation module 902 is configured to determine the pose change direction and the pose change rate according to historical pose information; determine the pose change amount according to the time delay information and the pose change rate.
  • the obtaining module 901 is also used to obtain time delay information; the compensation module is also used to respond to the time delay information being greater than the time delay threshold, perform a comparison of the first position based on the time delay information and the historical pose information. The posture and position indicated by the posture information are compensated to obtain the second posture information.
  • the device further includes a display module, configured to send a target instruction to the server, receive the image to be displayed returned by the server according to the target instruction, and display the image.
  • a display module configured to send a target instruction to the server, receive the image to be displayed returned by the server according to the target instruction, and display the image.
  • the device further includes: a sending module, configured to obtain the image to be displayed according to the target instruction, and send the image to be displayed to the terminal for display.
  • a sending module configured to obtain the image to be displayed according to the target instruction, and send the image to be displayed to the terminal for display.
  • An embodiment of the present application provides a communication device, which includes a transceiver, a memory, and a processor.
  • the transceiver, the memory, and the processor communicate with each other through an internal connection path, the memory is used to store instructions, and the processor is used to execute the instructions stored in the memory to control the transceiver to receive signals and control the transceiver to send signals And when the processor executes the instructions stored in the memory, the processor is caused to execute the method provided in any one of the exemplary embodiments of steps 201-204.
  • An embodiment of the present application provides a communication device, which includes a transceiver, a memory, and a processor.
  • the transceiver, the memory, and the processor communicate with each other through an internal connection path, the memory is used to store instructions, and the processor is used to execute the instructions stored in the memory to control the transceiver to receive signals and control the transceiver to send signals , And when the processor executes the instructions stored in the memory, the processor is caused to execute the method provided in any one of the exemplary embodiments of steps 701 to 704.
  • processors there are one or more processors, and one or more memories.
  • the memory may be integrated with the processor, or the memory and the processor may be provided separately.
  • the memory can be a non-transitory (non-transitory) memory, such as a read only memory (ROM), which can be integrated with the processor on the same chip, or can be set in different On the chip, the embodiment of the present application does not limit the type of the memory and the setting mode of the memory and the processor.
  • ROM read only memory
  • the embodiment of the present application provides a computer program (product), the computer program (product) includes: computer program code, when the computer program code is run by a computer, the computer executes any one of the exemplary embodiments of the present application. method.
  • the embodiments of the present application provide a readable storage medium, and the readable storage medium stores a program or instruction.
  • the program or instruction runs on a computer, the methods in the foregoing aspects are executed.
  • An embodiment of the present application provides a chip, including a processor, configured to call and execute instructions stored in the memory from a memory, so that a communication device installed with the chip executes any of the exemplary embodiments of the present application The method provided.
  • the embodiment of the present application provides a chip including: an input interface, an output interface, a processor, and a memory.
  • the input interface, output interface, the processor, and the memory are connected by an internal connection path, and the processor uses To execute the code in the memory, when the code is executed, the processor is used to execute the method provided in any exemplary embodiment of the present application.
  • processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processing (digital signal processing, DSP), and application specific integrated circuits. ASIC), field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc.
  • the general-purpose processor may be a microprocessor or any conventional processor. It is worth noting that the processor may be a processor that supports an advanced reduced instruction set machine (advanced RISC machines, ARM) architecture.
  • the foregoing memory may include a read-only memory and a random access memory, and provide instructions and data to the processor.
  • the memory may also include non-volatile random access memory.
  • the memory can also store device type information.
  • the memory may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory.
  • the non-volatile memory can be read-only memory (ROM), programmable read-only memory (programmable ROM, PROM), erasable programmable read-only memory (erasable PROM, EPROM), and electrically available Erase programmable read-only memory (electrically EPROM, EEPROM) or flash memory.
  • the volatile memory may be random access memory (RAM), which is used as an external cache. By way of exemplary but not limiting illustration, many forms of RAM are available.
  • static random access memory static random access memory
  • dynamic random access memory dynamic random access memory
  • DRAM dynamic random access memory
  • SDRAM synchronous dynamic random access memory
  • double data rate synchronous dynamic random access Memory double data date SDRAM, DDR SDRAM
  • enhanced synchronous dynamic random access memory enhanced SDRAM, ESDRAM
  • serial link DRAM SLDRAM
  • direct memory bus random access memory direct rambus RAM
  • This application provides a computer program.
  • the computer program When the computer program is executed by a computer, it can cause a processor or computer to execute the corresponding steps and/or processes in the foregoing method embodiments.
  • the computer program product includes one or more computer instructions.
  • the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.
  • the computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website, computer, server, or data center.
  • the computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or a data center integrated with one or more available media.
  • the usable medium may be a magnetic medium (for example, a floppy disk, a hard disk, and a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk).

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Signal Processing (AREA)
  • Computer Graphics (AREA)
  • Software Systems (AREA)
  • Computer Hardware Design (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Human Computer Interaction (AREA)
  • Multimedia (AREA)
  • Information Transfer Between Computers (AREA)
  • Processing Or Creating Images (AREA)

Abstract

本申请公开了图像处理的方法和装置，属于网络传输技术领域。该方法应用于网络设备，方法包括：获取用于指示目标对象在第一时刻的姿态与位置的第一位姿信息，以及用于指示目标对象在历史时刻的姿态与位置的历史位姿信息。之后，根据时延信息及历史位姿信息对第一位姿信息所指示的姿态与位置进行补偿，从而得到第二位姿信息。接着，便可获取与第二位姿信息匹配的目标指令，该目标指令用于获取待显示的图像。本申请校准了由于时延而引起的指令误差，提高了对象动作响应的精准度。不仅自适应性较强，而且能够处理得到精准度较高的图像，提升了用户的操作体验。

Description

图像处理的方法和装置
本申请要求于2020年4月30日提交的申请号为202010366426.4、发明名称为“图像处理的方法和装置”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及网络传输技术领域，特别涉及一种图像处理的方法和装置。
背景技术
随着网络传输技术的发展，基于网络传输的云化虚拟现实(cloud virtual reality，Cloud VR)逐渐被应用于人们的生活中。在Cloud VR的应用过程中，第一设备往往会通过网络将指令传输至第二设备，第二设备根据接收到的指令进行图像处理，将得到的图像返回给第一设备，以便于第一设备对图像进行显示。可见，第一设备所发送的指令是进行图像处理的关键。
相关技术中，将Cloud VR应用于人机交互场景中。以人机交互游戏为例，第二设备首先根据游戏逻辑处理得到第一图像，第一设备对第一图像进行显示，用户可针对第一图像中的游戏对象进行动作。第一设备根据检测到的用户动作生成动作指令，通过网络将动作指令传输至第二设备。第二设备根据接收到的动作指令及游戏逻辑处理得到第二图像，将第二图像返回给第一设备。之后，第一设备再对接收到的第二图像进行显示，从而完成一次人机交互的游戏过程。
然而，由于网络传输速率及传输距离的限制，通过网络将动作指令传输至第二设备的过程不可避免地存在延迟，也就是说，第二设备接收到的动作指令是在时间上滞后的指令。由于第二设备不间断地运行游戏逻辑，因而第二设备延迟接收到动作指令时，游戏对象已经发生偏离，从而使得第二设备根据动作指令及游戏逻辑渲染得到的第二图像中用户动作与游戏对象之间互不匹配，图像的精准度较低、显示效果较差。因此，亟待提供一种图像处理的方法，以提高图像的精准度，避免对用户的使用体验造成影响。
发明内容
本申请实施例提供了一种图像处理的方法和装置，以解决相关技术提供的图像精准度较低的问题，技术方案如下：
第一方面，提供了一种图像处理的方法，该方法应用于网络设备，该方法包括：获取用于指示目标对象在第一时刻的姿态与位置的第一位姿信息，以及用于指示目标对象在历史时刻的姿态与位置的历史位姿信息。之后，根据时延信息及历史位姿信息对第一位姿信息所指示的姿态与位置进行补偿，从而得到第二位姿信息。接着，便可获取与第二位姿信息匹配的目标指令，该目标指令用于获取待显示的图像。
本申请实施例通过时延信息及历史位姿信息对第一位姿信息进行调整，改善了基于第一位姿信息得到的目标指令的滞后性，校准了由于时延而引起的指令误差，使得目标指令由于时延的存在而延迟到达第二设备后，目标指令携带的第二位姿信息所描述的动作仍能与第二 设备运行的逻辑相匹配，从而提高了动作响应的精准度。该方案自适应性较强，能够生成精准度较高的图像或视频流，提升了用户的操作体验。
在示例性实施例中，根据时延信息及历史位姿信息对第一位姿信息所指示的姿态与位置进行补偿，得到第二位姿信息，包括：根据时延信息及历史位姿信息确定位姿变化信息，位姿变化信息包括位姿变化量及位姿变化方向。基于位姿变化方向，按照位姿变化量补偿第一位姿信息所指示的姿态与位置，根据补偿后的姿态与位置得到第二位姿信息。该实施例在时延信息较大的情况下才进行第一姿态信息所指示的姿态与位置的补偿，避免了处理资源的浪费。
在示例性实施例中，根据时延信息及历史位姿信息确定位姿变化信息，包括：根据历史位姿信息确定位姿变化方向及位姿变化率。根据时延信息及位姿变化率确定位姿变化量。
在示例性实施例中，获取目标对象的第一位姿信息之后，方法还包括：获取时延信息；响应于时延信息大于时延阈值，执行根据时延信息及历史位姿信息对第一位姿信息所指示的姿态与位置进行补偿，得到第二位姿信息的步骤。
在示例性实施例中，网络设备为终端，获取与第二位姿信息匹配的目标指令之后，方法还包括：向服务器发送目标指令，接收服务器根据目标指令返回的待显示的图像，对图像进行显示。
在示例性实施例中，网络设备为服务器，获取与第二位姿信息匹配的目标指令之后，方法还包括：根据目标指令获取待显示的图像，将待显示的图像发送给终端进行显示。
第二方面，提供了一种图像处理装置，该装置包括：获取模块，用于获取目标对象的第一位姿信息，所述第一位姿信息用于指示所述目标对象在第一时刻的姿态与位置；
所述获取模块，还用于获取所述目标对象的历史位姿信息，所述历史位姿信息用于指示所述目标对象在历史时刻的姿态与位置；
补偿模块，用于根据时延信息及所述历史位姿信息对所述第一位姿信息所指示的姿态与位置进行补偿，得到第二位姿信息，所述时延信息基于参考位姿信息的采集时刻与参考图像的显示时刻之间的时间间隔得到，所述参考图像是所述参考位姿信息对应的图像，所述参考位姿信息的采集时刻早于所述第一时刻；
所述获取模块，还用于获取与所述第二位姿信息匹配的目标指令，所述目标指令用于获取待显示的图像。
在示例性实施例中，所述补偿模块，用于根据所述时延信息及所述历史位姿信息确定位姿变化信息，所述位姿变化信息包括位姿变化量及位姿变化方向；基于所述位姿变化方向，按照所述位姿变化量补偿所述第一位姿信息所指示的姿态与位置，根据补偿后的姿态与位置得到第二位姿信息。
在示例性实施例中，所述补偿模块，用于根据所述历史位姿信息确定所述位姿变化方向及位姿变化率；根据所述时延信息及所述位姿变化率确定所述位姿变化量。
在示例性实施例中，所述获取模块，还用于获取所述时延信息；所述补偿模块，还用于响应于所述时延信息大于时延阈值，执行所述根据时延信息及所述历史位姿信息对所述第一位姿信息所指示的姿态与位置进行补偿，得到第二位姿信息的步骤。
在示例性实施例中，所述装置还包括：显示模块，用于向服务器发送所述目标指令，接收所述服务器根据所述目标指令返回的待显示的图像，对所述图像进行显示。
在示例性实施例中，所述装置还包括：发送模块，用于根据所述目标指令获取待显示的图像，将所述待显示的图像发送给终端进行显示。
第三方面，提供了一种通信装置，该装置包括：收发器、存储器和处理器。其中，该收发器、该存储器和该处理器通过内部连接通路互相通信，该存储器用于存储指令，该处理器用于执行该存储器存储的指令，以控制收发器接收信号，并控制收发器发送信号，并且当该处理器执行该存储器存储的指令时，使得该处理器执行第一方面或第一方面的任一种示例性实施例所提供的方法。
可选地，处理器为一个或多个，存储器为一个或多个。
可选地，存储器可以与处理器集成在一起，或者存储器与处理器分离设置。
在具体实现过程中，存储器可以为非瞬时性(non-transitory)存储器，例如只读存储器(read only memory，ROM)，其可以与处理器集成在同一块芯片上，也可以分别设置在不同的芯片上，本申请实施例对存储器的类型以及存储器与处理器的设置方式不做限定。
第四方面，提供了一种计算机程序(产品)，计算机程序(产品)包括：计算机程序代码，当计算机程序代码被计算机运行时，使得计算机执行上述各方面中的方法。
第五方面，提供了一种可读存储介质，可读存储介质存储程序或指令，当程序或指令在计算机上运行时，上述各方面中的方法被执行。
第六方面，提供了一种芯片，包括处理器，用于从存储器中调用并运行所述存储器中存储的指令，使得安装有所述芯片的通信设备执行上述各方面中的方法。
第七方面，提供一种芯片，包括：输入接口、输出接口、处理器和存储器，所述输入接口、输出接口、所述处理器以及所述存储器之间通过内部连接通路相连，所述处理器用于执行所述存储器中的代码，当所述代码被执行时，所述处理器用于执行上述各方面中的方法。
附图说明
图1为本申请实施例提供的图像处理的方法流程图；
图2为本申请实施例提供的图像处理的方法流程图；
图3为本申请实施例提供的图像处理的流程示意图；
图4为本申请实施例提供的图像处理的流程示意图；
图5为本申请实施例提供的位姿信息的示意图；
图6为本申请实施例提供的补偿第一位姿信息的示意图；
图7为本申请实施例提供的图像处理的方法流程图；
图8为本申请实施例提供的图像处理的流程示意图；
图9为本申请实施例提供的图像处理的装置结构图。
具体实施方式
本申请的实施方式部分使用的术语仅用于对本申请的具体实施例进行解释，而非旨在限定本申请。
虚拟现实(virtual reality，VR)技术是一种将计算机生成的虚拟内容与真实环境相结合的技术，目前已被应用于视频、游戏、教育及医疗等多个领域，具有庞大的市场空间。其中，Cloud VR将云计算的理念及技术引入VR技术，使得原本需要由第一设备执行的计算、渲染等 过程转由第二设备来执行，而第一设备仅需具备解码能力即可，从而降低了对第一设备的配置要求。因此，Cloud VR可以降低用户消费成本，有利于加快VR技术的普及速度，近年来受到广泛关注。
在Cloud VR的应用过程中，第一设备往往会通过网络将指令传输至第二设备，第二设备根据接收到的指令处理得到一帧或多帧图像，将一帧图像或由多帧图像构成的视频流返回给第一设备，以便于第一设备进行显示。然而，通过网络将指令传输至第二设备的过程不可避免地存在延迟，因而第二设备接收到的指令是在时间上滞后的指令，第二设备根据该种滞后的指令所生成的图像及视频流的精准度较低。因此，需要对上述应用过程进行调整，以便于获取具有较高精准度的图像及视频流。
本申请实施例提供了一种图像处理的***，参见图1，该***包括：终端101、网络节点102及服务器103，终端101通过网络节点102与服务器103连接。
其中，终端101包括但不限于虚拟现实(virtual reality，VR)头盔及手柄，VR头盔用于目标对象进行图像及视频流的浏览，手柄用于目标对象进行操作。VR头盔可以是VR一体机，可以是包括手机及VR头部显示器的分体式VR，还可以是个人计算机(personal computer，PC)VR。手柄上可以按照需求设置按钮、扳机等一种或多种部件。
网络节点102可以是光网络设备(optical network terminal，ONT)、交换机、路由器或者无线接入点(access point，AP)等能够将终端101接入通信网络的设备。示例性地，通信网络可以是互联网，可以是企业组织的内部网络，也可以是某一区域内的局部网络。
服务器103可以是终端本地的个人计算机(personal computer，PC)，可以是一个独立的服务器，也可以是由多个服务器构成的服务器集群，还可以是云服务提供的虚拟机或计算机等等。
终端101用于采集目标对象的位姿信息，位姿信息描述了目标对象所进行的动作。其中，位姿信息用于指示目标对象在某个时刻的姿态与位置，姿态描述了目标对象所进行的旋转动作，位置描述了目标对象所进行的静止动作或平移动作。示例性地，目标对象所进行的动作包括目标对象的身体部位进行运动所产生的动作以及目标对象针对手柄所进行的操作。目标对象的身体部位进行运动所产生的动作，例如为目标对象的头部转动、身体移动以及持有手柄的手部所进行的挥动等等。目标对象针对手柄所进行的操作，例如为目标对象按压手柄上设置的按钮、扣动手柄上设置的扳机等等。
需要说明的是，目标对象针对手柄所进行的操作直接通过手柄内置的传感器即可捕捉得到，而对于目标对象的身体部位进行运动所产生的动作，则需要采用inside-out(内-外)方式及outside-in(外-内)方式中的任一种进行捕捉，从而实现位姿信息的采集。响应于采用inside-out方式进行捕捉，则会通过VR头盔以及手柄的内置设备检测目标对象相对于目标对象所在环境的变化，再根据目标对象相对于目标对象所在环境的变化推算出目标对象身体部位进行运动所产生的动作。其中，内置设备包括但不限于传感器、摄像头等设备。例如，VR头盔可通过内置设备检测VR头盔发生的运动，通过VR头盔的运动可以推算出目标对象的头部动作。手柄可通过内置设备检测手柄发生的运动，通过手柄的运动可以推算出目标对象持有手柄的手部所进行的挥动。响应于采用outside-in方式进行捕捉，则除了终端以外，还需要借助外置的追踪设备，例如定位基点/基站、光学摄像头等追踪设备，对目标对象身体部位的运动进行追踪。
终端101还用于通过网络节点102接入通信网络，将所采集到的位姿信息发送至服务器103。相应地，服务器103用于获取与接收到的第一位姿信息相匹配的指令，从而根据与位姿信息相匹配的指令进行计算及渲染，生成位姿信息对应的图像或视频流。服务器103对图像或视频流进行编码后，再通过通信网络将编码后的图像或视频流返回给终端101。之后，终端101还用于对编码后的图像或视频流进行解码，将图像或视频流通过VR头盔进行投影，从而将图像或视频流显示，即目标对象可以通过VR头盔浏览到图像或视频流。
在示例性实施例中，终端101或网络节点102中的任意一个与服务器103构成时延检测子***，该时延检测子***用于检测时延信息。也就是说，本实施例可以由终端101与服务器103构成时延检测子***，也可以由网络节点102与服务器103构成时延检测子***。
其中，时延信息基于位姿信息的采集时刻与位姿信息对应的图像的显示时刻之间的时间间隔得到。由于终端101可用于对位姿信息进行采集，因而位姿信息的采集时刻即为终端101采集到位姿信息的时刻。示例性地，本实施例可以将终端101采集到位姿信息的时刻与位姿信息产生(即对象进行动作)的时刻视为同一时刻，或者说位姿信息的采集时刻与位姿信息的产生时刻之间的时间间隔可以忽略不计。
另外，如前所述，终端101在采集到位姿信息之后，会将位姿信息发送给服务器103，服务器103基于接收到的位姿信息进行处理，从而返回位姿信息对应的图像或视频流，终端101对服务器103所返回的图像或视频流进行显示的时刻即为上述位姿信息对应的图像的显示时刻。由于位姿信息的采集时刻及位姿信息对应的图像的显示时刻之间涉及到信息的传输、处理等过程，而传输、处理等过程需要耗费时间，因而采集时刻及显示时刻之间存在时间间隔，可以将该时间间隔作为上述时延信息。
需要说明的是，对目标对象在当前时刻的位姿信息对应的图像进行处理时，可以基于目标对象在历史时刻的位姿信息确定上述时延信息，也可以基于目标对象以外的其他对象在历史时刻的位姿信息确定上述时延信息。当然，本实施例中时延信息不仅可以根据基于历史时刻的位姿信息确定，也可以根据目标对象或其他对象在当前时刻的位姿信息实时确定，本实施例不对用于确定时延信息的位姿信息所在的时刻加以限定。
时延检测子***包括客户端与服务端，客户端用于触发时延信息的检测，服务端用于配合客户端完成时延信息的检测。在上述由终端101与服务器103构成的时延检测子***中，可以将终端101作为客户端而将服务器103作为服务端，也可以将终端101作为服务端而将服务器103作为客户端。在上述由网络节点101与服务器构成的时延检测子***中，可以将网络节点102作为客户端而将服务器103作为服务端，也可以将网络节点102作为服务端而将服务器103作为客户端。
可以理解的是，时延检测子***所检测的时延信息可以是位姿信息在终端101与服务器103之间传输所产生的网络时延，也可以是终端101或服务器103在运行过程中所产生的其他时延。其中，其他时延包括但不限于终端101将位姿信息发送给服务器103时，位姿信息在终端101的发送队列中等待发送的排队时延，以及终端101将位姿信息从发送队列中发送至网络中所产生的发送时延。
接下来，对检测得到网络时延及其他时延的方式分别进行说明：
网络时延：首先，由时延检测子***中的客户端将包含有发送时间的检测信号发送至服务端，以触发时延信息的检测。之后，服务端可计算接收到该检测信号的时间与该检测信号 所包含的发送时间之前的差值。由于该差值即为检测信号在通信网络中在终端101与服务器103之间进行传输所需的时间，因而该差值即可作为网络时延。
需要说明的是，由于所需检测的网络时延是终端101与服务器103之间的网络时延，因而在时延检测子***中的客户端或服务端为网络节点102的情况下，该网络节点102在通信网络中的位置需要贴近终端。由此，可以使得检测信号在网络节点102与服务器103间传输的时长与在终端101与服务器103间传输的时长之间的差值趋近于零，从而可以将网络节点102与服务器103之间的网络时延代替终端101与服务器103之间的网络时延。
其他时延：示例性地，本实施例可以采用软件探针进行其他时延的检测。其中，通过软件探针可以对设备所执行的业务添加时间戳，计算不同业务之间的时间戳差值即可计算得到其他时延。以上述排队时延为例，在终端101中，通过软件探针可以在终端101执行“将采集到的位姿信息加入发送队列中”这一业务时添加第一时间戳记录当前时间，在终端101执行“即将对发送队列中的位姿信息进行发送”这一业务添加第二时间戳记录当前时间。之后，计算第一时间戳及第二时间戳之间的时间戳差值，即可得到位姿信息在发送队列中等待发送的排队时延。对于排队时延以外的其他不同时延，此处不再加以赘述。
除了上述时延检测子***外，本实施例所提供的***中还包括计算节点，该计算节点用于获取根据时延信息及历史位姿信息对目标对象的位姿信息所指示的姿态与位置进行补偿，得到第二位姿信息，从而进一步获取与第二位姿信息匹配的、用于获取待显示的图像的目标指令。其中，历史位姿信息用于指示目标对象在历史时刻的姿态与位置。在示例性实施例中，该计算节点可以是终端101，也可以是服务器103。接下来，首先对获取历史位姿信息的过程加以说明，再对计算节点为终端101及服务器103的两种情况分别进行说明：
其中，历史位姿信息是由终端101采集得到的信息。终端101在采集得到历史位姿信息之后，可以将历史位姿信息存储于终端101本地，也可以将历史位姿信息发送至服务器103进行存储，还可以将历史位姿信息发送至服务器103以外的其他存储设备进行存储。因此，无论计算节点为终端101还是服务器103，均可以从存储有历史位姿信息的设备中获取到历史位姿信息。
如前所述，目标对象的位姿信息用于指示目标对象在当前时刻的姿态与位置，历史位姿信息用于指示目标对象在历史时刻的姿态与位置。需要说明的是，“当前时间”、“当前时刻”及“历史时刻”可以是指一个瞬时时刻，也可以是指由多个瞬时时刻所组成的一段时间。
以目标对象进行挥手动作为例，进行挥手动作时，目标对象的手部会从一处移动至另一处，该移动过程往往不会在一个瞬时时刻内完成，而是需要占用多个瞬时时刻所组成的一段时间。因此，在当前时刻及历史时刻所代表的是一个瞬时时刻的情况下，姿态信息所描述的动作是手部在进行移动的过程中所经过的某一点处的动作。在该情况下，历史姿态信息及目标对象的姿态信息可以均为挥动手部这一过程中某一点处的动作，只不过历史姿态信息所描述的动作发生在目标对象的姿态信息所描述的动作之前。
或者，在当前时刻及历史时刻所代表的是一段时间的情况下，目标对象的姿态信息所描述的是目标对象的手部从一处移动至另一处的完整动作。因此在该情况下，历史姿态信息所描述的是发生在目标对象的手部挥动这一完整动作的前一个或多个其他完整动作。本实施例不对时刻所指的时间加以限定，上述两种情况可以根据需要进行选择。接下来，分别对终端101及服务器103进行计算的过程加以说明：
第一种情况：计算节点为终端101。终端101在本地采集到目标对象的位姿信息之后，在终端101获取时延信息时，响应于终端101为时延检测子***中的服务端，则终端101可接收客户端发送的包含有发送时间的检测信号，从而通过计算得到时延信息。响应于终端101为时延检测子***中的客户端，则终端101可以从时延检测子***中的服务端获取时延信息。另外，对于终端101不构成时延检测子***的情况，终端101还可以从时延检测子***中的网络节点或者服务器中的任意一个获取时延信息。
另外，在终端101获取历史位姿信息时，终端101可以从本地、服务器103或者其他存储有历史位姿信息的设备中进行获取即可。接着，终端可以根据获取到的时延信息及历史位姿信息对采集到的位姿信息所指示的位姿与状态进行补偿，得到新的位姿信息，从而确定与新的位姿信息相匹配的目标指令。
在终端101得到目标指令之后，终端101还需要将目标指令发送给服务器103，以便于服务器103根据目标指令进行计算及渲染，并将渲染得到的图像或视频流编码返回给终端101。之后，终端101可对接收的图像或视频流进行显示。
第二种情况：计算节点为服务器103。其中，目标对象的位姿信息仍由终端采集，因而服务器103需要接收终端101发送的位姿信息，从而实现位姿信息的获取。在获取时延信息时，响应于服务器103为时延检测子***中的服务端，则服务器103可接收客户端发送的包含有发送时间的检测信号，从而通过计算得到时延信息。响应于服务器103为时延检测子***中的客户端，则服务器103可以从时延检测子***中的服务端获取时延信息。对于目标对象的历史动作信息，基于上述第一种情况中的说明可知，服务器103可能从终端101、本地或者存储有历史位姿信息的其他设备中获取到该历史动作信息。
服务器103通过时延信息及历史位姿信息对位姿信息所指示的姿态与位置进行补偿得到新的位姿信息之后，可获取与新的位姿信息相匹配的目标指令，从而直接根据目标指令进行计算及渲染，得到图像或视频流，将图像或视频流编码推送至终端101。需要说明的是，服务器103可以调用一个或多个旁挂设备来进行上述时延信息的检测以及位姿信息的补偿，也可以不调用旁挂设备直接进行检测及补偿，本实施例对于服务器103是否调用旁挂设备不加以限定。
基于上述说明，图1所示的终端101、网络节点102以及服务器103可以构成如下的四种***。可以理解的是，如下的四种***仅为示例。在实际应用中，本实施例所提供的***并不局限于如下示例的四种***，还可以包括基于终端101、网络节点102及服务器103组合得到的其他不同***：
第一种***：终端101与服务器103用于构成时延检测子***，以检测时延，终端101用于进行位姿信息的补偿计算。考虑到时延检测子***包括客户端和服务器，因而第一种***可进一步分为表1中的两种组合形式：
表1
时延检测子***客户端 时延检测子***服务端 位姿信息的补偿计算
终端 服务器 终端
服务器 终端 终端
第二种***：终端101与服务器103用于构成时延检测子***，服务器103用于进行位姿信息的补偿计算。第二种***可分为如表2所示的两种组合形式：
表2
时延检测子***客户端 时延检测子***服务端 位姿信息的补偿计算
终端 服务器 服务器
服务器 终端 服务器
第三种***：网络节点102与服务器103构成时延检测子***，终端101用于进行位姿信息的补偿计算。第三种***可分为如表3所示的两种组合形式：
表3
时延检测子***客户端 时延检测子***服务端 位姿信息的补偿计算
网络节点 服务器 终端
服务器 网络节点 终端
第四种***：网络节点102与服务器103构成时延检测子***，服务器103用于进行位姿信息的补偿计算。第四种***可分为表4所示的两种组合形式：
表4
时延检测子***客户端 时延检测子***服务端 位姿信息的补偿计算
网络节点 服务器 服务器
服务器 网络节点 服务器
基于上述图1所示的图像处理***，本实施例还提供了一种图像处理的方法，该方法可应用于网络设备中，网络设备可以是图1所示的终端，或者是图1所示的服务器。参见图2，响应于网络设备为终端，则该方法包括：
步骤201，获取目标对象的第一位姿信息，第一位姿信息用于指示目标对象在第一时刻的姿态与位置。
其中，第一位姿信息通过指示目标对象在第一时刻的姿态与位置，描述了目标对象在第一时刻所进行的动作。目标对象所进行的动作包括目标对象的身体部位进行运动所产生的动作以及目标对象针对手柄等VR设备所进行的操作。目标对象的身体部位进行运动所产生的动作例如为目标对象的头部转动、身体移动以及持有手柄的手部所进行的挥动等等，目标对象针对VR设备所进行的操作例如为目标对象按压设备上设置的按钮等等。
需要说明的是，目标对象所进行的动作可归纳为包括静止动作、平移动作及旋转动作三种不同的类型。例如目标对象的手部挥动可以看作是平移动作，目标对象的头部转动可以看作是旋转动作。示例性地，第一位姿信息可以包括第一姿态信息及第一位置信息中的一种或两种，第一姿态信息用于描述目标对象所进行的旋转动作，第一位置信息用于描述目标对象所进行的静止动作及平移动作。
其中，目标对象所在空间可以通过三维坐标进行表示，该三维坐标系包括两两相垂直的X、Y及Z轴。当目标对象的任一身体部位处于静止不同的状态时，该任一身体部位固定于一个位置，此时第一位置信息可以通过三维坐标系中的一个坐标点进行表示。当目标对象的任一身体部位进行平移变化时，该任一身体部位的位置不断发生变化，此时第一位置信息包括三维坐标系中的多个坐标点。可以理解的是，该平移变化可看作是在三维坐标系中沿X轴、Y轴及Z轴中的至少一轴所进行的平移变化。当目标对象的任一身体部位所在的位置不变，但该任一身体部位发生旋转时，第一姿态信息包括一个坐标点处各个方向上的旋转角度。可 以理解的是，该旋转过程可以看作是三维坐标系中围绕X轴、Y轴及Z轴中的至少一轴所进行的旋转。
步骤202，获取目标对象的历史位姿信息，历史位姿信息用于指示目标对象在历史时刻的姿态与位置。
其中，历史时刻是早于上述第一时刻的任一时刻。因此，历史位姿信息所描述的动作是在第一位姿信息所描述的动作之前已经由目标对象完成的动作。示例性地，第一时刻及历史时刻可以代表一个瞬时时刻，也可以代表由多个瞬时时刻所组成的一段时间。
以目标对象进行挥手动作为例，进行挥手动作时，目标对象的手部会从一处移动至另一处，该移动过程往往不会在一个瞬时时刻内完成，而是需要占用多个瞬时时刻所组成的一段时间来完成。因此，在第一时刻及历史时刻所代表的是一个瞬时时刻的情况下，第一姿态信息及历史姿态信息所描述的动作均是手部在进行移动的过程中所经过的某一点处的动作。假设目标对象的手部在进行移动的过程中先经过第一点、后经过第二点，由于历史位姿信息所描述的动作早于第一位姿信息所描述的动作完成，因而历史位姿信息描述了第一点处的动作，第一位姿信息描述了第二点处的动作。
或者，在当前时刻及历史时刻所代表的是一段时间的情况下，第一姿态信息所描述的则是目标对象的手部从一处移动至另一处的完整动作。因此，历史姿态信息所描述的是发生在目标对象的手部挥动这一完整动作的前一个或多个其他完整动作。对于以上两种情况本实施例不加以限定，可以根据目标对象的动作类型及实际需要进行选择。
步骤203，根据时延信息及历史位姿信息对第一位姿信息所指示的姿态与位置进行补偿，得到第二位姿信息，时延信息基于参考位姿信息的采集时刻与参考图像的显示时刻之间的时间间隔得到，参考图像是参考位姿信息对应的图像，参考位姿信息的采集时刻早于第一时刻。
其中，参考位姿信息可以用于指示目标对象在早于第一时刻的某个时刻的姿态与位置，或者，参考位姿信息还可以用于指示目标对象以外的其他对象在早于第一时刻的某个时刻的姿态与位姿，本实施例不对参考位姿信息用于指示何种对象的姿态与位置加以限定。需要说明的是，本实施例可以将参考位姿信息的采集时刻与参考位姿信息的产生时刻视为同一时刻，或者说采集时刻与产生时刻之间的时间间隔可以忽略不计。
参考位姿信息在早于第一时刻的某一时刻被采集之后，终端会获取与参考位姿信息(或者补偿后的参考位姿信息)相匹配的指令，将指令发送给服务器，服务器基于指令获取参考位姿信息对应的参考图像(或视频流)，终端可接收服务器返回的参考图像并进行显示。终端对参考图像进行显示的时刻即为上述参考图像的显示时刻。由于采集时刻与显示时刻之间至少涉及到获取指令、指令传输、生成参考图像以及参考图像传输等过程，还可能涉及到对参考位姿信息的补偿过程，因而采集时刻与显示时刻之间会存在时间间隔，该时间间隔即可作为用于对第一位姿信息所指示的姿态与位置进行补偿的时延信息。
基于上述说明，在对第一姿态信息所指示的姿态与位置进行补偿时，可以根据第一姿态信息之前的参考位姿信息来确定时延信息。示例性地，本实施例可以在获取到参考位姿信息的采集时刻及参考图像的显示时刻之后计算得到时延信息，对时延信息进行存储。之后，在需要对第一姿态信息所指示的姿态与位置进行补偿时，直接读取存储的时延信息，以便于完成补偿过程。
当然，除了上述情况以外，本实施例也可以在需要对第一姿态信息所指示的姿态与位置 进行补偿时，实时的对时延信息进行检测。其中，时延信息可以是终端与服务器之间的网络延迟，也可以是终端及服务器在运行过程中所产生的设备延迟。示例性地，终端与服务器之间的时延信息由时延检测子***确定。参见图3，该时延检测子***可以由终端与服务器构成。或者，参见图4，该时延检测子***也可以由网络节点与服务器构成。
对于图3所示的终端与服务器作为时延检测子***的情况，响应于终端为时延检测客户端、服务器为服务端，则终端向服务器发送检测信号，服务器计算终端发送检测信号的发送时间与本地接收检测信号的接收时间之间的差值，将该差值作为终端与服务器之间的时延信息。之后，服务器可以将计算得到的时延信息发送给终端，从而使得终端实现时延信息的获取。
响应于终端为时延检测服务端、服务器为客户端的情况，则终端会接收到服务器发送的检测信号，终端计算服务器发送检测信号的发送时间与本地接收到检测信号的接收时间之间的差值。在终端计算得到差值之后，便可将差值作为时延信息，从而实现了时延信息的获取。
对于图4所示的网络节点与服务器作为时延检测子***的情况，则由网络节点或服务器计算上述检测信号的发送时间与接收时间的差值，从而得到时延信息。之后，计算得到时延信息的网络节点或服务器可以直接将时延信息发送给终端，以便于终端获取到时延信息。或者，计算得到时延信息的网络节点或服务器也可以等待终端发送的获取请求，再根据获取请求将时延信息返回给终端，同样使得终端能够获取到时延信息。
在获取到时延信息之后，终端便可以根据时延信息及步骤202中获取到的历史位姿信息对第一位姿信息所指示的姿态与位置进行补偿，以避免基于第一位姿信息所生成的指令在时间上存在滞后，从而影响后续的图像处理过程，使得生成的图像精确度较低。
示例性地，根据时延信息及历史位姿信息对第一位姿信息所指示的姿态与位置进行补偿，得到第二位姿信息，包括如下的步骤2031及2032：
步骤2031，根据时延信息及历史位姿信息确定位姿变化信息，位姿变化信息包括位姿变化量及位姿变化方向。
在示例性实施例中，根据时延信息及历史位姿信息确定位姿变化信息，包括：根据历史位姿信息确定位姿变化方向及位姿变化率，根据时延信息及位姿变化率确定位姿变化方向。其中，本实施例可以根据目标对象的历史位姿信息确定目标对象的动作趋势，从而根据动作趋势来确定位姿变化方向。示例性地，响应于根据历史位姿信息确定的动作趋势呈沿某一方向的直线形、弧形等，则可以将历史位姿信息所指示的姿态与位置的方向作为位姿变化方向。响应于根据历史位姿信息确定的动作趋势呈方向不断改变甚至调转的波浪形、锯齿形等，则位姿变化方向可能与历史位姿信息所指示的姿态与位置的方向相同或相反，还可能与历史位姿信息所指示的姿态与位置的方向存在偏离。示例性地，对于后一种情况，可以根据至少三个历史位姿信息归纳出动作趋势的方向变化规律，例如方向发生改变或调转的周期，再结合方向变化规律确定位姿变化方向。
以图5为例，对上述确定位姿变化方向的过程进行说明：t1及t2是两个相邻历史时刻，且t1为早于t2的时刻。其中，t1时刻目标对象所在的位置A在三维坐标系中的坐标为(x1，y1，z1)，t2时刻目标对象所在的位置B在三维坐标系中的坐标为(x2，y2，z2)。响应于动作趋势为沿某一方向的直线形、弧形等轨迹，则可以直接将由位置A指向位置B的方向作为位姿变化方向。响应于动作趋势为不断发生改变的锯齿形、波浪形等轨迹，则还需要获取早 于t1时刻的其他一个或多个历史位姿信息，从而通过历史位姿信息确定动作趋势的方向变化规律，再根据方向变化规律确定出位姿变化方向。在后一种情况中，位姿变化方向可能是位置A指向位置B的方向，可能是位置B指向位置A的方向，还可能是位置B指向位置A以外的任意位置的方向。
在根据t1及t2时刻对应的历史位姿信息确定位姿变化率时，可按照如下的公式进行确定：
位姿变化率＝[(x2，y2，z2)-(x1，y1，z1)]/(t2-t1)。
在确定位姿变化率之后，通过计算时延信息及位姿变化率的乘积，即可得到发生时延的时间长度内位姿所发生的变化量。也就是说，位姿变化量可按照如下的公式进行计算：
位姿变化量＝[(x2，y2，z2)-(x1，y1，z1)]×(t3-t2)/(t2-t1)，其中，(t3-t2)即为上述时延信息。
步骤2032，基于位姿变化方向，按照位姿变化量补偿第一位姿信息所指示的姿态与位置，根据补偿后的姿态与位置得到第二位姿信息。
示例性地，本实施例在位姿变化方向上，对第一位姿信息及位姿变化量信息进行求和，将第一位姿信息及位姿变化量之和作为第二位姿信息，从而实现第一位姿信息的补偿，得到第二位姿信息。基于上述说明，第二位姿信息可按照如下的公式进行计算：
Figure PCTCN2021088120-appb-000001
其中，(x3，y3，z3)即为第二位姿信息。
如前所述，对第一位姿信息进行补偿的目的在于：改善后续生成的指令的滞后性，避免指令滞后降低后续生成的图像或视频流的精确度，从而对用户的使用体验造成的影响。但是，若指令的滞后性较弱，则即使指令存在滞后情况，也不会影响到图像或视频流的精确度。因此，若在指令滞后性较弱、不会影响到图像或视频流的精确度的情况下仍对第一位姿信息进行补偿，则会造成终端、网络节点及服务器的处理资源的浪费。
因此，在示例性实施例中，获取目标对象的第一位姿信息之后，方法还包括：获取时延信息，响应于时延信息大于时延阈值，响应于时延信息大于时延阈值，执行根据时延信息及历史位姿信息对第一位姿信息所指示的姿态与位置进行补偿，得到第二位姿信息的步骤。
其中，响应于时延信息不大于时延阈值，则可证明指令的滞后性较弱，因而无需进行补偿计算，从而避免了处理资源的浪费。而在时延信息大于时延阈值的情况下，则说明指令具有较强的滞后性，若服务器直接应用该指令进行图像殴视频流的生成，则会生成精确度较低的图像或视频流。因此，需要对第一指令进行调整。本实施例不对上述时延阈值的数值加以限定，可以根据实际需要或经验设置该时延阈值。例如，时延阈值可以为30毫秒，也可以为30毫秒以外的其他数值。
步骤204，获取与第二位姿信息匹配的目标指令，目标指令用于获取待显示的图像。
其中，终端在通过步骤203获取到第二位姿信息之后，可以将第二位姿信息及传输过程中需要使用的其他信息封装得到与第二位姿信息匹配的目标指令。传输过程中需要使用的其他信息包括但不限于传输协议信息、路由信息等等。
在示例性实施例中，获取与第二位姿信息匹配的目标指令之后，方法还包括：向服务器发送目标指令，接收服务器根据目标指令返回的待显示的图像，对图像进行显示。如图6所示，在未进行对第一位姿信息的补偿过程之前，第一位姿信息是t2时刻的位姿信息。由于终 端与服务器之间存在时延，而服务器上的逻辑不间断的运行，因而在服务器接收到与第一位姿信息匹配的指令时，服务器所运行的逻辑为t3时刻的逻辑，t2时刻的第一位姿信息相比于服务器所运行的逻辑存在滞后。因此，服务器根据t3时刻运行的逻辑与t2时刻的第一位姿信息生成图像或视频流，则会导致t2时刻的第一位姿信息所描述的动作与逻辑存在不匹配，图像或视频流的精准度较低。例如在人机交互游戏中，动作与逻辑存在不匹配则会导致目标对象本可击中的游戏对象无法被击中，影响了目标对象的使用体验。
而通过上述计算过程，终端在t2时刻发送给服务器的目标指令中所包括的目标位姿信息即为t3时刻对应的第二位姿信息。在该目标指令的传输过程中，仍然会有时延的存在，使得该目标指令于t3时刻才能被服务器所接收，此时服务器运行的逻辑为t3时刻的逻辑。由于该目标指令中所包括的第二位姿信息即为t3时刻的位姿信息，因而与服务器运行的逻辑相匹配，从而使得目标对象的对象动作与逻辑相匹配，视频流的精准度较高，保证了目标对象的使用体验。
在服务器接收到终端发送的目标指令之后，会基于目标指令进行计算、渲染，从而得到图像或视频流，再对图像或视频流进行编码，将编码后的图像或视频流发送给终端，以便于终端进行显示。由于服务器对编码后的图像或视频流进行了发送，因而终端可相应地对编码后的图像或视频流进行接收。终端进行接收之后，可对编码后的图像或视频流进行解码，从而得到原始的目标视频流。终端接收服务器发送的图像或视频流之后，还对目标视频流进行显示。也就是说，终端可以通过VR头盔或其他显示设备对目标视频流进行显示，以便于用户对精确度较高的目标视频流进行浏览。
基于以上说明，参见图3，本实施例可提供一种终端作为时延检测子***客户端、服务器作为时延检测子***服务端且终端用于补偿第一位姿信息所指示的姿态与位置的***。在步骤301中，终端可采集第一位姿信息，在步骤302中，第一设备还需要通过与服务器的交互来检测与服务器之间的时延信息，从而实现时延信息的获取。除了获取时延信息以外，参见步骤303，由于终端需要进行补偿计算，因而终端还需要获取补偿计算所需的历史位姿信息。在终端获取到时延信息及历史位姿信息之后，终端可以在步骤304中根据时延信息及历史位姿信息对第一位姿信息进行补偿调整，从而得到第二位姿信息。
接着，由于图像及视频流需要在服务器侧生成，因而在步骤305中，终端获取与第二位姿信息匹配的目标指令，将目标指令封装为数据包，将数据包发送给服务器。在步骤306中，服务器可基于接收到的目标指令进行计算及渲染，生成用于进行显示的图像或视频流。服务器生成图像或视频流之后，参见步骤307，还需要对图像或视频流进行编码，将编码后的图像或视频流返回给终端。在后续过程中，终端还可以对接收到的编码后的图像或视频流进行解码、显示等处理。
另外，参见图4，本实施例还可以提供一种网络节点作为时延检测子***客户端、服务器作为时延检测子***服务端且终端用于补偿第一位姿信息所指示的姿态与位置的***。在获取过程中，参见步骤401，终端可采集第一位姿信息。由于网络节点和服务器分别为时延检测子***中的客户端及服务端，因而终端及服务器之间的时延信息在步骤402中通过网络节点及服务器之间的交互得到。在步骤403中，终端可从网络节点或服务器中的任一设备获取到时延信息，终端还需要另外获取历史位姿信息，从而在步骤404中根据时延信息及历史位姿信息对第一位姿信息指示的姿态与位置进行调整，得到第二位姿信息。
接着，由于视频流需要在服务器侧生成，因而参见步骤405，终端获取第二位姿信息匹配的目标指令，将目标指令封装为数据包，将数据包发送给服务器。在步骤406中，服务器可基于接收到的目标指令进行计算及渲染，生成用于进行显示的图像或视频流。在步骤407中，服务器生成图像或视频流之后，还需要对图像或视频流进行编码，将编码后的图像或视频流返回给终端。在后续过程中，终端还可以对接收到的编码后的图像或视频流进行解码、显示等处理。
综上所述，本申请实施例通过时延信息及历史位姿信息对第一位姿信息进行补偿，改善了目标指令的滞后性，校准了由于时延而引起的指令误差，使得目标指令由于时延的存在而延迟到达服务器后，目标指令匹配的第二位姿信息所指示的姿态与位置仍能与服务器运行的逻辑相匹配，从而提高了动作响应的精准度。该方案自适应性较强，能够生成精准度较高的图像或视频流，提升了用户的操作体验。
基于上述图1所示的图像处理***，本实施例提供了一种图像处理的方法，该方法应用于网络设备中，网络设备可以是图1所示的服务器。参见图7，该方法包括：
步骤701，获取第一位姿信息，第一位姿信息用于指示目标对象在第一时刻的姿态与位置。
其中，由于终端具有采集第一位姿信息的功能，因而第一位姿信息由终端侧获取。由于服务器需要对第一位姿信息进行补偿，因而服务器需要首先从终端获取该待调整的第一位姿信息。示例性地，服务器所获取的第一位姿信息可以是终端推送的第一位姿信息，也开始是终端根据服务器发送的获取请求所返回的第一位姿信息。
步骤702，获取历史位姿信息，历史位姿信息用于指示目标对象在历史时刻的姿态与位置。
其中，历史位姿信息也由终端采集得到，响应于终端获取到历史位姿信息后将历史位姿信息存储于终端本地，则服务器需要从终端获取历史位姿信息。响应于终端获取到历史位姿信息后将历史位姿信息上传至服务器进行存储，服务器在本地即可获取到历史位姿信息。响应于终端将历史位姿信息存储于终端及服务器以外的其他设备，则服务器需要从该其他设备处获取到历史位姿信息。
步骤703，根据时延信息及历史位姿信息对第一位姿信息所指示的姿态与位置进行补偿，得到第二位姿信息。
其中，对时延信息的说明可以参见步骤203。示例性地，在时延信息是终端与服务器之间的网络延迟，或者是是终端及服务器在运行过程中所产生的设备延迟的情况下，时延信息可以额由时延检测子***确定。该时延检测子***可以由终端与服务器构成。或者，参见图8，该时延检测子***也可以由网络节点与服务器构成。
对于终端与服务器作为时延检测子***的情况，响应于终端为时延检测客户端、服务器为服务端，则终端向服务器发送检测信号，服务器计算终端发送检测信号的发送时间与本地接收检测信号的接收时间之间的差值，将该差值作为终端与服务器之间的时延信息，从而使得服务器实现时延信息的获取。
响应于终端为时延检测服务端、服务器为客户端的情况，则服务器首先向终端发送检测信号，终端会接收到服务器发送的检测信号，终端计算服务器发送检测信号的发送时间与本 地接收到检测信号的接收时间之间的差值。在终端计算得到差值之后，便可将差值返回给服务器，从而使得服务器实现了时延信息的获取。
如图8所示，对于网络节点与服务器作为时延检测子***的情况，响应于网络节点为时延检测客户端、服务器为服务端，则网络节点向服务器发送检测信号，服务器计算网络节点发送检测信号的发送时间与本地接收检测信号的接收时间之间的差值，将该差值作为网络节点与服务器之间的时延信息，从而使得服务器实现时延信息的获取。
响应于网络节点为时延检测服务端、服务器为客户端的情况，则服务器首先向网络节点发送检测信号，网络节点会接收到服务器发送的检测信号，网络节点计算服务器发送检测信号的发送时间与本地接收到检测信号的接收时间之间的差值。在网络节点计算得到差值之后，便可将差值返回给服务器，从而使得服务器实现了时延信息的获取。
对于根据时延信息及历史位姿信息对第一位姿信息所指示的姿态与位置进行补偿的过程，可以参见步骤203中的说明，此处不再加以赘述。
步骤704，获取与第二位姿信息匹配的目标指令，目标指令用于获取待显示的图像。
获取目标指令的过程可参见上文步骤204中的说明，此处不再加以赘述。在示例性实施例中，服务器获取与第二位姿信息匹配的目标指令之后，方法还包括：根据目标指令获取待显示的图像，将待显示的图像发送给终端进行显示。其中，服务器基于目标指令进行计算、渲染等过程可以得到待显示的图像或视频流。终端在接收服务器发送的图像或视频流之后，可以用于对图像或视频流进行显示。
基于以上说明，参见图8，本实施例可以提供一种网络节点作为时延检测子***客户端、服务器作为时延检测子***服务端且服务器用于对第一位姿信息所指示的姿态与位置进行补偿的***。在步骤801中，由于终端是可以与目标对象进行人机交互的设备，因而采集第一位姿信息由终端执行。在步骤802中，终端将该第一位姿信息发送给服务器，相应地，服务器便可以接收到待终端发送的第一位姿信息及历史位姿信息。另外，在步骤803中，服务器还需要通过与网络节点的交互来检测与终端之间的时延信息，从而实现时延信息的获取。在步骤804中，除了获取时延信息以外，服务器还需要获取补偿第一位姿信息所需的历史位姿信息。在步骤805中，在服务器获取到时延信息及历史位姿信息之后，可以根据时延信息及历史位姿信息对第一位姿信息进行补偿，从而得到第二位姿信息。
在步骤806中，由于图像或视频流在服务器侧即可生成，因而服务器获取与第二位姿信息匹配的目标指令之后，便可继续基于目标指令进行计算及渲染，生成用于进行显示的图像视频流。服务器生成图像视频流之后，参见步骤807，服务器还需要对图像或视频流进行编码，将编码后的图像或视频流返回给终端。在后续过程中，终端还可以对接收到的编码后的图像或视频流进行解码、显示等处理。
综上所述，本申请实施例通过时延信息及历史位姿信息对第一位姿信息进行补偿，改善了目标指令的滞后性，校准了由于时延而引起的指令误差，使得目标指令由于时延的存在而延迟到达服务器后，目标指令匹配的第二位姿信息所指示的姿态与位置仍能与服务器运行的逻辑相匹配，从而提高了动作响应的精准度。该方案自适应性较强，能够生成精准度较高的图像或视频流，提升了用户的操作体验。
如图9所示，本申请实施例还提供了一种图像处理的装置，该装置包括：
获取模块901，用于获取目标对象的第一位姿信息，第一位姿信息用于指示目标对象在第一时刻的姿态与位置；
该获取模块901，还用于获取目标对象的历史位姿信息，历史位姿信息用于指示目标对象在历史时刻的姿态与位置；
补偿模块902，用于根据时延信息及历史位姿信息对第一位姿信息所指示的姿态与位置进行补偿，得到第二位姿信息，时延信息基于参考位姿信息的采集时刻与参考图像的显示时刻之间的时间间隔得到，参考图像是参考位姿信息对应的图像，参考位姿信息的采集时刻早于第一时刻；
该获取模块901，用于获取与第二位姿信息匹配的目标指令，目标指令用于获取待显示的图像。
在示例性实施例中，补偿模块902，用于根据时延信息及历史位姿信息确定位姿变化信息，位姿变化信息包括位姿变化量及位姿变化方向；基于位姿变化方向，按照位姿变化量补偿第一位姿信息所指示的姿态与位置，根据补偿后的姿态与位置得到第二位姿信息。
在示例性实施例中，补偿模块902，用于根据历史位姿信息确定位姿变化方向及位姿变化率；根据时延信息及位姿变化率确定位姿变化量。
在示例性实施例中，该获取模块901，还用于获取时延信息；补偿模块，还用于响应于时延信息大于时延阈值，执行根据时延信息及历史位姿信息对第一位姿信息所指示的姿态与位置进行补偿，得到第二位姿信息的步骤。
在示例性实施例中，装置还包括：显示模块，用于向服务器发送目标指令，接收服务器根据目标指令返回的待显示的图像，对图像进行显示。
在示例性实施例中，装置还包括：发送模块，用于根据目标指令获取待显示的图像，将待显示的图像发送给终端进行显示。
应理解的是，上述图9提供的装置在实现其功能时，仅以上述各功能模块的划分进行举例说明，实际应用中，可以根据需要而将上述功能分配由不同的功能模块完成，即将设备的内部结构划分成不同的功能模块，以完成以上描述的全部或者部分功能。另外，上述实施例提供的装置与方法实施例属于同一构思，其具体实现过程详见方法实施例，这里不再赘述。
本申请实施例提供了一种通信装置，该装置包括：收发器、存储器和处理器。其中，该收发器、该存储器和该处理器通过内部连接通路互相通信，该存储器用于存储指令，该处理器用于执行该存储器存储的指令，以控制收发器接收信号，并控制收发器发送信号，并且当该处理器执行该存储器存储的指令时，使得该处理器执行步骤201-204中任一种示例性实施例所提供的方法。
本申请实施例提供了一种通信装置，该装置包括：收发器、存储器和处理器。其中，该收发器、该存储器和该处理器通过内部连接通路互相通信，该存储器用于存储指令，该处理器用于执行该存储器存储的指令，以控制收发器接收信号，并控制收发器发送信号，并且当该处理器执行该存储器存储的指令时，使得该处理器执行步骤701-704中任一种示例性实施例所提供的方法。
可选地，处理器为一个或多个，存储器为一个或多个。
可选地，存储器可以与处理器集成在一起，或者存储器与处理器分离设置。
在具体实现过程中，存储器可以为非瞬时性(non-transitory)存储器，例如只读存储器(read only memory，ROM)，其可以与处理器集成在同一块芯片上，也可以分别设置在不同的芯片上，本申请实施例对存储器的类型以及存储器与处理器的设置方式不做限定。
本申请实施例提供了一种计算机程序(产品)，计算机程序(产品)包括：计算机程序代码，当计算机程序代码被计算机运行时，使得计算机执行本申请的任一种示例性实施例所提供的方法。
本申请实施例提供了一种可读存储介质，可读存储介质存储程序或指令，当程序或指令在计算机上运行时，上述各方面中的方法被执行。
本申请实施例提供了一种芯片，包括处理器，用于从存储器中调用并运行所述存储器中存储的指令，使得安装有所述芯片的通信设备执行本申请的任一种示例性实施例所提供的方法。
本申请实施例提供一种芯片，包括：输入接口、输出接口、处理器和存储器，所述输入接口、输出接口、所述处理器以及所述存储器之间通过内部连接通路相连，所述处理器用于执行所述存储器中的代码，当所述代码被执行时，所述处理器用于执行本申请的任一种示例性实施例所提供的方法。
应理解的是，上述处理器可以是中央处理器(Central Processing Unit，CPU)，还可以是其他通用处理器、数字信号处理器(digital signal processing，DSP)、专用集成电路(application specific integrated circuit，ASIC)、现场可编程门阵列(field－programmable gate array，FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件等。通用处理器可以是微处理器或者是任何常规的处理器等。值得说明的是，处理器可以是支持进阶精简指令集机器(advanced RISC machines，ARM)架构的处理器。
进一步地，在一种可选的实施例中，上述存储器可以包括只读存储器和随机存取存储器，并向处理器提供指令和数据。存储器还可以包括非易失性随机存取存储器。例如，存储器还可以存储设备类型的信息。
该存储器可以是易失性存储器或非易失性存储器，或可包括易失性和非易失性存储器两者。其中，非易失性存储器可以是只读存储器(read-only memory，ROM)、可编程只读存储器(programmable ROM，PROM)、可擦除可编程只读存储器(erasable PROM，EPROM)、电可擦除可编程只读存储器(electrically EPROM，EEPROM)或闪存。易失性存储器可以是随机存取存储器(random access memory，RAM)，其用作外部高速缓存。通过示例性但不是限制性说明，许多形式的RAM可用。例如，静态随机存取存储器(static RAM，SRAM)、动态随机存取存储器(dynamic random access memory，DRAM)、同步动态随机存取存储器(synchronous DRAM，SDRAM)、双倍数据速率同步动态随机存取存储器(double data date SDRAM，DDR SDRAM)、增强型同步动态随机存取存储器(enhanced SDRAM，ESDRAM)、同步连接动态随机存取存储器(synchlink DRAM，SLDRAM)和直接内存总线随机存取存储器(direct rambus RAM，DR RAM)。
本申请提供了一种计算机程序，当计算机程序被计算机执行时，可以使得处理器或计算机执行上述方法实施例中对应的各个步骤和/或流程。
在上述实施例中，可以全部或部分地通过软件、硬件、固件或者其任意组合来实现。当使用软件实现时，可以全部或部分地以计算机程序产品的形式实现。所述计算机程序产品包括一个或多个计算机指令。在计算机上加载和执行所述计算机程序指令时，全部或部分地产生按照本申请所述的流程或功能。所述计算机可以是通用计算机、专用计算机、计算机网络、或者其他可编程装置。所述计算机指令可以存储在计算机可读存储介质中，或者从一个计算机可读存储介质向另一个计算机可读存储介质传输，例如，所述计算机指令可以从一个网站站点、计算机、服务器或数据中心通过有线(例如同轴电缆、光纤、数字用户线)或无线(例如红外、无线、微波等)方式向另一个网站站点、计算机、服务器或数据中心进行传输。所述计算机可读存储介质可以是计算机能够存取的任何可用介质或者是包含一个或多个可用介质集成的服务器、数据中心等数据存储设备。所述可用介质可以是磁性介质，(例如，软盘、硬盘、磁带)、光介质(例如，DVD)、或者半导体介质(例如固态硬盘Solid State Disk)等。
以上所述仅为本申请的实施例，并不用以限制本申请，凡在本申请的精神和原则之内，所作的任何修改、等同替换、改进等，均应包含在本申请的保护范围之内。

Claims (14)

  1. 一种图像处理的方法，其特征在于，所述方法应用于网络设备，所述方法包括：
    获取目标对象的第一位姿信息，所述第一位姿信息用于指示所述目标对象在第一时刻的姿态与位置；
    获取所述目标对象的历史位姿信息，所述历史位姿信息用于指示所述目标对象在历史时刻的姿态与位置；
    根据时延信息及所述历史位姿信息对所述第一位姿信息所指示的姿态与位置进行补偿，得到第二位姿信息，所述时延信息基于参考位姿信息的采集时刻与参考图像的显示时刻之间的时间间隔得到，所述参考图像是所述参考位姿信息对应的图像，所述参考位姿信息的采集时刻早于所述第一时刻；
    获取与所述第二位姿信息匹配的目标指令，所述目标指令用于获取待显示的图像。
  2. 根据权利要求1所述的方法，其特征在于，所述根据时延信息及所述历史位姿信息对所述第一位姿信息所指示的姿态与位置进行补偿，得到第二位姿信息，包括：
    根据所述时延信息及所述历史位姿信息确定位姿变化信息，所述位姿变化信息包括位姿变化量及位姿变化方向；
    基于所述位姿变化方向，按照所述位姿变化量补偿所述第一位姿信息所指示的姿态与位置，根据补偿后的姿态与位置得到第二位姿信息。
  3. 根据权利要求2所述的方法，其特征在于，所述根据所述时延信息及所述历史位姿信息确定位姿变化信息，包括：
    根据所述历史位姿信息确定所述位姿变化方向及位姿变化率；
    根据所述时延信息及所述位姿变化率确定所述位姿变化量。
  4. 根据权利要求1-3任一所述的方法，其特征在于，所述获取目标对象的第一位姿信息之后，所述方法还包括：
    获取所述时延信息；
    响应于所述时延信息大于时延阈值，执行所述根据时延信息及所述历史位姿信息对所述第一位姿信息所指示的姿态与位置进行补偿，得到第二位姿信息的步骤。
  5. 根据权利要求1-4任一所述的方法，其特征在于，所述网络设备为终端，所述获取与所述第二位姿信息匹配的目标指令之后，所述方法还包括：
    向服务器发送所述目标指令，接收所述服务器根据所述目标指令返回的待显示的图像，对所述图像进行显示。
  6. 根据权利要求1-4任一所述的方法，其特征在于，所述网络设备为服务器，所述获取与所述第二位姿信息匹配的目标指令之后，所述方法还包括：
    根据所述目标指令获取待显示的图像，将所述待显示的图像发送给终端进行显示。
  7. 一种图像处理的装置，其特征在于，所述装置包括：
    获取模块，用于获取目标对象的第一位姿信息，所述第一位姿信息用于指示所述目标对象在第一时刻的姿态与位置；
    所述获取模块，还用于获取所述目标对象的历史位姿信息，所述历史位姿信息用于指示所述目标对象在历史时刻的姿态与位置；
    补偿模块，用于根据时延信息及所述历史位姿信息对所述第一位姿信息所指示的姿态与位置进行补偿，得到第二位姿信息，所述时延信息基于参考位姿信息的采集时刻与参考图像的显示时刻之间的时间间隔得到，所述参考图像是所述参考位姿信息对应的图像，所述参考位姿信息的采集时刻早于所述第一时刻；
    所述获取模块，还用于获取与所述第二位姿信息匹配的目标指令，所述目标指令用于获取待显示的图像。
  8. 根据权利要求7所述的装置，其特征在于，所述补偿模块，用于根据所述时延信息及所述历史位姿信息确定位姿变化信息，所述位姿变化信息包括位姿变化量及位姿变化方向；基于所述位姿变化方向，按照所述位姿变化量补偿所述第一位姿信息所指示的姿态与位置，根据补偿后的姿态与位置得到第二位姿信息。
  9. 根据权利要求8所述的装置，其特征在于，所述补偿模块，用于根据所述历史位姿信息确定所述位姿变化方向及位姿变化率；根据所述时延信息及所述位姿变化率确定所述位姿变化量。
  10. 根据权利要求7-9任一所述的装置，其特征在于，所述获取模块，还用于获取所述时延信息；所述补偿模块，还用于响应于所述时延信息大于时延阈值，执行所述根据时延信息及所述历史位姿信息对所述第一位姿信息所指示的姿态与位置进行补偿，得到第二位姿信息的步骤。
  11. 根据权利要求7-10任一所述的装置，其特征在于，所述装置还包括：显示模块，用于向服务器发送所述目标指令，接收所述服务器根据所述目标指令返回的待显示的图像，对所述图像进行显示。
  12. 根据权利要求7-10任一所述的装置，其特征在于，所述装置还包括：发送模块，用于根据所述目标指令获取待显示的图像，将所述待显示的图像发送给终端进行显示。
  13. 一种网络设备，所述网络设备包括存储器及处理器；所述存储器中存储有至少一条指令，所述至少一条指令由所述处理器加载并执行，以实现权利要求1-6中任一所述的图像处理的方法。
  14. 一种计算机可读存储介质，其特征在于，所述存储介质中存储有至少一条指令，所述指令由处理器加载并执行以实现如权利要求1-6中任一所述的图像处理的方法。
PCT/CN2021/088120 2020-04-30 2021-04-19 图像处理的方法和装置 WO2021218683A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202010366426.4 2020-04-30
CN202010366426.4A CN113589919A (zh) 2020-04-30 2020-04-30 图像处理的方法和装置

Publications (1)

Publication Number Publication Date
WO2021218683A1 true WO2021218683A1 (zh) 2021-11-04

Family

ID=78237486

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/088120 WO2021218683A1 (zh) 2020-04-30 2021-04-19 图像处理的方法和装置

Country Status (2)

Country Link
CN (1) CN113589919A (zh)
WO (1) WO2021218683A1 (zh)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113804222A (zh) * 2021-11-16 2021-12-17 浙江欣奕华智能科技有限公司 一种定位精度的测试方法、装置、设备及存储介质
CN114219745A (zh) * 2021-12-03 2022-03-22 地平线(上海)人工智能技术有限公司 图像修正方法和装置、电子设备和存储介质
CN114833830A (zh) * 2022-04-27 2022-08-02 北京市商汤科技开发有限公司 抓取方法及装置、电子设备和存储介质

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150312494A1 (en) * 2014-04-28 2015-10-29 Lynx System Developers, Inc. Systems For Processing Event Timing Images
CN106293043A (zh) * 2015-06-29 2017-01-04 北京智谷睿拓技术服务有限公司 可视化内容传输控制方法、发送方法、及其装置
CN106998409A (zh) * 2017-03-21 2017-08-01 华为技术有限公司 一种图像处理方法、头戴显示器以及渲染设备
CN107566685A (zh) * 2017-09-26 2018-01-09 联想(北京)有限公司 一种图像数据处理方法及电子设备
CN110244840A (zh) * 2019-05-24 2019-09-17 华为技术有限公司 图像处理方法、相关设备及计算机存储介质

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10962780B2 (en) * 2015-10-26 2021-03-30 Microsoft Technology Licensing, Llc Remote rendering for virtual images

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150312494A1 (en) * 2014-04-28 2015-10-29 Lynx System Developers, Inc. Systems For Processing Event Timing Images
CN106293043A (zh) * 2015-06-29 2017-01-04 北京智谷睿拓技术服务有限公司 可视化内容传输控制方法、发送方法、及其装置
CN106998409A (zh) * 2017-03-21 2017-08-01 华为技术有限公司 一种图像处理方法、头戴显示器以及渲染设备
CN107566685A (zh) * 2017-09-26 2018-01-09 联想(北京)有限公司 一种图像数据处理方法及电子设备
CN110244840A (zh) * 2019-05-24 2019-09-17 华为技术有限公司 图像处理方法、相关设备及计算机存储介质

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113804222A (zh) * 2021-11-16 2021-12-17 浙江欣奕华智能科技有限公司 一种定位精度的测试方法、装置、设备及存储介质
CN113804222B (zh) * 2021-11-16 2022-03-04 浙江欣奕华智能科技有限公司 一种定位精度的测试方法、装置、设备及存储介质
CN114219745A (zh) * 2021-12-03 2022-03-22 地平线(上海)人工智能技术有限公司 图像修正方法和装置、电子设备和存储介质
CN114833830A (zh) * 2022-04-27 2022-08-02 北京市商汤科技开发有限公司 抓取方法及装置、电子设备和存储介质

Also Published As

Publication number Publication date
CN113589919A (zh) 2021-11-02

Similar Documents

Publication Publication Date Title
WO2021218683A1 (zh) 图像处理的方法和装置
EP2812770B1 (en) Image processing device, and computer program product
US9256986B2 (en) Automated guidance when taking a photograph, using virtual objects overlaid on an image
WO2020078250A1 (zh) 用于虚拟场景的数据处理方法以及设备
CN112333491B (zh) 视频处理方法、显示装置和存储介质
WO2021134960A1 (zh) 标定方法及装置、处理器、电子设备、存储介质
CN112783700A (zh) 用于基于网络的远程辅助***的计算机可读介质
CN112819860A (zh) 视觉惯性***初始化方法及装置、介质和电子设备
CN111327876A (zh) 目标跟踪显示方法、装置、电子设备及机器可读存储介质
JP2022531186A (ja) 情報処理方法、装置、電子機器、記憶媒体およびプログラム
CN111597466A (zh) 展示方法、装置和电子设备
CN112907652A (zh) 相机姿态获取方法、视频处理方法、显示设备和存储介质
US11694383B2 (en) Edge data network for providing three-dimensional character image to user equipment and method for operating the same
WO2022088989A1 (zh) 增强现实影像显示方法及相关装置
US20200045113A1 (en) Data processing method, computer device and storage medium
CN114240754A (zh) 投屏处理方法、装置、电子设备及计算机可读存储介质
CN107229055B (zh) 移动设备定位方法和移动设备定位装置
WO2023160445A1 (zh) 即时定位与地图构建方法、装置、电子设备及可读存储介质
CN110180185B (zh) 一种时间延迟测量方法、装置、***及存储介质
US20240019702A1 (en) Control method for head-mounted device and image rendering method
US11500455B2 (en) Video streaming system, video streaming method and apparatus
CN111064981A (zh) 一种视频串流的***及方法
WO2023201822A1 (zh) 多相机同步校正方法、装置及存储介质
CN113242398A (zh) 一种三维标注的音视频通话方法和***
US8755819B1 (en) Device location determination using images

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21796955

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21796955

Country of ref document: EP

Kind code of ref document: A1