CN114004957B

CN114004957B - Augmented reality picture generation method, device, equipment and storage medium

Info

Publication number: CN114004957B
Application number: CN202111276884.XA
Authority: CN
Inventors: 宋科科
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2021-10-29
Filing date: 2021-10-29
Publication date: 2024-07-09
Anticipated expiration: 2041-10-29
Also published as: CN114004957A

Abstract

The application provides a method, a device, equipment, a storage medium and a program product for generating an augmented reality picture, which can ensure that a virtual object is fused to a reality picture in a proper pose, and promote the sense of reality of the augmented reality picture; the method may be used to test an autopilot algorithm, comprising: acquiring a reporting pose which is different from the real pose of a real target object in a real scene; determining a target picture obtained by image acquisition of the digital twin scene according to the reporting pose; the target picture comprises a virtual object and a digital twin reference object which are positioned in the digital twin scene; acquiring a real picture which is acquired by a real target object in a real scene in a real pose and comprises a real reference object; determining a pose offset required for adjusting the pose of the digital twin reference object in the target picture to coincide with the pose of the real reference object in the real picture, and adjusting the pose of the virtual object according to the pose offset; and fusing the virtual object to the real picture in the adjusted pose to obtain the augmented reality picture.

Description

Augmented reality picture generation method, device, equipment and storage medium

Technical Field

The present application relates to the field of augmented reality technologies, and in particular, to a method, an apparatus, a computer device, a storage medium, and a computer program product for generating an augmented reality picture.

Background

Augmented reality technology, which is a technology that merges virtual objects with the real world, aims to expand information of the real world and enhance expression of the real world. The augmented reality technology can be applied to an optimized automatic driving algorithm, virtual objects such as virtual traffic flows are fused into the real images, and the augmented reality images are obtained for the automatic driving algorithm to perform perception and decision planning, so that the optimization of the automatic driving algorithm is realized.

In the generation process of the traditional augmented reality picture, the pose of the virtual object fused to the reality picture is difficult to determine, so that the reality sense of the fused augmented reality picture is lower.

Disclosure of Invention

In view of the foregoing, it is desirable to provide a method, an apparatus, a computer device, a storage medium, and a computer program product for generating an augmented reality picture.

A method of generating an augmented reality picture, the method comprising:

Acquiring a reporting pose of a real target object travelling in a real scene;

Determining a target picture obtained by carrying out image acquisition on the digital twin scene by using the reporting pose in the digital twin scene corresponding to the real scene; the digital twin scene comprises a virtual object which does not belong to the real scene and a digital twin reference object corresponding to a real reference object which belongs to the real scene, and the target picture comprises the virtual object and the digital twin reference object;

acquiring a real picture acquired by the real target object in the real scene in a real pose; the real picture comprises the real reference object, and the reporting pose and the real pose are different;

determining a pose offset required for adjusting the pose of the digital twin reference object in the target picture to be coincident with the pose of the real reference object in the real picture, and adjusting the pose of the virtual object in the target picture according to the pose offset;

And fusing the virtual object to the reality picture in the adjusted pose to obtain an augmented reality picture for guiding the reality target object.

An apparatus for generating an augmented reality picture, the apparatus comprising:

the pose acquisition module is used for acquiring the reporting pose of a real target object travelling in a real scene;

The image acquisition module of the digital twin scene is used for determining a target image obtained by image acquisition of the digital twin scene by the reporting pose in the digital twin scene corresponding to the real scene; the digital twin scene comprises a virtual object which does not belong to the real scene and a digital twin reference object corresponding to a real reference object which belongs to the real scene, and the target picture comprises the virtual object and the digital twin reference object;

The image acquisition module of the real scene is used for acquiring a real image acquired by the real target object in the real scene in a real pose; the real picture comprises the real reference object, and the reporting pose and the real pose are different;

The pose adjustment module is used for determining pose offset required for adjusting the pose of the digital twin reference object in the target picture to be coincident with the pose of the real reference object in the real picture, and adjusting the pose of the virtual object in the target picture according to the pose offset;

And the fusion module is used for fusing the virtual object to the real picture in the adjusted pose to obtain an augmented reality picture for guiding the real target object.

In some embodiments, the pose adjustment module is further configured to determine an amount of rotation required to adjust the pose of the digital twin reference object in the target picture to be consistent with the pose of the real reference object in the real picture; adjusting the posture of the digital twin reference object in the target picture according to the rotation quantity; determining a translation amount required for adjusting the position of the digital twin reference object in the target picture after the posture adjustment to be consistent with the position of the real reference object in the real picture; and taking the rotation amount and the translation amount as the pose offset amount.

In some embodiments, the pose adjustment module is further configured to determine a first linear function fitting the digital twin reference in the target picture, in a case where the digital twin reference belongs to a bar; determining a second linear function fitting the real reference object in the real picture in the case that the real reference object belongs to a bar; the rotation amount is determined based on a relative magnitude between an arctangent of a slope of the first linear function and an arctangent of a slope of the second linear function.

In some embodiments, the real reference is a real guide line located on both sides of the real target object, and the digital twinning reference is a digital twinning guide line corresponding to the real guide line; the pose adjustment module is further used for taking an arctangent value of a slope of a first linear function fitting the first side digital twin guide line as a first arctangent value and taking an arctangent value of a slope of a first linear function fitting the second side digital twin guide line as a second arctangent value; taking the arctangent value of the slope of the second linear function fitting the first side reality guide line as a third arctangent value and taking the arctangent value of the slope of the second linear function fitting the second side reality guide line as a fourth arctangent value; and determining the rotation amount by integrating the relative sizes of the first arctangent value and the third arctangent value and the relative sizes of the second arctangent value and the fourth arctangent value.

In some embodiments, the pose adjustment module is further configured to obtain the translation amount based on a relative magnitude between a constant of the first linear function and a constant of the second linear function.

In some embodiments, the real reference is a real guide line located on both sides of the real target object, and the digital twinning reference is a digital twinning guide line corresponding to the real guide line; the pose adjusting module is further used for taking a constant of a first linear function fitting the first side digital twin guide line as a first constant and taking a constant of a first linear function fitting the second side digital twin guide line as a second constant; taking a constant of a second linear function fitting the first side reality guide line as a third constant and taking a constant of a second linear function fitting the second side reality guide line as a fourth constant; and determining the translation amount by combining the relative magnitude between the first constant and the third constant and the relative magnitude between the second constant and the fourth constant.

In some embodiments, the image acquisition module of the digital twin scene is further configured to determine a digital twin target object corresponding to the real target object in the digital twin scene; and generating the target picture according to the virtual object and the digital twin reference object which are observed by the digital twin target object in the reporting pose.

In some embodiments, the image acquisition module of the digital twin scene is further configured to acquire camera parameters of a real camera provided to the real target object; and under the condition that the digital twin target object is in the digital twin scene in the reporting pose, projecting the virtual object and the digital twin reference object of the digital twin scene onto a picture according to the camera parameters to obtain the target picture.

In some embodiments, the pose acquisition module is configured to acquire a position and a traveling direction of the real target object in the real scene, where the position and the traveling direction are acquired by a satellite positioning device provided on the real target object; and taking the position and the advancing direction of the real target object in the real scene as the reporting pose.

In some embodiments, the virtual object is a virtual traffic flow, and the apparatus further comprises a digital twin scene processing module configured to construct the digital twin scene based on a high-precision map constructed for the real scene; determining the position of a digital twin road in the digital twin scene based on the high-precision map; and adding the virtual traffic flow into the digital twin scene according to the position of the digital twin road in the digital twin scene.

In some embodiments, the real target object is a real target vehicle, and the apparatus further comprises an automatic driving algorithm testing module for obtaining an augmented reality picture including a virtual traffic flow; and testing an automatic driving algorithm running on the real target vehicle based on the augmented reality picture comprising the virtual traffic flow.

A computer device comprising a memory storing a computer program and a processor executing the above-described method of generating an augmented reality picture.

A computer-readable storage medium having stored thereon a computer program for execution by a processor of the above-described method of generating an augmented reality picture.

A computer program product comprising a computer program which when executed by a processor implements the method of generating an augmented reality picture described above.

In the method, the device, the computer equipment, the storage medium and the computer program product for generating the augmented reality picture, because the reported pose of the real target object advancing in the real scene is different from the real pose thereof, the target picture obtained by reporting the pose in the digital twin scene is different from the real picture obtained by the real pose in the real scene, the pose of a virtual object in the target picture needs to be adjusted firstly, and then the virtual object is fused into the real picture in the adjusted pose; because the digital twin reference object and the real reference object are in one-to-one correspondence, the pose of the virtual object in the target image is adjusted based on the pose offset required by the superposition of the pose of the digital twin reference object in the target image and the pose of the real reference object in the real image, so that the virtual object can be fused into the real image in a proper pose, and the realism and the reality enhancement effect of the generated augmented reality image are improved.

Drawings

FIG. 1 is a schematic illustration of the pose of a real vehicle in a real scene and the pose of a digital twin vehicle in a digital twin scene from an overhead view in some embodiments;

FIG. 2 is a schematic diagram of images of a real host vehicle and a digital twinning host vehicle, each of which is seen in some embodiments;

fig. 3 is a flow chart of a method for generating an augmented reality picture in some embodiments;

FIG. 4 is a schematic diagram of a real picture and a target picture aligned in the same coordinate system in some embodiments;

FIG. 5 is a schematic view of images seen by a real host vehicle and a digital twinning host vehicle, respectively, with a street lamp as a reference in some embodiments;

FIG. 6 is a schematic diagram of an image seen by a real host vehicle in some embodiments where the reference object is a double-sided road line;

FIG. 7 is a schematic view of an image seen by a digital twin host vehicle in some embodiments where the reference is a double-sided road line;

FIG. 8 is an overall process framework diagram of a method of generating augmented reality pictures in some embodiments;

Fig. 9 is a flow chart of a method of generating an augmented reality picture in some embodiments;

FIG. 10 is a block diagram of an apparatus for generating augmented reality pictures in some embodiments;

FIG. 11 is an internal block diagram of a computer device in some embodiments.

Detailed Description

The present application will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present application more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the application.

Reference in the specification to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least some embodiments of the application. The appearances of such phrases in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Those of skill in the art will explicitly and implicitly appreciate that the described embodiments of the application may be combined with other embodiments.

The application provides a generation method of an augmented reality picture, and the related technologies comprise an augmented reality technology and a digital twin technology.

Augmented reality (Augmented Reality, AR), a technique that fuses virtual information with the real world, aims to expand the real world information and enhance the expression of the real world. The augmented reality technology mainly comprises two steps: (1) Determining the pose of a virtual object fused to a real picture (also called a real picture); (2) The virtual object is fused into the real picture in the determined pose.

In the related art, the method for determining the pose includes: the pose determining mode through image recognition and the pose determining mode through a sensor. The pose is determined through image recognition, mainly through an image recognition algorithm, the position of the real picture to be enhanced is determined, and a virtual object is placed at the position to be enhanced and fused in the real picture. In this way, the method performs better in the video with higher stability and relatively static state, but the video acquired by moving objects such as vehicles or robots in the moving process has low stability, the position to be enhanced identified from the video with low stability is inaccurate, the identification efficiency is low, and the omission phenomenon exists. The pose is determined by the sensor, mainly based on the sensor of the equipment, and the virtual object is fused into the real picture by directly utilizing the hardware positioning information. In this way, not only a high-precision sensor is needed to obtain the precise pose of a moving object such as a vehicle or a robot, but also precise camera internal and external parameters are needed to calculate the spatial position on a real picture; the internal and external parameters of the camera are realized by a calibration algorithm, and the calibration complexity of the external parameters is higher.

In the related art, a virtual object is fused to a real picture in a determined pose, and generally, the virtual object is fused to a corresponding position of the real picture in a certain pose and according to a certain viewing angle; the view angle processing mode mainly comprises view angle processing based on image geometry and view angle processing based on rendering. The view angle processing mode based on image geometry needs to predefine a series of view angle images with different angles, a closest image is selected according to positioning information, then residual view angle errors are compensated by geometric transformation, but detail distortion is caused due to the fact that local shielding of virtual objects is not processed, and the view angle images obtained through image geometric transformation are unnatural compared with real view angle images, and in addition, shielding relations among the virtual objects are difficult to process if the virtual objects are multiple. Based on the visual angle processing mode of rendering, a three-dimensional model (3D model) of a virtual object needs to be established, an image with a specific visual angle is acquired by using a graphic rendering tool, the requirement on hardware is high, and a rendering engine is needed.

In order to better determine the pose of the virtual object fused to the real picture and improve the sense of reality of the fused enhanced display picture, the application generates the enhanced real picture based on a digital twin technology.

The digital twin technology is a dynamic simulation technology integrating multiple disciplines, multiple physical quantities, multiple scales and multiple probabilities, and the mapping of the real world is completed in the virtual space, so that the whole life cycle process of corresponding entity equipment is reflected. The virtual scene simulating the real world created by the digital twinning technique may be referred to as a digital twinning scene. The digital twin technology is to sense, diagnose and predict the state of the physical object in real time through actual measurement, simulation and data analysis by using a digital model of the physical object in the real world, and regulate and control the behavior of the physical object through optimization and instructions; simulation techniques are one of the core techniques in creating and running digital twin scenes.

The digital twin scene has dynamics compared to a traditional simulation scene. Specifically, the digital twinning scene is not just a mirror image of the real world, but also accepts real-world real-time information and in turn drives the real world in real-time. The digital twinning scene also has bi-directionality. In particular, the data flow between the real scene and the digital twinning scene is bidirectional, the real scene can output data to the digital twinning scene, and the digital twinning scene can also feed back information to the real scene.

The augmented reality picture generated based on the digital twin technology provided by the application determines the pose of the virtual object fused into the reality picture by means of the pose of the digital twin reference object and the pose of the real reference object, has higher sense of reality, and can be applied to test scenes of decision and perception algorithms of moving objects such as vehicles, mobile robots, intelligent carrier and the like.

As shown in fig. 1, a digital twin scene corresponding to a real scene is created by a digital twin technique, and the digital twin scene is a digital representation of each real object in the real scene, so that a situation that a real object may happen can be seen in the digital twin scene.

In the autopilot application scenario, the real vehicle 101_r travels in a real scenario, and accordingly, the digital twin vehicle 101_s corresponding to the real vehicle 101_r travels in a digital twin scenario. The real camera of the real vehicle 101—r acquires a picture of the view ahead of the traveling in real time, which belongs to the real picture, as shown in the left picture of fig. 2. Correspondingly, in the digital twin scene, under the condition that the digital twin vehicle 101_s is in the reporting pose of the real vehicle 101_r, projecting the content observed by the digital twin vehicle 101_s in the digital twin scene into a picture to obtain a target picture, wherein the target picture can also be called a simulation picture; this calculation process can be abstracted as: and (3) carrying out image acquisition in the digital twin scene by using a digital twin camera on the digital twin vehicle 101_s corresponding to the real vehicle 101_r in the digital twin scene to obtain a target picture.

It can be appreciated that when there is no difference between the real pose of the real vehicle 101_r and the reporting pose thereof, then the viewing angle of the real vehicle 101_r in the real scene in the real pose is consistent with the viewing angle of the digital twin vehicle 101_s in the reporting pose in the digital twin scene. When there is a difference between the real pose of the real vehicle 101_r and the reporting pose thereof, since the pose of the digital twin vehicle 101_s in the digital twin scene is the reporting pose, there is a difference between the real pose of the real vehicle 101_r in the real scene and the pose of the digital twin vehicle 101_s in the digital twin scene, as shown in fig. 1, at this time, there is a difference between the viewing angle of the real vehicle 101_r in the real scene in the real pose and the viewing angle of the digital twin vehicle 101_s in the reporting pose in the digital twin scene, and in this case, the acquired real picture and target picture will also have a difference, as shown in fig. 2.

The server 102 may also set a virtual traffic flow in the digital twin scene, so when the digital twin vehicle 101_s runs in the digital twin scene, the pose of the digital twin vehicle 101_s in the digital twin scene is the reported pose of the real vehicle 101_r, and the digital twin camera set in the digital twin vehicle 101_s will collect the front virtual traffic flow, and the obtained target picture will include the virtual traffic flow; then, the virtual traffic flow is fused into the real picture, so that the automatic driving algorithm running in the real vehicle 101_r carries out obstacle sensing and decision planning, the 'spoofing' of the automatic driving algorithm by using an AR means is realized, and the test of the automatic driving algorithm is completed.

As described above, in an ideal situation, when the real pose of the real vehicle 101_r is consistent with the reported pose thereof, the viewing angle of the real vehicle 101_r in the real scene in the real pose is consistent with the viewing angle of the digital twin vehicle 101_s in the digital twin scene in the reported pose, so that the virtual traffic flow in the target picture can be perfectly fused with the real picture. However, if the reporting pose of the real vehicle 101_r is inconsistent with the real pose, and an error exists in the reporting pose, then when the real vehicle 101_r is in the real pose in the real scene, the real picture acquired from the perspective of the real vehicle 101_r and the target picture acquired from the perspective of the digital twin vehicle 101_s are deviated (as shown in fig. 2) when the digital twin vehicle 101_s is in the reporting pose in the digital twin scene, and if the virtual traffic flow in the target picture is directly fused into the real picture, the situation that the virtual traffic flow "drifts" to the road in the real picture occurs, so that the sense of realism of the fused augmented reality picture is lower, and when the real vehicle 101_r carries out obstacle avoidance on the input augmented reality picture based on a perception algorithm or a decision algorithm, the phenomenon of left-right drift occurs, so that the test of an automatic driving algorithm is affected.

In the method for generating the augmented reality picture provided by the embodiment of the application, the real target object travelling in the real scene can be communicated with the server 102 through the intelligent device arranged on the real target object. The data storage system may store data that the server 102 needs to process. The data storage system may be integrated on the server 102 or may be located on a cloud or other network server. In one embodiment, an intelligent device provided to a real target object reports a current pose of the real target object in a real scene to a server 102, the server 102 determines a target picture obtained by performing image acquisition on a digital twin scene corresponding to the real scene in the reported pose based on the reported pose, wherein the digital twin scene comprises a virtual object not belonging to the real scene and a digital twin reference object corresponding to a real reference object belonging to the real scene, and the target picture comprises the virtual object and the digital twin reference object; the server 102 also acquires a real picture acquired by the real target object in the real scene in a real pose; the real picture comprises the real reference object, and the reporting pose and the real pose are different; the server 102 determines a pose offset required for adjusting the pose of the digital twin reference object in the target picture to be coincident with the pose of the real reference object in the real picture, and adjusts the pose of the virtual object in the target picture according to the pose offset; and fusing the virtual object to the reality picture in the adjusted pose to obtain an augmented reality picture for guiding the reality target object. Optionally, the server 102 also sends the generated augmented reality picture to the smart device.

The intelligent device can be a personal computer, a notebook computer, a smart phone, a tablet personal computer, an Internet of things device and a portable wearable device which are arranged on a real target object, the Internet of things device can be an intelligent sound box, an intelligent television, an intelligent air conditioner and an intelligent vehicle-mounted device, and the real target object can be a vehicle or a mobile robot. The server 102 may be implemented as a stand-alone server or as a server cluster comprising a plurality of servers.

The application provides a method for generating an augmented reality picture, as shown in fig. 3, which can be applied to a server 102 in fig. 1, and mainly comprises the following steps:

step S301, acquiring a reporting pose of a real target object traveling in a real scene.

Wherein the real scene may be referred to as a real scene, which may be a real road scene in an application of the autopilot algorithm test. One of the real physical objects in the real scene can be set as a real target object to generate an augmented reality picture conforming to the viewing angle of the real target object. When the generated augmented reality pictures are used to test the application scenario of the autopilot algorithm, the real target object may travel vehicles in a real road scenario, also referred to as a real host vehicle.

The pose mainly comprises a position and a pose, and the position of a real target object in a real scene can be represented by longitude and latitude; if the real target object is in a moving state in the real scene, the gesture of the real target object in the real scene can be represented by a traveling direction (i.e. heading); if the gesture is characterized by a direction of travel, the direction of travel may be determined from the position at the previous time and the position at the subsequent time.

The reported pose can be determined according to the positioning information acquired by the positioning equipment arranged on the real target object, and the accuracy of the positioning information is inversely related to the difference between the reported pose and the real pose of the real target object in the real scene, namely, the higher the accuracy of the positioning information is, the smaller the pose difference is, the lower the accuracy of the positioning information is, and the larger the pose difference is. The positioning device may be a satellite positioning device such as GPS (Global Positioning System).

Taking a real target object as a real host vehicle as an example for introduction: when the real host vehicle runs in the real scene, the server 102 obtains the reporting pose according to the positioning information such as longitude, latitude, heading and the like acquired by the positioning equipment arranged on the real host vehicle.

Step S302, determining a target picture obtained by image acquisition of the digital twin scene according to the reported pose in the digital twin scene corresponding to the real scene.

The digital twin scene is a digital representation of each real object in a real scene, corresponding to the real scene. The digital twinning scene may comprise a digital twinning constructed from physical objects in the real scene, that is, there are physical objects in the real scene that correspond to the digital twinning, which may be referred to as simulators; wherein a digital twin constructed from the above-described real target object is referred to as a digital twin target object.

The digital twin scene may also include a virtual object that is not constructed from the physical objects of the real scene, that is, there are no physical objects in the real scene that correspond to the virtual object; the virtual object can be fused into a real picture acquired by taking a real target object as a visual angle so as to enhance the real picture; the virtual object is in the visual field range of a digital twin target object in a digital twin scene, and a target picture acquired by taking the digital twin target object as a visual angle comprises the virtual object; the target picture acquired by taking the digital twin target object as a viewing angle can be called a simulation picture. In the scenario of autopilot algorithm testing, the virtual object may be a virtual traffic flow.

In order to determine the pose offset required for fusing the virtual object to the real picture, it is necessary to set a real reference object in a physical object included in the real scene, set a digital twin reference object in a digital twin included in the digital twin scene, and then superimpose the pose of the digital twin reference object in the target picture to the pose offset required for fusing the real reference object to the pose of the real reference object in the real picture as the pose offset required for fusing the virtual object to the real picture.

The setting modes of the real reference object and the digital twin reference object can be as follows: determining a physical object set in the field of view of the real object in the real scene, and determining a digital twin object set in the field of view of the digital twin object in the digital twin scene; in the entity object set and the digital twin organism set, taking the entity objects and the digital twin organisms which are in one-to-one correspondence as one group, obtaining a plurality of groups, selecting a target group from the plurality of groups, taking the entity objects of the target group as a reality reference object and taking the digital twin organisms of the target group as a digital twin organism reference object.

The real reference object and the digital twin reference object selected in the above manner are respectively located in the visual field range of the real target object and the visual field range of the digital twin target object, so that the real picture acquired by taking the real target object as the visual angle comprises the real reference object, and the target picture acquired by taking the digital twin target object as the visual angle comprises the digital twin reference object.

Because both the entity object and the digital twin have a certain shape, and the pose of the digital twin reference object in the target picture is overlapped to the pose of the real reference object in the real picture, the pose offset required by the fusion of the virtual object to the real picture is determined; the more complex the shape of the digital twin reference object and the shape of the real reference object, the more complex the superposition processing between the pose of the digital twin reference object and the pose of the real reference object will be, and the lower the processing efficiency will be.

Therefore, in order to ensure processing efficiency, when selecting a target group from among a plurality of groups, the target group may be selected according to the shape complexity, specifically, a group having a shape complexity lower than a threshold may be taken as the target group or a group having a lowest shape complexity may be taken as the target group based on the shape complexity of each group.

The shape complexity of each group may be determined as follows:

① Synthesizing the shape complexity of the entity object included in the group and the shape complexity of the digital twin organism included in the group, and taking the synthesized result as the shape complexity of the group;

② Taking the shape complexity of the entity objects included in the group as the shape complexity of the group;

③ The shape complexity of the digital twins comprised by the set is taken as the shape complexity of the set.

The shape complexity of the physical object/digital twin can be determined according to the number of corner points of the physical object/digital twin, and the greater the number of corner points, the higher the shape complexity of the physical object.

In some scenes, the number of angular points of the entity object needs to be determined by image recognition processing, and a certain time is required; and because the digital twin is a digital model constructed according to the entity object, the number of the corner points of the digital twin can be obtained directly according to the corner points used in constructing the digital model, and compared with the image recognition processing, the processing efficiency is higher. Correspondingly, in the processing manner of ③, the number of corner points used in constructing the digital twin included in the group can be directly read, the shape complexity of the digital twin included in the group is determined based on the number of the read corner points, and the shape complexity of the digital twin included in the group is taken as the shape complexity of the group.

After the real reference object and the digital twin reference object are set in the mode, under the condition that the digital twin target object is in the reporting pose in the digital twin scene, acquiring pictures from the view angle of the digital twin target object, and obtaining target pictures comprising the virtual object and the digital twin reference object.

Step S303, acquiring a real picture acquired by a real target object in a real scene in a real pose.

The reporting pose is determined according to positioning information acquired by positioning equipment arranged on the real target object, and the reporting pose is different from the real pose of the real target object in a real scene.

By setting the real reference object in the above manner, since the real reference object is within the visual field of the real target object, when the real target object is in the real pose in the real scene, the image is acquired from the perspective of the real target object, and a real image including the real reference object can be obtained.

Step S304, determining a pose offset required for adjusting the pose of the digital twin reference object in the target picture to be coincident with the pose of the real reference object in the real picture, and adjusting the pose of the virtual object in the target picture according to the pose offset.

The pose of the digital twin reference object in the target picture is adjusted to the pose offset required by the superposition with the pose of the real reference object in the real picture, mainly comprising the following steps: the pose of the real reference object in the real picture is maintained unchanged, the pose of the digital twin reference object in the target picture is adjusted, and when the pose of the digital twin reference object in the target picture is overlapped to the pose of the real reference object in the real picture, the pose adjustment quantity of the digital twin reference object is taken as the required pose offset quantity; the desired pose offset may include a translation amount and a rotation amount.

When the required pose offset is obtained, if the required pose offset includes the rotation amount: rotating 10 degrees to the left, and the required pose offset includes a translation amount of 0, then adjusting the pose of the virtual object in the target picture according to the required pose offset to: the virtual object in the target picture is rotated 10 deg. to the left without translation.

If the required pose offset includes the rotation amount: rotated 10 ° to the left, and the required pose offset includes the translation amounts of: and translating the virtual object left by 4 pixels, and adjusting the pose of the virtual object in the target picture according to the required pose offset to be: the virtual object in the target picture is rotated 10 deg. to the left and the rotated virtual object is translated 4 pixels to the left.

Step S305, fusing the virtual object to the real picture in the adjusted pose, and obtaining an augmented reality picture for guiding the real target object.

After the pose of the virtual object in the target picture is adjusted according to the required pose offset, fusing the virtual object into a real picture according to the adjusted pose, and obtaining an augmented reality picture comprising the virtual object; after the augmented reality image is obtained by the real target object, a virtual object in the augmented reality image is taken as a real entity object, and the virtual object is used for responding, such as stopping running, right-right running and the like, so that the augmented reality image is guided to the real target object.

In the method for generating the augmented reality picture, since the reporting pose of the target object travelling in the real scene is different from the real pose, the target picture obtained by reporting the pose in the digital twin scene is different from the real picture obtained by the real pose in the real scene, the pose of the virtual object in the target picture needs to be adjusted firstly, and then the virtual object is fused into the real picture in the adjusted pose; because the digital twin reference object and the real reference object are in one-to-one correspondence, the pose of the virtual object in the target image is adjusted based on the pose offset required by the superposition of the pose of the digital twin reference object in the target image and the pose of the real reference object in the real image, so that the virtual object can be fused into the real image in a proper pose, and the sense of realism of the fused augmented reality image is improved.

In some embodiments, the determining the pose offset required for adjusting the pose of the digital twin reference object in the target picture to be coincident with the pose of the real reference object in the real picture specifically includes the following steps: determining a rotation amount required for adjusting the posture of the digital twin reference object in the target picture to be consistent with the posture of the real reference object in the real picture; adjusting the posture of the digital twin reference object in the target image according to the rotation quantity; determining a translation amount required for adjusting the position of the digital twin reference object in the target picture after the posture adjustment to be consistent with the position of the real reference object in the real picture; the rotation amount and the translation amount are taken as pose offset amounts.

From the perspective of the real target object, the real reference object has a certain posture in the real scene, so that the real reference object exists in a real picture acquired from the perspective of the real target object in a certain posture; likewise, from the perspective of the digital twin target object, the digital twin reference object has a certain pose in the digital twin scene, and therefore, the digital twin reference object may exist in a target picture acquired from the perspective of the digital twin target object in a certain pose.

The manner of determining the pose offset may be: the real picture and the target picture are aligned and placed in the same coordinate system as shown in fig. 4, the real picture is kept motionless, the target picture is rotated until the posture of the digital twin reference object in the target picture is consistent with the posture of the real reference object in the real picture, and the corresponding rotation quantity is obtained; translating the rotated target picture until the position of the digital twin reference object in the target picture after posture adjustment is consistent with the position of the real reference object in the real picture, and obtaining a corresponding translation amount; the obtained rotation amount and translation amount are taken as pose offset amounts.

In the mode, the gesture is determined to be consistent through rotation, and then the position is determined to be consistent through translation, so that the accuracy and the processing efficiency of the pose offset are ensured.

The above-described amounts of translation may include an amount of translation in the up-down direction (i.e., an amount of translation in the y direction shown in fig. 4) in the field of view of the real target object, and an amount of translation in the left-right direction (an amount of translation in the x direction shown in fig. 4) in the field of view of the real target object.

In some scenes, the translation amount along the up-down direction in the visual field range of the real target object has a larger influence on the size of the virtual objects such as the virtual traffic flow in the visual field of the real target object, and has a smaller influence on the 'drift' of the virtual objects such as the virtual traffic flow in the visual field of the real target object; the amount of translation in the left-right direction in the field of view of the real target object has a large influence on the "drift" of the virtual object such as the virtual traffic flow in the field of view of the real target object, and has a small influence on the size of the virtual object such as the virtual traffic flow in the field of view of the real target object.

Therefore, in some scenarios, to reduce the "drift" of the virtual object such as the virtual traffic flow and improve the processing efficiency, the above-mentioned translational amount may include only the translational amount in the left-right direction of the field of view of the real target object, that is, the translational amount in the x-direction shown in fig. 4. In this case, the posture-adjusted digital twin reference object is translated in the x-direction until the posture-adjusted digital twin reference object is positioned at the same position as the real reference object in the x-direction.

When the set real reference object and the digital twin reference object are bar-shaped objects such as road lines, lamp poles and the like, the shape complexity is low, and the fitting can be performed by using straight lines.

Thus, in some embodiments, the determining adjusts the pose of the digital twin reference in the target picture to an amount of rotation required to be consistent with the pose of the real reference in the real picture, specifically includes the steps of: determining a first linear function of the digital twin reference in the fitted target picture in the case that the digital twin reference belongs to a bar; determining a second linear function fitting the real reference object in the real picture under the condition that the real reference object belongs to the bar object; the amount of rotation is determined based on the relative magnitude between the arctangent of the slope of the first linear function and the arctangent of the slope of the second linear function.

Wherein the arctangent value of the slope of the linear function fitting the reference is characterized: the reference object is shown as a picture with respect to a tilt angle in the left-right direction along the visual field of the target object (the target object is a real target object or a digital twin target object).

Taking the real reference object shown in fig. 5 as the real light pole 103_r and the digital twin reference object as the digital twin light pole 103_s for example, the following description will be made:

From a target picture acquired from the perspective of the digital twin vehicle 101_s, determining a linear function y=a ₁x+b₁ fitting the digital twin lamp post 103_s in the target picture, and taking the linear function as a first linear function; in a real picture acquired from the perspective of the real vehicle 101_r, determining a linear function y=a ₂x+b₂ fitting the real lamp post 103_r in the real picture, and taking the linear function as a second linear function; the arctangent value atan (a ₁) of the slope a ₁ in the first linear function y=a ₁x+b₁ characterizes the inclination angle of the digital twin light pole relative to the x-axis direction in the target picture, and the arctangent value atan (a ₂) of the slope a ₂ in the second linear function y=a ₂x+b₂ characterizes the inclination angle of the real light pole relative to the x-axis direction in the real picture; then, the difference between atan (a ₁) and atan (a ₂) can be used as the rotation amount.

In the above manner, when the reference object belongs to the bar object, the rotation amount can be determined directly according to the relative magnitude of the arctangent value of the slope of the linear function fitting each reference object, the calculation is simple, and the real-time requirement of enhancing the real picture is met.

When the set real reference object and the digital twin reference object are bar-shaped objects such as road lines and street lamp poles, and straight line fitting processing can be utilized, the determining step of adjusting the position of the digital twin object in the target picture after the posture adjustment to the translation amount required to be consistent with the position of the real reference object in the real picture specifically comprises the following steps: the amount of translation is obtained based on the relative magnitude between the constant of the first linear function and the constant of the second linear function.

From a target picture acquired from the perspective of the digital twin vehicle 101_s, determining a linear function y=a ₁x+b₁ fitting the digital twin lamp post 103_s in the target picture, and taking the linear function as a first linear function; in a real picture acquired from the perspective of the real vehicle 101_r, determining a linear function y=a ₂x+b₂ fitting the real lamp post 103_r in the real picture, and taking the linear function as a second linear function; the difference between the constant b ₁ in the first linear function y=a ₁x+b₁ and the constant b ₂ in the second linear function y=a ₂x+b₂ characterizes the amount of translation in the left-right direction within the field of view of the target object (which is a real target object or a digital twin target object), that is, the amount of translation in the x-direction shown in fig. 4, and translates the rotated digital twin light pole 103_s in the x-direction in accordance with the difference so that the digital twin light pole 103_s coincides with the real light pole 103_r.

In the above manner, under the condition that the reference objects belong to the bar objects, the translation amount can be determined directly according to the relative magnitude of the constants of the linear functions fitting the reference objects, the calculation is simple, and the real-time requirement of enhancing the real picture is met.

In some embodiments, the portion of the reference object belongs to the strip object, and the portion of the reference object is in a curved state, then only the portion of the strip object can be linearly fitted, for example, in the road shown in fig. 1, the road closer to the front of the vehicle is a straight road, the road farther to the front of the vehicle is a turning road, and when the road route is fitted by using a linear function, only the road line of the straight road can be fitted; if a turn road and a straight road appear in the picture at the same time, a road line of the entire road in the picture, which is close to the quarter range of the vehicle (20 meters may be selected), may be selected and a straight line fitting may be performed on the selected road line.

In case the reference object (which is a real reference object and a digital twinning reference object) belongs to a strip, the reference object may comprise a plurality of strip objects, such as a plurality of guide lines, such as a plurality of light poles, etc.; in case the reference comprises a plurality of strip objects, each strip may be located on either side of the target object travel direction, e.g. a plurality of guide lines each located on the left side of the target object travel direction, and e.g. a plurality of light poles each located on the right side of the target object travel direction.

In order to further improve the accuracy of the pose offset, the reference object comprises strip objects which are respectively positioned at two sides of the advancing direction of the target object.

In some embodiments, the real reference is a real guide line located on both sides of the real target object, and the digital twinning reference is a digital twinning guide line corresponding to the real guide line; wherein the guide line is a line for guiding the travel of the target object, in particular can be characterized by a road line. Illustratively, as shown in fig. 6, road lines 104_r1 and 104_r2 located on both sides in the traveling direction of the real vehicle 101_r are set as real references; correspondingly, as shown in fig. 7, the road lines 104_s1 and 104_s2 located on both sides of the traveling direction of the digital twin vehicle 101_s are set as digital twin references.

In this case, the determining the rotation amount based on the relative magnitude between the arctangent of the slope of the first linear function and the arctangent of the slope of the second linear function may specifically include the steps of: taking the arctangent value of the slope of the first linear function fitting the first side digital twin guide line as a first arctangent value and taking the arctangent value of the slope of the first linear function fitting the second side digital twin guide line as a second arctangent value; taking the arctangent value of the slope of the second linear function fitting the first side reality guide line as a third arctangent value and taking the arctangent value of the slope of the second linear function fitting the second side reality guide line as a fourth arctangent value; the relative magnitudes between the first and third arctangent values and the second and fourth arctangent values are combined to determine the rotation amount.

Taking the first side digital twin guide line as the digital twin road line 104_s1, the second side digital twin guide line as the digital twin road line 104_s2, the first side real guide line as the real road line 104_r1, and the second side real guide line as the real road line 104_r2 as examples, the above description is presented:

In determining the fitting digital twin road route 104_s1, the first linear function of digital twin road route 104_s2 is: y=a ₁₁x+b₁₁、y＝a₁₂x+b₁₂, the second linear functions fitting the real road line 104_r1, the real road line 104_r2 are: after y=a ₂₁x+b₂₁、y＝a₂₂x+b₂₂, the difference between atan (a ₁₁) and atan (a ₂₁) and the difference between atan (a ₁₂) and atan (a ₂₂) are determined, and the difference between atan (a ₁₁) and atan (a ₂₁) and the difference between atan (a ₁₂) and atan (a ₂₂) are integrated to obtain the rotation amount.

Wherein, the difference between atan (a ₁₁) and atan (a ₂₁) and the difference between atan (a ₁₂) and atan (a ₂₂) are integrated, and the rotation amount may be obtained by: the difference between atan (a ₁₁) and atan (a ₂₁) and the difference between atan (a ₁₂) and atan (a ₂₂) are summed and averaged to obtain an average valueAs the amount of rotation. The difference between atan (a ₁₁) and atan (a ₂₁) and the difference between atan (a ₁₂) and atan (a ₂₂) are integrated, and the rotation amount may be obtained by: based on the distances between the real vehicle 101_r and the real road routes on the respective sides, the differences on the respective sides are given weights in negative correlation with the distances on the respective sides, the differences on the respective sides are weighted and summed, and the obtained average value is used as the rotation amount, for example, in the real road routes 104_r1 and 104_r2, the real road route 104_r2 is closer to the real vehicle 101_r than the real road route 104_r1, the weight given to the difference between atan (a ₁₂) and atan (a ₂₂) is greater than the weight given to the difference between atan (a ₁₁) and atan (a ₂₁), and the rotation amount is obtained by weighted and summed the differences according to the weights. It will be appreciated that the differences corresponding to each side may also be given a weight that is inversely related to the distance of each side based on the distance of the digital twin vehicle 101_s from the digital twin road route of each side, respectively.

In the above manner, the guideline positioned at both sides of the advancing direction of the target object is used as a reference object, and the relative sizes of the arctangent value of the slope of the first linear function and the arctangent value of the slope of the second linear function of the guideline on the similar side are synthesized based on the relative sizes, so that the rotation amount is obtained, and the accuracy of rotating the virtual object is improved.

In some embodiments, in a case where the real reference object is a real guide line located at both sides of the real target object and the digital twin reference object is a digital twin guide line corresponding to the real guide line, the obtaining the translation amount based on the relative magnitude between the constant of the first linear function and the constant of the second linear function may specifically include the following steps: taking the constant of the first linear function fitting the first side digital twin guide line as a first constant and taking the constant of the first linear function fitting the second side digital twin guide line as a second constant; taking a constant of a second linear function fitting the first side reality guide line as a third constant and taking a constant of a second linear function fitting the second side reality guide line as a fourth constant; the relative magnitudes between the first constant and the third constant and the second constant and the fourth constant are combined to determine the translation.

In determining the fitting digital twin road route 104_s1, the first linear function of digital twin road route 104_s2 is: y=a ₁₁x+b₁₁、y＝a₁₂x+b₁₂, the second linear functions fitting the real road line 104_r1, the real road line 104_r2 are: after y=a ₂₁x+b₂₁、y＝a₂₂x+b₂₂, the difference between b ₁₁ and b ₂₁ and the difference between b ₁₂ and b ₂₂ are determined, and the difference between b ₁₁ and b ₂₁ and the difference between b ₁₂ and b ₂₂ are combined to obtain a translation amount, which can represent a translation amount in the left-right direction within the field of view of the target object (the target object is a real target object or a digital twin target object), that is, a translation amount in the x direction shown in fig. 4.

The way to obtain the translation amount may be to integrate the difference between b ₁₁ and b ₂₁ and the difference between b ₁₂ and b ₂₂: the difference between b ₁₁ and b ₂₁ and the difference between b ₁₂ and b ₂₂ are summed and averaged to obtain an average valueAs the amount of translation. The way to obtain the translation amount by combining the difference between b ₁₁ and b ₂₁ and the difference between b ₁₂ and b ₂₂ can also be: based on the distances between the real vehicle 101_r and the real road routes on the respective sides, the differences corresponding to the respective sides are given weights in negative correlation with the distances on the respective sides, the differences corresponding to the respective sides are weighted and summed, and the obtained average value is used as the rotation amount, for example, in the real road routes 104_r1 and 104_r2, the real road route 104_r2 is closer to the real vehicle 101_r than the real road route 104_r1, the weights given to the differences between b ₁₂ and b ₂₂ are greater than the weights given to the differences between b ₁₁ and b ₂₁, and the respective differences are weighted and summed according to the weights, so as to obtain the translation amount. It will be appreciated that the differences corresponding to each side may also be given a weight that is inversely related to the distance of each side based on the distance of the digital twin vehicle 101_s from the digital twin road route of each side, respectively.

In the above manner, the guide lines positioned at two sides of the advancing direction of the target object are used as reference objects, the relative sizes of constants of the first linear functions of the guide lines at the similar sides are synthesized based on the relative sizes of constants of the first linear functions of the guide lines at the similar sides, and the translation amount is obtained, so that the accuracy of translating the virtual object is improved, and the operation efficiency is ensured.

In some embodiments, in the digital twin scene corresponding to the real scene, determining the target picture obtained by performing image acquisition on the digital twin scene with the reporting pose may specifically include the following steps: determining a digital twin target object corresponding to a real target object in the digital twin scene; and generating a target picture according to the virtual object and the digital twin reference object observed by the digital twin target object in the reporting pose.

The reporting pose may include a position and a heading; the real target object can report the pose in real time in the process of moving in the real scene, after receiving the report pose, the server 102 takes the report pose as the pose of the digital twin target object in the digital twin scene, and starts according to the observation view angle of the digital twin target object under the report pose, and generates a target picture according to the observed virtual object and the digital twin reference object.

In the mode, the pose of the digital twin target object in the digital twin scene is obtained based on the reported pose of the real target object, the target picture comprising the virtual object and the digital twin reference object is obtained, and the sense of reality of the augmented reality picture is ensured.

In some embodiments, the generating the target picture according to the virtual object and the digital twin reference object observed by the digital twin target object in the reporting pose specifically includes the following steps: acquiring camera parameters of a real camera arranged on a real target object; under the condition that the digital twin target object is in the digital twin scene according to the reported pose, projecting a virtual object of the digital twin scene and a digital twin reference object onto a picture according to camera parameters to obtain a target picture.

The camera parameters include camera internal parameters and camera external parameters; the real camera arranged on the real target object is mainly used for collecting pictures in the visual field range of the real target object to form a real picture.

After obtaining the camera internal and external parameters of the real camera provided for the real target object, the server 102 projects the virtual objects and the digital twin reference objects within the visual field range of the digital twin target object onto the picture according to the camera internal and external parameters under the condition that the digital twin target object is in the digital twin scene according to the reported pose, so as to obtain the target picture.

In the process of projecting a real reference object from a real scene to a picture to obtain the real picture, conversion among a pixel coordinate system, an image coordinate system, a camera coordinate system and a world coordinate system exists, and a specific conversion relation is shown in the following formula:

Wherein the method comprises the steps of

Wherein the method comprises the steps ofR ²＝x′²+y′² Day Z _c noteq0.

In the above conversion formula, u and v are coordinates in a pixel coordinate system, x and y are coordinates in an image coordinate system, xc, yc, and Zc are coordinates in a camera coordinate system, and Xw, yw, and Zw are coordinates in a world coordinate system.

In the above mode, the target picture comprising the virtual object and the digital twin reference object is generated according to the camera parameters of the real camera arranged on the real target object, so that the consistency of the target picture and the real picture is ensured as much as possible, and the subsequent pose adjustment of the virtual object is facilitated.

In some embodiments, the acquiring the reporting pose of the real target object traveling in the real scene specifically includes the following steps: acquiring the position and the advancing direction of a real target object in a real scene, wherein the position and the advancing direction are acquired by a satellite positioning device arranged on the real target object; and taking the position and the traveling direction of the real target object in the real scene as the reporting pose.

The positioning device may be a satellite positioning device such as GPS (Global Positioning System); the satellite positioning equipment is easy to have errors in positioning information acquired in a building dense area due to errors and signal interference of the satellite positioning equipment. The position in the positioning information may be characterized by a longitude and latitude, and the traveling direction (i.e., heading) may be determined according to the position at the previous time and the position at the subsequent time.

After obtaining the position and the traveling direction of the real target object in the real scene acquired by the satellite positioning device provided on the real target object, the server 102 takes the position and the traveling direction as the reporting pose.

In the mode, the position and the advancing direction acquired by the satellite positioning equipment arranged on the real target object are used as the reporting pose, so that the determining efficiency of the pose of the digital twin target object in the digital twin scene is improved.

In some embodiments, the virtual object is a virtual traffic flow, and the virtual traffic flow includes a pose of a motor vehicle and a pose of a passive vehicle; the embodiment may further include the following steps: based on a high-precision map constructed for a real scene, constructing to obtain a digital twin scene; determining the position of the digital twin road in the digital twin scene based on the high-precision map; and adding the virtual traffic flow into the digital twin scene according to the position of the digital twin road in the digital twin scene.

The high-precision map is constructed for a real scene and is used for reflecting the real scene.

The server 102 may perform automated digital twinning scene construction based on the high-definition map, and then determine a position of the digital twinning road in the digital twinning scene based on the high-definition map, and add a virtual traffic flow on the digital twinning road according to the position of the digital twinning road in the digital twinning scene.

In the mode, the digital twin scene is automatically constructed according to the high-precision map, manual modeling is not needed, efficiency is improved, virtual traffic flows are added based on the high-precision map, and the virtual traffic flows are guaranteed to be attached to the road surfaces of roads in the digital twin scene.

In some embodiments, the above-mentioned real target object is a real target vehicle, and the present embodiment may further include the following steps: obtaining an augmented reality picture comprising a virtual traffic stream; and testing an automatic driving algorithm running on the real target vehicle based on the augmented reality picture comprising the virtual traffic flow.

The autopilot algorithm includes obstacle awareness and decision planning.

In the implementation, after the virtual traffic flow is fused to the real picture in the adjusted pose, an augmented reality picture comprising the virtual traffic flow is obtained, and an automatic driving algorithm running on a real target vehicle is tested by using the augmented reality picture.

In the mode, the augmented reality picture comprising the virtual traffic flow is utilized, the AR means is used for 'cheating' the perception algorithm, the test of the automatic driving algorithm is realized, the complexity of obstacle avoidance test simulation of the traffic flow is reduced, vehicles are not required to be placed in various real traffic flow scenes, and the test cost of the automatic driving algorithm is reduced.

In order to better understand the above method, an application example of the method for generating an augmented reality image according to the present application is described in detail below with reference to fig. 8 and 9. The scene of the application example is to enhance the virtual traffic flow on the picture acquired by the vehicle-mounted camera, namely, the virtual traffic flow is added in the vehicle-end video for testing and verification of an automatic driving algorithm.

The actual host vehicle running the autopilot algorithm is equipped with a plurality of sensors such as cameras, GPS, IMU (Inertial Measurement Unit ), lidar, etc. The front-view camera is used for collecting a real picture and is used as a background image of the AR; the IMU positioning device is used for generating traffic flow images. During running of a real host vehicle, a front-view camera acquires a real picture, the real picture is processed by a scheme provided by an application example and becomes an augmented reality picture containing virtual traffic flow, the augmented reality picture is used by a sensing unit in an automatic driving algorithm, and the automatic driving algorithm under the condition of simulating complex traffic flow is tested.

In general, the present application example includes the following:

1. digital twin road scene

The application example mainly establishes a digital twin scene of the road, and aims to render a lane line of the road for AR positioning information extraction. The road scene is derived from high-precision map automatic modeling, and manual modeling is not needed.

2. Traffic flow engine

The traffic flow engine is mainly used for generating virtual traffic flow data and comprises position and attitude information of a plurality of motor vehicles and non-motor vehicles, the traffic flow engine depends on a high-precision map, and the generated virtual traffic flow position is matched with a real road surface.

3. Positioning sensor GPS/IMU

The positioning sensor can acquire the positioning information of the real host vehicle in the real road scene in real time, including the position and the gesture (direction), but due to the error and the signal interference of the positioning sensor, the positioning sensor is easy to generate error in a dense building area, and if the virtual traffic flow acquired according to the positioning information is directly fused to a real picture, the virtual traffic flow floats on the real picture. In the digital twin road scene, according to the positioning information of the positioning sensor, a digital twin camera on the digital twin host vehicle can shoot images at corresponding positions and postures along with the positioning information to obtain a target picture.

4. Camera emulation

The mounting position and camera parameters of the digital twin camera on the digital twin host vehicle need to be consistent with, but not particularly accurate as, the mounting position and camera parameters of the real camera on the real vehicle. The camera parameters comprise camera internal parameters and camera external parameters, the camera internal parameters are consistent, the fact that the vehicle rendered each time accords with the distortion and perspective effect (of near-large and far-small) of a real camera is ensured, and when in later fusion, special treatment is not needed, and the direct pixel-level fusion is realized; the camera external parameters are consistent, so that the observation angle of the digital twin camera accords with the real vehicle angle, and the correct shielding relationship of the rendered traffic vehicle is ensured.

5. Lane line matching

The real road line is matched with the digital twin road line, the real road line is obtained by identifying an image of a real host vehicle, and the digital twin road line is obtained by rendering the lane line of the digital twin road scene. Because of the positioning precision of the GPS/IMU, errors exist between the digital twin road route rendered based on the positioning data and the real road route, and the errors need to be corrected in time, so that the left-right drift of the virtual traffic flow in the display picture is avoided. In the high-rise forest and urban environment, the magnetic field and the signals are easy to interfere, the accuracy of positioning information such as the position and the heading yaw provided by the GPS is reduced, and because the pitch angle and the roll angle roll provided by the IMU are obtained by means of gravity and a gyroscope, the accuracy of the pitch angle and the roll angle roll cannot be influenced by the magnetic field and the signals, and therefore the position and the heading yaw provided by the GPS need to be corrected.

6. And correcting the gesture and the position of the virtual traffic flow according to the determined correction quantity of the heading yaw and the determined correction quantity of the position.

7. And fusing the virtual traffic flow into the real picture with the corrected pose.

As shown in fig. 8, in the present application example generating an augmented reality picture including a virtual traffic flow, a digital twin road scene is constructed by a high-precision map, and the virtual traffic flow is generated by a traffic flow engine; the virtual traffic flow generated by the traffic flow engine comprises the position and posture information of a plurality of motor vehicles and non-motor vehicles, the traffic flow engine depends on a high-precision map, and the generated virtual traffic flow position is matched with a real road surface. Then, adding the virtual traffic flow into the digital twin road scene; based on the high-precision map, the GPS and the IMU, rendering a digital twin road scene to obtain a digital twin road route; and extracting a real road route from the real picture, correcting the pose of the virtual traffic flow in the target picture based on pose matching of the digital twin road route and the real road route, and fusing the virtual traffic flow into the real picture with the corrected pose to obtain an enhanced display picture comprising the virtual traffic flow.

Specifically, the present application embodiment includes the steps shown in fig. 9:

Step S901, constructing a digital twin road scene based on a high-precision map constructed for a real road scene;

Step S902, determining the position of a digital twin road in a digital twin road scene based on a high-precision map;

step S903, adding the virtual traffic flow into the digital twin road scene according to the position of the digital twin road in the digital twin road scene;

Step S904, under the condition that the real host vehicle is in a real scene in a real pose, projecting real road routes positioned at two sides of the real host vehicle to the pictures according to camera parameters of a real camera arranged on the real host vehicle to obtain real pictures;

Step S905, taking the position and the advancing direction of the real host vehicle in the real road scene acquired by the satellite positioning equipment arranged on the real host vehicle as the reporting pose of the real host vehicle; the reported pose is different from the real pose of the real host vehicle in the real road scene; the satellite positioning equipment is a GPS;

Step S906, under the condition that the reported pose of the digital twin host vehicle is in a digital twin road scene, projecting a digital twin road route positioned at two sides of the digital twin host vehicle and a virtual traffic stream positioned in front of the digital twin host vehicle to a picture according to camera parameters of a real camera arranged on the real host vehicle to obtain a target picture;

step S907, determining a first linear function fitting the double-sided digital twin road route in the target picture and a second linear function fitting the double-sided real road route in the real picture respectively;

Specifically, in this step, the distortion of the real image may be removed, then the real road routes (optionally, road lines within 20 meters of the real host vehicle) located on both sides of the real host vehicle are identified from the real image after the distortion is removed, and the linear function of the fitted left real road route 104_r1 is determined to be y=a ₂₁x+b₂₁, and the linear function of the fitted right real road route 104_r2 is determined to be y=a ₂₂x+b₂₂; the heading angles of the left-side real road route 104_r1 and the right-side real road route 104_r2 are respectively denoted by atan (a ₂₁)、atan(a₂₂).

Identifying digital twin channel routes on two sides of the digital twin host vehicle from the target picture, and determining that a linear function fitting the left digital twin channel route 104_s1 is y=a ₁₁x+b₁₁ and a linear function fitting the right digital twin channel route 104_s2 is y=a ₁₂x+b₁₂; the heading angles of the left digital twin road line 104_s1 and the right digital twin road line 104_s2 are respectively designated by atan (a ₁₁)、atan(a₁₂).

Where x and y in ,y＝a₁₁x+b₁₁、y＝a₁₂x+b₁₂、y＝a₂₁x+b₂₁、y＝a₂₂x+b₂₂ are the positions of the pixels corresponding to the road line in the image coordinate system.

Step S908, regarding the arctangent value of the slope of the first linear function fitting the first side digital twin road route as the first arctangent value, regarding the arctangent value of the slope of the first linear function fitting the second side digital twin road route as the second arctangent value, regarding the arctangent value of the slope of the second linear function fitting the first side real road route as the third arctangent value, and regarding the arctangent value of the slope of the second linear function fitting the second side real road route as the fourth arctangent value.

In step S909, the relative magnitudes between the first arctangent and the third arctangent and the relative magnitudes between the second arctangent and the fourth arctangent are combined to determine the rotation amount.

The difference between atan (a ₁₁) and atan (a ₂₁) and the difference between atan (a ₁₂) and atan (a ₂₂) are summed and averaged to obtain an average valueAs a rotation amount; the selected amount being a correction amount to the heading angle, i.e. a correction amount to the heading angle

Step S910, taking the constant of the first linear function fitting the first side digital twin path line as a first constant, and taking the constant of the first linear function fitting the second side digital twin path line as a second constant;

Step S911, taking the constant of the second linear function fitting the first side real road route as a third constant, and taking the constant of the second linear function fitting the second side real road route as a fourth constant;

step S912, combining the relative magnitudes of the first constant and the third constant and the second constant and the fourth constant to determine the translation amount;

Specifically, the differences between b ₁₁ and b ₂₁ and the differences between b ₁₂ and b ₂₂ can be summed and averaged to obtain an average value As the amount of translation. The translation being in the x-direction, i.e. the correction in the x-direction

Step S913, the rotation amount and the translation amount are used as pose adjustment amounts required for adjusting the pose of the digital twin road route in the target picture to be overlapped with the pose of the real road route in the real picture;

Step S914, adjusting the pose of the virtual traffic flow in the target picture according to the rotation amount and the translation amount;

After the correction d _yaw of the heading angle and the correction d _x along the x direction are obtained through the above steps, the virtual traffic flow in the target picture can be rotated and translated according to x '=cos (d _yaw)*x-sin(d_yaw)*y+d_x and y' =cos (d _yaw)*y+sin(d_yaw) ×x (x, y is the position of the virtual traffic flow in the image coordinate system).

Step S915, fusing the virtual traffic flow to the real picture with the adjusted pose, and obtaining an augmented reality picture comprising the virtual traffic flow;

Step S916, testing an autopilot algorithm running on the real host vehicle based on the augmented reality picture including the virtual traffic flow.

The application example is based on the method for generating the augmented reality picture of the digital twin technology, the positioning information acquired by the GPS sensor of the real host vehicle is adopted to determine the position and the gesture of the digital twin host vehicle in the digital twin scene, and the pose error caused by the positioning information is compensated according to the method for matching the road route between images, so that the whole calculation process is simple, and the real-time requirement can be met. Specifically, the virtual traffic flow is ensured not to be in a state of left-right offset in the real picture and to be in a stable state, the pose correction of the virtual traffic flow is directly carried out on the image according to road line matching, the virtual traffic flow is ensured to be stably positioned in the road in the real picture, and the virtual traffic flow has a good stabilizing effect no matter in cities or suburbs; the calculation related to the application example mainly comprises road line identification, road line fitting, correction amount calculation and correction of pose, the calculated amount is not large, and real-time AR requirements are guaranteed.

It should be understood that, although the steps in the flowcharts of fig. 1 to 9 are sequentially shown as indicated by arrows, these steps are not necessarily sequentially performed in the order indicated by the arrows. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least a portion of the steps of fig. 1-9 may include steps or stages that are not necessarily performed at the same time, but may be performed at different times, nor does the order in which the steps or stages are performed necessarily occur sequentially, but may be performed alternately or alternately with other steps or at least a portion of the steps or stages in other steps.

In some embodiments, as shown in fig. 10, there is provided a generating apparatus of an augmented reality picture, including:

The pose acquisition module 1001 is configured to acquire a reporting pose of a real target object traveling in a real scene;

A picture obtaining module 1002 of a digital twin scene, configured to determine, in a digital twin scene corresponding to the real scene, a target picture obtained by performing image acquisition on the digital twin scene with the reporting pose; the digital twin scene comprises a virtual object which does not belong to the real scene and a digital twin reference object corresponding to a real reference object which belongs to the real scene, and the target picture comprises the virtual object and the digital twin reference object;

A picture acquisition module 1003 of a real scene, configured to acquire a real picture acquired by the real target object in the real scene in a real pose; the real picture comprises the real reference object, and the reporting pose and the real pose are different;

a pose adjustment module 1004, configured to determine a pose offset required for adjusting a pose of the digital twin reference object in the target picture to be coincident with a pose of the real reference object in the real picture, and adjust a pose of the virtual object in the target picture according to the pose offset;

And a fusion module 1005, configured to fuse the virtual object to the real image with the adjusted pose, to obtain an augmented reality image for guiding the real target object.

In some embodiments, the pose adjustment module 1004 is further configured to determine an amount of rotation required to adjust the pose of the digital twin reference object in the target picture to be consistent with the pose of the real reference object in the real picture; adjusting the posture of the digital twin reference object in the target picture according to the rotation quantity; determining a translation amount required for adjusting the position of the digital twin reference object in the target picture after the posture adjustment to be consistent with the position of the real reference object in the real picture; and taking the rotation amount and the translation amount as the pose offset amount.

In some embodiments, the pose adjustment module 1004 is further configured to determine a first linear function fitting the digital twin reference in the target picture, in case the digital twin reference belongs to a bar; determining a second linear function fitting the real reference object in the real picture in the case that the real reference object belongs to a bar; the rotation amount is determined based on a relative magnitude between an arctangent of a slope of the first linear function and an arctangent of a slope of the second linear function.

In some embodiments, the real reference is a real guide line located on both sides of the real target object, and the digital twinning reference is a digital twinning guide line corresponding to the real guide line; the pose adjustment module 1004 is further configured to take an arctangent value of a slope of a first linear function fitting a first side digital twin guide line as a first arctangent value and an arctangent value of a slope of a first linear function fitting a second side digital twin guide line as a second arctangent value; taking the arctangent value of the slope of the second linear function fitting the first side reality guide line as a third arctangent value and taking the arctangent value of the slope of the second linear function fitting the second side reality guide line as a fourth arctangent value; and determining the rotation amount by integrating the relative sizes of the first arctangent value and the third arctangent value and the relative sizes of the second arctangent value and the fourth arctangent value.

In some embodiments, the pose adjustment module 1004 is further configured to obtain the translation amount based on a relative magnitude between a constant of the first linear function and a constant of the second linear function.

In some embodiments, the real reference is a real guide line located on both sides of the real target object, and the digital twinning reference is a digital twinning guide line corresponding to the real guide line; the pose adjustment module 1004 is further configured to take a constant of a first linear function fitting the first side digital twin guide line as a first constant and a constant of a first linear function fitting the second side digital twin guide line as a second constant; taking a constant of a second linear function fitting the first side reality guide line as a third constant and taking a constant of a second linear function fitting the second side reality guide line as a fourth constant; and determining the translation amount by combining the relative magnitude between the first constant and the third constant and the relative magnitude between the second constant and the fourth constant.

In some embodiments, the picture obtaining module 1002 of the digital twin scene is further configured to determine a digital twin object corresponding to the real object in the digital twin scene; and generating the target picture according to the virtual object and the digital twin reference object which are observed by the digital twin target object in the reporting pose.

In some embodiments, the picture obtaining module 1002 of the digital twin scene is further configured to obtain camera parameters of a real camera provided to the real target object; and under the condition that the digital twin target object is in the digital twin scene in the reporting pose, projecting the virtual object and the digital twin reference object of the digital twin scene onto a picture according to the camera parameters to obtain the target picture.

In some embodiments, the pose obtaining module 1001 is configured to obtain a position and a traveling direction of the real target object in the real scene, where the position and the traveling direction are collected by a satellite positioning device provided on the real target object; and taking the position and the advancing direction of the real target object in the real scene as the reporting pose.

In the generating device of the augmented reality picture, because the reporting pose of the target object travelling in the real scene is different from the real pose, the target picture obtained by reporting the pose in the digital twin scene is different from the real picture obtained by the real pose in the real scene, the pose of a virtual object in the target picture needs to be adjusted firstly, and then the virtual object is fused into the real picture in the adjusted pose; because the digital twin reference object and the real reference object are in one-to-one correspondence, the pose of the virtual object in the target image is adjusted based on the pose offset required by the superposition of the pose of the digital twin reference object in the target image and the pose of the real reference object in the real image, so that the virtual object can be fused into the real image in a proper pose, and the sense of realism of the fused augmented reality image is improved.

For specific limitations regarding the generation device of the augmented reality picture, reference may be made to the above limitation of the generation method of the augmented reality picture, and no further description is given here. The above-described respective modules in the generation apparatus of the augmented reality picture may be implemented in whole or in part by software, hardware, and a combination thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.

In some embodiments, a computer device is provided, which may be a server, the internal structure of which may be as shown in fig. 11. The computer device includes a processor, a memory, and a network interface connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database of the computer device is used for storing the generation data of the augmented reality picture. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program, when executed by a processor, implements a method of generating an augmented reality picture.

It will be appreciated by those skilled in the art that the structure shown in FIG. 11 is merely a block diagram of some of the structures associated with the present inventive arrangements and is not limiting of the computer device to which the present inventive arrangements may be applied, and that a particular computer device may include more or fewer components than shown, or may combine some of the components, or have a different arrangement of components.

In some embodiments, a computer device is provided, comprising a memory storing a computer program and a processor implementing the steps of the method embodiments described above when the processor executes the computer program.

In some embodiments, a computer readable storage medium is provided, on which a computer program is stored which, when executed by a processor, implements the steps of the various method embodiments described above.

In some embodiments, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the steps of the various method embodiments described above.

Those skilled in the art will appreciate that implementing all or part of the above-described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed may comprise the steps of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in embodiments provided herein may include at least one of non-volatile and volatile memory. The nonvolatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical Memory, or the like. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory. By way of illustration, and not limitation, RAM can be in various forms such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), etc.

The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.

The foregoing examples illustrate only a few embodiments of the application, which are described in detail and are not to be construed as limiting the scope of the application. It should be noted that it will be apparent to those skilled in the art that several variations and modifications can be made without departing from the spirit of the application, which are all within the scope of the application. Accordingly, the scope of protection of the present application is to be determined by the appended claims.

Claims

1. A method for generating an augmented reality picture, the method comprising:

Acquiring a reporting pose of a real target object travelling in a real scene;

In a digital twin scene corresponding to the real scene, determining a digital twin target object corresponding to the real target object in the digital twin scene, and generating a target picture according to a virtual object and a digital twin reference object observed by the digital twin target object in the reporting pose; the digital twin scene comprises a virtual object which does not belong to the real scene and a digital twin reference object corresponding to a real reference object which belongs to the real scene, and the target picture comprises the virtual object and the digital twin reference object;

2. The method of claim 1, wherein the determining a pose offset required to adjust the pose of the digital twin reference in the target picture to coincide with the pose of the real reference in the real picture comprises:

Determining an amount of rotation required to adjust the pose of the digital twin reference object in the target picture to be consistent with the pose of the real reference object in the real picture;

Adjusting the posture of the digital twin reference object in the target picture according to the rotation quantity;

Determining a translation amount required for adjusting the position of the digital twin reference object in the target picture after the posture adjustment to be consistent with the position of the real reference object in the real picture;

And taking the rotation amount and the translation amount as the pose offset amount.

3. The method of claim 2, wherein the determining an amount of rotation required to adjust the pose of the digital twin reference in the target picture to be consistent with the pose of the real reference in the real picture comprises:

determining a first linear function fitting the digital twin reference in the target picture, in case the digital twin reference belongs to a bar;

Determining a second linear function fitting the real reference object in the real picture in the case that the real reference object belongs to a bar;

the rotation amount is determined based on a relative magnitude between an arctangent of a slope of the first linear function and an arctangent of a slope of the second linear function.

4. A method according to claim 3, wherein the real reference is a real guide line located on both sides of the real target object, and the digital twin reference is a digital twin guide line corresponding to the real guide line;

Determining the rotation amount based on a relative magnitude between an arctangent of a slope of the first linear function and an arctangent of a slope of the second linear function, comprising:

taking the arctangent value of the slope of the first linear function fitting the first side digital twin guide line as a first arctangent value and taking the arctangent value of the slope of the first linear function fitting the second side digital twin guide line as a second arctangent value;

Taking the arctangent value of the slope of the second linear function fitting the first side reality guide line as a third arctangent value and taking the arctangent value of the slope of the second linear function fitting the second side reality guide line as a fourth arctangent value;

and determining the rotation amount by integrating the relative sizes of the first arctangent value and the third arctangent value and the relative sizes of the second arctangent value and the fourth arctangent value.

5. A method according to claim 3, wherein said determining an amount of translation required to adjust the position of the posed digital twin in the target picture to coincide with the position of the real reference in the real picture comprises:

The translation amount is obtained based on a relative magnitude between a constant of the first linear function and a constant of the second linear function.

6. The method of claim 5, wherein the real reference is a real guide line located on both sides of the real target object, and the digital twin reference is a digital twin guide line corresponding to the real guide line;

-deriving said translation based on a relative magnitude between a constant of said first linear function and a constant of said second linear function, comprising:

Taking the constant of the first linear function fitting the first side digital twin guide line as a first constant and taking the constant of the first linear function fitting the second side digital twin guide line as a second constant;

Taking a constant of a second linear function fitting the first side reality guide line as a third constant and taking a constant of a second linear function fitting the second side reality guide line as a fourth constant;

And determining the translation amount by combining the relative magnitude between the first constant and the third constant and the relative magnitude between the second constant and the fourth constant.

7. The method of claim 1, wherein the generating the target picture in accordance with the virtual object and the digital twin reference observed by the digital twin target object in the reporting pose comprises:

acquiring camera parameters of a real camera arranged on the real target object;

and under the condition that the digital twin target object is in the digital twin scene in the reporting pose, projecting the virtual object and the digital twin reference object of the digital twin scene onto a picture according to the camera parameters to obtain the target picture.

8. The method of claim 1, wherein the acquiring the reported pose of the real target object traveling in the real scene comprises:

Acquiring the position and the advancing direction of the real target object in the real scene, which are acquired by the satellite positioning equipment arranged on the real target object;

and taking the position and the advancing direction of the real target object in the real scene as the reporting pose.

9. The method of claim 1, wherein the virtual object is a virtual traffic flow, the method further comprising:

the digital twin scene is constructed based on a high-precision map constructed for the real scene;

determining the position of a digital twin road in the digital twin scene based on the high-precision map;

and adding the virtual traffic flow into the digital twin scene according to the position of the digital twin road in the digital twin scene.

10. The method according to any one of claims 1 to 9, wherein the real target object is a real target vehicle, the method further comprising:

Obtaining an augmented reality picture comprising a virtual traffic stream;

and testing an automatic driving algorithm running on the real target vehicle based on the augmented reality picture comprising the virtual traffic flow.

11. An apparatus for generating an augmented reality picture, the apparatus comprising:

the image acquisition module of the digital twin scene is used for determining a digital twin target object corresponding to the real target object in the digital twin scene corresponding to the real scene, and generating a target image according to a virtual object and a digital twin reference observed by the digital twin target object in the reporting pose; the digital twin scene comprises a virtual object which does not belong to the real scene and a digital twin reference object corresponding to a real reference object which belongs to the real scene, and the target picture comprises the virtual object and the digital twin reference object;

12. The apparatus of claim 11, wherein the pose adjustment module is configured to determine an amount of rotation required to adjust the pose of the digital twin reference object in the target picture to be consistent with the pose of the real reference object in the real picture; adjusting the posture of the digital twin reference object in the target picture according to the rotation quantity; determining a translation amount required for adjusting the position of the digital twin reference object in the target picture after the posture adjustment to be consistent with the position of the real reference object in the real picture; and taking the rotation amount and the translation amount as the pose offset amount.

13. The apparatus of claim 12, wherein the pose adjustment module is further configured to determine a first linear function fitting the digital twin reference in the target picture if the digital twin reference belongs to a bar; determining a second linear function fitting the real reference object in the real picture in the case that the real reference object belongs to a bar; the rotation amount is determined based on a relative magnitude between an arctangent of a slope of the first linear function and an arctangent of a slope of the second linear function.

14. The apparatus of claim 13, wherein the real reference is a real guide line located on both sides of the real target object, and the digital twin reference is a digital twin guide line corresponding to the real guide line;

The pose adjustment module is further used for taking an arctangent value of a slope of a first linear function fitting the first side digital twin guide line as a first arctangent value and taking an arctangent value of a slope of a first linear function fitting the second side digital twin guide line as a second arctangent value; taking the arctangent value of the slope of the second linear function fitting the first side reality guide line as a third arctangent value and taking the arctangent value of the slope of the second linear function fitting the second side reality guide line as a fourth arctangent value; and determining the rotation amount by integrating the relative sizes of the first arctangent value and the third arctangent value and the relative sizes of the second arctangent value and the fourth arctangent value.

15. The apparatus of claim 13, wherein the pose adjustment module is further configured to obtain the translation based on a relative magnitude between a constant of the first linear function and a constant of the second linear function.

16. The apparatus of claim 15, wherein the real reference is a real guide line located on both sides of the real target object, and the digital twin reference is a digital twin guide line corresponding to the real guide line;

the pose adjusting module is further used for taking a constant of a first linear function fitting the first side digital twin guide line as a first constant and taking a constant of a first linear function fitting the second side digital twin guide line as a second constant; taking a constant of a second linear function fitting the first side reality guide line as a third constant and taking a constant of a second linear function fitting the second side reality guide line as a fourth constant; and determining the translation amount by combining the relative magnitude between the first constant and the third constant and the relative magnitude between the second constant and the fourth constant.

17. The apparatus of claim 11, wherein the picture acquisition module of the digital twinning scene is further configured to acquire camera parameters of a real camera provided to the real target object; and under the condition that the digital twin target object is in the digital twin scene in the reporting pose, projecting the virtual object and the digital twin reference object of the digital twin scene onto a picture according to the camera parameters to obtain the target picture.

18. The apparatus according to claim 11, wherein the pose acquisition module is configured to acquire a position and a traveling direction of the real target object in the real scene acquired by a satellite positioning device provided on the real target object; and taking the position and the advancing direction of the real target object in the real scene as the reporting pose.

19. The apparatus of claim 11, wherein the virtual object is a virtual traffic stream, the apparatus further comprising a digital twin scene processing module configured to construct the digital twin scene based on a high-precision map constructed for the real scene; determining the position of a digital twin road in the digital twin scene based on the high-precision map; and adding the virtual traffic flow into the digital twin scene according to the position of the digital twin road in the digital twin scene.

20. The apparatus according to any one of claims 11 to 19, wherein the real target object is a real target vehicle, the apparatus further comprising an autopilot algorithm testing module for obtaining an augmented reality picture comprising a virtual traffic flow; and testing an automatic driving algorithm running on the real target vehicle based on the augmented reality picture comprising the virtual traffic flow.

21. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the method of any one of claims 1 to 10 when executing the computer program.

22. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the method of any one of claims 1 to 10.

23. A computer program product comprising a computer program, characterized in that the computer program, when executed by a processor, implements the method of any one of claims 1 to 10.