CN113132708B

CN113132708B - Method and apparatus for acquiring three-dimensional scene image using fisheye camera, device and medium

Info

Publication number: CN113132708B
Application number: CN202110438633.0A
Authority: CN
Inventors: 陆泽辉; 汪少俊; 王志华
Original assignee: Seashell Housing Beijing Technology Co Ltd
Current assignee: Seashell Housing Beijing Technology Co Ltd
Priority date: 2021-04-22
Filing date: 2021-04-22
Publication date: 2022-02-22
Anticipated expiration: 2041-04-22
Also published as: CN113132708A

Abstract

The present disclosure provides a method and apparatus, a device and a medium for acquiring a three-dimensional scene image using a fisheye camera, the method comprising: acquiring a video image of a target scene by using a fisheye camera; fitting the video image into the three-dimensional space model based on the corresponding relation between the three-dimensional coordinates of each vertex in the three-dimensional space model of the target scene and the corresponding two-dimensional coordinates to obtain a three-dimensional scene effect image of the target scene, wherein the three-dimensional coordinates of the vertex are the space position coordinates of the vertex in a three-dimensional coordinate system corresponding to the three-dimensional space model, and the two-dimensional coordinates are the coordinates of projection points of a mapping point corresponding to the vertex on the cross section of a spherical coordinate system corresponding to the fisheye camera; the spherical coordinate system is obtained by taking the fisheye camera as a spherical center and constructing based on internal reference and external reference of the fisheye camera; the cross section is a plane which passes through the central point of the spherical coordinate system and is vertical to the shooting positive direction of the fisheye camera. The method and the device can reduce the image acquisition cost, improve the imaging effect of the 3D scene, and meet the real-time requirement of video streaming.

Description

Method and apparatus for acquiring three-dimensional scene image using fisheye camera, device and medium

Technical Field

The present disclosure relates to video image processing technologies, and in particular, to a method and apparatus, a device, and a medium for acquiring a three-dimensional scene video image using a fisheye camera.

Background

At present, imaging modes in applications such as Augmented Reality (AR) technology and the like collect model data and material data of a whole scene in a data modeling mode, and restore the whole 3D scene by using the data model and the material data through a certain three-dimensional (3D) technology.

In the course of implementing the present disclosure, the inventors found through research that the imaging mode in the above AR application and the like has at least the following problems: model data and material data of all objects in the whole scene space need to be collected in advance for modeling, and real imaging of the whole 3D scene cannot be restored in real time by using video stream; at present, common cameras are mostly used for shooting images to acquire the images, due to the limitation of shooting angles of the common cameras, the images shot by the multiple cameras are usually fused to achieve the imaging of the whole space, the image acquisition cost required by deploying the multiple cameras is high, and if an object or a human moving in the space exists, the splitting phenomenon can occur at the fusion gap of the images shot by the multiple cameras, so that the imaging effect of a 3D scene is influenced.

Disclosure of Invention

One technical problem to be solved by the embodiments of the present disclosure is: the method, the device, the equipment and the medium for acquiring the three-dimensional scene image by using the fisheye camera are provided, so that the image acquisition cost is reduced, the imaging effect of a 3D scene is improved, and the real-time requirement of a video stream can be met.

The method for acquiring the three-dimensional scene image by using the fisheye camera provided by the embodiment of the disclosure comprises the following steps:

acquiring a video image of a target scene by using a fisheye camera;

fitting the video image into the three-dimensional space model based on the corresponding relation between the three-dimensional coordinates of each vertex in the three-dimensional space model of the target scene and the corresponding two-dimensional coordinates to obtain a three-dimensional scene effect graph of the target scene;

the three-dimensional coordinate of the vertex is a space position coordinate of the vertex in a three-dimensional coordinate system corresponding to the three-dimensional space model, and the two-dimensional coordinate is a coordinate of a projection point of a mapping point corresponding to the vertex on a cross section of a spherical coordinate system corresponding to the fisheye camera; the spherical coordinate system is constructed by taking the fisheye camera as a spherical center and based on internal parameters and external parameters of the fisheye camera; the cross section is through the central point of spherical coordinate system, and with the perpendicular plane in shooting positive direction of fisheye camera.

In another embodiment of the foregoing method based on the present disclosure, the establishing of the corresponding relationship includes:

respectively mapping each vertex in the three-dimensional space model from the three-dimensional coordinate system to the spherical coordinate system to obtain spherical coordinates of each mapping point corresponding to each vertex in the spherical coordinate system;

projecting the mapping points from the spherical coordinate system to a cross section of the spherical coordinate system to obtain two-dimensional coordinates of projection points of the mapping points on the cross section;

and establishing a corresponding relation between the three-dimensional coordinates of each vertex and the two-dimensional coordinates of the corresponding projection point.

In another embodiment of the foregoing method based on the present disclosure, before establishing a correspondence between the three-dimensional coordinates and the corresponding two-dimensional coordinates of each vertex, the method further includes:

calibrating the fisheye camera to obtain internal parameters and external parameters of the fisheye camera; wherein the internal parameters include at least one of an imaging center, a radius of the fisheye camera, and a shooting field angle, and the external parameters include at least one of a position and an orientation of the fisheye camera in the three-dimensional coordinate system;

and constructing a spherical coordinate system with the fisheye camera as a sphere center by using the internal reference and the external reference of the fisheye camera.

In another embodiment of the above method according to the present disclosure, the spherical coordinate system is specifically a unit spherical coordinate system.

In another embodiment of the above method based on the present disclosure, the mapping the vertices from the three-dimensional coordinate system to the spherical coordinate system to obtain spherical coordinates of mapping points corresponding to the vertices in the spherical coordinate system includes:

moving the vertexes from the three-dimensional coordinate system to a three-dimensional Cartesian coordinate system with a central point of the spherical coordinate system as an origin according to the position of the fisheye camera in the three-dimensional coordinate system;

and mapping each vertex from the three-dimensional Cartesian coordinate system to the spherical coordinate system according to the orientation of the fisheye camera in the three-dimensional coordinate system to obtain the spherical coordinates of each mapping point corresponding to each vertex in the spherical coordinate system.

In another embodiment of the foregoing method based on the present disclosure, after the establishing a correspondence between the three-dimensional coordinates of each vertex and the two-dimensional coordinates of the corresponding projection point, the method further includes:

taking a frame of video image in the video stream of the target scene acquired by the fisheye camera as a reference image, and fitting the reference image into the three-dimensional space model based on the corresponding relation to obtain a three-dimensional fusion effect picture of the target scene;

responding to points with inaccurate fitting in the three-dimensional fusion effect graph, collecting at least one point with inaccurate fitting, correcting and calculating the two-dimensional coordinates of all vertexes in the corresponding relation by utilizing a least square method based on the current two-dimensional coordinates of each point in the at least one point with inaccurate fitting and the reference two-dimensional coordinates corresponding to the point with accurate fitting, obtaining the correction relation information between the two-dimensional coordinates of all vertexes in the corresponding relation and the reference two-dimensional coordinates corresponding to the point with accurate fitting, and storing the correction relation information.

In another embodiment of the above method based on the present disclosure, the storing the correction relation information includes:

respectively storing the two-dimensional coordinates of each vertex in all the vertexes and the corresponding reference two-dimensional coordinates when the lamination is accurate in an RGBA format to form a corrected texture picture; the correction relationship information is specifically the correction texture picture.

In another embodiment of the above method based on the present disclosure, before the storing the correction relation information, the method further includes:

responding to the existence of noise points in the corrected texture picture, and denoising the noise points by using pixel points adjacent to the noise points;

the storing the correction relation information includes:

and storing the corrected texture picture obtained after denoising.

In another embodiment of the above method based on the present disclosure, the attaching the video image to the three-dimensional space model based on a correspondence between three-dimensional coordinates of each vertex in the three-dimensional space model of the target scene and corresponding two-dimensional coordinates to obtain a three-dimensional scene effect map of the target scene includes:

acquiring two-dimensional coordinates corresponding to the three-dimensional coordinates of each vertex based on the corresponding relation between the three-dimensional coordinates of each vertex and the corresponding two-dimensional coordinates;

correcting the two-dimensional coordinates corresponding to the three-dimensional coordinates of the vertexes based on the reference two-dimensional coordinates corresponding to the correction relation information by using the two-dimensional coordinates corresponding to the three-dimensional coordinates of the vertexes to obtain the reference two-dimensional coordinates corresponding to the three-dimensional coordinates of the vertexes;

and fitting the video image into the three-dimensional space model based on the reference two-dimensional coordinates corresponding to the three-dimensional coordinates of each vertex to obtain a three-dimensional scene effect graph of the target scene.

The device for acquiring three-dimensional scene images by using the fisheye camera provided by the embodiment of the disclosure comprises:

the fisheye camera is used for acquiring a video image of a target scene;

the fusion processing module is used for fitting the video images into the three-dimensional space model based on the corresponding relation between the three-dimensional coordinates of each vertex in the three-dimensional space model of the target scene and the corresponding two-dimensional coordinates to obtain a three-dimensional scene effect graph of the target scene;

In another embodiment of the above apparatus according to the present disclosure, further comprising:

the first processing module is used for mapping each vertex in the three-dimensional space model from the three-dimensional coordinate system to the spherical coordinate system respectively to obtain spherical coordinates of each mapping point corresponding to each vertex in the spherical coordinate system;

the second processing module is used for projecting the mapping points from the spherical coordinate system to a cross section of the spherical coordinate system to obtain two-dimensional coordinates of projection points of the mapping points on the cross section;

and the establishing module is used for establishing the corresponding relation between the three-dimensional coordinates of each vertex and the two-dimensional coordinates of the corresponding projection point.

the calibration module is used for calibrating the fisheye camera to obtain internal parameters and external parameters of the fisheye camera; wherein the internal parameters include at least one of an imaging center, a radius of the fisheye camera, and a shooting field angle, and the external parameters include at least one of a position and an orientation of the fisheye camera in the three-dimensional coordinate system;

and the building module is used for building a spherical coordinate system with the fisheye camera as the sphere center by using the internal reference and the external reference of the fisheye camera.

In another embodiment of the above apparatus according to the present disclosure, the spherical coordinate system is specifically a unit spherical coordinate system.

In another embodiment of the above apparatus according to the present disclosure, the first processing module is specifically configured to, for each vertex in the three-dimensional space model, move the vertex from the three-dimensional coordinate system into a three-dimensional cartesian coordinate system with a central point of the spherical coordinate system as an origin according to a position of the fisheye camera in the three-dimensional coordinate system; and mapping each vertex from the three-dimensional Cartesian coordinate system to the spherical coordinate system according to the orientation of the fisheye camera in the three-dimensional coordinate system to obtain the spherical coordinates of each mapping point corresponding to each vertex in the spherical coordinate system.

In another embodiment of the above apparatus according to the present disclosure, further comprising: a rectification module;

the fusion processing module is further configured to use a frame of video image in the video stream of the target scene acquired by the fisheye camera as a reference image, and fit the reference image into the three-dimensional space model based on the correspondence to obtain a three-dimensional fusion effect map of the target scene;

the correction module is used for responding to points which are not accurately attached in the three-dimensional fusion effect graph, acquiring at least one point which is not accurately attached, and performing correction calculation on the two-dimensional coordinates of all vertexes in the corresponding relation by using a least square method based on the current two-dimensional coordinates of each point in the at least one point which is not accurately attached and the reference two-dimensional coordinates corresponding to the point when the attachment is accurate to obtain correction relation information between the two-dimensional coordinates of all vertexes in the corresponding relation and the corresponding reference two-dimensional coordinates when the attachment is accurate;

and the storage module is used for storing the correction relation information.

In another embodiment of the above apparatus based on the present disclosure, the storage module is specifically configured to store the two-dimensional coordinates of each vertex in all the vertices and the corresponding reference two-dimensional coordinates when the fitting is accurate in an RGBA format, so as to form a texture correction picture; the correction relationship information is specifically the correction texture picture.

the de-noising module is used for responding to the existence of noise points in the corrected texture picture and de-noising the noise points by using pixel points adjacent to the noise points;

the storage module is specifically used for storing the corrected texture picture obtained after denoising.

In another embodiment of the above apparatus based on the present disclosure, the fusion processing module is specifically configured to:

In another aspect of the disclosed embodiments, an electronic device is provided, including:

a memory for storing a computer program;

a processor for executing the computer program stored in the memory, and the computer program, when executed, implements the method of any of the above embodiments of the present disclosure.

In yet another aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and the computer program, when executed by a processor, implements the method according to any of the above embodiments of the present disclosure.

Based on the method, the device, the equipment and the medium for acquiring the three-dimensional scene image by using the fisheye camera, which are provided by the embodiments of the present disclosure, a spherical coordinate system is constructed in advance by using the fisheye camera as a spherical center and based on the internal reference and the external reference of the fisheye camera, a video image of a target scene is acquired by using the fisheye camera, then based on the corresponding relationship between the three-dimensional coordinates of each vertex in the three-dimensional space model of the target scene and the corresponding two-dimensional coordinates, the video image acquired by the fisheye camera is attached to the three-dimensional space model to obtain a three-dimensional scene effect map of the target scene, wherein the three-dimensional coordinates of the vertex are space position coordinates of the vertex in the three-dimensional coordinate system corresponding to the three-dimensional space model, the two-dimensional coordinates are coordinates of a projection point on a cross section of the spherical coordinate system corresponding to the fisheye camera and corresponding to the vertex, and the cross section is a projection point passing through the center point of the spherical coordinate system, And a plane perpendicular to the shooting positive direction of the fisheye camera. Based on the embodiment of the disclosure, the video images of the target scene collected by the fisheye camera can be fitted into the three-dimensional space model, and the fusion of the video images collected by the fisheye camera and the three-dimensional space model is realized, so that the three-dimensional scene effect image of the target scene can be obtained by the fisheye camera, due to the characteristics of large imaging angle and large imaging range of the fisheye camera, the imaging of one fisheye camera can completely cover the whole target scene without shielding in the view field range, compared with the prior art, a plurality of cameras are not required to be deployed, the image collection cost can be reduced, the split phenomenon at the fusion gap of the images shot by the plurality of cameras can be avoided, the imaging effect of the 3D scene is improved, in addition, the model data and the material data of all objects in the whole scene space are not required to be collected in advance for modeling, the operation of the embodiment is executed by sequentially aiming at each video image in the video stream collected by the fisheye camera, therefore, the fusion of the video stream collected by the fisheye camera and the three-dimensional space model can be realized, the dynamic and static states of all objects in the real scene are really restored through the video stream of the fisheye camera, the real-time requirement of the video stream can be met, and the real-time problem in AR imaging is solved. In addition, real imaging of the whole 3D space model can be restored in real time by utilizing video streams based on the fisheye camera, virtual data information (such as a standard decoration wiring diagram) can be added into the video streams and combined with real data information in imaging of the fisheye camera, and the effect of virtual-real combination of the whole AR scene can be achieved.

The technical solution of the present disclosure is further described in detail by the accompanying drawings and examples.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the disclosure and together with the description, serve to explain the principles of the disclosure.

The present disclosure may be more clearly understood from the following detailed description, taken with reference to the accompanying drawings, in which:

fig. 1 is a flowchart illustrating an embodiment of a method for acquiring an image of a three-dimensional scene using a fisheye camera according to the present disclosure.

Fig. 2 is a flowchart of an embodiment of establishing a correspondence between three-dimensional coordinates and corresponding two-dimensional coordinates of each vertex in the embodiment of the present disclosure.

Fig. 3 is a schematic diagram of a cross section of a spherical coordinate system in an embodiment of the disclosure.

Fig. 4 is a flowchart illustrating another embodiment of the method for acquiring an image of a three-dimensional scene using a fisheye camera according to the disclosure.

Fig. 5 is a flowchart illustrating a method for acquiring an image of a three-dimensional scene using a fisheye camera according to another embodiment of the disclosure.

Fig. 6 is a schematic diagram of a corrected texture picture according to an embodiment of the disclosure.

Fig. 7 is a flowchart illustrating a method for acquiring an image of a three-dimensional scene using a fisheye camera according to still another embodiment of the disclosure.

Fig. 8(a) is a schematic diagram of an image of a target scene captured by a fisheye camera in an embodiment of the present disclosure.

Fig. 8(b) is a schematic diagram of a three-dimensional scene effect graph obtained based on the embodiment of the present disclosure.

Fig. 9 is a schematic structural diagram of an embodiment of an apparatus for acquiring a three-dimensional scene image by using a fisheye camera according to the disclosure.

Fig. 10 is a schematic structural diagram of another embodiment of the apparatus for acquiring an image of a three-dimensional scene by using a fisheye camera according to the disclosure.

Fig. 11 is a schematic structural diagram of an embodiment of an electronic device according to the present disclosure.

Detailed Description

Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that: the relative arrangement of the components and steps, the numerical expressions, and numerical values set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.

It will be understood by those of skill in the art that the terms "first," "second," and the like in the embodiments of the present disclosure are used merely to distinguish one element from another, and are not intended to imply any particular technical meaning, nor is the necessary logical order between them.

It is also understood that in embodiments of the present disclosure, "a plurality" may refer to two or more and "at least one" may refer to one, two or more.

It is also to be understood that any reference to any component, data, or structure in the embodiments of the disclosure, may be generally understood as one or more, unless explicitly defined otherwise or stated otherwise.

In addition, the term "and/or" in the present disclosure is only one kind of association relationship describing an associated object, and means that three kinds of relationships may exist, for example, a and/or B may mean: a exists alone, A and B exist simultaneously, and B exists alone. In addition, the character "/" in the present disclosure generally indicates that the former and latter associated objects are in an "or" relationship.

It should also be understood that the description of the various embodiments of the present disclosure emphasizes the differences between the various embodiments, and the same or similar parts may be referred to each other, so that the descriptions thereof are omitted for brevity.

Meanwhile, it should be understood that the sizes of the respective portions shown in the drawings are not drawn in an actual proportional relationship for the convenience of description.

The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the disclosure, its application, or uses.

Techniques, methods, and apparatus known to those of ordinary skill in the relevant art may not be discussed in detail but are intended to be part of the specification where appropriate.

It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, further discussion thereof is not required in subsequent figures.

The disclosed embodiments may be applied to electronic devices such as terminal devices, computer systems, servers, etc., which are operational with numerous other general purpose or special purpose computing system environments or configurations. Examples of well known terminal devices, computing systems, environments, and/or configurations that may be suitable for use with electronic devices, such as terminal devices, computer systems, servers, and the like, include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, hand-held or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, networked personal computers, minicomputer systems, mainframe computer systems, distributed cloud computing environments that include any of the above, and the like.

Electronic devices such as terminal devices, computer systems, servers, etc. may be described in the general context of computer system-executable instructions, such as program modules, being executed by a computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types. The computer system/server may be practiced in distributed cloud computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computer system storage media including memory storage devices.

Fig. 1 is a flowchart illustrating an embodiment of a method for acquiring an image of a three-dimensional scene using a fisheye camera according to the present disclosure. As shown in fig. 1, the method for acquiring a three-dimensional scene image by using a fisheye camera in the embodiment includes:

and 102, acquiring a video image of the target scene by using the fisheye camera.

In the embodiment of the disclosure, the target scene is a scene needing video monitoring. The video images are images in a video stream shot aiming at a target scene.

For example, taking the application in a decoration site as an example, the fisheye camera is installed in each room of the decoration site at a proper position and angle, the proper position is preferably the center of the room as much as possible, the fisheye camera is not close to any side wall, the monitoring range covers the room as complete as possible, and the proper angle is preferably the fisheye camera has the lens surface (i.e., the positive direction and the front direction) facing the ground as positive as possible. The fisheye camera is installed based on the proper position and angle, so that the fisheye camera can sense the whole scene most accurately, and depth loss is caused.

And 104, fitting the video images into a three-dimensional (3D) space model of the target scene based on the corresponding relation between the three-dimensional coordinates of each vertex in the three-dimensional space model and the corresponding two-dimensional coordinates to obtain a three-dimensional scene effect graph of the target scene.

The vertex is the vertex of a projection patch (i.e. a mesh intersection) when the three-dimensional space model is visually rendered.

The three-dimensional coordinates of the vertex are space position coordinates of the vertex in a three-dimensional coordinate system corresponding to the three-dimensional space model, and the two-dimensional coordinates are coordinates of projection points of a mapping point corresponding to the vertex on a cross section of a spherical coordinate system corresponding to the fisheye camera; the spherical coordinate system is obtained by taking the fisheye camera as a spherical center and constructing based on internal parameters and external parameters of the fisheye camera; the cross section is a plane which passes through the central point of the spherical coordinate system and is vertical to the shooting positive direction of the fisheye camera.

Optionally, in some embodiments, the three-dimensional space model of the target scene may be obtained by performing three-dimensional reconstruction on a real scene (for example, a house type in an entire decoration project), and generating a usable three-dimensional space model as three-dimensional stereo space information by performing computer simulation on the real scene. Based on the embodiment of the disclosure, the video image of the target scene acquired by the fisheye camera can be fitted into the three-dimensional space model, and the fusion of the video image acquired by the fisheye camera and the three-dimensional space model is realized, so that the three-dimensional scene effect image of the target scene can be obtained by the fisheye camera, because of the characteristics of large imaging angle and large imaging range of the fisheye camera, the imaging of one fisheye camera can completely cover the whole target scene without shielding in the view field range, compared with the prior art, a plurality of cameras are not required to be deployed, the image acquisition cost can be reduced, the split phenomenon at the fusion gap of the images shot by the plurality of cameras can be avoided, the imaging effect of the 3D scene is improved, in addition, the model data and the material data of all objects in the whole scene space are not required to be acquired in advance for modeling, and the operation 102 and 104 are executed for each video image in the video stream acquired by the fisheye camera in sequence, therefore, the fusion of the video stream collected by the fisheye camera and the three-dimensional space model can be realized, the dynamic and static states of all objects in the real scene are really restored through the video stream of the fisheye camera, the real-time requirement of the video stream can be met, and the real-time problem in AR imaging is solved. In addition, real imaging of the whole 3D space model can be restored in real time by utilizing video streams based on the fisheye camera, virtual data information (such as a standard decoration wiring diagram) can be added into the video streams and combined with real data information in imaging of the fisheye camera, and the effect of virtual-real combination of the whole AR scene can be achieved.

Optionally, in some embodiments, the correspondence between the three-dimensional coordinates of each vertex in the three-dimensional space model and the corresponding two-dimensional coordinates may be established in advance for subsequent direct retrieval.

Fig. 2 is a flowchart of an embodiment of establishing a correspondence between three-dimensional coordinates and corresponding two-dimensional coordinates of each vertex in a three-dimensional space model according to the embodiment of the present disclosure. As shown in fig. 2, in this embodiment, the correspondence between the three-dimensional coordinates and the corresponding two-dimensional coordinates of each vertex in the three-dimensional space model can be established as follows:

202, for each vertex in the three-dimensional space model, mapping each vertex from the three-dimensional coordinate system to the spherical coordinate system to obtain the spherical coordinates of each mapping point corresponding to each vertex in the spherical coordinate system.

Optionally, in some embodiments, the spherical coordinates of each vertex in the spherical coordinate system corresponding to the fisheye camera may be obtained based on external parameters of the fisheye camera, that is, the spherical coordinates of each mapping point corresponding to each vertex in the spherical coordinate system may be obtained.

And 204, projecting each mapping point from the spherical coordinate system to the cross section of the spherical coordinate system to obtain the two-dimensional coordinates of the projection point of each mapping point on the cross section.

The two-dimensional coordinates (U, V) of the projection point are sampling coordinates (i.e., texture coordinates) corresponding to the vertex position in the 3D space model. And sequentially carrying out the calculation on all vertexes in the three-dimensional coordinate system, so that the whole fisheye camera can be imaged and attached to the whole 3D space model.

Fig. 3 is a schematic diagram of a cross section of a spherical coordinate system according to an embodiment of the disclosure.

In the embodiment of the present disclosure, the cross section of the spherical coordinate system, i.e., the imaging plane of the fisheye camera, is a plane passing through the center point (i.e., the spherical center) of the spherical coordinate system and perpendicular to the positive direction (i.e., the front direction) of the fisheye camera.

206, establishing a corresponding relation between the three-dimensional coordinates of each vertex and the two-dimensional coordinates of the corresponding projection point.

Fig. 4 is a flowchart illustrating another embodiment of the method for acquiring an image of a three-dimensional scene using a fisheye camera according to the disclosure. As shown in fig. 4, before establishing a correspondence relationship between the three-dimensional coordinates and the corresponding two-dimensional coordinates of each vertex, that is, before 202, the method may further include:

302, calibrating the fisheye camera to obtain the internal reference and the external reference of the fisheye camera.

The internal parameters of the fisheye camera may include at least one of an imaging center, a fisheye camera radius (also referred to as a radius of a circular surface), and a shooting angle, for example. The external parameters of the fisheye camera may for example comprise at least one of a position and an orientation of the fisheye camera in the three-dimensional coordinate system. The position of the fisheye camera in the three-dimensional coordinate system O-XYZ can be represented as three-dimensional spatial position coordinates, and the orientation of the fisheye camera in the three-dimensional coordinate system, i.e., the rotational euler angle of the fisheye camera in the three-dimensional coordinate system, can be represented as a rotational component vector based on three rotational directions (Pitch, Yaw, Roll). Wherein Pitch is Pitch, i.e. the object rotates around the X axis in the three-dimensional coordinate system; the Yaw is the course, namely the object rotates around the Y axis in the three-dimensional coordinate system; roll is a Roll and rotates the object around the Z axis in the three-dimensional coordinate system.

In the embodiment of the disclosure, the fisheye camera can be calibrated in various ways to obtain the internal reference and the external reference of the fisheye camera. For example, in some of these embodiments, the internal and external parameters of the fisheye camera may be calibrated by a capture plate (i.e., a calibration plate).

And 304, constructing a spherical coordinate system with the fisheye camera as the spherical center by using the internal reference and the external reference of the fisheye camera.

Based on the embodiment of the disclosure, the internal reference and the external reference of the fisheye camera can be obtained through calibration of the fisheye camera, so that the internal reference and the external reference of the fisheye camera generated by the calibration can be extracted to establish a spherical coordinate system with the fisheye camera as a spherical center.

Optionally, in some embodiments, the spherical coordinate system is specifically a unit spherical coordinate system in which the distance from a point on the spherical surface to the spherical center (also referred to as a center point, i.e., a position of the fisheye camera) (i.e., a radius length of the sphere) is 1, which helps to improve the calculation efficiency.

Fig. 5 is a flowchart illustrating a method for acquiring an image of a three-dimensional scene using a fisheye camera according to another embodiment of the disclosure. As shown in fig. 5, based on the embodiment shown in fig. 2, in operation 202, mapping each vertex from the three-dimensional coordinate system to a spherical coordinate system to obtain a spherical coordinate of each mapping point corresponding to each vertex in the spherical coordinate system, may include:

2022, moving each vertex from the three-dimensional coordinate system to a three-dimensional cartesian coordinate system with the center point of the spherical coordinate system as the origin, according to the position of the fisheye camera in the three-dimensional coordinate system.

Since the O point of the three-dimensional coordinate system may be located at a different position from the center point of the spherical coordinate system, based on the operation 2022, the O point of the three-dimensional coordinate system is moved to the center point of the spherical coordinate system, and the three-dimensional coordinate system after the O point is moved is referred to as a three-dimensional cartesian coordinate system for convenience of distinction.

2024, according to the orientation of the fisheye camera in the three-dimensional coordinate system, mapping each vertex from the three-dimensional cartesian coordinate system to the spherical coordinate system, and obtaining the spherical coordinates of each mapping point corresponding to each vertex in the spherical coordinate system.

As shown in FIG. 3, A is the vertex in the three-dimensional coordinate system, F is the point where A is moved into the three-dimensional Cartesian coordinate system, and G is the projected point of F on the cross-section.

On the basis of this operation 2024, it is,the spatial position coordinates (x, y, z) of each vertex in a three-dimensional Cartesian coordinate system can be converted into spherical coordinates in a spherical coordinate system

Wherein r is the distance from the vertex to the central point of the spherical coordinate system, and the value of r is always 1 when a unit spherical coordinate system is adopted;

the angle is rotated from the positive z-axis to the point in the xy plane by the projection of the connecting line of the point and the origin in the counterclockwise direction; theta is an included angle between a connecting line of the vertex and the central point of the spherical coordinate system and the positive direction of the z axis.

Optionally, in some embodiments, after the correspondence between the three-dimensional coordinates of each vertex and the two-dimensional coordinates of the corresponding projection point is established, a frame of video image in a video stream of the target scene acquired by a fish-eye camera may be used as a reference image, and based on the correspondence, the reference video image is fitted into the three-dimensional space model to obtain a three-dimensional fusion effect map of the target scene.

Responding to points with inaccurate laminating in the three-dimensional fusion effect graph, collecting at least one point with inaccurate laminating, correcting and calculating the two-dimensional coordinates of all vertexes in the corresponding relation by utilizing a least square method based on the current two-dimensional coordinates of each point in the at least one point with inaccurate laminating and the reference two-dimensional coordinates corresponding to the point with accurate laminating, obtaining the correction relation information between the two-dimensional coordinates of all vertexes in the corresponding relation and the reference two-dimensional coordinates corresponding to the points with accurate laminating, and storing the correction relation information.

Based on the embodiment, aiming at some points with inaccurate fusion of fisheye camera imaging and a 3D space model, the current two-dimensional coordinates (U, V) and the reference two-dimensional coordinates (namely, the two-dimensional coordinates corresponding to the point fitting when the point fitting is accurate) of the points are collected as input, the two-dimensional coordinates of all the points are corrected and calculated by using a least square method, and the correction relation information between the two-dimensional coordinates of all the vertexes and the corresponding reference two-dimensional coordinates when the point fitting is accurate is stored so as to realize the corrected fusion effect based on the correction relation information.

Optionally, in some embodiments, before storing the correction relationship information, it may further be configured to identify a noise point existing in the correction texture picture, that is, a pixel point whose difference between pixel values of adjacent pixel points found up, down, left, and right is greater than a preset threshold, if the noise point exists in the correction texture picture, the noise point may be denoised by using the pixel point adjacent to the noise point, and when storing the correction relationship information, the correction texture picture obtained after denoising is specifically stored. Fig. 6 is a schematic diagram of a texture correcting picture according to an embodiment of the disclosure.

Optionally, in some embodiments, adjacent non-noise point pixel points may be searched for up, down, left, and right around the noise point, and one adjacent non-noise point pixel point is selected according to a preset manner and filled in the noise point, so as to achieve an effect of removing the noise point.

Optionally, in some embodiments, when the rectification relationship information is stored, the two-dimensional coordinates of each vertex in all the vertices and the corresponding reference two-dimensional coordinates when the fitting is accurate may be stored in an RGBA format in a 32-bit format, so as to form a rectification texture picture, so as to store a data algorithm. In this embodiment, the correction relationship information is specifically a correction texture picture.

Where RGBA is a color space representing Red (Red) Green (Green) Blue (Blue) and Alpha. The Alpha channel and the RGB channel are juxtaposed and stored together in the image information. Thus, as an image is created, its Alpha channel is also generated or stored.

In the field of computer graphics, Alpha composition (English: Alpha composition) is a process of combining an image with a background, which can produce a partially or fully transparent visual effect. Alpha compositing is also called Alpha compositing or transparent compositing. When rendering an image, a plurality of sub-elements in a target image are usually rendered separately, and finally, pictures of the plurality of sub-elements are synthesized into a single image. For example, television live broadcasts may combine a large number of computer-generated image elements into a live shot.

Fig. 7 is a flowchart illustrating a method for acquiring an image of a three-dimensional scene using a fisheye camera according to still another embodiment of the disclosure. As shown in fig. 7, on the basis of the above embodiment, the operation 104 may include:

1042, obtaining the two-dimensional coordinates corresponding to the three-dimensional coordinates of each vertex based on the corresponding relationship between the three-dimensional coordinates of each vertex and the corresponding two-dimensional coordinates.

1044, correcting the two-dimensional coordinates corresponding to the three-dimensional coordinates of each vertex by using the two-dimensional coordinates corresponding to the three-dimensional coordinates of each vertex based on the reference two-dimensional coordinates corresponding to the correction relationship information to obtain the reference two-dimensional coordinates corresponding to the three-dimensional coordinates of each vertex.

1046, fitting the video image to the three-dimensional space model based on the reference two-dimensional coordinates corresponding to the three-dimensional coordinates of each vertex to obtain a three-dimensional scene effect graph of the target scene.

Fig. 8(a) is a schematic diagram of an image of a target scene captured by a fisheye camera in an application embodiment of the present disclosure. As shown in fig. 8(b), the schematic diagram of the three-dimensional scene effect map obtained based on the embodiment of the present disclosure is a three-dimensional scene effect map obtained by fitting a video image of a target scene acquired by a fisheye camera to a three-dimensional space model of the target scene.

Any of the methods provided by the embodiments of the present disclosure for acquiring a three-dimensional scene image using a fisheye camera may be performed by any suitable device with data processing capabilities, including but not limited to: terminal equipment, a server and the like. Alternatively, any method for acquiring a three-dimensional scene image by using a fisheye camera provided by the embodiments of the present disclosure may be executed by a processor, for example, the processor may execute any method for acquiring a three-dimensional scene image by using a fisheye camera by calling corresponding instructions stored in a memory. And will not be described in detail below.

Those of ordinary skill in the art will understand that: all or part of the steps for implementing the method embodiments may be implemented by hardware related to program instructions, and the program may be stored in a computer readable storage medium, and when executed, the program performs the steps including the method embodiments; and the aforementioned storage medium includes: various media that can store program codes, such as ROM, RAM, magnetic or optical disks.

Fig. 9 is a schematic structural diagram of an embodiment of an apparatus for acquiring a three-dimensional scene image by using a fisheye camera according to the disclosure. The apparatus of this embodiment may be used to implement the method embodiments of the present disclosure described above. As shown in fig. 9, the apparatus of this embodiment includes: fisheye camera and fuse processing module. Wherein:

the fisheye camera is used for acquiring a video image of a target scene.

And the fusion processing module is used for fitting the video image into the three-dimensional space model based on the corresponding relation between the three-dimensional coordinates of each vertex in the three-dimensional space model of the target scene and the corresponding two-dimensional coordinates to obtain the three-dimensional scene effect graph of the target scene.

Based on the embodiment of the disclosure, the video image of the target scene collected by the fisheye camera can be fitted into the three-dimensional space model, and the fusion of the video image collected by the fisheye camera and the three-dimensional space model is realized, so that the three-dimensional scene effect image of the target scene obtained by the fisheye camera is realized.

Fig. 10 is a schematic structural diagram of another embodiment of the apparatus for acquiring an image of a three-dimensional scene by using a fisheye camera according to the disclosure. As shown in fig. 10, on the basis of the embodiment shown in fig. 9, the method may further include: the device comprises a first processing module and a second processing module. Wherein:

and the first processing module is used for mapping each vertex in the three-dimensional space model from the three-dimensional coordinate system to the spherical coordinate system respectively to obtain the spherical coordinates of each mapping point corresponding to each vertex in the spherical coordinate system.

And the second processing module is used for projecting each mapping point from the spherical coordinate system to the cross section of the spherical coordinate system to obtain the two-dimensional coordinates of the projection point of each mapping point on the cross section.

Optionally, referring to fig. 10 again, in the apparatus of each of the above embodiments, the apparatus may further include: the device comprises a calibration module and a construction module. Wherein:

and the calibration module is used for calibrating the fisheye camera to obtain the internal parameters and the external parameters of the fisheye camera. Wherein the internal parameters may include at least one of an imaging center, a radius of the fisheye camera, and a photographing field angle, and the external parameters may include at least one of a position and an orientation of the fisheye camera in a three-dimensional coordinate system.

And the building module is used for building a spherical coordinate system taking the fisheye camera as a sphere center by utilizing the internal reference and the external reference of the fisheye camera.

Optionally, in some of the embodiments, the spherical coordinate system is specifically a unit spherical coordinate system.

Optionally, in some embodiments, the first processing module is specifically configured to, for each vertex in the three-dimensional space model, move each vertex from the three-dimensional coordinate system into a three-dimensional cartesian coordinate system with the center point of the spherical coordinate system as an origin according to the position of the fisheye camera in the three-dimensional coordinate system; and mapping each vertex from the three-dimensional Cartesian coordinate system to the spherical coordinate system according to the orientation of the fisheye camera in the three-dimensional coordinate system to obtain the spherical coordinates of each mapping point corresponding to each vertex in the spherical coordinate system.

Optionally, referring to fig. 10 again, in the apparatus of each of the above embodiments, the apparatus may further include: the device comprises a correction module and a storage module. In this embodiment, the fusion processing module is further configured to use a frame of video image in the video stream of the target scene acquired by the fisheye camera as a reference image, and fit the reference image to the three-dimensional space model based on the correspondence, so as to obtain a three-dimensional fusion effect map of the target scene.

and the storage module is used for storing the correction relation information.

Optionally, in some embodiments, the storage module is specifically configured to store the two-dimensional coordinates of each vertex in all the vertices and the corresponding reference two-dimensional coordinates when the fitting is accurate in an RGBA format, so as to form a corrected texture picture; the correction relation information is specifically a correction texture picture.

Optionally, referring to fig. 10 again, in the apparatus of each of the above embodiments, the apparatus may further include: and the denoising module is used for responding to the existence of the noise point in the corrected texture picture and denoising the noise point by using the pixel point adjacent to the noise point. Correspondingly, in this embodiment, the storage module is specifically configured to store the corrected texture picture obtained after denoising.

Optionally, in some embodiments, the fusion processing module is specifically configured to: acquiring two-dimensional coordinates corresponding to the three-dimensional coordinates of each vertex based on the corresponding relation between the three-dimensional coordinates of each vertex and the corresponding two-dimensional coordinates; correcting the two-dimensional coordinates corresponding to the three-dimensional coordinates of each vertex by using the two-dimensional coordinates corresponding to the three-dimensional coordinates of each vertex based on the reference two-dimensional coordinates corresponding to the correction relation information to obtain the reference two-dimensional coordinates corresponding to the three-dimensional coordinates of each vertex; and fitting the video image into the three-dimensional space model based on the reference two-dimensional coordinates corresponding to the three-dimensional coordinates of each vertex to obtain a three-dimensional scene effect graph of the target scene.

In addition, an embodiment of the present disclosure also provides an electronic device, including:

a memory for storing a computer program;

a processor configured to execute the computer program stored in the memory, and when the computer program is executed, the method for acquiring a three-dimensional scene image by using a fisheye camera according to any of the above embodiments of the present disclosure is implemented.

Next, an electronic apparatus according to an embodiment of the present disclosure is described with reference to fig. 11. The electronic device may be either or both of the first device and the second device, or a stand-alone device separate from them, which stand-alone device may communicate with the first device and the second device to receive the acquired input signals therefrom.

Fig. 11 is a schematic structural diagram of an embodiment of an electronic device according to the present disclosure. As shown in fig. 11, the electronic device includes one or more processors and memory.

The processor may be a Central Processing Unit (CPU) or other form of processing unit having data processing capabilities and/or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

The memory may include one or more computer program products that may include various forms of computer-readable storage media, such as volatile memory and/or non-volatile memory. The volatile memory may include, for example, Random Access Memory (RAM), cache memory (cache), and/or the like. The non-volatile memory may include, for example, Read Only Memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium and executed by a processor to implement the method for acquiring three-dimensional scene images using a fisheye camera according to the various embodiments of the disclosure described above and/or other desired functions.

In one example, the electronic device may further include: an input device and an output device, which are interconnected by a bus system and/or other form of connection mechanism (not shown).

The input device may also include, for example, a keyboard, a mouse, and the like.

The output device may output various information including the determined distance information, direction information, and the like to the outside. The output devices may include, for example, a display, speakers, a printer, and a communication network and remote output devices connected thereto, among others.

Of course, for simplicity, only some of the components of the electronic device relevant to the present disclosure are shown in fig. 11, omitting components such as buses, input/output interfaces, and the like. In addition, the electronic device may include any other suitable components, depending on the particular application.

In addition to the above methods and apparatus, embodiments of the present disclosure may also be a computer program product comprising computer program instructions which, when executed by a processor, cause the processor to perform the steps in the method of acquiring images of a three-dimensional scene with a fisheye camera according to various embodiments of the present disclosure described in the above section of this specification.

The computer program product may write program code for carrying out operations for embodiments of the present disclosure in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server.

Furthermore, embodiments of the present disclosure may also be a computer-readable storage medium having stored thereon computer program instructions which, when executed by a processor, cause the processor to perform the steps in the method of acquiring an image of a three-dimensional scene with a fisheye camera according to various embodiments of the present disclosure described in the above section of this specification.

The computer-readable storage medium may take any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may include, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

The foregoing describes the general principles of the present disclosure in conjunction with specific embodiments, however, it is noted that the advantages, effects, etc. mentioned in the present disclosure are merely examples and are not limiting, and they should not be considered essential to the various embodiments of the present disclosure. Furthermore, the foregoing disclosure of specific details is for the purpose of illustration and description and is not intended to be limiting, since the disclosure is not intended to be limited to the specific details so described.

In the present specification, the embodiments are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same or similar parts in the embodiments are referred to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and for the relevant points, reference may be made to the partial description of the method embodiment.

The block diagrams of devices, apparatuses, systems referred to in this disclosure are only given as illustrative examples and are not intended to require or imply that the connections, arrangements, configurations, etc. must be made in the manner shown in the block diagrams. These devices, apparatuses, devices, systems may be connected, arranged, configured in any manner, as will be appreciated by those skilled in the art. Words such as "including," "comprising," "having," and the like are open-ended words that mean "including, but not limited to," and are used interchangeably therewith. The words "or" and "as used herein mean, and are used interchangeably with, the word" and/or, "unless the context clearly dictates otherwise. The word "such as" is used herein to mean, and is used interchangeably with, the phrase "such as but not limited to".

The methods and apparatus of the present disclosure may be implemented in a number of ways. For example, the methods and apparatus of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order for the steps of the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above unless specifically stated otherwise. Further, in some embodiments, the present disclosure may also be embodied as programs recorded in a recording medium, the programs including machine-readable instructions for implementing the methods according to the present disclosure. Thus, the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure.

It is also noted that in the devices, apparatuses, and methods of the present disclosure, each component or step can be decomposed and/or recombined. These decompositions and/or recombinations are to be considered equivalents of the present disclosure.

The previous description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects without departing from the scope of the disclosure. Thus, the present disclosure is not intended to be limited to the aspects shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

The foregoing description has been presented for purposes of illustration and description. Furthermore, this description is not intended to limit embodiments of the disclosure to the form disclosed herein. While a number of example aspects and embodiments have been discussed above, those of skill in the art will recognize certain variations, modifications, alterations, additions and sub-combinations thereof.

Claims

1. A method for acquiring a three-dimensional scene image by using a fisheye camera is characterized by comprising the following steps:

acquiring a video image of a target scene by using a fisheye camera;

acquiring two-dimensional coordinates corresponding to the three-dimensional coordinates of each vertex based on the corresponding relation between the three-dimensional coordinates of each vertex in the three-dimensional space model of the target scene and the corresponding two-dimensional coordinates;

correcting the two-dimensional coordinates corresponding to the three-dimensional coordinates of each vertex by using the two-dimensional coordinates corresponding to the three-dimensional coordinates of each vertex based on the reference two-dimensional coordinates corresponding to the correction relation information to obtain the reference two-dimensional coordinates corresponding to the three-dimensional coordinates of each vertex; the correction relation information is used for expressing the correction relation between the two-dimensional coordinates of all the vertexes in the corresponding relation and the corresponding reference two-dimensional coordinates when the fitting is accurate;

fitting the video image into the three-dimensional space model based on the reference two-dimensional coordinates corresponding to the three-dimensional coordinates of each vertex to obtain a three-dimensional scene effect graph of the target scene;

2. The method according to claim 1, wherein the establishing of the corresponding relationship comprises:

3. The method of claim 2, further comprising, prior to establishing a correspondence between the three-dimensional coordinates and the corresponding two-dimensional coordinates of the vertices:

constructing a spherical coordinate system with the fisheye camera as a sphere center by using the internal reference and the external reference of the fisheye camera;

the spherical coordinate system is specifically a unit spherical coordinate system.

4. The method according to claim 3, wherein the mapping the vertices from the three-dimensional coordinate system to the spherical coordinate system to obtain spherical coordinates of the mapping points corresponding to the vertices in the spherical coordinate system comprises:

5. The method according to any one of claims 2-4, wherein after establishing the correspondence between the three-dimensional coordinates of each vertex and the two-dimensional coordinates of the corresponding projection point, the method further comprises:

responding to points which are not accurately attached in the three-dimensional fusion effect graph, collecting at least one point which is not accurately attached, based on the current two-dimensional coordinates of each point in the at least one point which is not accurately attached and the corresponding reference two-dimensional coordinates when the attachment is accurate, performing correction calculation on the two-dimensional coordinates of all vertexes in the corresponding relation by using a least square method to obtain correction relation information between the two-dimensional coordinates of all vertexes in the corresponding relation and the corresponding reference two-dimensional coordinates when the attachment is accurate, and storing the correction relation information;

the storing the correction relation information includes:

6. The method of claim 5, wherein prior to storing the corrective relationship information, further comprising:

the storing the correction relation information includes:

and storing the corrected texture picture obtained after denoising.

7. An apparatus for acquiring images of a three-dimensional scene using a fisheye camera, comprising:

the fisheye camera is used for acquiring a video image of a target scene;

the fusion processing module is used for acquiring two-dimensional coordinates corresponding to the three-dimensional coordinates of each vertex based on the corresponding relation between the three-dimensional coordinates of each vertex in the three-dimensional space model of the target scene and the corresponding two-dimensional coordinates; correcting the two-dimensional coordinates corresponding to the three-dimensional coordinates of each vertex by using the two-dimensional coordinates corresponding to the three-dimensional coordinates of each vertex based on the reference two-dimensional coordinates corresponding to the correction relation information to obtain the reference two-dimensional coordinates corresponding to the three-dimensional coordinates of each vertex; the correction relation information is used for expressing the correction relation between the two-dimensional coordinates of all the vertexes in the corresponding relation and the corresponding reference two-dimensional coordinates when the fitting is accurate; fitting the video image into the three-dimensional space model based on the reference two-dimensional coordinates corresponding to the three-dimensional coordinates of each vertex to obtain a three-dimensional scene effect graph of the target scene;

the three-dimensional coordinates of the vertex are space position coordinates of the vertex in a three-dimensional coordinate system corresponding to the three-dimensional space model, and the two-dimensional coordinates are coordinates of a projection point on a cross section of a spherical coordinate system corresponding to the fisheye camera and corresponding to the vertex; the spherical coordinate system is constructed by taking the fisheye camera as a spherical center and based on internal parameters and external parameters of the fisheye camera; the cross section is through the central point of spherical coordinate system, and with the perpendicular plane in shooting positive direction of fisheye camera.

8. An electronic device, comprising:

a memory for storing a computer program;

a processor for executing a computer program stored in the memory, and when executed, implementing the method of any of the preceding claims 1-6.

9. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the method of any one of the preceding claims 1 to 6.