CN114066731A

CN114066731A - Method and device for generating panorama, electronic equipment and storage medium

Info

Publication number: CN114066731A
Application number: CN202111331704.3A
Authority: CN
Inventors: 杨海岩
Original assignee: Baidu Online Network Technology Beijing Co Ltd
Current assignee: Baidu Online Network Technology Beijing Co Ltd
Priority date: 2021-11-11
Filing date: 2021-11-11
Publication date: 2022-02-18

Abstract

The disclosure provides a method and a device for generating a panoramic image, electronic equipment and a storage medium, and relates to the technical field of image processing, in particular to the field of computer vision. The specific implementation scheme is as follows: collecting a plurality of pictures by using a panoramic camera shooting component; acquiring estimation parameter information corresponding to a plurality of pictures, wherein the estimation parameter information is used for estimating the expansion effect of an expansion diagram corresponding to each picture in the plurality of pictures and the position information of the expansion diagram; unfolding a plurality of pictures by adopting a spherical projection mode to obtain a picture unfolding result; and splicing the image unfolding results in a spherical splicing mode to generate a target panoramic image.

Description

Method and device for generating panorama, electronic equipment and storage medium

Technical Field

The present disclosure relates to the field of image processing technologies, and further relates to the field of computer vision, and in particular, to a method and an apparatus for generating a panorama, an electronic device, and a storage medium.

Background

The panoramic view can represent the surrounding environment as much as possible by means of a wide-angle representation and in the form of a drawing, a photograph, a video, a three-dimensional model, or the like.

In the related scheme, image materials are obtained through shooting hardware, a plurality of image materials are imported into panoramic software to be automatically spliced, and then a spliced panoramic image is obtained.

Disclosure of Invention

The disclosure provides a method and a device for generating a panoramic image, electronic equipment and a storage medium, so as to improve the image splicing effect and splicing efficiency when the panoramic image is generated.

According to an aspect of the present disclosure, there is provided a method of generating a panorama, including: collecting a plurality of pictures by using a panoramic camera shooting component; acquiring estimation parameter information corresponding to a plurality of pictures, wherein the estimation parameter information is used for estimating the expansion effect of an expansion diagram corresponding to each picture in the plurality of pictures and the position information of the expansion diagram; unfolding a plurality of pictures by adopting a spherical projection mode to obtain a picture unfolding result; and splicing the image unfolding results in a spherical splicing mode to generate a target panoramic image.

According to still another aspect of the present disclosure, there is provided an apparatus for generating a panorama, including: the acquisition module is used for acquiring a plurality of pictures by using the panoramic camera shooting assembly; the device comprises an acquisition module, a display module and a display module, wherein the acquisition module is used for acquiring estimation parameter information corresponding to a plurality of pictures, and the estimation parameter information is used for estimating the expansion effect of an expansion diagram corresponding to each picture in the plurality of pictures and the position of the expansion diagram; the unfolding module is used for unfolding a plurality of pictures in a spherical projection mode to obtain a picture unfolding result; and the generating module is used for splicing the image unfolding results in a spherical splicing mode to generate a target panoramic image.

According to still another aspect of the present disclosure, there is provided an electronic device including: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executable by the at least one processor to enable the at least one processor to perform the method of generating a panorama as set forth in the present disclosure.

According to yet another aspect of the present disclosure, there is provided a non-transitory computer readable storage medium having stored thereon computer instructions for causing a computer to perform the method of generating a panorama proposed by the present disclosure.

According to yet another aspect of the present disclosure, a computer program product is provided, comprising a computer program which, when executed by a processor, performs the method of generating a panorama proposed by the present disclosure.

In the present disclosure, a plurality of pictures are acquired by using a panoramic camera assembly; acquiring estimation parameter information corresponding to a plurality of pictures, wherein the estimation parameter information is used for estimating the expansion effect of an expansion diagram corresponding to each picture in the plurality of pictures and the position information of the expansion diagram; unfolding a plurality of pictures by adopting a spherical projection mode to obtain a picture unfolding result; the method has the advantages that the picture expansion result is spliced in a spherical splicing mode to generate the target panoramic picture, the purpose that multiple pictures collected by the panoramic camera assembly are used for rapidly generating the panoramic picture is achieved, and the picture splicing effect and the splicing efficiency in the process of generating the panoramic picture can be improved.

It should be understood that the statements in this section do not necessarily identify key or critical features of the embodiments of the present disclosure, nor do they limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description.

Drawings

The drawings are included to provide a better understanding of the present solution and are not to be construed as limiting the present disclosure. Wherein:

fig. 1 is a block diagram of a hardware structure of a computer terminal (or mobile device) for implementing a method of generating a panorama according to an embodiment of the present disclosure;

FIG. 2 is a flow chart of a method of generating a panorama according to an embodiment of the present disclosure;

FIG. 3 is a schematic diagram of a process for generating a panorama according to an embodiment of the present disclosure;

FIG. 4 is a schematic diagram of an input picture according to an embodiment of the present disclosure;

FIG. 5 is a schematic diagram of a black border of a picture according to an embodiment of the present disclosure;

FIG. 6 is a panorama generated according to an embodiment of the present disclosure;

fig. 7 is a block diagram of an apparatus for generating a panorama according to an embodiment of the present disclosure.

Detailed Description

Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, in which various details of the embodiments of the disclosure are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

It should be noted that the terms "first," "second," and the like in the description and claims of the present disclosure and in the above-described drawings are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the disclosure described herein are capable of operation in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

In the related scheme, when the image material acquired by the shooting hardware is a fisheye picture, the fisheye picture has large distortion, so that a panoramic image with good splicing effect is difficult to obtain by utilizing panoramic software processing.

In accordance with an embodiment of the present disclosure, there is provided a method of generating a panorama, it being noted that the steps illustrated in the flow chart of the drawings may be performed in a computer system such as a set of computer-executable instructions, and that while a logical order is illustrated in the flow chart, in some cases the steps illustrated or described may be performed in an order different than here.

The method embodiments provided by the embodiments of the present disclosure may be executed in a mobile terminal, a computer terminal or similar electronic devices. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the disclosure described and/or claimed herein. Fig. 1 shows a hardware configuration block diagram of a computer terminal (or mobile device) for implementing the method of generating a panorama.

As shown in fig. 1, the computer terminal 100 includes a computing unit 101 that can perform various appropriate actions and processes according to a computer program stored in a Read Only Memory (ROM)102 or a computer program loaded from a storage unit 108 into a Random Access Memory (RAM) 103. In the RAM103, various programs and data necessary for the operation of the computer terminal 100 can also be stored. The computing unit 101, the ROM 102, and the RAM103 are connected to each other via a bus 104. An input/output (I/O) interface 105 is also connected to bus 104.

A number of components in the computer terminal 100 are connected to the I/O interface 105, including: an input unit 106 such as a keyboard, a mouse, and the like; an output unit 107 such as various types of displays, speakers, and the like; a storage unit 108, such as a magnetic disk, optical disk, or the like; and a communication unit 109 such as a network card, modem, wireless communication transceiver, etc. The communication unit 109 allows the computer terminal 100 to exchange information/data with other devices through a computer network such as the internet and/or various telecommunication networks.

Computing unit 101 may be a variety of general purpose and/or special purpose processing components having processing and computing capabilities. Some examples of the computing unit 101 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, and so forth. The computing unit 101 performs the method of generating a panorama described herein. For example, in some embodiments, the method of generating a panorama may be implemented as a computer software program tangibly embodied in a machine-readable medium, such as storage unit 108. In some embodiments, part or all of the computer program may be loaded and/or installed onto the computer terminal 100 via the ROM 102 and/or the communication unit 109. When the computer program is loaded into RAM103 and executed by the computing unit 101, one or more steps of the method of generating a panorama described herein may be performed. Alternatively, in other embodiments, the computing unit 101 may be configured by any other suitable means (e.g., by means of firmware) to perform the method of generating the panorama.

Various implementations of the systems and techniques described here can be implemented in digital electronic circuitry, integrated circuitry, Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), system on a chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.

It should be noted here that in some alternative embodiments, the electronic device shown in fig. 1 may include hardware elements (including circuitry), software elements (including computer code stored on a computer-readable medium), or a combination of both hardware and software elements. It should be noted that fig. 1 is only one example of a particular specific example and is intended to illustrate the types of components that may be present in the electronic device described above.

In the operating environment described above, the present disclosure provides a method of generating a panorama as shown in fig. 2, which may be performed by a computer terminal or similar electronic device as shown in fig. 1. Fig. 2 is a flowchart of a method for generating a panorama according to an embodiment of the present disclosure. As shown in fig. 2, the method may include the steps of:

step S20, collecting a plurality of pictures by using the panoramic camera shooting component;

optionally, the panoramic camera module may be packaged in an image capturing device in the mobile terminal, or may be a detachable independent device that can be assembled on a rear camera of the mobile terminal for image capturing.

Optionally, the panoramic camera shooting and collecting assembly is used for shooting and collecting a plurality of pictures in a rotating mode, and the collected pictures can be used for being spliced to generate a panoramic image.

For example, the panoramic photographing component can be a fisheye lens, the focal length of the fisheye lens is less than or equal to 16mm, the viewing angle is close to or equal to 180 degrees, and a plurality of fisheye pictures can be collected by the fisheye lens. The fisheye lens is added on a rear camera of the mobile phone, the mobile phone is placed on the Bluetooth rotating holder, the mobile phone is controlled to shoot when the holder rotates, and then a plurality of pictures are collected.

Step S22, acquiring estimation parameter information corresponding to a plurality of pictures, wherein the estimation parameter information is used for estimating the expansion effect of the expansion diagram corresponding to each picture in the plurality of pictures and the position information of the expansion diagram;

the estimation parameter information includes: panoramic camera module model parameters and a rotation matrix.

Specifically, the implementation process of obtaining the estimated parameter information corresponding to the multiple pictures may refer to further description of the embodiments of the present disclosure, and is not repeated herein.

Step S26, unfolding a plurality of pictures by adopting a spherical projection mode to obtain a picture unfolding result;

optionally, a spherical projection mode is adopted to perform expansion processing on the multiple pictures.

And step S28, splicing the image expansion results in a spherical splicing mode to generate a target panoramic image.

Specifically, the image expansion result is spliced in a spherical splicing manner, and further description of the embodiment of the disclosure may be referred to for generating the target panorama, which is not repeated herein.

According to the present disclosure, the steps S20 to S24 are performed by capturing a plurality of pictures by using the panoramic camera assembly; acquiring estimation parameter information corresponding to a plurality of pictures, wherein the estimation parameter information is used for estimating the expansion effect of an expansion diagram corresponding to each picture in the plurality of pictures and the position information of the expansion diagram; unfolding a plurality of pictures by adopting a spherical projection mode to obtain a picture unfolding result; the method has the advantages that the picture expansion result is spliced in a spherical splicing mode to generate the target panoramic picture, the purpose that multiple pictures collected by the panoramic camera assembly are used for rapidly generating the panoramic picture is achieved, and the picture splicing effect and the splicing efficiency in the process of generating the panoramic picture can be improved.

The method for generating the panorama according to the above embodiment is further described below.

As an optional implementation manner, in step S22, the acquiring estimated parameter information corresponding to the multiple pictures includes:

step S220, carrying out graph detection on each of a plurality of pictures to obtain a target graph corresponding to each picture;

optionally, the performing of the graph detection on each of the plurality of pictures includes performing the circle detection on each of the plurality of pictures. The target graph corresponding to each picture can be a finally detected circle.

Specifically, the implementation process of performing circle detection on each of the multiple pictures to obtain the target graph corresponding to each picture may refer to further description of the following embodiments, and is not repeated.

Step S222, performing feature detection on the target graph to extract a plurality of feature points, performing pose estimation based on the plurality of feature points, and determining estimation parameter information, or determining estimation parameter information by using angular motion detection data acquired by an angular motion detection assembly, where the angular motion detection data is detection data that is synchronously recorded when a plurality of pictures are acquired by using a panoramic camera assembly.

Optionally, feature detection is performed on the target graph to obtain a plurality of feature descriptors, and a plurality of feature points are extracted from the plurality of feature descriptors. Wherein each feature descriptor is a vector for describing a corresponding feature point.

Specifically, pose estimation is performed based on a plurality of feature points, and the implementation process of determining estimation parameter information is as follows: and projecting the extracted characteristic points to a fish eye hemisphere model, and solving a rotation matrix by utilizing random sampling consistency. The fish-eye hemisphere model is a hemisphere of a unit sphere, the relation of a plurality of characteristic points on a fish-eye picture in a world coordinate system is also the rotation relation of the characteristic points on a 3D point of the fish-eye hemisphere model, namely a rotation matrix, the characteristic points are projected into the fish-eye hemisphere model, and the rotation matrix can be accurately estimated. Further, a projection mode of the fish-eye model and a calculation mode of a reprojection error are determined, and optimization of parameters and a rotation matrix of the fish-eye camera model is completed by using a beam adjustment optimization algorithm, so that more accurate estimated parameter information is obtained.

Specifically, the implementation process of determining the estimation parameter information by using the angular motion detection data acquired by the angular motion component is as follows: initializing camera internal parameters, acquiring circle center radius by using the circle detection in the step S220, setting a field of view (FOV) to be the view angle of the fisheye camera, generally to be 180 °, and initializing parameters of a fisheye camera model as follows: and a is 0, b is-0.2, c is 0, and the rotation matrix is directly calculated by using the Euler angles collected by the gyroscope, wherein the Euler angles comprise a pitch angle, a yaw angle and a roll angle. Wherein, the projection function of the fisheye lens is shown as the following formula (1):

r(θ)≈a*θ+b*θ³+c*θ⁵formula (1)

Optionally, when pose estimation based on a plurality of feature points fails, the estimation parameter information is determined by using angular motion detection data collected by the angular motion detection component. Specifically, if the bundle adjustment optimization algorithm is converged and the estimated parameter value is within a reasonable range, the pose estimation is considered to be successful, otherwise, the pose estimation is considered to be failed.

Based on the above steps S220 to S222, image detection may be performed on each of the multiple images, so as to obtain a target image corresponding to each image, further perform feature detection and extraction on the target image, perform pose estimation by using the extracted multiple feature points to determine estimation parameter information, or determine estimation parameter information by using the angular motion acquisition data acquired by the angular motion detection assembly, so as to obtain accurate estimation parameter information for subsequent image processing.

As an optional implementation, estimating the parameter information includes: the panoramic shooting assembly comprises panoramic shooting assembly model parameters and a rotation matrix, wherein the panoramic shooting assembly model parameters are used for estimating the expansion effect of an expansion image corresponding to each of a plurality of images, and the rotation matrix is used for estimating the position of the expansion image in a target panoramic image.

The panoramic shooting component model is a fish-eye camera model parameter.

As an optional implementation manner, in step S220, performing graph detection on each of the multiple pictures, and acquiring a target graph corresponding to each picture includes:

step S2200, adjusting the brightness and contrast of each of the plurality of pictures by utilizing the color level adjustment to obtain an adjustment result;

specifically, the adjusting the brightness and the contrast of each of the plurality of pictures by using the tone scale adjustment includes: the brightness of each picture in the plurality of pictures is reduced, and the contrast of each picture in the plurality of pictures is increased. The color level adjustment comprises a group of parameters obtained by adjusting a white field value and a black field value of the color level through experiments, so that the circular detection accuracy can be the highest.

Step S2202, searching a first center position from the adjustment result by a first search method, and searching a second center position from the adjustment result by a second search method;

the first searching mode is that a random sampling consistency method is adopted for searching, the first center position is the outline shape of a main body area of the fisheye picture, the second searching mode is that an interface in opencv is adopted for searching, and the second center position is a closed circle in the fisheye picture.

Step S2204, comparing a first distance with a second distance to obtain a comparison result, wherein the first distance is a distance between the first center position and the center position of the corresponding picture, and the second distance is a distance between the second center position and the center position of the corresponding picture;

in step S2206, a target graphic corresponding to each picture is acquired based on the comparison result.

Optionally, when the first distance is smaller than the second distance, the first center position is determined as the target graphic corresponding to the picture. For example, when the outline shape found by the first search means is closer to the center position of the corresponding picture, the outline shape is regarded as a finally detected circle.

Optionally, when the first distance is greater than the second distance, the second center position is determined as the target graphic corresponding to the picture. For example, when the closed circle found by the second search means is closer to the center position of the corresponding picture, the closed circle is taken as the finally detected circle.

Based on the above steps S2200 to S2206, the color level of each of the plurality of pictures can be adjusted, and then the first center position is searched from the adjustment result by using the first searching method, the second center position is searched from the adjustment result by using the second searching method, and finally the target graph corresponding to each picture is obtained based on the comparison result of the first distance and the second distance, thereby realizing the circle detection for the plurality of pictures.

As an optional implementation manner, in step S26, performing a stitching process on the image expansion result by using a spherical stitching method, and generating the target panorama includes:

step S260, determining an overlapping part between each expansion diagram and the rest expansion diagrams by using the image expansion result, and cutting the overlapping part to obtain a cutting result;

in order to estimate the rotation matrix more accurately, in the process of acquiring a plurality of pictures by rotating the panoramic camera component by 360 degrees, the overlapping area between two adjacent pictures is large, and finally the leftmost area and the rightmost area of the equidistant rectangular projection result cannot be perfectly connected after splicing.

Optionally, when the cutting processing is performed on the overlapped part between each expansion diagram and the rest of the expansion diagrams, the width of the area needing to be cut is determined through experimental parameter adjustment. In the cutting result, the overlapping between every two adjacent expansion diagrams is reduced, and only one splicing seam exists between the adjacent expansion diagrams. And cutting the unfolded picture which is split by the splicing seam into two pictures, and respectively calculating projection points at the upper left corners of the two pictures, so that the subsequent cutting and image fusion are facilitated.

Step S262, finding out a splicing seam between adjacent expansion graphs in the cutting result;

specifically, an image segmentation and splicing Seam searching algorithm (Graph Cut team Finder) in opencv is adopted to search for a splicing Seam between two adjacent expansion images.

Step S264, splicing the spliced seams by adopting a spherical splicing mode to generate an initial panoramic image;

specifically, image fusion can be performed by utilizing multiband fusion in the stitching processing to generate an initial panorama.

And step S266, cutting the area to be cut in the initial panoramic image to obtain the target panoramic image.

The area to be cut is an area where black areas exist at the upper end and the lower end of the initial panorama, namely, a black edge exists, the black areas represent areas which are not collected by the camera shooting collecting assembly, and the black edge needs to be cut for the attractiveness of the final panorama.

Specifically, the implementation process of performing clipping processing on the region to be clipped in the initial panorama to obtain the target panorama can refer to the description of the following embodiments, which are not repeated.

As an optional implementation manner, in step S266, performing clipping processing on the region to be clipped in the initial panorama, and obtaining the target panorama includes:

step 2660, obtaining a mask corresponding to the initial panorama;

step S2662, searching for a first height and a second height in the height direction of the initial panorama based on the mask;

the first height is the minimum height of the upper black edge, and the second height is the maximum height of the lower black edge.

Specifically, the minimum height of the upper black edge and the maximum height of the lower black edge are searched in a traversal manner in the height direction of the initial panorama based on the mask (mask). Because the traversal operation consumes time, the initial panoramic image after the image fusion is divided into small images to be input in the traversal operation, and the calculation time can be effectively reduced.

Step 2664, determining an area to be cut by using the first height and the second height;

the area to be cut comprises an upper end black edge area and a lower end black edge area. The origin of coordinates of the initial panorama is at the upper left corner, and is in the positive x-axis direction towards the right and is in the positive y-axis direction downwards, so that in the y-axis direction, a black edge at the upper end is positioned below the minimum height in the initial panorama, and a black edge at the lower end is positioned above the maximum height in the initial panorama.

And step 2666, performing clipping processing on the region to be clipped to obtain a target panoramic image.

Based on the steps S2660 to S2666, the black border in the initial panorama can be accurately cropped, so that the black border of the black object in the image can be prevented from being cropped as the black border by mistake, and the visual effect of the panorama can be further improved.

The method for generating a panorama provided by the embodiments of the present disclosure is described below with reference to schematic diagrams.

Fig. 3 is a schematic diagram of a process of generating a panorama, as shown in fig. 3, the process including the following steps:

step S301, carrying out image reading operation;

the pictures used for the picture reading operation are a plurality of pictures collected by the panoramic camera shooting assembly. For example, fig. 4 is a schematic diagram of an input picture according to an embodiment of the present disclosure, and as shown in fig. 4, 8 fisheye pictures are selected from a plurality of pictures captured by a panoramic camera component to generate a panoramic picture, and the fisheye pictures are in a compressed format JPG format, and each fisheye picture has a size of 1536 × 2048.

Step S302, performing circle detection;

specifically, the step S220 may be referred to for the process of implementing the circle detection, and is not described in detail.

Step S303, carrying out feature detection;

specifically, Scale-Invariant Feature Transform (SIFT) features are adopted for Feature detection and Feature descriptor extraction.

Step S304, estimating the pose of the camera based on the characteristics, and acquiring estimation parameter information corresponding to a plurality of pictures;

optionally, in step S304, the obtaining estimated parameter information corresponding to multiple pictures based on the estimation of the pose of the camera based on the features further includes the following steps:

step S3041, performing feature matching;

step S3042, projecting the feature points to a spherical surface;

step S3043, estimating the internal and external parameters of the camera based on the coordinates of the spherical projection points;

step S3044, optimizing the internal and external parameters of the camera;

step S305, judging whether the camera pose estimation based on the characteristics is successful;

specifically, if the bundle adjustment optimization algorithm is converged and the estimated parameter value is within a reasonable range, the pose estimation is considered to be successful, otherwise, the pose estimation is considered to be failed.

Step S306, when the camera pose estimation based on the characteristics is determined to be unsuccessful, camera internal and external parameter estimation is carried out based on gyroscope data;

wherein the gyroscope data comprises a pitch angle, a yaw angle, and a roll angle.

Step S307, when the camera pose estimation based on the characteristics is determined to be successful, spherical projection and deformation operation are carried out;

specifically, a spherical projection mode is adopted to expand a plurality of pictures to obtain a picture expansion result.

Step S308, cutting an overlapping area;

specifically, the step S26 may be referred to for implementing the clipping overlapping area, and is not described in detail.

Step S309, performing image fusion operation;

step S310, performing black edge cutting operation;

for example, fig. 5 is a schematic diagram of a black border of a picture according to an embodiment of the present disclosure, and specifically, the step S266 may be referred to for implementing the black border clipping operation, which is not repeated herein.

In step S311, a panorama is generated.

For example, fig. 6 is a panorama generated according to an embodiment of the present disclosure.

Based on the steps S301 to S311, fisheye pictures with large distortion can be spliced, so that a better panorama splicing effect can be obtained, and meanwhile, because the viewing angle of the fisheye pictures is large, fewer input pictures are required for splicing the scene graph of one scene, fewer pictures need to be processed by the algorithm, and the efficiency of generating the panorama by splicing is higher.

In the technical scheme of the disclosure, the collection, storage, use, processing, transmission, provision, disclosure and other processing of the personal information of the related user are all in accordance with the regulations of related laws and regulations and do not violate the good customs of the public order.

Through the above description of the embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but the former is a better implementation mode in many cases. Based on such understanding, the technical solutions of the present disclosure may be embodied in the form of a software product, which is stored in a storage medium and includes several instructions to enable a terminal device (which may be a mobile phone, a computer, a server, or a network device) to execute the methods according to the embodiments of the present disclosure.

The present disclosure further provides an apparatus for generating a panorama, which is used to implement the foregoing embodiments and preferred embodiments, and the description of the apparatus is omitted here. As used below, the term "module" may be a combination of software and/or hardware that implements a predetermined function. Although the means described in the embodiments below are preferably implemented in software, an implementation in hardware, or a combination of software and hardware is also possible and contemplated.

Fig. 7 is a block diagram of an apparatus for generating a panorama according to an embodiment of the present disclosure, and as shown in fig. 7, an apparatus 700 for generating a panorama includes:

the acquisition module 701 is used for acquiring a plurality of pictures by using the panoramic camera shooting assembly;

an obtaining module 702, configured to obtain estimated parameter information corresponding to multiple pictures, where the estimated parameter information is used to estimate an expansion effect and a position of an expansion map corresponding to each of the multiple pictures;

the unfolding module 703 is configured to unfold a plurality of pictures in a spherical projection manner to obtain a picture unfolding result;

and the generating module 704 is configured to perform splicing processing on the image expansion result by using a spherical splicing manner, so as to generate a target panorama.

Optionally, the obtaining module 702 is configured to obtain the estimated parameter information corresponding to multiple pictures, including: carrying out graph detection on each of the plurality of pictures to obtain a target graph corresponding to each picture; the method comprises the steps of carrying out feature detection on a target graph to extract a plurality of feature points, carrying out pose estimation based on the feature points, and determining estimation parameter information, or determining the estimation parameter information by using angular motion detection data acquired by an angular motion detection assembly, wherein the angular motion detection data is detection data synchronously recorded when a plurality of pictures are acquired by using a panoramic camera assembly.

Optionally, estimating the parameter information comprises: the panoramic shooting assembly comprises panoramic shooting assembly model parameters and a rotation matrix, wherein the panoramic shooting assembly model parameters are used for estimating the expansion effect of an expansion image corresponding to each of a plurality of images, and the rotation matrix is used for estimating the position of the expansion image in a target panoramic image.

Optionally, the obtaining module 702 is further configured to perform image detection on each of the multiple pictures, where obtaining the target image corresponding to each picture includes: adjusting the brightness and contrast of each of the plurality of pictures by utilizing the color level adjustment to obtain an adjustment result; searching a first center position from the adjustment result by adopting a first searching mode, and searching a second center position from the adjustment result by adopting a second searching mode; comparing the first distance with the second distance to obtain a comparison result, wherein the first distance is the distance between the first center position and the center position of the corresponding picture, and the second distance is the distance between the second center position and the center position of the corresponding picture; and acquiring a target graph corresponding to each picture based on the comparison result.

Optionally, the generating module 704 is configured to perform a splicing process on the image expansion result in a spherical splicing manner, and generating the target panorama includes: determining an overlapping part between each expanded image and the rest of the expanded images by using the image expansion result, and cutting the overlapping part to obtain a cutting result; searching for a splicing seam between adjacent expansion graphs in the cutting result; splicing the spliced seams in a spherical splicing mode to generate an initial panoramic image; and cutting the area to be cut in the initial panoramic image to obtain the target panoramic image.

Optionally, the generating module 704 is further configured to perform clipping processing on the region to be clipped in the initial panorama, and obtaining the target panorama includes: acquiring a mask corresponding to the initial panorama; searching for a first height and a second height in the height direction of the initial panorama based on the mask; determining an area to be cut by utilizing the first height and the second height; and cutting the area to be cut to obtain the target panoramic image.

It should be noted that, the above modules may be implemented by software or hardware, and for the latter, the following may be implemented, but not limited to: the modules are all positioned in the same processor; alternatively, the modules are respectively located in different processors in any combination.

According to an embodiment of the present disclosure, there is also provided an electronic device including a memory having stored therein computer instructions and at least one processor configured to execute the computer instructions to perform the steps in any of the above method embodiments.

Optionally, the electronic device may further include a transmission device and an input/output device, wherein the transmission device is connected to the processor, and the input/output device is connected to the processor.

Alternatively, in the present disclosure, the processor may be configured to execute the following steps by a computer program:

s1, collecting a plurality of pictures by using the panoramic camera shooting assembly;

s2, acquiring estimation parameter information corresponding to the multiple pictures, wherein the estimation parameter information is used for estimating the expansion effect of the expansion diagram corresponding to each picture in the multiple pictures and the position information of the expansion diagram;

s3, unfolding a plurality of pictures by adopting a spherical projection mode to obtain a picture unfolding result;

and S4, splicing the image expansion results in a spherical splicing mode to generate a target panoramic image.

Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments and optional implementation manners, and this embodiment is not described herein again.

According to an embodiment of the present disclosure, there is also provided a non-transitory computer readable storage medium having stored therein computer instructions, wherein the computer instructions are arranged to perform the steps in any of the above method embodiments when executed.

Alternatively, in the present embodiment, the above-mentioned non-transitory computer-readable storage medium may be configured to store a computer program for executing the steps of:

Alternatively, in the present embodiment, the non-transitory computer readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

The present disclosure also provides a computer program product according to an embodiment of the present disclosure. Program code for implementing the audio processing methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the program codes, when executed by the processor or controller, cause the functions/operations specified in the flowchart and/or block diagram to be performed. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.

In the above embodiments of the present disclosure, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.

In the embodiments provided in the present disclosure, it should be understood that the disclosed technology can be implemented in other ways. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units may be a logical division, and in actual implementation, there may be another division, for example, multiple units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, units or modules, and may be in an electrical or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present disclosure may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.

The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present disclosure may be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present disclosure. And the aforementioned storage medium includes: various media capable of storing program codes, such as a U disk, a Read Only Memory (ROM), a Random Access Memory (RAM), a removable hard disk, a magnetic disk, or an optical disk.

The foregoing is merely a preferred embodiment of the present disclosure, and it should be noted that modifications and embellishments could be made by those skilled in the art without departing from the principle of the present disclosure, and these should also be considered as the protection scope of the present disclosure.

Claims

1. A method of generating a panorama, comprising:

collecting a plurality of pictures by using a panoramic camera shooting component;

acquiring estimation parameter information corresponding to the plurality of pictures, wherein the estimation parameter information is used for estimating the expansion effect of the expansion diagram corresponding to each picture in the plurality of pictures and the position information of the expansion diagram;

unfolding the plurality of pictures by adopting a spherical projection mode to obtain a picture unfolding result;

and splicing the image unfolding results in a spherical splicing mode to generate a target panoramic image.

2. The method of claim 1, wherein obtaining the estimated parameter information corresponding to the plurality of pictures comprises:

carrying out graph detection on each of the plurality of pictures to obtain a target graph corresponding to each picture;

and performing feature detection on the target graph to extract a plurality of feature points, performing pose estimation based on the plurality of feature points, and determining the estimation parameter information, or determining the estimation parameter information by using angular motion detection data acquired by an angular motion detection assembly, wherein the angular motion detection data is detection data synchronously recorded when the panoramic camera assembly is used for acquiring the plurality of pictures.

3. The method of claim 1 or 2, wherein the estimating parameter information comprises: the panoramic shooting assembly model parameters are used for estimating the expansion effect of the expansion image corresponding to each picture in the multiple pictures, and the rotation matrix is used for estimating the position of the expansion image in the target panoramic image.

4. The method according to claim 2, wherein the performing the pattern detection on each of the plurality of pictures, and the obtaining the target pattern corresponding to each picture comprises:

adjusting the brightness and contrast of each of the plurality of pictures by utilizing the color level adjustment to obtain an adjustment result;

searching a first central position from the adjustment result by adopting a first searching mode, and searching a second central position from the adjustment result by adopting a second searching mode;

comparing a first distance with a second distance to obtain a comparison result, wherein the first distance is a distance between the first center position and the center position of the corresponding picture, and the second distance is a distance between the second center position and the center position of the corresponding picture;

and acquiring a target graph corresponding to each picture based on the comparison result.

5. The method of claim 1, wherein the stitching the image expansion result by using the spherical stitching method to generate the target panorama comprises:

determining an overlapping part between each expansion diagram and the rest expansion diagrams by using the image expansion result, and cutting the overlapping part to obtain a cutting result;

searching for a splicing seam between adjacent expansion graphs in the cutting result;

splicing the splicing seams by adopting the spherical splicing mode to generate an initial panoramic image;

and cutting the area to be cut in the initial panoramic image to obtain the target panoramic image.

6. The method of claim 5, wherein the cropping the area to be cropped in the initial panorama to obtain the target panorama comprises:

acquiring a mask corresponding to the initial panorama;

finding a first height and a second height in a height direction of the initial panorama based on the mask;

determining the area to be cut by utilizing the first height and the second height;

and cutting the area to be cut to obtain the target panoramic image.

7. An apparatus to generate a panorama, comprising:

the acquisition module is used for acquiring a plurality of pictures by using the panoramic camera shooting assembly;

an obtaining module, configured to obtain estimated parameter information corresponding to the multiple pictures, where the estimated parameter information is used to estimate an expansion effect of an expansion map corresponding to each of the multiple pictures and a position of the expansion map;

the unfolding module is used for unfolding the plurality of pictures in a spherical projection mode to obtain a picture unfolding result;

and the generating module is used for splicing the image unfolding result in a spherical splicing mode to generate a target panoramic image.

8. An electronic device, comprising:

a panoramic camera component;

an angular motion detection assembly;

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,

the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-6.

9. A non-transitory computer readable storage medium having stored thereon computer instructions for causing the computer to perform the method of any one of claims 1-6.

10. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1-6.