WO2022037285A1

WO2022037285A1 - Camera extrinsic calibration method and apparatus

Info

Publication number: WO2022037285A1
Application number: PCT/CN2021/104424
Authority: WO
Inventors: 曹炎培
Original assignee: 北京达佳互联信息技术有限公司
Priority date: 2020-08-20
Filing date: 2021-07-05
Publication date: 2022-02-24
Also published as: CN114170302A

Abstract

A camera extrinsic calibration method and apparatus. The method comprises: performing multi-camera photographing on a target object to obtain a plurality of target object images at different camera viewing angles, each of the camera viewing angles corresponding to one of the plurality of cameras (S210); inputting the plurality of target object images into a pre-trained three-dimensional reconstruction network to generate a three-dimensional mesh model of the target object in each of the target object images (S220); determining a vertex coordinate set of the three-dimensional mesh model at each of the camera viewing angles in a corresponding camera coordinate system (S230); and according to a relative position relationship of the vertex coordinate sets corresponding to each of the camera viewing angles in the same coordinate system, determining an extrinsic calibration value of a camera corresponding to each of the camera viewing angles (S240).

Description

相机外参标定方法及装置Camera external parameter calibration method and device

本申请要求于2020年8月20日提交至中国专利局、申请号为202010844425.6的中国专利申请的优先权，其全部内容通过引用结合在本申请中。This application claims the priority of the Chinese Patent Application No. 202010844425.6 filed with the China Patent Office on August 20, 2020, the entire contents of which are incorporated herein by reference.

技术领域technical field

本公开涉及计算机技术领域，尤其涉及一种相机外参标定方法、装置、电子设备及存储介质。The present disclosure relates to the field of computer technology, and in particular, to a method, device, electronic device and storage medium for calibrating external parameters of a camera.

背景技术Background technique

增强现实(AugmentedReality，AR)是将虚拟物体叠加在真实场景中并加以显示、交互的***。目前的AR技术已经成为虚拟现实研究中的一个重要领域,也是人机界面技术发展的一个重要方向。Augmented Reality (AR) is a system that superimposes virtual objects in real scenes and displays and interacts with them. At present, AR technology has become an important field in virtual reality research, and it is also an important direction of human-machine interface technology development.

现有技术往往采用利用外部标记物对多相机外参进行人工标定方案：首先需要人工设计并搭建外部标记物；在确定多相机***的安装配置后，需要操作员或机械臂带动外部标记物在拍摄场景范围内移动，并同步拍摄相机数据；接着在离线处理阶段，在采集到的各视角标定数据中检测标记物中的特征点，并利用光束平差法(BundleAdjustment)对各相机的外参进行求解。Existing technologies often use external markers to manually calibrate multi-camera external parameters: first, external markers need to be manually designed and built; after the installation configuration of the multi-camera system is determined, the operator or robotic arm needs to drive the external markers to Move within the scope of the shooting scene, and shoot the camera data synchronously; then in the offline processing stage, detect the feature points in the markers in the collected calibration data of each viewing angle, and use the beam adjustment method (BundleAdjustment) to adjust the external parameters of each camera. to solve.

发明内容SUMMARY OF THE INVENTION

本公开提供一种相机外参标定方法、装置、电子设备及存储介质。The present disclosure provides a camera external parameter calibration method, device, electronic device and storage medium.

根据本公开实施例的第一方面，提供一种相机外参标定方法，所述方法包括：According to a first aspect of the embodiments of the present disclosure, there is provided a method for calibrating external parameters of a camera, the method comprising:

对目标对象进行多相机拍摄，以获得不同相机视角下的多张目标对象图像；每个所述相机视角对应于所述多相机中的一个相机；Shooting the target object with multiple cameras to obtain multiple images of the target object under different camera perspectives; each of the camera perspectives corresponds to one camera in the multiple cameras;

将多张所述目标对象图像输入预训练的三维重建网络，生成所述目标对象在每张所述目标对象图像中的三维网格模型；Inputting multiple images of the target object into a pre-trained 3D reconstruction network to generate a 3D mesh model of the target object in each of the target object images;

确定各个所述相机视角下的所述三维网格模型在对应的相机坐标系中的顶点坐标集合；determining a vertex coordinate set of the three-dimensional mesh model in the corresponding camera coordinate system under each of the camera perspectives;

根据各个所述相机视角对应的顶点坐标集合在同一坐标系中的相对位置关系，确定各个所述相机视角对应相机的外参标定值。According to the relative positional relationship of the vertex coordinate sets corresponding to each of the camera angles of view in the same coordinate system, the external parameter calibration values of the cameras corresponding to each of the camera angles of view are determined.

在一些实施例中，所述根据各个所述相机视角对应的顶点坐标集合在同一坐标系中的相对位置关系，确定各个所述相机视角对应相机的外参标定值，包括；In some embodiments, the determining, according to the relative positional relationship of the vertex coordinate sets corresponding to each of the camera perspectives in the same coordinate system, determines the external parameter calibration value of the camera corresponding to each of the camera perspectives, including;

将各个所述相机视角中的其中一个相机视角作为基准相机视角；Using one of the camera angles of view as the reference camera angle of view;

基于所述基准相机视角对应的顶点坐标集合与各个所述相机视角对应的顶点坐标集合之间的顶点位置差异，确定各个所述相机视角对应相机的外参标定值。Based on the vertex position difference between the vertex coordinate set corresponding to the reference camera angle of view and the vertex coordinate set corresponding to each of the camera angles of view, the external parameter calibration value of the camera corresponding to each of the camera angles of view is determined.

在一些实施例中，所述基于所述基准相机视角对应的顶点坐标集合与各个所述相机视角对应的顶点坐标集合之间的顶点位置差异，确定各个所述相机视角对应相机的外参标定值，包括；In some embodiments, the extrinsic parameter calibration value of the camera corresponding to each camera perspective is determined based on the vertex position difference between the vertex coordinate set corresponding to the reference camera perspective and the vertex coordinate set corresponding to each of the camera perspectives ,include;

分别求解将各个所述相机视角对应的顶点坐标集合对齐到所述基准相机视角对应的顶点坐标集合所在坐标系上时所需的刚体变换；respectively solving the rigid body transformation required when aligning the vertex coordinate sets corresponding to each of the camera perspectives to the coordinate system where the vertex coordinate sets corresponding to the reference camera perspectives are located;

将求解得到的各个所述相机视角对应的刚体变换，作为各个所述相机视角对应相机的外参标定值。The rigid body transformation corresponding to each of the camera angles obtained through the solution is used as an external parameter calibration value of the camera corresponding to each of the camera angles.

在一些实施例中，该相机外参标定方法还包括；In some embodiments, the camera extrinsic parameter calibration method further includes;

将多张所述目标对象图像输入预训练的关键点检测网络，得到所述目标对象在每张所述目标对象图像中的二维关键点；Inputting a plurality of the target object images into a pre-trained key point detection network to obtain two-dimensional key points of the target object in each of the target object images;

基于各个所述相机视角对应的所述二维关键点的位置信息，对各个所述相机视角对应相机的外参标定值进行优化，得到各个所述相机视角对应相机的优化外参标定值。Based on the position information of the two-dimensional key points corresponding to each of the camera angles of view, the external parameter calibration values of the cameras corresponding to each of the camera angles of view are optimized, and the optimized external parameter calibration values of the cameras corresponding to each of the camera angles of view are obtained.

在一些实施例中，所述基于各个所述相机视角对应的所述二维关键点的位置信息，对各个所述相机视角对应相机的外参标定值进行优化，得到各个所述相机视角对应相机的优化外参标定值，包括；In some embodiments, based on the position information of the two-dimensional key points corresponding to each of the camera perspectives, the external parameter calibration values of the cameras corresponding to each of the camera perspectives are optimized to obtain the camera corresponding to each of the camera perspectives. The optimized external parameter calibration values, including;

基于多张所述目标对象图像，确定各个所述相机视角下的所述二维关键点在对应的图像坐标系中的关键点二维坐标；Determine, based on a plurality of the target object images, the two-dimensional coordinates of the key points in the corresponding image coordinate system of the two-dimensional key points under each of the camera perspectives;

在各个所述相机视角对应的顶点坐标集合中，确定各个所述相机视角下的所述二维关键点在对应的相机坐标系中的关键点三维坐标；In the vertex coordinate set corresponding to each of the camera perspectives, determine the three-dimensional coordinates of the key points in the corresponding camera coordinate system of the two-dimensional key points under each of the camera perspectives;

根据各个所述相机视角对应的所述关键点三维坐标和各个所述相机视角下对应的所述关键点二维坐标，对各个所述相机视角对应相机的外参标定值进行优化，得到各个所述相机视角对应相机的优化外参标定值。According to the three-dimensional coordinates of the key points corresponding to each of the camera perspectives and the two-dimensional coordinates of the key points corresponding to each of the camera perspectives, the external parameter calibration values of the cameras corresponding to each of the camera perspectives are optimized to obtain each of the camera perspectives. The camera angle of view corresponds to the optimized external parameter calibration value of the camera.

在一些实施例中，所述根据各个所述相机视角对应的所述关键点三维坐标和各个所述相机视角下对应的所述关键点二维坐标，对各个所述相机视角对应相机的外参标定值进行优化，得到各个所述相机视角对应相机的优化外参标定值，包括；In some embodiments, according to the three-dimensional coordinates of the key points corresponding to the camera perspectives and the two-dimensional coordinates of the key points corresponding to the camera perspectives, the external parameters of the cameras corresponding to the camera perspectives are The calibration value is optimized to obtain the optimized external parameter calibration value of the camera corresponding to each of the camera angles of view, including;

根据各个所述相机视角对应的所述关键点三维坐标和各个所述相机视角下对应的所述关键点二维坐标，确定各个所述相机视角对应的相机在采用所述外参标定值时的投影误差；所述投影误差为所述相机视角对应的相机将所述关键点三维坐标投影至所述相机的像平面而得到的二维坐标与所述关键点二维坐标之间的误差；According to the three-dimensional coordinates of the key points corresponding to the respective camera perspectives and the two-dimensional coordinates of the key points corresponding to the respective camera perspectives, determine the camera corresponding to each camera perspective when the external parameter calibration value is used. Projection error; the projection error is the error between the two-dimensional coordinates obtained by projecting the three-dimensional coordinates of the key point onto the image plane of the camera and the two-dimensional coordinates of the key point by the camera corresponding to the camera angle of view;

基于各个所述相机视角对应的相机在采用所述外参标定值时的投影误差，对各个所述相机视角对应相机的所述外参标定值进行调整，得到各个所述相机视角对应相机的调整后外参标定值，作为各个所述相机视角对应相机的优化外参标定值；Based on the projection errors of the cameras corresponding to the camera perspectives when the external parameter calibration values are used, the external parameter calibration values of the cameras corresponding to the camera perspectives are adjusted to obtain the adjustment of the cameras corresponding to the camera perspectives. The back external parameter calibration value is used as the optimized external parameter calibration value of each camera corresponding to the camera angle of view;

其中，所述相机视角对应的相机在采用所述优化外参标定值时的投影误差满足预设条件。Wherein, the projection error of the camera corresponding to the camera angle of view when the optimized external parameter calibration value is adopted satisfies a preset condition.

在一些实施例中，所述根据各个所述相机视角对应的所述关键点三维坐标和各个所述相机视角下对应的所述关键点二维坐标，确定各个所述相机视角对应的相机的投影误差，包括：In some embodiments, the projection of the camera corresponding to each of the camera perspectives is determined according to the three-dimensional coordinates of the key points corresponding to the respective camera perspectives and the two-dimensional coordinates of the key points corresponding to the respective camera perspectives errors, including:

通过预设的投影函数和各个所述相机视角对应相机的外参标定值，分别将各个所述相机视角对应的所述关键点三维坐标投影至对应相机的像平面上，得到各个所述相机视角对应的所述二维关键点在对应所述相机的像平面上的投影点；According to the preset projection function and the external parameter calibration values of the cameras corresponding to the camera angles of view, the three-dimensional coordinates of the key points corresponding to the camera angles of view are respectively projected onto the image plane of the corresponding camera to obtain the camera angles of view. the projection point of the corresponding two-dimensional key point on the image plane corresponding to the camera;

确定各个所述相机视角对应的所述投影点在对应的图像坐标系中的投影点二维坐标；determining the two-dimensional coordinates of the projection point in the corresponding image coordinate system of the projection point corresponding to each of the camera perspectives;

根据每个所述相机视角对应的所述投影点二维坐标和对应的所述关键点二维坐标之间的差异，确定各个所述相机视角对应的相机的投影误差。According to the difference between the two-dimensional coordinates of the projection point corresponding to each of the camera perspectives and the corresponding two-dimensional coordinates of the key points, the projection errors of the cameras corresponding to the respective camera perspectives are determined.

根据本公开实施例的第二方面，提供一种相机外参标定装置，包括：According to a second aspect of the embodiments of the present disclosure, there is provided an apparatus for calibrating external parameters of a camera, including:

获取单元，被配置为执行对目标对象进行多相机拍摄，以获得不同相机视角下的多张目标对象图像；每个所述相机视角对应于所述多相机中的一个相机；an acquisition unit, configured to perform multi-camera shooting of the target object, so as to obtain a plurality of target object images under different camera perspectives; each of the camera perspectives corresponds to one camera in the multi-camera;

重建单元，被配置为执行将多张所述目标对象图像输入预训练的三维重建网络，生成所述目标对象在每张所述目标对象图像中的三维网格模型；a reconstruction unit, configured to input multiple images of the target object into a pre-trained 3D reconstruction network, and generate a 3D mesh model of the target object in each of the target object images;

确定单元，被配置为确定各个所述相机视角下的所述三维网格模型在对应的相机坐标系中的顶点坐标集合；a determining unit, configured to determine a vertex coordinate set of the three-dimensional mesh model in the corresponding camera coordinate system under each of the camera perspectives;

标定单元，被配置为执行根据各个所述相机视角对应的顶点坐标集合在同一坐标系中的相对位置关系，确定各个所述相机视角对应相机的外参标定值。The calibration unit is configured to determine the extrinsic parameter calibration value of the camera corresponding to each of the camera perspectives according to the relative positional relationship of the vertex coordinate sets corresponding to each of the camera perspectives in the same coordinate system.

在一些实施例中，所述标定单元，被配置为执行将各个所述相机视角中的其中一个相机视角作为基准相机视角；基于所述基准相机视角对应的顶点坐标集合与各个所述相机视角对应的顶点坐标集合之间的顶点位置差异，确定各个所述相机视角对应相机的外参标定值。In some embodiments, the calibration unit is configured to perform using one of the camera perspectives as a reference camera perspective; a vertex coordinate set corresponding to the reference camera perspective corresponds to each of the camera perspectives The vertex position difference between the vertex coordinate sets of the vertices is determined to determine the external parameter calibration value of the camera corresponding to each of the camera angles of view.

在一些实施例中，所述标定单元，被配置为执行分别求解将各个所述相机视角对应的顶点坐标集合对齐到所述基准相机视角对应的顶点坐标集合所在坐标系上时所需的刚体变换；将求解得到的各个所述相机视角对应的刚体变换，作为各个所述相机视角对应相机的外参标定值。In some embodiments, the calibration unit is configured to perform a rigid body transformation required to separately solve the rigid body transformation required when aligning the vertex coordinate sets corresponding to each of the camera viewpoints to the coordinate system where the vertex coordinate sets corresponding to the reference camera viewpoints are located ; Use the obtained rigid body transformation corresponding to each of the camera angles of view as the external parameter calibration values of the cameras corresponding to each of the camera angles of view.

在一些实施例中，所述装置还包括；关键点检测单元，被配置为执行将多张所述目标对象图像输入预训练的关键点检测网络，得到所述目标对象在每张所述目标对象图像中的二维关键点；优化单元，被配置为执行基于各个所述相机视角对应的所述二维关键点的位置信息，对各个所述相机视角对应相机的外参标定值进行优化，得到各个所述相机视角对应相机的优化外参标定值。In some embodiments, the apparatus further includes: a key point detection unit, configured to input a plurality of images of the target object into a pre-trained key point detection network, and obtain the target object in each image of the target object. The two-dimensional key points in the image; the optimization unit is configured to perform, based on the position information of the two-dimensional key points corresponding to the respective camera angles of view, optimize the external parameter calibration values of the cameras corresponding to the respective camera angles of view, and obtain Each of the camera angles of view corresponds to an optimized external parameter calibration value of the camera.

在一些实施例中，所述优化单元，被配置为执行基于多张所述目标对象图像，确定各个所述相机视角下的所述二维关键点在对应的图像坐标系中的关键点二维坐标；在各个所述相机视角对应的顶点坐标集合中，确定各个所述相机视角下的所述二维关键点在对应的相机坐标系中的关键点三维坐标；根据各个所述相机视角对应的所述关键点三维坐标和各个所述相机视角下对应的所述关键点二维坐标，对各个所述相机视角对应相机的外参标定值进行优化，得到各个所述相机视角对应相机的优化外参标定值。In some embodiments, the optimization unit is configured to determine, based on a plurality of images of the target object, a two-dimensional key point of the two-dimensional key point in the corresponding image coordinate system under each of the camera perspectives coordinates; in the vertex coordinate set corresponding to each of the camera perspectives, determine the three-dimensional coordinates of the key points in the corresponding camera coordinate system of the two-dimensional key points under each of the camera perspectives; according to the corresponding camera perspectives The three-dimensional coordinates of the key points and the two-dimensional coordinates of the key points corresponding to each of the camera perspectives are optimized, and the external parameter calibration values of the cameras corresponding to each of the camera perspectives are optimized to obtain the optimized external parameters of the cameras corresponding to each of the camera perspectives. parameter calibration value.

在一些实施例中，所述优化单元，被配置为执行根据各个所述相机视角对应的所述关键点三维坐标和各个所述相机视角下对应的所述关键点二维坐标，确定各个所述相机视角对应的相机在采用所述外参标定值时的投影误差；所述投影误差为所述相机视角对应的相机将所述关键点三维坐标投影至所述相机的像平面而得到的二维坐标与所述关键点二维坐标之间的误差；基于各个所述相机视角对应的相机在采用所述外参标定值时的投影误差，对各个所述相机视角对应相机的所述外参标定值进行调整，得到各个所述相机视角对应相机的调整后外参标定值，作为各个所述相机视角对应相机的优化外参标定值；其中，所述相机视角对应的相机在采用所述优化外参标定值时的投影误差满足预设条件。In some embodiments, the optimization unit is configured to determine each of the key points according to the three-dimensional coordinates of the key points corresponding to the respective camera perspectives and the two-dimensional coordinates of the key points corresponding to the respective camera perspectives. The projection error of the camera corresponding to the camera angle of view when the external parameter calibration value is used; the projection error is the two-dimensional image obtained by projecting the three-dimensional coordinates of the key point to the image plane of the camera by the camera corresponding to the camera angle of view The error between the coordinates and the two-dimensional coordinates of the key point; based on the projection error of the camera corresponding to each of the camera perspectives when the external parameter calibration value is used, the external parameter calibration of the camera corresponding to each of the camera perspectives to obtain the adjusted extrinsic parameter calibration values of the cameras corresponding to the camera angles of view, as the optimized external parameter calibration values of the cameras corresponding to the camera angles of view; wherein, the cameras corresponding to the camera angles of view use the optimized external parameter calibration values. The projection error of the parameter calibration value satisfies the preset condition.

在一些实施例中，所述优化单元，被配置为执行通过预设的投影函数和各个所述相机视角对应相机的外参标定值，分别将各个所述相机视角对应的所述关键点三维坐标投影至对应相机的像平面上，得到各个所述相机视角对应的所述二维关键点在对应所述相机的像平面上的投影点；确定各个所述相机视角对应的所述投影点在对应的图像坐标系中的投影点二维坐标；根据每个所述相机视角对应的所述投影点二维坐标和对应的所述关键点二维坐标之间的差异，确定各个所述相机视角对应的相机的投影误差。In some embodiments, the optimization unit is configured to perform a preset projection function and an external parameter calibration value of the camera corresponding to each of the camera perspectives, respectively, the three-dimensional coordinates of the key points corresponding to each of the camera perspectives Projecting onto the image plane of the corresponding camera to obtain the projection points of the two-dimensional key points corresponding to the camera perspectives on the image plane corresponding to the cameras; determining that the projection points corresponding to the camera perspectives are in the corresponding The two-dimensional coordinates of the projection point in the image coordinate system of the The projection error of the camera.

根据本公开实施例的第三方面，提供一种电子设备，包括存储器和处理器，所述存储器存储有计算机程序，所述处理器执行所述计算机程序时实现如第一方面或第一方面的任意实施例所述的相机外参标定方法。According to a third aspect of the embodiments of the present disclosure, an electronic device is provided, including a memory and a processor, the memory stores a computer program, and the processor implements the first aspect or the first aspect when executing the computer program The camera extrinsic parameter calibration method described in any embodiment.

根据本公开实施例的第四方面，提供一种存储介质，其上存储有计算机程序，所述计算机程序被处理器执行时实现如第一方面或第一方面的任意实施例所述的相机外参标定方法。According to a fourth aspect of the embodiments of the present disclosure, there is provided a storage medium on which a computer program is stored, and when the computer program is executed by a processor, implements the external camera according to the first aspect or any embodiment of the first aspect. parameter calibration method.

根据本公开实施例的第五方面，提供一种计算机程序产品，所述程序产品包括计算机程序，所述计算机程序存储在可读存储介质中，设备的至少一个处理器从所述可读存储介质读取并执行所述计算机程序，使得设备执行第一方面的任一项实施例中所述的相机外参标定方法。According to a fifth aspect of the embodiments of the present disclosure, there is provided a computer program product, the program product comprising a computer program, the computer program being stored in a readable storage medium, and at least one processor of a device from the readable storage medium The computer program is read and executed, so that the device executes the camera extrinsic parameter calibration method described in any one of the embodiments of the first aspect.

本公开的实施例通过对目标对象进行多相机拍摄，以获得不同相机视角下的多张目标对象图像；并通过预训练的三维重构网络，生成所述目标对象在每张所述目标对象图像中的三维网格模型；再确定各个所述相机视角下的所述三维网格模型在对应的相机坐标系中的顶点坐标集合；最后，根据各个所述相机视角对应的顶点坐标集合在同一坐标系中的相对位置关系，即可确定出各个所述相机视角对应相机的外参标定值；如此，无需额外设计并搭建外部标记物，并无需在确定多相机***的安装配置后，带动外部标记物在拍摄场景范围内移动，并同步拍摄相机数据；并进行一系列地离线标定处理，避免了传统对多相机***进行标定时的一系列繁琐且耗时耗力的操作，提高了在每次改变相机配置后对多相机外参进行标定时的标定效率。The embodiment of the present disclosure obtains multiple images of the target object under different camera perspectives by shooting the target object with multiple cameras; Then determine the vertex coordinate set in the corresponding camera coordinate system of the three-dimensional mesh model under each of the camera perspectives; finally, according to the vertex coordinates corresponding to each of the camera perspectives, set at the same coordinate The relative positional relationship in the system can be used to determine the external parameter calibration value of the camera corresponding to each of the camera angles; in this way, there is no need to additionally design and build external markers, and it is not necessary to drive the external markers after determining the installation configuration of the multi-camera system. The object moves within the shooting scene, and the camera data is captured synchronously; and a series of offline calibration processing is performed, which avoids a series of tedious and time-consuming operations when traditionally calibrating a multi-camera system, and improves the performance of each time. Calibration efficiency when calibrating multi-camera extrinsic parameters after changing camera configuration.

应当理解的是，以上的一般描述和后文的细节描述仅是示例性和解释性的，并不能限制本公开。It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure.

附图说明Description of drawings

图1是根据一示例性实施例示出的一种相机外参标定方法的应用环境图。FIG. 1 is an application environment diagram of a method for calibrating external parameters of a camera according to an exemplary embodiment.

图2是根据一示例性实施例示出的一种相机外参标定方法的流程图。Fig. 2 is a flow chart of a method for calibrating external parameters of a camera according to an exemplary embodiment.

图3是根据一示例性实施例示出的另一种相机外参标定方法的流程图。Fig. 3 is a flow chart of another method for calibrating external parameters of a camera according to an exemplary embodiment.

图4是根据一示例性实施例示出的一种相机外参标定方法的流程框图。Fig. 4 is a flowchart of a method for calibrating external parameters of a camera according to an exemplary embodiment.

图5是根据一示例性实施例示出的一种相机外参标定装置的框图。Fig. 5 is a block diagram of an apparatus for calibrating external parameters of a camera according to an exemplary embodiment.

图6是根据一示例性实施例示出的一种电子设备的内部结构图。Fig. 6 is an internal structure diagram of an electronic device according to an exemplary embodiment.

具体实施方式detailed description

为了使本领域普通人员更好地理解本公开的技术方案，下面将结合附图，对本公开实施例中的技术方案进行清楚、完整地描述。In order to make those skilled in the art better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

需要说明的是，本公开的说明书和权利要求书及上述附图中的术语“第一”、“第二”等是用于区别类似的对象，而不必用于描述特定的顺序或先后次序。应该理解这样使用的数据在适当情况下可以互换，以便这里描述的本公开的实施例能够以除了在这里图示或描述的那些以外的顺序实施。以下示例性实施例中所描述的实施方式并不代表与本公开相一致的所有实施方式。相反，它们仅是与如所附权利要求书中所详述的、本公开的一些方面相一致的装置和方法的例子。It should be noted that the terms "first", "second" and the like in the description and claims of the present disclosure and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific sequence or sequence. It is to be understood that the data so used may be interchanged under appropriate circumstances such that the embodiments of the disclosure described herein can be practiced in sequences other than those illustrated or described herein. The implementations described in the illustrative examples below are not intended to represent all implementations consistent with this disclosure. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present disclosure as recited in the appended claims.

本公开所提供的相机外参标定方法，可以应用于如图1所示的应用环境中。其中，多相机120通过网络与电子设备110进行通信。其中，电子设备110可以是但不限于各种个人计算机、笔记本电脑、智能手机、平板电脑和便携式可穿戴设备。The camera external parameter calibration method provided by the present disclosure can be applied to the application environment shown in FIG. 1 . The multi-camera 120 communicates with the electronic device 110 through the network. The electronic device 110 may be, but not limited to, various personal computers, notebook computers, smart phones, tablet computers and portable wearable devices.

图2是根据一示例性实施例示出的一种相机外参标定方法的流程图，该相机外参标定方法可以由图1的电子设备110执行，如图2所示，该相机外参标定方法包括以下步骤。FIG. 2 is a flowchart of a method for calibrating external parameters of a camera according to an exemplary embodiment. The method for calibrating external parameters of a camera may be executed by the electronic device 110 in FIG. 1 . As shown in FIG. 2 , the method for calibrating external parameters of a camera Include the following steps.

在步骤S210中，对目标对象进行多相机拍摄，以获得不同相机视角下的多张目标对象图像。In step S210, the target object is photographed with multiple cameras to obtain multiple images of the target object under different camera angles of view.

其中，目标对象可以是指需要进行多视角动作捕捉的运动物体。例如，动物、人体等运动物体。The target object may refer to a moving object that needs multi-view motion capture. For example, moving objects such as animals and humans.

其中，目标对象图像可以是指包含有目标对象的彩色图像。The target object image may refer to a color image containing the target object.

在一些实施例中，以目标对象为人体为例，当被采集人员进入多视角动作捕捉***的采集区域后，电子设备控制多视角动作捕捉***的多个拍摄相机开始对被采集人员进行多相机拍摄，进而供电子设备获得被采集人员在不同相机视角下的多张彩色图像即目标对象图像。其中，每个相机视角对应于多相机中的一个拍摄相机。In some embodiments, taking the target object as a human body as an example, when the person to be captured enters the collection area of the multi-view motion capture system, the electronic device controls multiple cameras of the multi-view motion capture system to start multi-camera scanning of the captured person. Shooting, and then for the electronic device to obtain a plurality of color images of the collected person under different camera perspectives, that is, the target object image. Wherein, each camera angle of view corresponds to one shooting camera in the multi-camera.

在步骤S220中，通过预训练的三维重构网络，生成目标对象在每张目标对象图像中的三维网格模型。In step S220, a 3D mesh model of the target object in each target object image is generated through the pre-trained 3D reconstruction network.

其中，预训练的三维重构网络可以是指基于深度学习训练得到的用于对输入图像中的物体进行三维网格模型重构的神经网络。实际应用中，电子设备通过预训练的三维重构网络对输入彩色图像中人体的三维形状及姿态进行估计与重建，得到图像中人体在相机坐标系下的三维空间表示。The pre-trained 3D reconstruction network may refer to a neural network trained based on deep learning and used to reconstruct a 3D mesh model of an object in an input image. In practical applications, the electronic device estimates and reconstructs the three-dimensional shape and posture of the human body in the input color image through the pre-trained three-dimensional reconstruction network, and obtains the three-dimensional spatial representation of the human body in the image in the camera coordinate system.

在一些实施例中，当电子设备获得被采集人员在不同相机视角下的多张彩色图像即目标对象图像后，电子设备则将各张目标对象图像输入至预训练的三维重构网络，通过预训练的三维重构网络，对被采集人员的三维形状及姿态进行估计与重建，得到被采集人员在每张目标对象图像中的三维网格模型。In some embodiments, after the electronic device obtains multiple color images of the captured person under different camera angles, that is, the target object image, the electronic device inputs each target object image into a pre-trained three-dimensional reconstruction network, through the pre-trained 3D reconstruction network. The trained 3D reconstruction network estimates and reconstructs the 3D shape and posture of the collected person, and obtains the 3D mesh model of the collected person in each target object image.

在步骤S230中，确定各个相机视角下的三维网格模型在对应的相机坐标系中的顶点坐标集合。In step S230, a vertex coordinate set in the corresponding camera coordinate system of the three-dimensional mesh model under each camera viewing angle is determined.

在一些实施例中，当电子设备得到被采集人员在每张目标对象图像中的三维网格模型后，电子设备则确定各个相机视角下的三维网格模型在对应的相机坐标系中的顶点坐标集合。In some embodiments, after the electronic device obtains the 3D mesh model of the captured person in each target object image, the electronic device determines the vertex coordinates of the 3D mesh model in the corresponding camera coordinate system under each camera view angle gather.

其中，各个相机视角下的三维网格模型在对应的相机坐标系中的顶点坐标集合可以表示为{M ₀,M ₁,...,M _i}： Among them, the vertex coordinate set of the three-dimensional mesh model in the corresponding camera coordinate system under each camera perspective can be expressed as {M ₀ , M ₁ ,...,M _i }:

其中，M _i表示第i个相机坐标系(或相机视角)下的三维网格模型的顶点坐标集合。 Wherein, M _i represents the vertex coordinate set of the 3D mesh model in the ith camera coordinate system (or camera view angle).

在步骤S240中，根据各个相机视角对应的顶点坐标集合在同一坐标系中的相对位置关系，确定各个相机视角对应相机的外参标定值。In step S240, according to the relative positional relationship of the vertex coordinate sets corresponding to each camera angle of view in the same coordinate system, the external parameter calibration value of the camera corresponding to each camera angle of view is determined.

其中，相机外参可以是指与对应相机视角对应的相机的外参数，即相机位姿(R,t)。The camera extrinsic parameter may refer to an extrinsic parameter of the camera corresponding to the corresponding camera angle of view, that is, the camera pose (R, t).

在一些实施例中，基于电子设备获取到的各个相机视角下的三维网格模型在对应的相机坐标系中的顶点坐标集合，电子设备确定各个相机视角对应的顶点坐标集合在同一坐标系中的相对位置关系，并基于各个相机视角对应的顶点坐标集合在同一坐标系中的相对位置关系，确定各个相机视角对应相机的外参标定值。In some embodiments, based on the vertex coordinate sets of the three-dimensional mesh models in the corresponding camera coordinate systems obtained by the electronic device under each camera perspective, the electronic device determines the vertex coordinates corresponding to each camera perspective in the same coordinate system. The relative positional relationship is determined, and based on the relative positional relationship of the vertex coordinate sets corresponding to each camera perspective in the same coordinate system, the external parameter calibration values of the cameras corresponding to each camera perspective are determined.

在一些实施例中，电子设备可以将各个相机视角对应的顶点坐标集合统一至同一坐标系下，然后，电子设备在基于各个相机视角对应的顶点坐标集合在同一坐标系中的相对位置关系，确定出各个相机视角对应的相机之间的相对位姿关系，进而确定各个相机视角对应相机的外参标定值。In some embodiments, the electronic device may unify the vertex coordinate sets corresponding to each camera angle of view into the same coordinate system, and then the electronic device determines the relative positional relationship between the vertex coordinate sets corresponding to each camera angle of view in the same coordinate system to determine The relative pose relationship between the cameras corresponding to each camera angle of view is obtained, and then the external parameter calibration value of the camera corresponding to each camera angle of view is determined.

实际应用中，电子设备利用计算机图形处理器(GPU)可实时在线完成上述相机外参标定方法，如此，在每次改变多视角动作捕捉***的相机配置后，被采集人员可直接进入采集区域拍摄，而无需进行繁琐的预标定过程。In practical applications, the electronic device can use the computer graphics processing unit (GPU) to complete the above-mentioned camera external parameter calibration method online in real time. In this way, after each change of the camera configuration of the multi-view motion capture system, the person being collected can directly enter the collection area to shoot. , without the need for a tedious pre-calibration process.

本申请实施例的技术方案，通过对目标对象进行多相机拍摄，以获得不同相机视角下的多张目标对象图像；并通过预训练的三维重构网络，生成所述目标对象在每张所述目标对象图像中的三维网格模型；再确定各个所述相机视角下的所述三维网格模型在对应的相机坐标系中的顶点坐标集合；最后，根据各个所述相机视角对应的顶点坐标集合在同一坐标系中的相对位置关系，即可确定出各个所述相机视角对应相机的外参标定值；如此，无需额外设计并搭建外部标记物，并无需在确定多相机***的安装配置后，带动外部标记物在拍摄场景范围内移动，并同步拍摄相机数据；并进行一系列地离线标定处理，避免了传统对多相机***进行标定时的一系列繁琐且耗时耗力的操作，提高了在每次改变相机配置后对多相机外参进行标定时的标定效率。The technical solution of the embodiment of the present application is to obtain multiple images of the target object from different camera perspectives by shooting the target object with multiple cameras; The three-dimensional mesh model in the target object image; then determine the vertex coordinate set in the corresponding camera coordinate system of the three-dimensional mesh model under each of the camera perspectives; finally, according to the vertex coordinate set corresponding to each of the camera perspectives The relative positional relationship in the same coordinate system can determine the external parameter calibration value of each camera corresponding to the camera; in this way, there is no need to additionally design and build external markers, and there is no need to determine the installation configuration of the multi-camera system. It drives the external markers to move within the shooting scene, and shoots camera data synchronously; and performs a series of offline calibration processing, which avoids a series of tedious and time-consuming operations when calibrating traditional multi-camera systems. Calibration efficiency when calibrating multi-camera extrinsic parameters after each camera configuration change.

在一些实施例中，根据各个相机视角对应的顶点坐标集合在同一坐标系中的相对位置关系，确定各个相机视角对应相机的外参标定值，包括：将各个相机视角中的其中一个相机视角作为基准相机视角；基于基准相机视角对应的顶点坐标集合与各个相机视角对应的顶点坐标集合之间的顶点位置差异，确定各个相机视角对应相机的外参标定值。In some embodiments, determining the extrinsic parameter calibration value of the camera corresponding to each camera perspective according to the relative positional relationship of the vertex coordinate sets corresponding to each camera perspective in the same coordinate system, including: using one of the camera perspectives as the camera perspective. The reference camera angle of view; based on the vertex position difference between the vertex coordinate set corresponding to the reference camera angle of view and the vertex coordinate set corresponding to each camera angle of view, the external parameter calibration value of the camera corresponding to each camera angle of view is determined.

在一些实施例中，电子设备在根据各个相机视角对应的顶点坐标集合在同一坐标系中的相对位置关系，确定各个相机视角对应相机的外参标定值的过程中，包括：电子设备可以将各个相机视角中的其中一个相机视角作为基准相机视角。在一些实施例中，电子设备可以将各个相机视角中第0个相机视角作为基准相机视角。In some embodiments, when the electronic device determines the calibration value of the external parameters of the camera corresponding to each camera perspective according to the relative positional relationship of the vertex coordinate sets corresponding to each camera perspective in the same coordinate system, the process includes: the electronic device can One of the camera perspectives is used as the base camera perspective. In some embodiments, the electronic device may use the 0th camera angle of view among the respective camera angles as the reference camera angle of view.

然后，电子设备再根据各个相机视角对应的顶点坐标集合在同一坐标系中的相对位置关系，确定各个相机视角对应相机的外参标定值。在一些实施例中，电子设备可以将各个相机视角对应的顶点坐标集合统一至基准相机视角对应的顶点坐标集合所在的坐标系中，并基于基准相机视角对应的顶点坐标集合与各个相机视角对应的顶点坐标集合在基准相机视角对应的坐标系中的顶点位置差异，确定出各个相机视角对应的相机之间的相对位姿关系，进而初步确定各个相机视角对应相机的外参标定值。Then, the electronic device determines the external parameter calibration value of the camera corresponding to each camera angle of view according to the relative positional relationship of the vertex coordinate sets corresponding to each camera angle of view in the same coordinate system. In some embodiments, the electronic device may unify the vertex coordinate set corresponding to each camera perspective into the coordinate system where the vertex coordinate set corresponding to the reference camera perspective is located, and based on the vertex coordinate set corresponding to the reference camera perspective and each camera perspective corresponding to the coordinate system The vertex position difference of the vertex coordinate set in the coordinate system corresponding to the reference camera angle of view determines the relative pose relationship between the cameras corresponding to each camera angle of view, and then preliminarily determines the external parameter calibration value of the camera corresponding to each camera angle of view.

本申请实施例的技术方案，通过将各个相机视角中的其中一个相机视角作为基准相机视角；并基于基准相机视角对应的顶点坐标集合与各个相机视角对应的顶点坐标集合之间的顶点位置差异，确定各个相机视角对应相机的外参标定值，从而实现了准确地确定出各个相机视角对应的顶点坐标集合在同一坐标系中的相对位置关系，进而便于后续确定出各个相机视角对应的外参标定值。In the technical solution of the embodiment of the present application, by using one of the camera perspectives as the reference camera perspective; and based on the vertex position difference between the vertex coordinate set corresponding to the reference camera perspective and the vertex coordinate set corresponding to each camera perspective, Determine the external parameter calibration values of the cameras corresponding to each camera angle of view, so as to accurately determine the relative positional relationship of the vertex coordinate sets corresponding to each camera angle of view in the same coordinate system, thereby facilitating the subsequent determination of the external parameter calibration corresponding to each camera angle of view value.

在一些实施例中，基于基准相机视角对应的顶点坐标集合与各个相机视角对应的顶点坐标集合之间的顶点位置差异，确定各个相机视角对应相机的外参标定值，包括：分别求解将各个相机视角对应的顶点坐标集合对齐到基准相机视角对应的顶点坐标集合所在坐标系上时所需的刚体变换；将求解得到的各个相机视角对应的刚体变换，作为各个相机视角对应相机的外参标定值。In some embodiments, determining the extrinsic parameter calibration value of the camera corresponding to each camera perspective based on the vertex position difference between the vertex coordinate set corresponding to the reference camera perspective and the vertex coordinate set corresponding to each camera perspective, including: separately solving for each camera perspective The rigid body transformation required when the vertex coordinate set corresponding to the viewing angle is aligned to the coordinate system where the vertex coordinate set corresponding to the reference camera viewing angle is located; the rigid body transformation corresponding to each camera viewing angle obtained by the solution is used as the external parameter calibration value of the camera corresponding to each camera viewing angle .

在一些实施例中，在电子设备基于基准相机视角对应的顶点坐标集合与各个相机视角对应的顶点坐标集合之间的顶点位置差异，确定各个相机视角对应相机的外参标定值的过程中，电子设备需要分别求解将各个相机视角对应的顶点坐标集合对齐到基准相机视角对应的顶点坐标集合坐标系上时所需的刚体变换；将求解得到的各个相机视角对应的刚体变换，作为各个相机视角对应相机的外参标定值。在一些实施例中，电子设备可以采用SVD 法(奇异值分解算法)，计算出基准相机视角对应的顶点坐标集合对齐到基准相机视角对应的顶点坐标集合所在坐标系上时所需的刚体变换，得到各个相机视角对应的刚体变换。在一些实施例中，第i个相机视角对应的刚体变换(相机外参标定值)可以表示为T _I＝{R _i,t _i}，即第i个相机视角对应的相机外参标定值可以相当于将M _i对齐至M ₀所在坐标系所需的刚体变换。 In some embodiments, in the process of determining the external parameter calibration value of the camera corresponding to each camera angle of view by the electronic device based on the vertex position difference between the vertex coordinate set corresponding to the reference camera angle of view and the vertex coordinate set corresponding to each camera angle of view, the electronic device The device needs to solve the rigid body transformation required to align the vertex coordinate set corresponding to each camera perspective to the coordinate system of the vertex coordinate set corresponding to the reference camera perspective; the rigid body transformation corresponding to each camera perspective obtained by the solution is used as the corresponding camera perspective The external parameter calibration value of the camera. In some embodiments, the electronic device can use the SVD method (singular value decomposition algorithm) to calculate the rigid body transformation required when the vertex coordinate set corresponding to the reference camera view angle is aligned to the coordinate system where the vertex coordinate set corresponding to the reference camera view angle is located, Get the rigid body transformation corresponding to each camera perspective. In some embodiments, the rigid body transformation (calibration value of camera extrinsic parameters) corresponding to the ith camera angle of view can be expressed as T _I ={R _i ,t _i }, that is, the calibration value of camera extrinsic parameters corresponding to the ith camera angle of view can be Equivalent to the rigid body transformation required to align _Mi to the coordinate system where M ₀ resides.

其中，

t _i＝c _i-(R _i·c ₀)， in,

t _i = _ci -(R _i ·c ₀ ),

其中，U _i,S _i,

c _i为第i个相机视角下的三维网格模型在对应的相机坐标系中的中心点。 Among them, U _i , S _i ,

c _i is the center point of the 3D mesh model in the corresponding camera coordinate system under the ith camera view angle.

其中，SVD求解刚体变换的中间变量L _i＝M ₀·M _i。 Among them, SVD solves the intermediate variable Li ₌ M ₀ ·M _i of rigid body transformation.

电子设备将求解得到的各个相机视角对应的刚体变换，作为各个相机视角对应相机的外参标定值。The electronic device uses the obtained rigid body transformation corresponding to each camera angle of view as the external parameter calibration value of each camera angle corresponding to the camera.

本申请实施例的技术方案，通过将各个相机视角中的其中一个相机视角作为基准相机视角，并分别求解将各个相机视角对应的顶点坐标集合对齐到基准相机视角对应的顶点坐标集合所在坐标系上时所需的刚体变换；使得求解得到的各个相机视角对应的刚体变换可以准确地表征各个相机视角对应的顶点坐标集合在同一坐标系中的相对位置关系。The technical solution of the embodiment of the present application is to use one of the camera angles of view as the reference camera angle of view, and solve the problem of aligning the vertex coordinate set corresponding to each camera angle of view to the coordinate system where the vertex coordinate set corresponding to the reference camera angle of view is located. The rigid body transformation required for each camera view angle can accurately represent the relative positional relationship of the vertex coordinate sets corresponding to each camera view angle in the same coordinate system.

在一些实施例中，该相机外参标定方法还包括：将多张目标对象图像输入预训练的关键点检测网络，得到目标对象在每张目标对象图像中的二维关键点；基于各个相机视角对应的二维关键点的位置信息，对各个相机视角对应相机的外参标定值进行优化，得到各个相机视角对应相机的优化外参标定值。In some embodiments, the camera extrinsic parameter calibration method further includes: inputting multiple target object images into a pre-trained keypoint detection network to obtain two-dimensional keypoints of the target object in each target object image; Based on the position information of the corresponding two-dimensional key points, the external parameter calibration values of the cameras corresponding to each camera angle of view are optimized, and the optimized external parameter calibration values of the cameras corresponding to each camera angle of view are obtained.

其中，当目标对象为人体时，目标对象图像中的二维关键点可以是目标对象在目标对象图像中的关节点。Wherein, when the target object is a human body, the two-dimensional key points in the target object image may be joint points of the target object in the target object image.

其中，预训练的关键点检测网络可以是指基于深度学习训练并利用海量标注数据训练得到的用于对输入图像中的物体关键点识别的神经网络。实际应用中，电子设备通过预训练的关键点检测网络对输入彩色图片中人体各个关节在图像中的位置进行检测与定位。The pre-trained key point detection network may refer to a neural network trained based on deep learning and trained with massive labeled data for recognizing key points of objects in an input image. In practical applications, the electronic device detects and locates the position of each joint of the human body in the input color image through the pre-trained key point detection network.

在一些实施例中，该相机外参标定方法还包括：电子设备可以将各张目标对象图像输入至预训练的关键点检测网络，通过预训练的关键点检测网络，确定目标对象在每张目标对象图像中的二维关键点。电子设备在基于各个相机视角对应的二维关键点的位置信息，对各个相机视角对应相机的外参标定值进行优化，得到各个相机视角对应相机的优化外参标定值。In some embodiments, the camera external parameter calibration method further includes: the electronic device may input each target object image into a pre-trained key point detection network, and through the pre-trained key point detection network, determine that the target object is in each target image 2D keypoints in the object image. Based on the position information of the two-dimensional key points corresponding to each camera perspective, the electronic device optimizes the external parameter calibration values of the cameras corresponding to each camera perspective, and obtains the optimized external parameter calibration values of the cameras corresponding to each camera perspective.

本申请实施例的技术方案，通过进一步地将多张目标对象图像输入预训练的关键点检测网络，得到目标对象在每张目标对象图像中的二维关键点；基于各个相机视角对应的二维关键点的位置信息，对各个相机视角对应相机的外参标定值进行进一步地优化，使得得到的各个相机视角对应相机的优化外参标定值可以更为准确地描述出各个相机视角对应相机的实际相机位姿。In the technical solution of the embodiment of the present application, by further inputting multiple target object images into a pre-trained key point detection network, two-dimensional key points of the target object in each target object image are obtained; The position information of key points is used to further optimize the external parameter calibration value of each camera angle corresponding to the camera, so that the obtained optimized external parameter calibration value of each camera angle corresponding to the camera can more accurately describe the actual camera angle corresponding to each camera angle. camera pose.

在一些实施例中，基于各个相机视角对应的二维关键点的位置信息，对各个相机视角对应相机的外参标定值进行优化，得到各个相机视角对应相机的优化外参标定值，包括：基于多张目标对象图像，确定各个相机视角下的二维关键点在对应的图像坐标系中的关键点二维坐标；在各个相机视角对应的顶点坐标集合中，确定各个相机视角下的二维关键点在对应的相机坐标系中的关键点三维坐标；根据各个相机视角对应的关键点三维坐标和各个相机视角下对应的关键点二维坐标，对各个相机视角对应相机的外参标定值进行优化，得到各个相机视角对应相机的优化外参标定值。In some embodiments, based on the position information of the two-dimensional key points corresponding to each camera perspective, the external parameter calibration values of the cameras corresponding to each camera perspective are optimized, and the optimized external parameter calibration values of the cameras corresponding to each camera perspective are obtained, including: based on: For multiple target object images, determine the 2D coordinates of the 2D key points in the corresponding image coordinate system under each camera perspective; in the vertex coordinate set corresponding to each camera perspective, determine the 2D key points in each camera perspective. The three-dimensional coordinates of the key points in the corresponding camera coordinate system; according to the three-dimensional coordinates of the key points corresponding to each camera perspective and the two-dimensional coordinates of the key points corresponding to each camera perspective, the external parameter calibration values of the cameras corresponding to each camera perspective are optimized. , to obtain the optimized external parameter calibration value of each camera corresponding to the camera.

在一些实施例中，电子设备基于各个所述相机视角对应的所述二维关键点的位置信息，对各个所述相机视角对应相机的外参标定值进行优化，得到各个所述相机视角对应相机的优化外参标定值的过程中，包括：基于多张目标对象图像，确定各个相机视角下的二维关键点在对应的图像坐标系中的关键点二维坐标。其中，各个相机视角下的关键点在对应的图像坐标系中的关键点二维坐标可以表示为{P ₀,P ₁,...,P _i}： In some embodiments, the electronic device optimizes, based on the position information of the two-dimensional key points corresponding to the respective camera angles of view, the external parameter calibration values of the cameras corresponding to the respective camera angles of view, and obtains the camera corresponding to each of the camera angles of view. The process of optimizing the calibration value of the external parameters includes: determining the two-dimensional coordinates of the two-dimensional key points in the corresponding image coordinate system of the two-dimensional key points under each camera perspective based on multiple target object images. Among them, the two-dimensional coordinates of the key points in the corresponding image coordinate system of the key points under each camera perspective can be expressed as {P ₀ , P ₁ ,...,P _i }:

其中，P _i表示第i个相机坐标系(或相机视角)下的二维关键点在对应的图像坐标系中的关键点二维坐标。 Among them, P _i represents the two-dimensional coordinates of the two-dimensional key point in the corresponding image coordinate system of the two-dimensional key point in the ith camera coordinate system (or camera angle of view).

电子设备在各个相机视角对应的顶点坐标集合中，确定各个相机视角下的二维关键点在对应的相机坐标系中的关键点三维坐标；将关键点二维坐标和关键点三维坐标，作为关键点位置信息。其中，各个相机视角下的关键点在对应的相机坐标系中的关键点三维坐标可以表示为{X ₀,X ₁,...,X _i}： The electronic device determines the three-dimensional coordinates of the two-dimensional key points in the corresponding camera coordinate system of the two-dimensional key points under each camera perspective in the vertex coordinate set corresponding to each camera perspective; the two-dimensional coordinates of the key points and the three-dimensional coordinates of the key points are regarded as the key point location information. Among them, the three-dimensional coordinates of the key points in the corresponding camera coordinate system of the key points under each camera perspective can be expressed as {X ₀ , X ₁ ,...,X _i }:

其中，X _i表示第i个相机坐标系(或相机视角)下的二维关键点在对应的相机坐标系中的关键点三维坐标。 Wherein, X _i represents the three-dimensional coordinates of the key point in the corresponding camera coordinate system of the two-dimensional key point in the ith camera coordinate system (or camera angle of view).

电子设备再基于各个相机视角对应的关键点位置信息即根据各个相机视角对应的关键点三维坐标和各个相机视角下对应的关键点二维坐标，对各个相机视角对应相机的外参标定值进行优化，得到各个相机视角对应相机的优化外参标定值。The electronic device then optimizes the external parameter calibration value of the camera corresponding to each camera perspective based on the position information of the key points corresponding to each camera perspective, that is, according to the three-dimensional coordinates of the key points corresponding to each camera perspective and the two-dimensional coordinates of the key points corresponding to each camera perspective. , to obtain the optimized external parameter calibration value of each camera corresponding to the camera.

本申请实施例的技术方案，在对各个相机视角对应相机的外参标定值进行优化的过程中，通过预训练的关键点检测网络，确定目标对象在每张目标对象图像中的关键点；分别确定各个相机视角下的关键点在对应的相机坐标系和对应的图像坐标系中的关键点位置信息；基于各个相机视角对应的关键点位置信息，准确地对各个相机视角对应相机的外参标定值进行进一步优化，使得得到各个相机视角对应的相机的优化外参标定值具有较高的精度。In the technical solutions of the embodiments of the present application, in the process of optimizing the calibration values of the external parameters of the cameras corresponding to the camera angles of view, the key points of the target object in each target object image are determined through a pre-trained key point detection network; respectively; Determine the key point position information of the key points under each camera perspective in the corresponding camera coordinate system and the corresponding image coordinate system; based on the key point position information corresponding to each camera perspective, accurately calibrate the external parameters of the camera corresponding to each camera perspective The value is further optimized, so that the optimized external parameter calibration value of the camera corresponding to each camera angle of view can be obtained with high accuracy.

在一些实施例中，根据各个相机视角对应的关键点三维坐标和各个相机视角下对应的关键点二维坐标，对各个相机视角对应相机的外参标定值进行优化，得到各个相机视角对应相机的优化外参标定值，包括：根据各个相机视角对应的关键点三维坐标和各个相机视角下对应的关键点二维坐标，确定各个相机视角对应的相机在采用外参标定值时的投影误差；基于各个相机视角对应的相机在采用外参标定值时的投影误差，对各个相机视角对应相机的外参标定值进行调整，得到各个相机视角对应相机的调整后外参标定值，作为各个相机视角对应相机的优化外参标定值。其中，相机视角对应的相机在采用优化外参标定值时的投影误差满足预设条件。In some embodiments, according to the three-dimensional coordinates of the key points corresponding to the camera perspectives and the two-dimensional coordinates of the key points corresponding to the camera perspectives, the external parameter calibration values of the cameras corresponding to the camera perspectives are optimized to obtain the camera perspectives corresponding to the camera perspectives. Optimizing the external parameter calibration value includes: determining the projection error of the camera corresponding to each camera perspective when using the external parameter calibration value according to the three-dimensional coordinates of the key points corresponding to each camera perspective and the two-dimensional coordinates of the key points corresponding to each camera perspective; The projection error of the camera corresponding to each camera perspective when the external parameter calibration value is used, adjust the external parameter calibration value of the camera corresponding to each camera perspective, and obtain the adjusted external parameter calibration value of each camera perspective corresponding to the camera, as the corresponding camera perspective The optimized extrinsic parameter calibration value of the camera. Among them, the projection error of the camera corresponding to the camera angle of view when the optimized external parameter calibration value is adopted satisfies the preset condition.

其中，投影误差为相机视角对应的相机将关键点三维坐标投影至相机的像平面而得到的二维坐标与关键点二维坐标之间的误差。The projection error is the error between the two-dimensional coordinates of the key point obtained by projecting the three-dimensional coordinates of the key point onto the image plane of the camera and the two-dimensional coordinates of the key point.

在一些实施例中，电子设备在根据各个相机视角对应的关键点三维坐标和各个相机视角下对应的关键点二维坐标，对各个相机视角对应相机的外参标定值进行优化，得到各个相机视角对应相机的优化外参标定值的过程中，包括：电子设备通过根据各个相机视角对应的关键点三维坐标和各个相机视角下对应的关键点二维坐标，确定各个相机视角对应的相机在采用该外参标定值时，将关键点三维坐标投影至相机的像平面而得到的二维坐标与关键点二维坐标之间的误差。In some embodiments, the electronic device optimizes the calibration value of the external parameters of the camera corresponding to each camera perspective according to the three-dimensional coordinates of the key points corresponding to each camera perspective and the two-dimensional coordinates of the key points corresponding to each camera perspective to obtain each camera perspective. The process of optimizing the external parameter calibration value corresponding to the camera includes: the electronic device determines, according to the three-dimensional coordinates of the key points corresponding to each camera perspective and the two-dimensional coordinates of the key points corresponding to each camera perspective, that the camera corresponding to each camera perspective is using the camera. When the external parameter is calibrated, the error between the two-dimensional coordinates obtained by projecting the three-dimensional coordinates of the key points to the image plane of the camera and the two-dimensional coordinates of the key points.

在一些实施例中，根据各个相机视角对应的关键点三维坐标和各个相机视角下对应的关键点二维坐标，确定各个相机视角对应的相机的投影误差，包括：通过预设的投影函数，基于各个相机视角对应相机的外参标定值和各个相机视角对应的关键点三维坐标，将各个相机视角对应的关键点投影至对应相机的像平面上，得到各个相机视角对应的关键点在对应相机的像平面上的投影点；确定各个相机视角对应的投影点在对应的图像坐标系中的投影点二维坐标；根据每个相机视角对应的投影点二维坐标和对应的关键点二维坐标之间的差异，确定各个相机视角对应的相机的投影误差。In some embodiments, determining the projection error of the camera corresponding to each camera perspective according to the three-dimensional coordinates of the key points corresponding to the respective camera perspectives and the two-dimensional coordinates of the key points corresponding to the respective camera perspectives includes: using a preset projection function, based on Each camera perspective corresponds to the external parameter calibration value of the camera and the three-dimensional coordinates of the key points corresponding to each camera perspective, project the key points corresponding to each camera perspective to the image plane of the corresponding camera, and obtain the key points corresponding to each camera perspective in the corresponding camera The projection point on the image plane; determine the two-dimensional coordinates of the projection point corresponding to each camera perspective in the corresponding image coordinate system; according to the difference between the two-dimensional coordinates of the projection point corresponding to each camera perspective and the corresponding key point The difference between the two cameras determines the projection error of the camera corresponding to each camera angle of view.

换句话说，电子设备可以通过预设的投影函数，基于各个相机视角对应相机的外参标定值和各个相机视角对应的关键点三维坐标，将各个相机视角对应的关键点在对应相机坐标系的关键点三维坐标转换为各个相机视角对应的关键点在对应图像坐标系的关键点三维坐标，作为各个相机视角对应的投影点二维坐标。In other words, the electronic device can use the preset projection function, based on the calibration value of the external parameters of the camera corresponding to each camera perspective and the three-dimensional coordinates of the key points corresponding to each camera perspective, the key points corresponding to each camera perspective are in the corresponding camera coordinate system. The three-dimensional coordinates of the key points are converted into the three-dimensional coordinates of the key points corresponding to the respective camera perspectives in the corresponding image coordinate system, as the two-dimensional coordinates of the projection points corresponding to the respective camera perspectives.

电子设备基于各个相机视角对应的相机在采用外参标定值时的投影误差，对各个相机视角对应相机的外参标定值进行调整，得到各个相机视角对应相机的调整后外参标定值，作为各个相机视角对应相机的优化外参标定值。The electronic device adjusts the external parameter calibration value of each camera angle corresponding to the camera based on the projection error of the camera corresponding to each camera angle of view when the external parameter calibration value is used, and obtains the adjusted extrinsic parameter calibration value of each camera angle corresponding to the camera, as each camera angle. The camera angle of view corresponds to the optimized external parameter calibration value of the camera.

其中，相机视角对应的相机在采用优化外参标定值时的投影误差满足预设条。Among them, the projection error of the camera corresponding to the camera angle of view when the optimized external parameter calibration value is adopted satisfies the preset criteria.

实际应用中，电子设备在基于各个相机视角对应的投影误差，对各个相机视角对应相机的外参标定值进行调整，得到各个相机视角对应的优化外参标定值的过程中，可以采用将各个相机视角对应的关键点三维坐标和各个相机视角下对应的关键点二维坐标作为约束，以待标定的各相机外参作为目标变量，建立非线性最小二乘问题，并利用最优化数学方法对其进行求解，最终得到多视角动作捕捉***中的各个相机的优化外参标定值。在一些实施例中，电子设备可以将第i个相机视角对应相机的外参标定值T _I＝{R _i,t _i}，同时结合各个相机视角下的关键点在对应的图像坐标系中的关键点二维坐标{P ₀,P ₁,...,P _i}：可建立如下最小二乘问题： In practical applications, in the process of adjusting the external parameter calibration value of each camera angle corresponding to the camera based on the projection error corresponding to each camera angle of view, the electronic device can obtain the optimized external parameter calibration value corresponding to each camera angle of view. The three-dimensional coordinates of the key points corresponding to the angle of view and the two-dimensional coordinates of the key points corresponding to each camera angle are used as constraints, and the external parameters of each camera to be calibrated are used as the target variables to establish a nonlinear least squares problem, and use the optimization mathematical method to solve it. After solving, the optimized external parameter calibration values of each camera in the multi-view motion capture system are finally obtained. In some embodiments, the electronic device may set the i-th camera angle of view to correspond to the camera's extrinsic parameter calibration value T _I ={R _i ,t _i }, and at the same time combine the key points of each camera angle of view in the corresponding image coordinate system Two-dimensional coordinates of key points {P ₀ , P ₁ ,...,P _i }: The following least squares problem can be established:

其中,π _i为各相机内参所对应的投影函数。 Among them, π _i is the projection function corresponding to the internal parameters of each camera.

最后，电子设备通过使用最优化数学方法对上述非线性最小二乘问题进行求解，求解得出的{R _i,t _i}即为各相机外参的最终结果。实际应用中，可以使用高斯-牛顿法、Levenberg–Marquardt算法、牛顿法等进行求解。本方案不对具体的求解方法进行约束。 Finally, the electronic device solves the above nonlinear least squares problem by using the optimization mathematical method, and the obtained {R _i ,t _i } is the final result of the external parameters of each camera. In practical applications, Gauss-Newton method, Levenberg-Marquardt algorithm, Newton method, etc. can be used to solve the problem. This scheme does not constrain the specific solution method.

本申请实施例的技术方案，通过根据各个相机视角对应的关键点三维坐标和各个相机视角下对应的关键点二维坐标，确定各个相机视角对应的相机的投影误差；并基于各个相机视角对应的投影误差，对各个相机视角对应相机的外参标定值进行调整，得到各个相机视角对应的调整后外参标定值，作为各个相机视角对应相机的优化外参标定值，使得各个相机视角对应的相机在采用优化外参标定值时将该二维关键点投影至相机的像平面时产生的误差满足预设条件。According to the technical solutions of the embodiments of the present application, the projection errors of the cameras corresponding to the respective camera perspectives are determined according to the three-dimensional coordinates of the key points corresponding to the respective camera perspectives and the two-dimensional coordinates of the key points corresponding to the respective camera perspectives; Projection error, adjust the external parameter calibration value of each camera perspective corresponding to the camera, and obtain the adjusted external parameter calibration value corresponding to each camera perspective, as the optimized external parameter calibration value of each camera perspective corresponding to the camera, so that the camera corresponding to each camera perspective The error generated when the two-dimensional key point is projected to the image plane of the camera when the optimal external parameter calibration value is adopted satisfies the preset condition.

图3是根据一示例性实施例示出的另一种相机外参标定方法的流程图，该相机外参标定方法可以由图1的电子设备110执行，如图3所示，该相机外参标定方法包括以下步骤。在步骤S302中，对目标对象进行多相机拍摄，以获得不同相机视角下的多张目标对象图像。在步骤S304中，通过预训练的三维重构网络，生成所述目标对象在每张所述目标对象图像中的三维网格模型。在步骤S306中，确定各个所述相机视角下的所述三维网格模型在对应的相机坐标系中的顶点坐标集合。在步骤S308中，将各个所述相机视角中的其中一个相机视角作为基准相机视角。在步骤S310中，分别求解将各个所述相机视角对应的顶点坐标集合对齐到所述基准相机视角对应的顶点坐标集合所在坐标系上时所需的刚体变换。在步骤S312中，将求解得到的各个所述相机视角对应的刚体变换，作为各个所述相机视角对应相机的外参标定值。在步骤S314中，通过预训练的关键点检测网络，确定所述目标对象在每张所述目标对象图像中的二维关键点。在步骤S316中，基于多张所述目标对象图像，确定各个所述相机视角下的所述二维关键点在对应的图像坐标系中的关键点二维坐标。在步骤S318中，在各个所述相机视角对应的顶点坐标集合中，确定各个所述相机视角下的所述二维关键点在对应的相机坐标系中的关键点三维坐标。在步骤S320中，根据各个所述相机视角对应的所述关键点三维坐标和各个所述相机视角下对应的所述关键点二维坐标，确定各个所述相机视角对应的相机在采用所述外参标定值时的投影误差。在步骤S322中，基于各个所述相机视角对应的相机在采用所述外参标定值时的投影误差，对各个所述相机视角对应相机的所述外参标定值进行调整，得到各个所述相机视角对应相机的调整后外参标定值，作为各个所述相机视角对应相机的优化外参标定值。需要说明的是，上述步骤的具体限定可以参见上文对一种相机外参标定方法的具体限定，在此不再赘述。FIG. 3 is a flowchart illustrating another method for calibrating external parameters of a camera according to an exemplary embodiment. The method for calibrating external parameters of a camera may be executed by the electronic device 110 in FIG. 1 . As shown in FIG. 3 , the method for calibrating external parameters of a camera is The method includes the following steps. In step S302, multi-camera photography is performed on the target object to obtain multiple target object images from different camera angles of view. In step S304, a three-dimensional mesh model of the target object in each image of the target object is generated through a pre-trained three-dimensional reconstruction network. In step S306, a vertex coordinate set in the corresponding camera coordinate system of the three-dimensional mesh model under each camera perspective is determined. In step S308, one of the camera angles of view is used as the reference camera angle of view. In step S310, the rigid body transformation required for aligning the vertex coordinate set corresponding to each camera angle of view to the coordinate system where the vertex coordinate set corresponding to the reference camera angle of view is located is solved separately. In step S312, the obtained rigid body transformation corresponding to each of the camera angles of view is used as an external parameter calibration value of the camera corresponding to each of the camera angles of view. In step S314, the two-dimensional key points of the target object in each image of the target object are determined through the pre-trained key point detection network. In step S316, based on a plurality of images of the target object, the two-dimensional coordinates of the key points in the corresponding image coordinate system of the two-dimensional key points under each of the camera perspectives are determined. In step S318, in the vertex coordinate set corresponding to each of the camera perspectives, determine the three-dimensional coordinates of the key points in the corresponding camera coordinate system of the two-dimensional key points under each of the camera perspectives. In step S320, according to the three-dimensional coordinates of the key points corresponding to the respective camera perspectives and the two-dimensional coordinates of the key points corresponding to the respective camera perspectives, it is determined that the cameras corresponding to the camera perspectives are using the external The projection error when the parameter is calibrated. In step S322, based on the projection errors of the cameras corresponding to the camera perspectives when the external parameter calibration values are used, the external parameter calibration values of the cameras corresponding to the camera perspectives are adjusted to obtain each camera. The angle of view corresponds to the adjusted external parameter calibration value of the camera, which is used as the optimized external parameter calibration value of the camera corresponding to each of the camera angles of view. It should be noted that, for the specific limitations of the above steps, reference may be made to the specific limitations on a camera external parameter calibration method above, which will not be repeated here.

应该理解的是，虽然图2和图3的流程图中的各个步骤按照箭头的指示依次显示，但是这些步骤并不是必然按照箭头指示的顺序依次执行。除非本文中有明确的说明，这些步骤的执行并没有严格的顺序限制，这些步骤可以以其它的顺序执行。而且，图2和图3中的至少一部分步骤可以包括多个步骤或者多个阶段，这些步骤或者阶段并不必然是在同一时刻执行完成，而是可以在不同的时刻执行，这些步骤或者阶段的执行顺序也不必然是依次进行，而是可以与其它步骤或者其它步骤中的步骤或者阶段的至少一部分轮流或者交替地执行。It should be understood that although the respective steps in the flowcharts of FIG. 2 and FIG. 3 are shown in sequence according to the arrows, these steps are not necessarily executed in the sequence shown by the arrows. Unless explicitly stated herein, the execution of these steps is not strictly limited to the order, and these steps may be performed in other orders. Moreover, at least a part of the steps in FIG. 2 and FIG. 3 may include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but may be executed at different times. The order of execution is also not necessarily sequential, but may be performed alternately or alternately with other steps or at least a portion of the steps or stages within the other steps.

为了便于本领域技术人员理解本申请实施例的逻辑，图4提供了一种相机外参标定方法的流程框图；其中，对目标对象进行多相机拍摄，以获得不同相机视角下的多张目标对象图像；然后，将多张目标对象图像输入至人体二维关键点检测模块，对输入彩色图片中人体各个关节在图像中的位置进行检测与定位。同时，将多张目标对象图像输入至人体三维网格模型估计模块对输入彩色图像中人体的三维形状及姿态进行估计与重建，得到图像中人体在相机坐标系下的三维空间表示。再然后，将各相机各帧中检测到的人体关键点及三维网格模型数据输入至多帧多视角联合优化模块。该模块利用多帧多视角下的对应二维及三维关键点位置作为约束，以待标定的各相机外参作为目标变量，建立非线性最小二乘问题，并利用最优化数学方法对其进行求解，最终得到多视角动补***中各相机的外参标定值。In order to facilitate those skilled in the art to understand the logic of the embodiments of the present application, FIG. 4 provides a flowchart of a method for calibrating external parameters of a camera; wherein, the target object is photographed with multiple cameras to obtain multiple target objects from different camera perspectives Then, input multiple target object images to the human body two-dimensional key point detection module to detect and locate the position of each joint of the human body in the input color image. At the same time, multiple target object images are input to the human body 3D mesh model estimation module to estimate and reconstruct the 3D shape and posture of the human body in the input color image, and obtain the 3D space representation of the human body in the image in the camera coordinate system. Then, the human body key points and 3D mesh model data detected in each frame of each camera are input to the multi-frame multi-view joint optimization module. This module uses the corresponding two-dimensional and three-dimensional key point positions under multiple frames and multiple perspectives as constraints, and takes the external parameters of each camera to be calibrated as the target variable, establishes a nonlinear least squares problem, and uses the optimization mathematical method to solve it , and finally the calibration values of the external parameters of each camera in the multi-view motion compensation system are obtained.

图5是根据一示例性实施例示出的一种相机外参标定装置框图。参照图5，该装置包括：Fig. 5 is a block diagram of an apparatus for calibrating external parameters of a camera according to an exemplary embodiment. Referring to Figure 5, the device includes:

获取单元510，被配置为执行对目标对象进行多相机拍摄，以获得不同相机视角下的多张目标对象图像；每个所述相机视角对应于所述多相机中的一个相机；The acquiring unit 510 is configured to perform multi-camera shooting of the target object, so as to obtain multiple images of the target object under different camera perspectives; each of the camera perspectives corresponds to one camera in the multi-cameras;

重建单元520，被配置为执行将多张所述目标对象图像输入预训练的三维重建网络，生成所述目标对象在每张所述目标对象图像中的三维网格模型；The reconstruction unit 520 is configured to input a plurality of images of the target object into a pre-trained 3D reconstruction network, and generate a 3D mesh model of the target object in each image of the target object;

确定单元530，被配置为确定各个所述相机视角下的所述三维网格模型在对应的相机坐标系中的顶点坐标集合；a determining unit 530, configured to determine a vertex coordinate set of the three-dimensional mesh model in the corresponding camera coordinate system under each of the camera perspectives;

标定单元540，被配置为执行根据各个所述相机视角对应的顶点坐标集合在同一坐标系中的相对位置关系，确定各个所述相机视角对应相机的外参标定值。The calibration unit 540 is configured to determine the extrinsic parameter calibration value of the camera corresponding to each camera angle of view according to the relative positional relationship of the vertex coordinate sets corresponding to each of the camera angles of view in the same coordinate system.

在一些实施例中，所述标定单元540，被配置为执行将各个所述相机视角中的其中一个相机视角作为基准相机视角；基于所述基准相机视角对应的顶点坐标集合与各个所述相机视角对应的顶点坐标集合之间的顶点位置差异，确定各个所述相机视角对应相机的外参标定值。In some embodiments, the calibration unit 540 is configured to perform taking one of the camera perspectives as a reference camera perspective; based on the vertex coordinate set corresponding to the reference camera perspective and each of the camera perspectives The vertex position difference between the corresponding vertex coordinate sets determines the external parameter calibration value of the camera corresponding to each of the camera angles of view.

在一些实施例中，所述标定单元540，被配置为执行分别求解将各个所述相机视角对应的顶点坐标集合对齐到所述基准相机视角对应的顶点坐标集合所在坐标系上时所需的刚体变换；将求解得到的各个所述相机视角对应的刚体变换，作为各个所述相机视角对应相机的外参标定值。In some embodiments, the calibration unit 540 is configured to separately solve the rigid body required for aligning the vertex coordinate sets corresponding to the camera viewpoints to the coordinate system where the vertex coordinate sets corresponding to the reference camera viewpoints are located. Transform; take the obtained rigid body transformation corresponding to each of the camera angles of view as an external parameter calibration value of the camera corresponding to each of the camera angles of view.

在一些实施例中，所述优化单元，被配置为执行通过预设的投影函数和各个所述相机视角对应相机的外参标定值，分别将各个所述相机视角对应的所述关键点三维坐标投影至对应相机的像平面上，得到各个所述相机视角对应的所述二维关键点在对应所述相机的像平面上的投影点；确定各个所述相机视角对应的所述投影点在对应的图像坐标系中的投影点二维坐标；根据每个所述相机视角对应的所述投影点二维坐标和对应的所述关键点二维坐标之间的差异，确定各个所述相机视角对应的相机的投影误差In some embodiments, the optimization unit is configured to perform a preset projection function and an external parameter calibration value of the camera corresponding to each of the camera perspectives, respectively, the three-dimensional coordinates of the key points corresponding to each of the camera perspectives Projecting onto the image plane of the corresponding camera to obtain the projection points of the two-dimensional key points corresponding to the camera perspectives on the image plane corresponding to the cameras; determining that the projection points corresponding to the camera perspectives are in the corresponding The two-dimensional coordinates of the projection point in the image coordinate system of the The projection error of the camera

关于上述实施例中的装置，其中各个模块执行操作的具体方式已经在有关该方法的实施例中进行了详细描述，此处将不做详细阐述说明。Regarding the apparatus in the above-mentioned embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be described in detail here.

图6是根据一示例性实施例示出的一种用于执行相机外参标定方法的设备600的框图。例如，设备600可以是移动电话、计算机、数字广播终端、消息收发设备、游戏控制台、平板设备、医疗设备、健身设备、个人数字助理等。FIG. 6 is a block diagram of a device 600 for performing a camera extrinsic parameter calibration method according to an exemplary embodiment. For example, device 600 may be a mobile phone, computer, digital broadcast terminal, messaging device, game console, tablet device, medical device, fitness device, personal digital assistant, or the like.

参照图6，设备600可以包括以下一个或多个组件：处理组件602、存储器604、电力组件606、多媒体组件608、音频组件610、输入/输出(I/O)的接口612、传感器组件614以及通信组件616。6, device 600 may include one or more of the following components: processing component 602, memory 604, power component 606, multimedia component 608, audio component 610, input/output (I/O) interface 612, sensor component 614, and Communication component 616 .

处理组件602通常控制设备600的整体操作，诸如与显示、电话呼叫、数据通信、相机操作和记录操作相关联的操作。处理组件602可以包括一个或多个处理器620来执行指令，以完成上述的方法的全部或部分步骤。此外，处理组件602可以包括一个或多个模块，便于处理组件602和其他组件之间的交互。例如，处理组件602可以包括多媒体模块，以方便多媒体组件608和处理组件602之间的交互。 Processing component 602 generally controls the overall operation of device 600, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 602 may include one or more processors 620 to execute instructions to perform all or some of the steps of the methods described above. Additionally, processing component 602 may include one or more modules that facilitate interaction between processing component 602 and other components. For example, processing component 602 may include a multimedia module to facilitate interaction between multimedia component 608 and processing component 602.

存储器604被配置为存储各种类型的数据以支持在设备600的操作。这些数据的示例包括用于在设备600上操作的任何应用程序或方法的指令、联系人数据、电话簿数据、消息、图片、视频等。存储器604可以由任何类型的易失性或非易失性存储设备或者它们的组合实现，如静态随机存取存储器(SRAM)、电可擦除可编程只读存储器(EEPROM)、可擦除可编程只读存储器(EPROM)、可编程只读存储器(PROM)、只读存储器(ROM)、磁存储器、快闪存储器、磁盘或光盘。 Memory 604 is configured to store various types of data to support operation at device 600 . Examples of such data include instructions for any application or method operating on device 600, contact data, phonebook data, messages, pictures, videos, and the like. Memory 604 may be implemented by any type of volatile or nonvolatile storage device or combination thereof, such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable Programmable Read Only Memory (EPROM), Programmable Read Only Memory (PROM), Read Only Memory (ROM), Magnetic Memory, Flash Memory, Magnetic or Optical Disk.

电源组件606为设备600的各种组件提供电力。电源组件606可以包括电源管理***，一个或多个电源，及其他与为设备600生成、管理和分配电力相关联的组件。 Power supply assembly 606 provides power to various components of device 600 . Power components 606 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to device 600 .

多媒体组件608包括在所述设备600和用户之间的提供一个输出接口的屏幕。在一些实施例中，屏幕可以包括液晶显示器(LCD)和触摸面板(TP)。如果屏幕包括触摸面板，屏幕可以被实现为触摸屏，以接收来自用户的输入信号。触摸面板包括一个或多个触摸传感器以感测触摸、滑动和触摸面板上的手势。所述触摸传感器可以不仅感测触摸或滑动动作的边界，而且还检测与所述触摸或滑动操作相关的持续时间和压力。在一些实施例中，多媒体组件608包括一个前置摄像头和/或后置摄像头。当设备600处于操作模式，如拍摄模式或视频模式时，前置摄像头和/或后置摄像头可以接收外部的多媒体数据。每个前置摄像头和后置摄像头可以是一个固定的光学透镜***或具有焦距和光学变焦能力。 Multimedia component 608 includes a screen that provides an output interface between the device 600 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, swipe, and gestures on the touch panel. The touch sensor may not only sense the boundaries of a touch or swipe action, but also detect the duration and pressure associated with the touch or swipe action. In some embodiments, the multimedia component 608 includes a front-facing camera and/or a rear-facing camera. When the device 600 is in an operation mode, such as a shooting mode or a video mode, the front camera and/or the rear camera may receive external multimedia data. Each of the front and rear cameras can be a fixed optical lens system or have focal length and optical zoom capability.

音频组件610被配置为输出和/或输入音频信号。例如，音频组件610包括一个麦克风(MIC)，当设备600处于操作模式，如呼叫模式、记录模式和语音识别模式时，麦克风被配置为接收外部音频信号。所接收的音频信号可以被进一步存储在存储器604或经由通信组件616发送。在一些实施例中，音频组件610还包括一个扬声器，用于输出音频信号。 Audio component 610 is configured to output and/or input audio signals. For example, audio component 610 includes a microphone (MIC) that is configured to receive external audio signals when device 600 is in operating modes, such as call mode, recording mode, and voice recognition mode. The received audio signal may be further stored in memory 604 or transmitted via communication component 616 . In some embodiments, audio component 610 also includes a speaker for outputting audio signals.

I/O接口612为处理组件602和***接口模块之间提供接口，上述***接口模块可以是键盘，点击轮，按钮等。这些按钮可包括但不限于：主页按钮、音量按钮、启动按钮和锁定按钮。The I/O interface 612 provides an interface between the processing component 602 and a peripheral interface module, which may be a keyboard, a click wheel, a button, or the like. These buttons may include, but are not limited to: home button, volume buttons, start button, and lock button.

传感器组件614包括一个或多个传感器，用于为设备600提供各个方面的状态评估。例如，传感器组件614可以检测到设备600的打开/关闭状态，组件的相对定位，例如所述组件为设备600的显示器和小键盘，传感器组件614还可以检测设备600或设备600一个组件的位置改变，用户与设备600接触的存在或不存在，设备600方位或加速/减速和设备600的温度变化。传感器组件614可以包括接近传感器，被配置用来在没有任何的物理接触时检测附近物体的存在。传感器组件614还可以包括光传感器，如CMOS或CCD图像传感器，用于在成像应用中使用。在一些实施例中，该传感器组件614还可以包括加速度传感器、陀螺仪传感器、磁传感器、压力传感器或温度传感器。 Sensor assembly 614 includes one or more sensors for providing status assessments of various aspects of device 600 . For example, the sensor component 614 can detect the open/closed state of the device 600, the relative positioning of components, such as the display and keypad of the device 600, and the sensor component 614 can also detect a change in the position of the device 600 or a component of the device 600 , the presence or absence of user contact with the device 600 , the orientation or acceleration/deceleration of the device 600 and the temperature change of the device 600 . Sensor assembly 614 may include a proximity sensor configured to detect the presence of nearby objects in the absence of any physical contact. Sensor assembly 614 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 614 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

通信组件616被配置为便于设备600和其他设备之间有线或无线方式的通信。设备600可以接入基于通信标准的无线网络，如WiFi，运营商网络(如2G、3G、4G或5G)，或它们的组合。在一个示例性实施例中，通信组件616经由广播信道接收来自外部广播管理***的广播信号或广播相关信息。在一个示例性实施例中，所述通信组件616还包括近场通信(NFC)模块，以促进短程通信。例如，在NFC模块可基于射频识别(RFID)技术，红外数据协会(IrDA)技术，超宽带(UWB)技术，蓝牙(BT)技术和其他技术来实现。Communication component 616 is configured to facilitate wired or wireless communication between device 600 and other devices. Device 600 may access wireless networks based on communication standards, such as WiFi, carrier networks (eg, 2G, 3G, 4G, or 5G), or a combination thereof. In one exemplary embodiment, the communication component 616 receives broadcast signals or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 616 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

在示例性实施例中，设备600可以被一个或多个应用专用集成电路(ASIC)、数字信号处理器(DSP)、数字信号处理设备(DSPD)、可编程逻辑器件(PLD)、现场可编程门阵列(FPGA)、控制器、微控制器、微处理器或其他电子元件实现，用于执行上述方法。In an exemplary embodiment, device 600 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable A gate array (FPGA), controller, microcontroller, microprocessor or other electronic component implementation is used to perform the above method.

在示例性实施例中，还提供了一种包括指令的非临时性计算机可读存储介质，例如包括指令的存储器604，上述指令可由设备600的处理器620执行以完成上述方法。例如，所述非临时性计算机可读存储介质可以是ROM、随机存取存储器(RAM)、CD-ROM、磁带、软盘和光数据存储设备等。In an exemplary embodiment, there is also provided a non-transitory computer readable storage medium including instructions, such as memory 604 including instructions, executable by processor 620 of device 600 to perform the above method. For example, the non-transitory computer-readable storage medium may be ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like.

本公开所有实施例均可以单独被执行，也可以与其他实施例相结合被执行，均视为本公开要求的保护范围。All the embodiments of the present disclosure can be implemented independently or in combination with other embodiments, which are all regarded as the protection scope required by the present disclosure.

本领域技术人员在考虑说明书及实践本公开后，将容易想到本公开的其它实施方案。本申请旨在涵盖本公开的任何变型、用途或者适应性变化，这些变型、用途或者适应性变化遵循本公开的一般性原理并包括本公开未公开的本技术领域中的公知常识或惯用技术手段。说明书和实施例仅被视为示例性的，本公开的真正范围和精神由下面的权利要求指出。Other embodiments of the present disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the present disclosure. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or techniques in the technical field not disclosed by the present disclosure . The specification and examples are to be regarded as exemplary only, with the true scope and spirit of the disclosure being indicated by the following claims.

应当理解的是，本公开并不局限于上面已经描述并在附图中示出的精确结构，并且可以在不脱离其范围进行各种修改和改变。本公开的范围仅由所附的权利要求来限制。It is to be understood that the present disclosure is not limited to the precise structures described above and illustrated in the accompanying drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

一种相机外参标定方法，其特征在于，所述方法包括：A method for calibrating external parameters of a camera, characterized in that the method comprises:

对目标对象进行多相机拍摄，以获得不同相机视角下的多张目标对象图像；每个所述相机视角对应于所述多相机中的一个相机；Shooting the target object with multiple cameras to obtain multiple images of the target object under different camera perspectives; each of the camera perspectives corresponds to one camera in the multiple cameras;

将多张所述目标对象图像输入预训练的三维重建网络，生成所述目标对象在每张所述目标对象图像中的三维网格模型；Inputting multiple images of the target object into a pre-trained 3D reconstruction network to generate a 3D mesh model of the target object in each of the target object images;

确定各个所述相机视角下的所述三维网格模型在对应的相机坐标系中的顶点坐标集合；determining a vertex coordinate set of the three-dimensional mesh model in the corresponding camera coordinate system under each of the camera perspectives;

根据各个所述相机视角对应的顶点坐标集合在同一坐标系中的相对位置关系，确定各个所述相机视角对应相机的外参标定值。According to the relative positional relationship of the vertex coordinate sets corresponding to each of the camera angles of view in the same coordinate system, the external parameter calibration values of the cameras corresponding to each of the camera angles of view are determined.
根据权利要求1所述的相机外参标定方法，其特征在于，所述根据各个所述相机视角对应的顶点坐标集合在同一坐标系中的相对位置关系，确定各个所述相机视角对应相机的外参标定值，包括：The camera extrinsic parameter calibration method according to claim 1, wherein, according to the relative positional relationship of the vertex coordinate sets corresponding to each of the camera perspectives in the same coordinate system, the external parameters of the camera corresponding to each of the camera perspectives are determined. Parameter calibration values, including:

将各个所述相机视角中的其中一个相机视角作为基准相机视角；Using one of the camera angles of view as the reference camera angle of view;

基于所述基准相机视角对应的顶点坐标集合与各个所述相机视角对应的顶点坐标集合之间的顶点位置差异，确定各个所述相机视角对应相机的外参标定值。Based on the vertex position difference between the vertex coordinate set corresponding to the reference camera angle of view and the vertex coordinate set corresponding to each of the camera angles of view, the external parameter calibration value of the camera corresponding to each of the camera angles of view is determined.
根据权利要求2所述的相机外参标定方法，其特征在于，所述基于所述基准相机视角对应的顶点坐标集合与各个所述相机视角对应的顶点坐标集合之间的顶点位置差异，确定各个所述相机视角对应相机的外参标定值，包括：The camera extrinsic parameter calibration method according to claim 2, wherein the determination of each vertex coordinate based on the vertex position difference between the vertex coordinate set corresponding to the reference camera perspective and the vertex coordinate set corresponding to each of the camera perspectives is performed. The camera angle of view corresponds to the external parameter calibration value of the camera, including:

分别求解将各个所述相机视角对应的顶点坐标集合对齐到所述基准相机视角对应的顶点坐标集合所在坐标系上时所需的刚体变换；respectively solving the rigid body transformation required when aligning the vertex coordinate sets corresponding to each of the camera perspectives to the coordinate system where the vertex coordinate sets corresponding to the reference camera perspectives are located;

将求解得到的各个所述相机视角对应的刚体变换，作为各个所述相机视角对应相机的外参标定值。The rigid body transformation corresponding to each of the camera angles obtained through the solution is used as an external parameter calibration value of the camera corresponding to each of the camera angles.
根据权利要求1-3中任一项所述的相机外参标定方法，其特征在于，所述相机外参标定方法还包括：The camera extrinsic parameter calibration method according to any one of claims 1-3, wherein the camera extrinsic parameter calibration method further comprises:

将多张所述目标对象图像输入预训练的关键点检测网络，得到所述目标对象在每张所述目标对象图像中的二维关键点；Inputting a plurality of the target object images into a pre-trained key point detection network to obtain two-dimensional key points of the target object in each of the target object images;

基于各个所述相机视角对应的所述二维关键点的位置信息，对各个所述相机视角对应相机的外参标定值进行优化，得到各个所述相机视角对应相机的优化外参标定值。Based on the position information of the two-dimensional key points corresponding to each of the camera angles of view, the external parameter calibration values of the cameras corresponding to each of the camera angles of view are optimized, and the optimized external parameter calibration values of the cameras corresponding to each of the camera angles of view are obtained.
根据权利要求4所述的相机外参标定方法，其特征在于，所述基于各个所述相机视角对应的所述二维关键点的位置信息，对各个所述相机视角对应相机的外参标定值进行优化，得到各个所述相机视角对应相机的优化外参标定值，包括：The camera extrinsic parameter calibration method according to claim 4, wherein the extrinsic parameter calibration value of the camera corresponding to each of the camera perspectives is based on the position information of the two-dimensional key points corresponding to each of the camera perspectives Perform optimization to obtain the optimized external parameter calibration values of the cameras corresponding to each of the camera angles of view, including:

基于多张所述目标对象图像，确定各个所述相机视角下的所述二维关键点在对应的图像坐标系中的关键点二维坐标；Determine, based on a plurality of the target object images, the two-dimensional coordinates of the key points in the corresponding image coordinate system of the two-dimensional key points under each of the camera perspectives;

在各个所述相机视角对应的顶点坐标集合中，确定各个所述相机视角下的所述二维关键点在对应的相机坐标系中的关键点三维坐标；In the vertex coordinate set corresponding to each of the camera perspectives, determine the three-dimensional coordinates of the key points in the corresponding camera coordinate system of the two-dimensional key points under each of the camera perspectives;

根据各个所述相机视角对应的所述关键点三维坐标和各个所述相机视角下对应的所述关键点二维坐标，对各个所述相机视角对应相机的外参标定值进行优化，得到各个所述相机视角对应相机的优化外参标定值。According to the three-dimensional coordinates of the key points corresponding to each of the camera perspectives and the two-dimensional coordinates of the key points corresponding to each of the camera perspectives, the external parameter calibration values of the cameras corresponding to each of the camera perspectives are optimized to obtain each of the camera perspectives. The camera angle of view corresponds to the optimized external parameter calibration value of the camera.
根据权利要求5所述的相机外参标定方法，其特征在于，所述根据各个所述相机视角对应的所述关键点三维坐标和各个所述相机视角下对应的所述关键点二维坐标，对各个所述相机视角对应相机的外参标定值进行优化，得到各个所述相机视角对应相机的优化外参标定值，包括：The camera extrinsic parameter calibration method according to claim 5, wherein the three-dimensional coordinates of the key points corresponding to each of the camera perspectives and the two-dimensional coordinates of the key points corresponding to each of the camera perspectives, Optimizing the external parameter calibration values of the cameras corresponding to each of the camera angles of view, to obtain the optimized external parameter calibration values of the cameras corresponding to each of the camera angles of view, including:

根据各个所述相机视角对应的所述关键点三维坐标和各个所述相机视角下对应的所述关键点二维坐标，确定各个所述相机视角对应的相机在采用所述外参标定值时的投影误差；所述投影误差为所述相机视角对应的相机将所述关键点三维坐标投影至所述相机的像平面而得到的二维坐标与所述关键点二维坐标之间的误差；According to the three-dimensional coordinates of the key points corresponding to the respective camera perspectives and the two-dimensional coordinates of the key points corresponding to the respective camera perspectives, determine the camera corresponding to each camera perspective when the external parameter calibration value is used. Projection error; the projection error is the error between the two-dimensional coordinates obtained by projecting the three-dimensional coordinates of the key point onto the image plane of the camera and the two-dimensional coordinates of the key point by the camera corresponding to the camera angle of view;

基于各个所述相机视角对应的相机在采用所述外参标定值时的投影误差，对各个所述相机视角对应相机的所述外参标定值进行调整，得到各个所述相机视角对应相机的调整后外参标定值，作为各个所述相机视角对应相机的优化外参标定值；Based on the projection errors of the cameras corresponding to the camera perspectives when the external parameter calibration values are used, the external parameter calibration values of the cameras corresponding to the camera perspectives are adjusted to obtain the adjustment of the cameras corresponding to the camera perspectives. The back external parameter calibration value is used as the optimized external parameter calibration value of each camera corresponding to the camera angle of view;

其中，所述相机视角对应的相机在采用所述优化外参标定值时的投影误差满足预设条件。Wherein, the projection error of the camera corresponding to the camera angle of view when the optimized external parameter calibration value is adopted satisfies a preset condition.
根据权利要求6所述的相机外参标定方法，其特征在于，所述根据各个所述相机视角对应的所述关键点三维坐标和各个所述相机视角下对应的所述关键点二维坐标，确定各个所述相机视角对应的相机的投影误差，包括：The camera external parameter calibration method according to claim 6, wherein the three-dimensional coordinates of the key points corresponding to each of the camera perspectives and the two-dimensional coordinates of the key points corresponding to each of the camera perspectives, Determining the projection error of the camera corresponding to each of the camera angles of view includes:

通过预设的投影函数和各个所述相机视角对应相机的外参标定值，分别将各个所述相机视角对应的所述关键点三维坐标投影至对应相机的像平面上，得到各个所述相机视角对应的所述二维关键点在对应所述相机的像平面上的投影点；According to the preset projection function and the external parameter calibration values of the cameras corresponding to the camera angles of view, the three-dimensional coordinates of the key points corresponding to the camera angles of view are respectively projected onto the image plane of the corresponding camera to obtain the camera angles of view. the projection point of the corresponding two-dimensional key point on the image plane corresponding to the camera;

确定各个所述相机视角对应的所述投影点在对应的图像坐标系中的投影点二维坐标；determining the two-dimensional coordinates of the projection point in the corresponding image coordinate system of the projection point corresponding to each of the camera perspectives;

根据每个所述相机视角对应的所述投影点二维坐标和对应的所述关键点二维坐标之间的差异，确定各个所述相机视角对应的相机的投影误差。According to the difference between the two-dimensional coordinates of the projection point corresponding to each of the camera perspectives and the corresponding two-dimensional coordinates of the key points, the projection errors of the cameras corresponding to the respective camera perspectives are determined.
一种相机外参标定装置，其特征在于，包括：A camera external parameter calibration device, characterized in that it includes:

获取单元，被配置为执行对目标对象进行多相机拍摄，以获得不同相机视角下的多张目标对象图像；每个所述相机视角对应于所述多相机中的一个相机；an acquisition unit, configured to perform multi-camera shooting of the target object, so as to obtain a plurality of target object images under different camera perspectives; each of the camera perspectives corresponds to one camera in the multi-camera;

重建单元，被配置为执行将多张所述目标对象图像输入预训练的三维重建网络，生成所述目标对象在每张所述目标对象图像中的三维网格模型；a reconstruction unit, configured to input multiple images of the target object into a pre-trained 3D reconstruction network, and generate a 3D mesh model of the target object in each of the target object images;

确定单元，被配置为确定各个所述相机视角下的所述三维网格模型在对应的相机坐标系中的顶点坐标集合；a determining unit, configured to determine a vertex coordinate set of the three-dimensional mesh model in the corresponding camera coordinate system under each of the camera perspectives;

标定单元，被配置为执行根据各个所述相机视角对应的顶点坐标集合在同一坐标系中的相对位置关系，确定各个所述相机视角对应相机的外参标定值。The calibration unit is configured to determine the extrinsic parameter calibration value of the camera corresponding to each of the camera perspectives according to the relative positional relationship of the vertex coordinate sets corresponding to each of the camera perspectives in the same coordinate system.
根据权利要求1所述的相机外参标定装置，其特征在于，所述标定单元，被配置为执行将各个所述相机视角中的其中一个相机视角作为基准相机视角；基于所述基准相机视角对应的顶点坐标集合与各个所述相机视角对应的顶点坐标集合之间的顶点位置差异，确定各个所述相机视角对应相机的外参标定值。The camera extrinsic parameter calibration device according to claim 1, wherein the calibration unit is configured to use one of the camera angles of view as a reference camera angle of view; The vertex position difference between the vertex coordinate set corresponding to each of the camera angles of view and the vertex coordinate sets corresponding to each of the camera angles of view determines the external parameter calibration value of the camera corresponding to each of the camera angles of view.
根据权利要求2所述的相机外参标定装置，其特征在于，所述标定单元，被配置为执行分别求解将各个所述相机视角对应的顶点坐标集合对齐到所述基准相机视角对应的顶点坐标集合所在坐标系上时所需的刚体变换；将求解得到的各个所述相机视角对应的刚体变换，作为各个所述相机视角对应相机的外参标定值。The camera extrinsic parameter calibration device according to claim 2, wherein the calibration unit is configured to perform a separate solution to align the vertex coordinate sets corresponding to each of the camera angles of view to the vertex coordinates corresponding to the reference camera angle of view The rigid body transformation required when the set is located on the coordinate system; the rigid body transformation corresponding to each of the camera perspectives obtained through the solution is used as the external parameter calibration value of the camera corresponding to each of the camera perspectives.
根据权利要求8-10中任一项所述的相机外参标定装置，其特征在于，所述装置还包括：关键点检测单元，被配置为执行将多张所述目标对象图像输入预训练的关键点检测网络，得到所述目标对象在每张所述目标对象图像中的二维关键点；The camera extrinsic parameter calibration device according to any one of claims 8-10, wherein the device further comprises: a key point detection unit, configured to perform inputting a plurality of the target object images into a pre-training a key point detection network to obtain two-dimensional key points of the target object in each image of the target object;

优化单元，被配置为执行基于各个所述相机视角对应的所述二维关键点的位置信息，对各个所述相机视角对应相机的外参标定值进行优化，得到各个所述相机视角对应相机的优化外参标定值。The optimization unit is configured to perform optimization based on the position information of the two-dimensional key points corresponding to each of the camera angles of view, to optimize the external parameter calibration values of the cameras corresponding to the respective camera angles of view, and to obtain the corresponding camera angles of each of the camera angles. Optimize the external parameter calibration value.
根据权利要求11所述的相机外参标定装置，其特征在于，所述优化单元，被配置为执行基于多张所述目标对象图像，确定各个所述相机视角下的所述二维关键点在对应的图像坐标系中的关键点二维坐标；在各个所述相机视角对应的顶点坐标集合中，确定各个所述相机视角下的所述二维关键点在对应的相机坐标系中的关键点三维坐标；根据各个所述相机视角对应的所述关键点三维坐标和各个所述相机视角下对应的所述关键点二维坐标，对各个所述相机视角对应相机的外参标定值进行优化，得到各个所述相机视角对应相机的优化外参标定值。The camera extrinsic parameter calibration device according to claim 11, wherein the optimization unit is configured to perform, based on a plurality of images of the target object, determine that the two-dimensional key points in each of the camera perspectives are in The two-dimensional coordinates of the key points in the corresponding image coordinate system; in the vertex coordinate set corresponding to each of the camera perspectives, determine the key points of the two-dimensional key points in the corresponding camera coordinate system under each of the camera perspectives Three-dimensional coordinates; according to the three-dimensional coordinates of the key points corresponding to each of the camera perspectives and the two-dimensional coordinates of the key points corresponding to each of the camera perspectives, the external parameter calibration values of the cameras corresponding to each of the camera perspectives are optimized, The optimized external parameter calibration values of the cameras corresponding to each of the camera angles of view are obtained.
根据权利要求12所述的相机外参标定装置，其特征在于，所述优化单元，被配置为执行根据各个所述相机视角对应的所述关键点三维坐标和各个所述相机视角下对应的所述关键点二维坐标，确定各个所述相机视角对应的相机在采用所述外参标定值时的投影误差；所述投影误差为所述相机视角对应的相机将所述关键点三维坐标投影至所述相机的像平面而得到的二维坐标与所述关键点二维坐标之间的误差；基于各个所述相机视角对应的相机在采用所述外参标定值时的投影误差，对各个所述相机视角对应相机的所述外参标定值进行调整，得到各个所述相机视角对应相机的调整后外参标定值，作为各个所述相机视角对应相机的优化外参标定值；其中，所述相机视角对应的相机在采用所述优化外参标定值时的投影误差满足预设条件。The camera extrinsic parameter calibration device according to claim 12, wherein the optimization unit is configured to execute the three-dimensional coordinates of the key points corresponding to each of the camera perspectives and the corresponding three-dimensional coordinates of each of the camera perspectives. The two-dimensional coordinates of the key points are determined, and the projection errors of the cameras corresponding to the camera perspectives when using the external parameter calibration values are determined; the projection errors are the three-dimensional coordinates of the key points projected by the cameras corresponding to the camera perspectives to The error between the two-dimensional coordinates obtained from the image plane of the camera and the two-dimensional coordinates of the key point; based on the projection error of the camera corresponding to each of the camera angles when the external parameter calibration value is used, for each The camera angle of view is adjusted corresponding to the extrinsic parameter calibration value of the camera, and the adjusted extrinsic parameter calibration value of each camera angle of view corresponding to the camera is obtained as the optimized extrinsic parameter calibration value of the camera corresponding to each camera angle of view; wherein, the The projection error of the camera corresponding to the camera angle of view when the optimized external parameter calibration value is adopted satisfies the preset condition.
根据权利要求13所述的相机外参标定装置，其特征在于，所述优化单元，被配置为执行通过预设的投影函数和各个所述相机视角对应相机的外参标定值，分别将各个所述相机视角对应的所述关键点三维坐标投影至对应相机的像平面上，得到各个所述相机视角对应的所述二维关键点在对应所述相机的像平面上的投影点；确定各个所述相机视角对应的所述投影点在对应的图像坐标系中的投影点二维坐标；根据每个所述相机视角对应的所述投影点二维坐标和对应的所述关键点二维坐标之间的差异，确定各个所述相机视角对应的相机的投影误差。The camera extrinsic parameter calibration device according to claim 13, wherein the optimization unit is configured to perform a preset projection function and an extrinsic parameter calibration value of the camera corresponding to each of the camera angles of view, respectively The three-dimensional coordinates of the key points corresponding to the camera perspective are projected onto the image plane of the corresponding camera, and the projection points of the two-dimensional key points corresponding to the camera perspectives on the image plane corresponding to the camera are obtained; The two-dimensional coordinates of the projection point corresponding to the camera perspective in the corresponding image coordinate system; according to the difference between the two-dimensional coordinates of the projection point corresponding to each of the camera perspectives and the corresponding two-dimensional coordinates of the key points The difference between the two is to determine the projection error of the camera corresponding to each of the camera angles of view.
一种电子设备，其特征在于，包括：An electronic device, comprising:

处理器；processor;

用于存储所述处理器可执行指令的存储器；a memory for storing the processor-executable instructions;

其中，所述处理器被配置为执行所述指令，以实现以下步骤：wherein the processor is configured to execute the instructions to implement the following steps:

对目标对象进行多相机拍摄，以获得不同相机视角下的多张目标对象图像；每个所述相机视角对应于所述多相机中的一个相机；Shooting the target object with multiple cameras to obtain multiple images of the target object under different camera perspectives; each of the camera perspectives corresponds to one camera in the multiple cameras;

将多张所述目标对象图像输入预训练的三维重建网络，生成所述目标对象在每张所述目标对象图像中的三维网格模型；Inputting multiple images of the target object into a pre-trained 3D reconstruction network to generate a 3D mesh model of the target object in each of the target object images;

确定各个所述相机视角下的所述三维网格模型在对应的相机坐标系中的顶点坐标集合；determining a vertex coordinate set of the three-dimensional mesh model in the corresponding camera coordinate system under each of the camera perspectives;

根据各个所述相机视角对应的顶点坐标集合在同一坐标系中的相对位置关系，确定各个所述相机视角对应相机的外参标定值。According to the relative positional relationship of the vertex coordinate sets corresponding to each of the camera angles of view in the same coordinate system, the external parameter calibration values of the cameras corresponding to each of the camera angles of view are determined.
根据权利要求15所述的电子设备，其特征在于，所述处理器被配置为执行所述指令，以实现以下步骤：16. The electronic device of claim 15, wherein the processor is configured to execute the instructions to implement the following steps:

将各个所述相机视角中的其中一个相机视角作为基准相机视角；Using one of the camera angles of view as the reference camera angle of view;

基于所述基准相机视角对应的顶点坐标集合与各个所述相机视角对应的顶点坐标集合之间的顶点位置差异，确定各个所述相机视角对应相机的外参标定值。Based on the vertex position difference between the vertex coordinate set corresponding to the reference camera angle of view and the vertex coordinate set corresponding to each of the camera angles of view, the external parameter calibration value of the camera corresponding to each of the camera angles of view is determined.
根据权利要求16所述的电子设备，其特征在于，所述处理器被配置为执行所述指令，以实现以下步骤：17. The electronic device of claim 16, wherein the processor is configured to execute the instructions to implement the following steps:

分别求解将各个所述相机视角对应的顶点坐标集合对齐到所述基准相机视角对应的顶点坐标集合所在坐标系上时所需的刚体变换；respectively solving the rigid body transformation required when aligning the vertex coordinate sets corresponding to each of the camera perspectives to the coordinate system where the vertex coordinate sets corresponding to the reference camera perspectives are located;

将求解得到的各个所述相机视角对应的刚体变换，作为各个所述相机视角对应相机的外参标定值。The rigid body transformation corresponding to each of the camera angles obtained through the solution is used as an external parameter calibration value of the camera corresponding to each of the camera angles.
根据权利要求15-17中任一项所述的电子设备，其特征在于，所述处理器被配置为执行所述指令，以实现以下步骤：The electronic device of any one of claims 15-17, wherein the processor is configured to execute the instructions to implement the following steps:

将多张所述目标对象图像输入预训练的关键点检测网络，得到所述目标对象在每张所述目标对象图像中的二维关键点；Inputting a plurality of the target object images into a pre-trained key point detection network to obtain two-dimensional key points of the target object in each of the target object images;

基于各个所述相机视角对应的所述二维关键点的位置信息，对各个所述相机视角对应相机的外参标定值进行优化，得到各个所述相机视角对应相机的优化外参标定值。Based on the position information of the two-dimensional key points corresponding to each of the camera angles of view, the external parameter calibration values of the cameras corresponding to each of the camera angles of view are optimized, and the optimized external parameter calibration values of the cameras corresponding to each of the camera angles of view are obtained.
根据权利要求18所述的电子设备，其特征在于，所述处理器被配置为执行所述指令，以实现以下步骤：The electronic device of claim 18, wherein the processor is configured to execute the instructions to implement the following steps:

基于多张所述目标对象图像，确定各个所述相机视角下的所述二维关键点在对应的图像坐标系中的关键点二维坐标；Determine, based on a plurality of the target object images, the two-dimensional coordinates of the key points in the corresponding image coordinate system of the two-dimensional key points under each of the camera perspectives;

在各个所述相机视角对应的顶点坐标集合中，确定各个所述相机视角下的所述二维关键点在对应的相机坐标系中的关键点三维坐标；In the vertex coordinate set corresponding to each of the camera perspectives, determine the three-dimensional coordinates of the key points in the corresponding camera coordinate system of the two-dimensional key points under each of the camera perspectives;

根据各个所述相机视角对应的所述关键点三维坐标和各个所述相机视角下对应的所述关键点二维坐标，对各个所述相机视角对应相机的外参标定值进行优化，得到各个所述相机视角对应相机的优化外参标定值。According to the three-dimensional coordinates of the key points corresponding to each of the camera perspectives and the two-dimensional coordinates of the key points corresponding to each of the camera perspectives, the external parameter calibration values of the cameras corresponding to each of the camera perspectives are optimized to obtain each of the camera perspectives. The camera angle of view corresponds to the optimized external parameter calibration value of the camera.
根据权利要求19所述的电子设备，其特征在于，所述处理器被配置为执行所述指令，以实现以下步骤：19. The electronic device of claim 19, wherein the processor is configured to execute the instructions to implement the following steps:

根据各个所述相机视角对应的所述关键点三维坐标和各个所述相机视角下对应的所述关键点二维坐标，确定各个所述相机视角对应的相机在采用所述外参标定值时的投影误差；所述投影误差为所述相机视角对应的相机将所述关键点三维坐标投影至所述相机的像平面而得到的二维坐标与所述关键点二维坐标之间的误差；According to the three-dimensional coordinates of the key points corresponding to the respective camera perspectives and the two-dimensional coordinates of the key points corresponding to the respective camera perspectives, determine the camera corresponding to each camera perspective when the external parameter calibration value is used. Projection error; the projection error is the error between the two-dimensional coordinates obtained by projecting the three-dimensional coordinates of the key point onto the image plane of the camera and the two-dimensional coordinates of the key point by the camera corresponding to the camera angle of view;

基于各个所述相机视角对应的相机在采用所述外参标定值时的投影误差，对各个所述相机视角对应相机的所述外参标定值进行调整，得到各个所述相机视角对应相机的调整后外参标定值，作为各个所述相机视角对应相机的优化外参标定值；Based on the projection errors of the cameras corresponding to the camera perspectives when the external parameter calibration values are used, the external parameter calibration values of the cameras corresponding to the camera perspectives are adjusted to obtain the adjustment of the cameras corresponding to the camera perspectives. The back external parameter calibration value is used as the optimized external parameter calibration value of each camera corresponding to the camera angle of view;

其中，所述相机视角对应的相机在采用所述优化外参标定值时的投影误差满足预设条件。Wherein, the projection error of the camera corresponding to the camera angle of view when the optimized external parameter calibration value is adopted satisfies a preset condition.
根据权利要求20所述的电子设备，其特征在于，所述处理器被配置为执行所述指令，以实现以下步骤：21. The electronic device of claim 20, wherein the processor is configured to execute the instructions to implement the following steps:

通过预设的投影函数和各个所述相机视角对应相机的外参标定值，分别将各个所述相机视角对应的所述关键点三维坐标投影至对应相机的像平面上，得到各个所述相机视角对应的所述二维关键点在对应所述相机的像平面上的投影点；According to the preset projection function and the external parameter calibration values of the cameras corresponding to the camera angles of view, the three-dimensional coordinates of the key points corresponding to the camera angles of view are respectively projected onto the image plane of the corresponding camera to obtain the camera angles of view. the projection point of the corresponding two-dimensional key point on the image plane corresponding to the camera;

确定各个所述相机视角对应的所述投影点在对应的图像坐标系中的投影点二维坐标；determining the two-dimensional coordinates of the projection point in the corresponding image coordinate system of the projection point corresponding to each of the camera perspectives;

根据每个所述相机视角对应的所述投影点二维坐标和对应的所述关键点二维坐标之间的差异，确定各个所述相机视角对应的相机的投影误差。According to the difference between the two-dimensional coordinates of the projection point corresponding to each of the camera perspectives and the corresponding two-dimensional coordinates of the key points, the projection errors of the cameras corresponding to the respective camera perspectives are determined.
一种非暂时性机器可读存储介质，存储有指令，当所述指令由电子设备的处理器执行时，使得所述电子设备能够执行以下步骤：A non-transitory machine-readable storage medium storing instructions that, when executed by a processor of an electronic device, enable the electronic device to perform the following steps:

对目标对象进行多相机拍摄，以获得不同相机视角下的多张目标对象图像；每个所述相机视角对应于所述多相机中的一个相机；Shooting the target object with multiple cameras to obtain multiple images of the target object under different camera perspectives; each of the camera perspectives corresponds to one camera in the multiple cameras;

将多张所述目标对象图像输入预训练的三维重建网络，生成所述目标对象在每张所述目标对象图像中的三维网格模型；Inputting multiple images of the target object into a pre-trained 3D reconstruction network to generate a 3D mesh model of the target object in each of the target object images;

确定各个所述相机视角下的所述三维网格模型在对应的相机坐标系中的顶点坐标集合；Determine the vertex coordinate set in the corresponding camera coordinate system of the three-dimensional mesh model under each of the camera perspectives;

根据各个所述相机视角对应的顶点坐标集合在同一坐标系中的相对位置关系，确定各个所述相机视角对应相机的外参标定值。According to the relative positional relationship of the vertex coordinate sets corresponding to each of the camera angles of view in the same coordinate system, the external parameter calibration values of the cameras corresponding to each of the camera angles of view are determined.