WO2024108393A1 - 相机标定方法和装置、增强现实设备、可读存储介质 - Google Patents
相机标定方法和装置、增强现实设备、可读存储介质 Download PDFInfo
- Publication number
- WO2024108393A1 WO2024108393A1 PCT/CN2022/133551 CN2022133551W WO2024108393A1 WO 2024108393 A1 WO2024108393 A1 WO 2024108393A1 CN 2022133551 W CN2022133551 W CN 2022133551W WO 2024108393 A1 WO2024108393 A1 WO 2024108393A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- image
- camera
- corner point
- virtual rendering
- virtual
- Prior art date
Links
- 230000003190 augmentative effect Effects 0.000 title claims abstract description 65
- 238000000034 method Methods 0.000 title claims abstract description 55
- 238000004422 calculation algorithm Methods 0.000 claims description 31
- 238000012545 processing Methods 0.000 claims description 28
- 239000011159 matrix material Substances 0.000 claims description 20
- 238000001514 detection method Methods 0.000 claims description 18
- 238000006243 chemical reaction Methods 0.000 claims description 13
- 238000004590 computer program Methods 0.000 claims description 11
- 238000001914 filtration Methods 0.000 claims description 10
- 230000004044 response Effects 0.000 claims description 4
- 230000001131 transforming effect Effects 0.000 claims description 4
- 230000008569 process Effects 0.000 description 15
- 238000009877 rendering Methods 0.000 description 15
- 238000004891 communication Methods 0.000 description 10
- 238000005516 engineering process Methods 0.000 description 8
- 238000010586 diagram Methods 0.000 description 6
- 238000006073 displacement reaction Methods 0.000 description 6
- 238000003384 imaging method Methods 0.000 description 6
- 230000003993 interaction Effects 0.000 description 6
- 230000009466 transformation Effects 0.000 description 4
- 230000000694 effects Effects 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 230000001133 acceleration Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000005094 computer simulation Methods 0.000 description 2
- 238000007726 management method Methods 0.000 description 2
- 101001121408 Homo sapiens L-amino-acid oxidase Proteins 0.000 description 1
- 102100026388 L-amino-acid oxidase Human genes 0.000 description 1
- 230000009471 action Effects 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 239000007788 liquid Substances 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 239000010409 thin film Substances 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
Definitions
- the present disclosure relates to the field of augmented reality technology, and in particular to a camera calibration method and apparatus, an augmented reality device, and a readable storage medium.
- Augmented reality (AR) technology aims to enhance the user's view of the real world by superimposing computer-generated virtual elements on the real world. Therefore, it is necessary to ensure accurate spatial alignment between real-world scenes and virtual elements, which is particularly challenging in optical see-through head-mounted displays.
- Existing manual calibration procedures require user interaction to accurately align real-world scenes and virtual elements. This interaction process and the calibration effect are highly dependent on the operator's skills. The calibration process is cumbersome and time-consuming, and the head-mounted display should be repeatedly calibrated once it moves on the user's head, which seriously affects the user experience.
- the present disclosure provides a camera calibration method and apparatus, an augmented reality device, and a readable storage medium to address the deficiencies of related technologies.
- a camera calibration method which is applied to an augmented reality device, and includes:
- the first image refers to a rendered image rendered by a virtual rendering camera in the augmented reality device
- the second image refers to an image obtained by photographing the rendered image by a preset viewpoint camera when the augmented reality device displays the rendered image
- the preset viewpoint camera and the virtual rendering camera are calibrated according to the first image and the second image to obtain a calibration relationship between the preset viewpoint camera and the virtual rendering camera.
- calibrating the preset viewpoint camera and the virtual rendering camera according to the first image and the second image to obtain a calibration relationship between the preset viewpoint camera and the virtual rendering camera includes:
- a calibration relationship between the preset viewpoint camera and the virtual rendering camera is determined according to a preset pinhole camera model, the first relative posture, and the second relative posture.
- processing the first image to obtain a first corner point includes:
- processing the second image to obtain a second corner point includes:
- determining a first relative position between the virtual rendering camera and the virtual screen and a second relative position between the preset viewpoint camera and the virtual screen respectively according to the first corner point, the second corner point and the three-dimensional corner point includes:
- the first corner point and the three-dimensional corner point are input into the point pair motion algorithm, and a first relative position between the virtual rendering camera and the virtual screen output by the point pair motion algorithm is obtained; and the second corner point and the three-dimensional corner point are input into the point pair motion algorithm, and a second relative position between the preset viewpoint camera and the virtual screen output by the point pair motion algorithm is obtained.
- determining a calibration relationship between the preset viewpoint camera and the virtual rendering camera according to a preset pinhole camera model, the first relative pose, and the second relative pose includes:
- the product of the parameter matrix of the pinhole camera model, the conversion relationship and the adjustment relationship is obtained, and the product is used as the calibration relationship between the preset viewpoint camera and the virtual rendering camera.
- the method further comprises:
- the calibration relationship is adjusted according to the offset to obtain a target calibration relationship that matches the user.
- a camera calibration device which is applied to an augmented reality device, and the device includes:
- a first image acquisition module used to acquire a first image, where the first image refers to a rendered image rendered by a virtual rendering camera in the augmented reality device;
- a second image acquisition module used to acquire a second image, where the second image refers to an image obtained by taking the rendered image with a preset viewpoint camera when the augmented reality device displays the rendered image;
- a calibration relationship acquisition module is used to calibrate the preset viewpoint camera and the virtual rendering camera according to the first image and the second image to obtain a calibration relationship between the preset viewpoint camera and the virtual rendering camera.
- the calibration relationship acquisition module includes:
- a first corner point acquisition submodule used for processing the first image to obtain a first corner point and transforming the first corner point to obtain a three-dimensional corner point;
- a second corner point acquisition submodule used for processing the second image to obtain a second corner point
- a relative displacement acquisition submodule used to determine a first relative position between the virtual rendering camera and the virtual screen and a second relative position between the preset viewpoint camera and the virtual screen according to the first corner point, the second corner point and the three-dimensional corner point;
- a calibration relationship acquisition submodule is used to determine the calibration relationship between the preset viewpoint camera and the virtual rendering camera according to a preset pinhole camera model, the first relative posture and the second relative posture.
- the first corner point acquisition submodule includes:
- a first filtered image acquisition unit configured to perform filtering processing on the first image to obtain a first filtered image
- the first corner point acquisition unit is used to perform corner point detection on the first filtered image to obtain a first corner point.
- the second corner point acquisition submodule includes:
- a second filtered image acquisition unit configured to perform filtering on the second image to obtain a second filtered image
- the second corner point acquisition unit is used to perform corner point detection on the second filtered image to obtain second corner points.
- the relative displacement acquisition submodule includes:
- a motion algorithm acquisition unit used to acquire a preset point-to-point motion algorithm
- a relative posture acquisition unit is used to input the first corner point and the three-dimensional corner point into the point-to-motion algorithm, and obtain the first relative posture of the virtual rendering camera and the virtual screen output by the point-to-motion algorithm; and input the second corner point and the three-dimensional corner point into the point-to-motion algorithm, and obtain the second relative posture of the preset viewpoint camera and the virtual screen output by the point-to-motion algorithm.
- the calibration relationship acquisition submodule includes:
- a conversion relationship acquisition unit used for acquiring a conversion relationship between the preset viewpoint camera and the virtual rendering camera according to the first relative posture and the second relative posture;
- an adjustment relationship acquisition unit configured to acquire an adjustment relationship between the preset viewpoint camera and the virtual rendering camera according to the first relative posture and the second relative posture;
- a calibration relationship acquisition unit is used to acquire the product of the parameter matrix of the pinhole camera model, the conversion relationship and the adjustment relationship, and use the product as the calibration relationship between the preset viewpoint camera and the virtual rendering camera.
- the device further comprises:
- An offset acquisition module configured to acquire an offset corresponding to an adjustment operation in response to the user's adjustment operation
- the calibration relationship adjustment module is used to adjust the calibration relationship according to the offset to obtain a target calibration relationship that matches the user.
- an augmented reality device including:
- the memory is used to store a computer program executable by the processor
- the processor is used to execute the computer program in the memory to implement the above method.
- a non-transitory computer-readable storage medium is provided, and when an executable computer program in the storage medium is executed by a processor, the above-mentioned method can be implemented.
- a first image and a second image can be obtained, wherein the first image refers to a rendered image rendered by a virtual rendering camera in the augmented reality device; the second image refers to an image obtained by taking the rendered image by a preset viewpoint camera when the augmented reality device displays the rendered image; then, the preset viewpoint camera and the virtual rendering camera are calibrated according to the first image and the second image to obtain a calibration relationship between the preset viewpoint camera and the virtual rendering camera.
- the calibration relationship is automatically obtained through the images of the preset viewpoint camera and the virtual rendering camera, and the rendering image obtained by the virtual rendering camera can be corrected through the above calibration relationship, so that the rendering image can be accurately superimposed on the physical world, thereby improving the user experience.
- Fig. 1 is a flow chart showing a camera calibration method according to an exemplary embodiment.
- Fig. 2 is a flow chart showing a method of acquiring a calibration relationship according to an exemplary embodiment.
- Fig. 3 is a flow chart showing a method of obtaining a first corner point according to an exemplary embodiment.
- Fig. 4 is a flow chart showing a method of obtaining a second corner point according to an exemplary embodiment.
- Fig. 5 is a schematic diagram showing a position relationship of various cameras according to an exemplary embodiment.
- Fig. 6 is a block diagram showing a camera calibration method according to an exemplary embodiment.
- Fig. 7 is a block diagram showing a method of acquiring a calibration relationship according to an exemplary embodiment.
- Fig. 8 is a block diagram of a camera calibration device according to an exemplary embodiment.
- Fig. 9 is a block diagram showing an augmented reality device according to an exemplary embodiment.
- FIG1 is a camera calibration method shown according to an exemplary embodiment, which can be applied to an augmented reality device.
- the augmented reality device includes a preset viewpoint camera (viewpoint camera), including a left viewpoint camera and a right viewpoint camera.
- the left viewpoint camera and the right viewpoint camera can be used to capture images on the side to represent the images viewed by the human eye on the side.
- the augmented reality device has a rendering function, and outputs the rendered image to a display screen for display.
- the rendered image displayed on the display screen can be regarded as an image captured by a camera.
- the augmented reality device also includes a virtual rendering camera (rendering camera), which captures the virtual object to obtain the above-mentioned rendering image.
- a virtual rendering camera rendering camera
- the user of the augmented reality device he can see two images, namely the image actually projected by the physical world and the image rendered by the virtual rendering camera.
- a camera calibration method includes steps 11 and 12.
- step 11 a first image and a second image are obtained, wherein the first image refers to a rendered image rendered by a virtual rendering camera in the augmented reality device; and the second image refers to an image obtained by taking the rendered image by a preset viewpoint camera when the augmented reality device displays the rendered image.
- the virtual rendering camera in the augmented reality device can generate a rendering image and store the rendering image in a specified location.
- the processor in the augmented reality device can read the rendering image from the specified location and use the rendering image as the first image, that is, the processor can obtain the first image.
- the display screen in the augmented reality device can display the above-mentioned rendered image.
- the preset viewpoint camera can capture the display screen area to obtain a second image and store it in a specified location.
- the processor can read the above-mentioned second image from the above-mentioned specified location, that is, obtain the second image.
- step 12 the preset viewpoint camera and the virtual rendering camera are calibrated according to the first image and the second image to obtain a calibration relationship between the preset viewpoint camera and the virtual rendering camera.
- the display screen in the augmented reality device can calibrate the preset viewpoint camera and the virtual rendering camera according to the first image and the second image, see FIG. 2 , including steps 21 to 23.
- step 21 the first image is processed to obtain a first corner point and the first corner point is transformed to obtain a three-dimensional corner point; and the second image is processed to obtain a second corner point.
- the first image can be implemented using an Apriltag image, that is, the virtual rendering camera can process the rendered image into a two-dimensional code image and encode the corner points of the two-dimensional code image to obtain labels for each corner point.
- the Apriltag image since each corner point has a label, when the number of corner points of each first image exceeds a preset corner point number threshold, it can be used to solve the subsequent relative posture; or at least two first images are matched with labels to solve the subsequent relative posture, thereby ensuring the smooth implementation of the method provided in this embodiment.
- the processor may process the first image to obtain the first corner point, as shown in FIG. 3 , which includes steps 31 and 32 .
- the processor may perform filtering processing on the first image to obtain a first filtered image. It is understandable that the filtering processing is used to remove distortion in the first image, such as image deformation, rotation, stretching, etc., so as to obtain an accurate first filtered image.
- the processor may perform corner detection on the first filtered image to obtain a first corner point.
- the corner detection algorithm used in the above corner detection process may include but is not limited to Harris corner detection, Shi-Tomasi corner detection, FAST corner detection, scale-invariant SURF detection, scale-invariant SIFT detection and feature point description, etc.
- a suitable algorithm may be selected according to a specific scenario, and the corresponding solution falls within the protection scope of the present disclosure.
- the processor may process the second image to obtain the second corner point, as shown in FIG. 4 , which includes steps 41 and 42 .
- step 41 the processor may filter the second image to obtain a second filtered image.
- the solution of step 41 is the same as that of step 31, and the details may refer to the content of step 31, which will not be described here.
- step 42 the processor may perform corner point detection on the second filtered image to obtain a second corner point.
- the solution of step 42 is the same as that of step 32, and the details may refer to the content of step 32, which will not be repeated here.
- the parameter matrix of the display screen is stored in the augmented reality device, as shown in formula (1):
- w represents the width and height of the display screen
- h represents the height of the display screen
- vfov represents the vertical field of view of the display screen
- hfov represents the horizontal field of view of the display screen
- Kon represents the parameter matrix of the display screen.
- the processor can obtain the three-dimensional coordinates of the corner points on the virtual screen according to the height h, width w, vertical field angle vfov, horizontal field angle hfov and the first corner point of the parameter matrix of the display screen.
- the process of obtaining the three-dimensional coordinates can be seen as the inverse process of the physical world being mapped to the plane imaging process of the image sensor in the camera. According to the pinhole imaging principle, when the parameter matrix and the first corner point are known and the depth Z is set to 0, the coordinate data of the three-dimensional corner points corresponding to each corner point can be calculated.
- a first relative position between the virtual rendering camera and the virtual screen and a second relative position between the preset viewpoint camera and the virtual screen are determined respectively according to the first corner point, the second corner point and the three-dimensional corner point.
- W represents the world coordinate system
- S represents the tracking camera (tracking_camera), which can be implemented by a fisheye camera
- C represents the preset viewpoint camera, which can be implemented by a pineye camera; it is used to capture images of the virtual screen area
- E represents the virtual rendering camera, which is used to display the rendered image on the display screen.
- I R K eye ⁇ T _C _ S ⁇ T _S _ W ⁇ P W ;
- I V H _C _ E ⁇ P _E _ S ⁇ T _S _ W ⁇ P W ;
- Keye represents the intrinsic parameter matrix of the preset viewpoint camera, which is a known quantity and can be obtained through Zhang Zhengyou calibration method;
- T_S_W represents the posture from the physical world to the tracking camera, which is a known quantity;
- T_C_S is the extrinsic parameter matrix from the tracking camera to the preset viewpoint camera, which is a known quantity;
- H_C_E represents the transformation matrix (or Homegraph matrix) from the virtual rendering camera to the preset viewpoint camera imaging plane, which is an unknown quantity;
- P_E_S represents the transformation matrix from the tracking camera to the virtual rendering camera, which is a known quantity;
- PW represents a point in the physical world.
- the preset viewpoint camera and the display are modeled as an off-axis pinhole camera model, and its internal parameter matrix is set to Koff.
- the preset viewpoint camera corresponds to the projection center of the pinhole camera model
- the virtual screen corresponds to the camera imaging plane. Therefore, in this embodiment, any point in the preset viewpoint camera coordinate system can be projected to the virtual screen, as shown in formula (5):
- R _E_C represents the transformation matrix from the preset viewpoint camera to the virtual rendering camera, including data in yaw, pitch and roll dimensions;
- x _E_C , y _E_C , z _E_C represent the x, y and z axis components of the translation transformation from the preset viewpoint camera to the virtual rendering camera, respectively;
- d _screen_C represents the distance between the preset viewpoint camera and the imaging plane of the virtual rendering camera, that is, the distance from the human eye to the virtual image;
- K on is the parameter matrix of the virtual rendering camera (that is, an on-axis pinhole camera model), which is a known quantity.
- the above internal parameter matrix K off can be determined by determining R _E_C , x _E_C , y _E_C , and z _E_C .
- the preset point-to-point motion algorithm (PnP, Perspective-n-Point) stored in the augmented reality device is a method for solving the point-to-point motion from a 3D corner point to a 2D corner point, and the purpose is to solve the relative position and posture of the coordinate system where the 3D corner point is located relative to the coordinate system where the 2D corner point is located.
- the processor can obtain the preset point-to-point motion algorithm, input the above-mentioned first corner point and the above-mentioned three-dimensional corner point into the above-mentioned point-to-point motion algorithm, and the above-mentioned point-to-point motion algorithm processes the first corner point and the above-mentioned three-dimensional corner point and outputs the relative position and posture, and uses the relative displacement as the first relative position and posture of the virtual rendering camera and the virtual screen.
- the processor can obtain the preset point-to-point motion algorithm, input the above-mentioned second corner point and the above-mentioned three-dimensional corner point into the above-mentioned point-to-point motion algorithm, and the above-mentioned point-to-point motion algorithm processes the second corner point and the above-mentioned three-dimensional corner point and outputs the relative position and posture, and uses the relative displacement as the second relative position and posture of the preset viewpoint camera and the virtual screen.
- step 23 a calibration relationship between the preset viewpoint camera and the virtual rendering camera is determined according to the preset pinhole camera model, the first relative pose, and the second relative pose.
- the processor can determine the conversion relationship between the preset viewpoint camera and the virtual rendering camera according to the preset pinhole camera model, the first relative posture and the second relative posture.
- the conversion relationship R _E_C between the preset viewpoint camera and the virtual rendering camera is determined using the yaw, pitch and roll parameters in the first relative posture and the second relative posture.
- the processor can determine the adjustment relationship between the preset viewpoint camera and the virtual rendering camera according to the preset pinhole camera model, the first relative posture and the second relative posture. For example, the adjustment relationship between the preset viewpoint camera and the virtual rendering camera is determined by using the xyz component parameters in the first relative posture and the second relative posture.
- the processor can obtain the parameter matrix Kon of the virtual rendering camera, the product of the above conversion relationship and the above adjustment relationship, that is, calculate Koff according to formula (5), and use the above product as the calibration relationship between the preset viewpoint camera and the virtual rendering camera.
- the user can be allowed to perform an adjustment operation to adjust the parallax, such as allowing the user to manually move an object in the display screen to the correct position, such as the object from point A to point B.
- the processor responds to the user's adjustment operation and obtains the offset corresponding to the above adjustment operation, such as D_A_B .
- the processor can adjust the above calibration relationship according to the above offset, so as to obtain a target calibration relationship matching the above user, as shown in formula (6):
- K off ' represents the target calibration relationship
- dx, dy and dz represent the offset (or compensation) of the object in the xyz axis direction.
- the user only needs one operation to adjust the position and size of the object in the three-dimensional space, which can reduce the number of user interactions during the calibration process and improve the calibration efficiency; and the parallax of the viewpoint in the preset viewpoint camera can be compensated to achieve the effect of adjusting the calibration relationship to match the user, which is conducive to improving the calibration accuracy.
- the scheme provided in the embodiment of the present disclosure can obtain the first image and the second image, wherein the first image refers to the rendered image rendered by the virtual rendering camera in the augmented reality device; the second image refers to the image obtained by the preset viewpoint camera shooting the rendered image when the augmented reality device displays the rendered image; then, the preset viewpoint camera and the virtual rendering camera are calibrated according to the first image and the second image to obtain the calibration relationship between the preset viewpoint camera and the virtual rendering camera.
- the calibration relationship is automatically obtained through the images of the preset viewpoint camera and the virtual rendering camera, and the rendering image obtained by the virtual rendering camera can be corrected through the above calibration relationship, so that the rendering image can be accurately superimposed on the physical world, thereby improving the user experience.
- a camera calibration method provided by the present disclosure is described below in conjunction with FIG. 6 and FIG. 7 .
- the processor can receive the (undisplayed) rendering image generated by the rendering engine, i.e., the first image, and the displayed rendering image (or the upper screen image), and can obtain the upper screen image taken by the preset viewpoint camera to obtain the second image.
- the processor can perform calibration calculations based on the above-mentioned first image, second image, and upper screen image to obtain the external parameter matrix from the preset viewpoint camera to the virtual rendering camera, i.e., the above-mentioned calibration relationship, and apply it to the rendering display of the Unity3D tool.
- the preset degrees of freedom (such as movement in the xyz direction, yaw, pitch, and roll rotation) are adjusted through user interaction until a virtual-real combination effect that satisfies a specific user is achieved, thereby obtaining the compensation amount or offset on each degree of freedom, and obtaining the updated external parameter matrix from the preset viewpoint camera to the virtual rendering camera, i.e., the target calibration relationship.
- the preset viewpoint camera may acquire a first image (ie, an apriltag image), and the virtual rendering camera may acquire a second image (ie, an apriltag image).
- the image processing module filters the input apriltag image to obtain a filtered image; then performs corner point detection on the filtered image to obtain corner points; then, labels are matched on the corner points to obtain the first corner point or the second corner point. It is understandable that when the number of corner points in the apriltag image is greater than or equal to the preset corner point number threshold, the pose solution can be completed, avoiding the problem of being unable to calibrate when the black and white chess grids are not fully detected.
- the virtual 3D point generation module assigns a scale in the xy direction to each corner point on the virtual screen according to the vertical field of view, horizontal field of view, width and/or height of the virtual screen, sets Z to 0, and thereby obtains the three-dimensional coordinate data of the corner point on the virtual screen, that is, obtains the above-mentioned three-dimensional corner point.
- the relative posture solving module whose input includes the two-dimensional corner point output by the image processing module corresponding to the preset viewpoint camera, namely the second corner point, the two-dimensional corner point output by the image processing module corresponding to the virtual rendering camera, namely the first corner point, and the three-dimensional corner point output by the virtual 3D point generating module.
- the relative posture solving module can generate the first relative posture of the virtual rendering camera and the virtual screen according to the first corner point and the three-dimensional corner point, and generate the second relative posture of the preset viewpoint camera and the virtual screen according to the second corner point and the three-dimensional corner point.
- the system modeling and calibration quantity solving module is used to calculate the calibration relationship Koff according to the first relative posture and the second relative posture and the preset pinhole camera model. Alternatively, the system modeling and calibration quantity solving module adjusts the calibration relationship Koff according to the user's adjustment operation to obtain the target calibration relationship Koff'.
- an embodiment of the present disclosure further provides a camera calibration device, which is applicable to an augmented reality device.
- the device includes:
- a first image acquisition module 81 used to acquire a first image, where the first image refers to a rendered image rendered by a virtual rendering camera in the augmented reality device;
- a second image acquisition module 82 used to acquire a second image, where the second image refers to an image obtained by taking the rendered image with a preset viewpoint camera when the augmented reality device displays the rendered image;
- the calibration relationship acquisition module 83 is used to calibrate the preset viewpoint camera and the virtual rendering camera according to the first image and the second image to obtain a calibration relationship between the preset viewpoint camera and the virtual rendering camera.
- the calibration relationship acquisition module includes:
- a first corner point acquisition submodule used for processing the first image to obtain a first corner point and transforming the first corner point to obtain a three-dimensional corner point;
- a second corner point acquisition submodule used for processing the second image to obtain a second corner point
- a relative displacement acquisition submodule used to determine a first relative position between the virtual rendering camera and the virtual screen and a second relative position between the preset viewpoint camera and the virtual screen according to the first corner point, the second corner point and the three-dimensional corner point;
- a calibration relationship acquisition submodule is used to determine the calibration relationship between the preset viewpoint camera and the virtual rendering camera according to a preset pinhole camera model, the first relative posture and the second relative posture.
- the first corner point acquisition submodule includes:
- a first filtered image acquisition unit configured to perform filtering processing on the first image to obtain a first filtered image
- the first corner point acquisition unit is used to perform corner point detection on the first filtered image to obtain a first corner point.
- the second corner point acquisition submodule includes:
- a second filtered image acquisition unit configured to perform filtering on the second image to obtain a second filtered image
- the second corner point acquisition unit is used to perform corner point detection on the second filtered image to obtain second corner points.
- the relative displacement acquisition submodule includes:
- a motion algorithm acquisition unit used to acquire a preset point-to-point motion algorithm
- a relative pose acquisition unit is used to input the first corner point and the three-dimensional corner point into the point-to-motion algorithm, and obtain a first relative pose between the virtual rendering camera and the virtual screen output by the point-to-motion algorithm; and input the second corner point and the three-dimensional corner point into the point-to-motion algorithm, and obtain a second relative pose between the preset viewpoint camera and the virtual screen output by the point-to-motion algorithm.
- the calibration relationship acquisition submodule includes:
- a conversion relationship acquisition unit used for acquiring a conversion relationship between the preset viewpoint camera and the virtual rendering camera according to the first relative posture and the second relative posture;
- an adjustment relationship acquisition unit configured to acquire an adjustment relationship between the preset viewpoint camera and the virtual rendering camera according to the first relative posture and the second relative posture;
- a calibration relationship acquisition unit is used to acquire the product of the parameter matrix of the pinhole camera model, the conversion relationship and the adjustment relationship, and use the product as the calibration relationship between the preset viewpoint camera and the virtual rendering camera.
- the device further comprises:
- An offset acquisition module configured to acquire an offset corresponding to an adjustment operation in response to the user's adjustment operation
- the calibration relationship adjustment module is used to adjust the calibration relationship according to the offset to obtain a target calibration relationship that matches the user.
- the device embodiment shown in this embodiment matches the content of the above method embodiment, and the content of the above method embodiment can be referred to, which will not be repeated here.
- Fig. 9 is a block diagram of an augmented reality device according to an exemplary embodiment.
- the augmented reality device 900 may be a smart phone, a computer, a digital broadcast terminal, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.
- the augmented reality device 900 may include one or more of the following components: a processing component 902 , a memory 904 , a power component 906 , a multimedia component 908 , an audio component 910 , an input/output (I/O) interface 912 , a sensor component 914 , a communication component 916 , and an image acquisition component 918 .
- the processing component 902 generally controls the overall operation of the augmented reality device 900, such as operations associated with display, phone calls, data communications, camera operations, and recording operations.
- the processing component 902 may include one or more processors 920 to execute computer programs.
- the processing component 902 may include one or more modules to facilitate interaction between the processing component 902 and other components.
- the processing component 902 may include a multimedia module to facilitate interaction between the multimedia component 908 and the processing component 902.
- the memory 904 is configured to store various types of data to support the operation of the augmented reality device 900. Examples of such data include computer programs for any application or method operating on the augmented reality device 900, contact data, phone book data, messages, pictures, videos, etc.
- the memory 904 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.
- SRAM static random access memory
- EEPROM electrically erasable programmable read-only memory
- EPROM erasable programmable read-only memory
- PROM programmable read-only memory
- ROM read-only memory
- magnetic memory flash memory
- flash memory magnetic disk or optical disk.
- the power supply component 906 provides power to various components of the augmented reality device 900.
- the power supply component 906 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the augmented reality device 900.
- the power supply component 906 may include a power supply chip, and the controller may communicate with the power supply chip to control the power supply chip to turn on or off the switch device, so that the battery supplies power to the mainboard circuit or not.
- the multimedia component 908 includes a screen that provides an output interface between the augmented reality device 900 and the target object.
- the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input information from the target object.
- the touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.
- the audio component 910 is configured to output and/or input audio file information.
- the audio component 910 includes a microphone (MIC), and when the augmented reality device 900 is in an operation mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive external audio file information.
- the received audio file information can be further stored in the memory 904 or sent via the communication component 916.
- the audio component 910 also includes a speaker for outputting audio file information.
- the I/O interface 912 provides an interface between the processing component 902 and a peripheral interface module, which may be a keyboard, a click wheel, a button, etc.
- the sensor component 914 includes one or more sensors for providing various aspects of the state evaluation for the augmented reality device 900.
- the sensor component 914 can detect the on/off state of the augmented reality device 900, the relative positioning of the components, such as the display screen and the keypad of the augmented reality device 900, and the sensor component 914 can also detect the position change of the augmented reality device 900 or a component, the presence or absence of contact between the target object and the augmented reality device 900, the orientation or acceleration/deceleration of the augmented reality device 900, and the temperature change of the augmented reality device 900.
- the sensor component 914 can include a magnetic sensor, a gyroscope, and a magnetic field sensor, wherein the magnetic field sensor includes at least one of the following: a Hall sensor, a thin film magnetoresistive sensor, and a magnetic liquid acceleration sensor.
- the communication component 916 is configured to facilitate wired or wireless communication between the augmented reality device 900 and other devices.
- the augmented reality device 900 can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G, 5G, or a combination thereof.
- the communication component 916 receives broadcast information or broadcast-related information from an external broadcast management system via a broadcast channel.
- the communication component 916 also includes a near field communication (NFC) module to facilitate short-range communication.
- the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.
- RFID radio frequency identification
- IrDA infrared data association
- UWB ultra-wideband
- Bluetooth Bluetooth
- the augmented reality device 900 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components.
- ASICs application-specific integrated circuits
- DSPs digital signal processors
- DSPDs digital signal processing devices
- PLDs programmable logic devices
- FPGAs field programmable gate arrays
- controllers microcontrollers, microprocessors, or other electronic components.
- an augmented reality device comprising:
- the memory is used to store a computer program executable by the processor
- the processor is used to execute the computer program in the memory to implement the above method.
- a non-transitory computer readable storage medium such as a memory 904 including instructions, and the above executable computer program can be executed by a processor.
- the readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Computer Graphics (AREA)
- Computer Hardware Design (AREA)
- General Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Processing Or Creating Images (AREA)
Abstract
本公开是关于一种相机标定方法和装置、增强现实设备、可读存储介质。该方法包括：获取第一图像和第二图像，所述第一图像是指所述增强现实设备中虚拟渲染相机所渲染出的渲染图像；所述第二图像是指所述增强现实设备显示所述渲染图像时预设视点相机拍摄所述渲染图像得到的图像；根据所述第一图像和所述第二图像标定所述预设视点相机和所述虚拟渲染相机，得到所述预设视点相机和所述虚拟渲染相机之间的标定关系。本实施例中通过预设视点相机和虚拟渲染相机的图像进行自动获取标定关系，可以通过上述标定关系可以校正虚拟渲染相机所获取的渲染图像，使得渲染图像能够准确叠加到物理世界中，从而提升使用体验。
Description
本公开涉及增强现实技术领域，尤其涉及一种相机标定方法和装置、增强现实设备、可读存储介质。
视觉增强现实(augmented reality,AR)技术旨在通过在现实世界中叠加计算机生成的虚拟元素来增强用户对现实世界的看法，因此需要确保真实世界场景和虚拟元素之间的精确空间对齐，这个问题在光学透视式头戴显示屏中尤其具有挑战性。现有手动校准程序需要用户的交互完成，才能精确对齐真实世界场景和虚拟元素，此交互过程以及校准效果高度依赖于操作员的技能，校准过程繁琐耗时，并且头戴显示屏一旦在用户头上移动时都应该重复校准,严重影响用户体验。
发明内容
本公开提供一种相机标定方法和装置、增强现实设备、可读存储介质，以解决相关技术的不足。
根据本公开实施例的第一方面，提供一种相机标定方法，应用于增强现实设备，包括：
获取第一图像和第二图像，所述第一图像是指所述增强现实设备中虚拟渲染相机所渲染出的渲染图像；所述第二图像是指所述增强现实设备显示所述渲染图像时预设视点相机拍摄所述渲染图像得到的图像；
根据所述第一图像和所述第二图像标定所述预设视点相机和所述虚拟渲染相机，得到所述预设视点相机和所述虚拟渲染相机之间的标定关系。
可选地，根据所述第一图像和所述第二图像标定所述预设视点相机和所述虚拟渲染相机，得到所述预设视点相机和所述虚拟渲染相机之间的标定关系，包括：
对所述第一图像进行处理得到第一角点并对所述第一角点进行变换，得到三维角点；以及对所述第二图像进行处理得到第二角点；
根据所述第一角点、所述第二角点和所述三维角点分别确定所述虚拟渲染相机与虚拟屏幕的第一相对位姿以及预设视点相机与虚拟屏幕的第二相对位姿；
根据预设的针孔相机模型、所述第一相对位姿和所述第二相对位姿确定所述预设视点相机和所述虚拟渲染相机之间的标定关系。
可选地，对所述第一图像进行处理得到第一角点，包括：
对所述第一图像进行滤波处理，得到第一滤波图像；
对所述第一滤波图像进行角点检测，得到第一角点。
可选地，对所述第二图像进行处理得到第二角点，包括：
对所述第二图像进行滤波处理，得到第二滤波图像；
对所述第二滤波图像进行角点检测，得到第二角点。
可选地，根据所述第一角点、所述第二角点和所述三维角点分别确定所述虚拟渲染相机与虚拟屏幕的第一相对位姿以及预设视点相机与虚拟屏幕的第二相对位姿，包括：
获取预设的点对运动算法；
将所述第一角点和所述三维角点输入到所述点对运动算法，并获取所述点对运动算法输出的所述虚拟渲染相机与虚拟屏幕的第一相对位姿；以及将所述第二角点和所述三维角点输入到所述点对运动算法，并获取所述点对运动算法输出的所述预设视点相机与虚拟屏幕的第二相对位姿。
可选地，根据预设的针孔相机模型、所述第一相对位姿和所述第二相对位姿确定所述预设视点相机和所述虚拟渲染相机之间的标定关系，包括：
根据所述第一相对位姿和所述第二相对位姿获取所述预设视点相机和所述虚拟渲染相机之间的转换关系，以及根据所述第一相对位姿和所述第二相对位姿获取所述预设视点相机和所述虚拟渲染相机之间的调整关系；
获取所述针孔相机模型的参数矩阵、所述转换关系和所述调整关系的乘积，并将所述乘积作为所述预设视点相机和所述虚拟渲染相机之间的标定关系。
可选地，所述方法还包括：
响应于用户的调整操作，获取所述调整操作对应的偏移量；
根据所述偏移量调整所述标定关系，得到与所述用户相匹配的目标标定关系。
根据本公开实施例的第二方面，提供一种相机标定装置，应用于增强现实设备，所述装置包括：
第一图像获取模块，用于获取第一图像，所述第一图像是指所述增强现实设备中虚拟渲染相机所渲染出的渲染图像；
第二图像获取模块，用于获取第二图像，所述第二图像是指所述增强现实设备显示所述渲染图像时预设视点相机拍摄所述渲染图像得到的图像；
标定关系获取模块，用于根据所述第一图像和所述第二图像标定所述预设视点相机和所述虚拟渲染相机，得到所述预设视点相机和所述虚拟渲染相机之间的标定关系。
可选地，所述标定关系获取模块包括：
第一角点获取子模块，用于对所述第一图像进行处理得到第一角点并对所述第一角点进行变换，得到三维角点；
第二角点获取子模块，用于对所述第二图像进行处理得到第二角点；
相对位移获取子模块，用于根据所述第一角点、所述第二角点和所述三维角点分别确定所述虚拟渲染相机与虚拟屏幕的第一相对位姿以及预设视点相机与虚拟屏幕的第二相对位姿；
标定关系获取子模块，用于根据预设的针孔相机模型、所述第一相对位姿和所述第二相对位姿确定所述预设视点相机和所述虚拟渲染相机之间的标定关系。
可选地，所述第一角点获取子模块包括：
第一滤波图像获取单元，用于对所述第一图像进行滤波处理，得到第一滤波图像；
第一角点获取单元，用于对所述第一滤波图像进行角点检测，得到第一角点。
可选地，所述第二角点获取子模块包括：
第二滤波图像获取单元，用于对所述第二图像进行滤波处理，得到第二滤波图像；
第二角点获取单元，用于对所述第二滤波图像进行角点检测，得到第二角点。
可选地，所述相对位移获取子模块包括：
运动算法获取单元，用于获取预设的点对运动算法；
相对位姿获取单元，用于将所述第一角点和所述三维角点输入到所述点对运动算法，并获取所述点对运动算法输出的所述虚拟渲染相机与虚拟屏幕的第一相对位姿；以及将所述第二角点和所述三维角点输入到所述点对运动算法，并获取所述点对运动算法输出的所述预设视点相机与虚拟屏幕的第二相对位姿。
可选地，所述标定关系获取子模块，包括：
转换关系获取单元，用于根据所述第一相对位姿和所述第二相对位姿获取所述预设视点相机和所述虚拟渲染相机之间的转换关系；
调整关系获取单元，用于根据所述第一相对位姿和所述第二相对位姿获取所述预设视点相机和所述虚拟渲染相机之间的调整关系；
标定关系获取单元，用于获取所述针孔相机模型的参数矩阵、所述转换关系和所述调整关系的乘积，并将所述乘积作为所述预设视点相机和所述虚拟渲染相机之间的标定关系。
可选地，所述装置还包括：
偏移量获取模块，用于响应于用户的调整操作，获取所述调整操作对应的偏移量；
标定关系调整模块，用于根据所述偏移量调整所述标定关系，得到与所述用户相匹配的目标标定关系。
根据本公开实施例的第三方面，提供一种增强现实设备，包括：
预设视点相机；
显示模组；
存储器与处理器；
所述存储器用于存储所述处理器可执行的计算机程序；
所述处理器用于执行所述存储器中的计算机程序，以实现如上述的方法。
根据本公开实施例的第四方面，提供一种非暂态计算机可读存储介质，当所述存储介质中的可执行的计算机程序由处理器执行时，能够实现如上述的方法。
本公开的实施例提供的技术方案可以包括以下有益效果：
本公开实施例提供的方案中可以获取第一图像和第二图像，所述第一图像是指所述增强现实设备中虚拟渲染相机所渲染出的渲染图像；所述第二图像是指所述增强现实设备显示所述渲染图像时预设视点相机拍摄所述渲染图像得到的图像；然后，根据所述第一图像和所述第二图像标定所述预设视点相机和所述虚拟渲染相机，得到所述预设视点相机和所述虚拟渲染相机之间的标定关系。这样，本实施例中通过预设视点相机和虚拟渲染相机的图像进行自动获取标定关系，可以通过上述标定关系可以校正虚拟渲染相机所获取的渲染图像，使得渲染图像能够准确叠加到物理世界中，从而提升使用体验。
应当理解的是，以上的一般描述和后文的细节描述仅是示例性和解释性的，并不能限制本公开。
此处的附图被并入说明书中并构成本说明书的一部分，示出了符合本公开的实施例，并与说明书一起用于解释本公开的原理。
图1是根据一示例性实施例示出的一种相机标定方法的流程图。
图2是根据一示例性实施例示出的一种获取标定关系的流程图。
图3是根据一示例性实施例示出的一种获取第一角点的流程图。
图4是根据一示例性实施例示出的一种获取第二角点的流程图。
图5是根据一示例性实施例示出的一种各个相机位置关系的示意图。
图6是根据一示例性实施例示出的一种相机标定方法的框图。
图7是根据一示例性实施例示出的一种获取标定关系的框图。
图8是根据一示例性实施例示出的一种相机标定装置的框图。
图9是根据一示例性实施例示出的一种增强现实设备的框图。
这里将详细地对示例性实施例进行说明，其示例表示在附图中。下面的描述涉及附图时，除非另有表示，不同附图中的相同数字表示相同或相似的要素。以下示例性所描述的实施例并不代表与本公开相一致的所有实施例。相反，它们仅是与如所附权利要求书中所详述的、本公开的一些方面相一致的装置例子。需要说明的是，在不冲突的情况下，下述的实施例及实施方式中的特征可以相互组合。
为解决上述技术问题，本公开实施例提供了一种相机标定方法和装置、增强现实设备、可读存储介质。图1是根据一示例性实施例示出的一种相机标定方法，可以适用于增强现实设备。该增强现实设备包括预设视点相机(viewpoint camera)，包括左侧视点相机和右侧视点相机。左侧视点相机和右侧视点相机可以用于拍摄所在一侧的图像，以表示所在侧人眼所观看到的图像。并且增强现实设备具有渲染功能，并将渲染后的渲染图像输出到显示屏显示。本实施例中可以将显示屏中显示的渲染图像看成一个相机拍摄的图像，因此，增强现实设备还包括虚拟出的一个虚拟渲染相机(rendering camera)，该虚拟渲染相机对虚拟物体进行拍摄得到上述渲染图像。或者说，对于增强现实设备的用户而言，其可以看到两个图像即物理世界真实投射的图像和虚拟渲染相机渲染的图像。
参见图1，一种相机标定方法，包括步骤11～步骤12。
在步骤11中，获取第一图像和第二图像，所述第一图像是指所述增强现实设备中虚拟渲染相机所渲染出的渲染图像；所述第二图像是指所述增强现实设备显示所述渲染图像时预设视点相机拍摄所述渲染图像得到的图像。
本实施例中，增强现实设备中虚拟渲染相机可以生成渲染图像，并将渲染图像存储到指定位置。增强现实设备中处理器可以从上述指定位置读取上述渲染图像，并将上述 渲染图像作为第一图像，即处理器可以获取到第一图像。
本实施例中，增强现实设备中显示屏可以显示上述渲染图像，在显示图像过程中，预设视点相机可以拍摄显示屏区域得到第二图像，并存储到指定位置。处理器可以从上述指定位置读取上述第二图像，即获取到第二图像。
在步骤12中，根据所述第一图像和所述第二图像标定所述预设视点相机和所述虚拟渲染相机，得到所述预设视点相机和所述虚拟渲染相机之间的标定关系。
本实施例中，增强现实设备中显示屏可以根据所述第一图像和所述第二图像标定所述预设视点相机和所述虚拟渲染相机，参见图2，包括步骤21～步骤23。
在步骤21中，对所述第一图像进行处理得到第一角点并对所述第一角点进行变换，得到三维角点；以及对所述第二图像进行处理得到第二角点。
本实施例中，第一图像可以采用Apriltag图像实现，即虚拟渲染相机可以将渲染图像处理为二维码图像并将二维码图像的角点进行编码，得到各个角点的标签。可理解的是，本实施例中采用Apriltag图像实现时，由于各个角点具有标签，使得各张第一图像的角点数量超过预设角点数量阈值时即可用于求解后续的相对位姿；或者将至少2张第一图像进行标签匹配后求解后续的相对位姿，从而保证本实施例所提供方法的顺利实现。
本实施例中，处理器可以对第一图像进行处理得到第一角点，参见图3，包括步骤31～步骤32。
在步骤31中，处理器可以对所述第一图像进行滤波处理，得到第一滤波图像。可理解的是，上述滤波处理用于去除第一图像中的畸变，例如图像变形、旋转、拉伸等，从而得到准确的第一滤波图像。
在步骤32中，处理器可以对所述第一滤波图像进行角点检测，得到第一角点。上述角点检测过程使用的角点检测算法可以包括但不限于harris角点检测、Shi-Tomasi角点检测、FAST角点检测、尺度不变surf检测、尺度不变sift检测和特征点的描述等，可以根据具体场景选择合适的算法，相应方案落入本公开的保护范围。
本实施例中，处理器可以对第二图像进行处理得到第二角点，参见图4，包括步骤41～步骤42。
在步骤41中，处理器可以对所述第二图像进行滤波处理，得到第二滤波图像。步骤41的方案与步骤31的方案相同，具体可以参见步骤31的内容，在此不再赘述。
在步骤42中，处理器可以对所述第二滤波图像进行角点检测，得到第二角点。步骤42的方案与步骤32的方案相同，具体可以参见步骤32的内容，在此不再赘述。
在一示例中，增强现实设备内存储显示屏的参数矩阵，如式(1)所示：
式(1)，w表示显示屏的宽高，h表示显示屏的高度，vfov表示显示屏的垂直视场角，hfov表示显示屏的水平视场角，Kon表示显示屏的参数矩阵。
本实施例中，处理器可以根据显示屏的参数矩阵的高度h、宽度w、垂直视场角vfov、水平视场角hfov和第一角点可以获取虚拟屏幕上角点的三维坐标。可理解的是，获取三维坐标的过程可以看到物理世界映射到相机中图像传感器平面成像过程的逆过程，根据小孔成像原理，在参数矩阵已知且第一角点已知以及将深度Z设置0的情况下，可以计算出各个角点对应的三维角点的坐标数据。
在步骤22中，根据所述第一角点、所述第二角点和所述三维角点分别确定所述虚拟渲染相机与虚拟屏幕的第一相对位姿以及预设视点相机与虚拟屏幕的第二相对位姿。
本步骤中，参见图5，W表示世界坐标系；S表示跟踪相机(tracking_camera)，可以采用鱼眼相机实现；C表示预设视点相机，可以采用针眼相机实现；用于拍摄虚拟屏幕区域的图像；E表示虚拟渲染相机，用于将渲染图像显示在显示屏上。
其中，预设视点相机获取第二图像即真实图像成像过程如式(2)所示：
I
R＝K
eye×T
_C_S×T
_S_W×P
W； (2)
其中，虚拟渲染图像获取第一图像即虚拟图像成像的过程如式(3)所示：
I
V＝H
_C_E×P
_E_S×T
_S_W×P
W； (3)
其中，Keye表示预设视点相机的内参矩阵，为已知量，可以通过张正友标定法获取；T
_S_W表示物理世界到跟踪相机的姿态，为已知量；T
_C_S为跟踪相机到预设视点相机的外参矩阵，为已知量；H
_C_E表示虚拟渲染相机到预设视点相机成像平面(image plane)之间的转换矩阵(或者Homegraph矩阵)，为未知量；P
_E_S表示跟踪相机到虚拟渲染相机之间的变换矩阵，为已知量；P
W表示物理世界中的点。
当真实图像和虚拟图像结合时，真实图像上的实像像素与虚拟图像上的虚像像素相等，如式(4)所示：
K
eye×T
_C_S×T
_S_W×P
W＝H
_C_E×P
_E_S×T
_S_W×P
W； (4)
结合式(2)～式(4)可知，式(4)中仅存在H
_C_E这一个未知量。
为求解上述H
_C_E这个未知量，本实施例中将预设视点相机和显示器建模为一个离轴的针孔相机模型，其内参矩阵设置为Koff。在该针孔相机模型中，预设视点相机对应针孔相机模型的投影中心，虚拟屏幕对应相机成像平面。因此，本实施例中可以将预设视点相机坐标系下的任一点投影到虚拟屏幕，如式(5)所示：
式(5)中，R
_E_C表示预设视点相机到虚拟渲染相机的变换矩阵，包括yaw，pitch和roll维度的数据；x
_E_C、y
_E_C、z
_E_C分别表示预设视点相机到虚拟渲染相机的平移变换的x、y和z轴分量；d
_screen_C表示预设视点相机到虚拟渲染相机的成像平面之间距离，即人眼到虚像的距离；K
on是虚拟渲染相机(即一个在轴的针孔相机模型)的参数矩阵，为已知量。
基于上述式(5)可知，本实施例中确定出R
_E_C、x
_E_C、y
_E_C、z
_E_C即可 确定出上述内参矩阵K
off。
本实施例中，增强现实设备内存储预设的点对运动算法(PnP，Perspective-n-Point)是求解3D角点到2D角点点对运动的方法，目的是求解3D角点所在坐标系相对2D角点所在坐标系的相对位姿。因此，处理器可以获取预设的点对运动算法，将上述第一角点和上述三维角点输入到上述点对运动算法之内，由上述点对运动算法处理第一角点和上述三维角点并输出相对位姿，并将该相对位移作为虚拟渲染相机与虚拟屏幕的第一相对位姿。并且，处理器可以获取预设的点对运动算法，将上述第二角点和上述三维角点输入到上述点对运动算法之内，由上述点对运动算法处理第二角点和上述三维角点并输出相对位姿，并将该相对位移作为预设视点相机与虚拟屏幕的第二相对位姿。
在步骤23中，根据预设的针孔相机模型、所述第一相对位姿和所述第二相对位姿确定所述预设视点相机和所述虚拟渲染相机之间的标定关系。
本实施例中，处理器可以根据预设的针孔相机模型、第一相对位姿和第二相对位姿确定预设视点相机和虚拟渲染相机之间的转换关系。例如，利用第一相对位姿和第二相对位姿中的yaw，pitch和roll参数来确定预设视点相机和虚拟渲染相机之间的转换关系R
_E_C。
处理器可以根据预设的针孔相机模型、第一相对位姿和第二相对位姿确定预设视点相机和虚拟渲染相机之间的调整关系。例如，利用第一相对位姿和第二相对位姿中的xyz分量参数来确定预设视点相机和虚拟渲染相机之间的调整关系
本实施例中，处理器可以获取虚拟渲染相机的参数矩阵Kon、上述转换关系和上述调整关系的乘积，即根据式(5)计算Koff，并将上述乘积作为预设视点相机和虚拟渲染相机之间的标定关系。
由于不同用户的视眼距离不同，导致上述标定关系无法适配每一个用户。因此，在一实施例中，可以允许用户进行一次调整操作来调整视差，例如允许用户手动显示屏 中的某一个对象至正确位置，例如对象从A点到B点。此时，处理器响应于用户的调整操作，获取上述调整操作对应的偏移量，如D
_A_B。然后，处理器可以根据上述偏移量调整上述标定关系，从而得到与上述用户相匹配的目标标定关系，如式(6)所示：
式(6)中，K
off'表示目标标定关系，dx、dy和dz表示对象在xyz轴方向的偏移量(或者补偿量)。
本实施例中仅需要用户一次操作可以调整对象在三维空间中的位置和尺寸，可以降低标定过程中用户交互次数，提高标定效率；并且，可以补偿预设视点相机中视点的视差，达到调整标定关系以使之与用户相匹配的效果，有利于提高标定精度。
至此，本公开实施例提供的方案中可以获取第一图像和第二图像，所述第一图像是指所述增强现实设备中虚拟渲染相机所渲染出的渲染图像；所述第二图像是指所述增强现实设备显示所述渲染图像时预设视点相机拍摄所述渲染图像得到的图像；然后，根据所述第一图像和所述第二图像标定所述预设视点相机和所述虚拟渲染相机，得到所述预设视点相机和所述虚拟渲染相机之间的标定关系。这样，本实施例中通过预设视点相机和虚拟渲染相机的图像进行自动获取标定关系，可以通过上述标定关系可以校正虚拟渲染相机所获取的渲染图像，使得渲染图像能够准确叠加到物理世界中，从而提升使用体验。
下面结合图6和图7来描述本公开提供的一种相机标定方法。
参见图6，处理器可以接收渲染引擎生成的(未显示的)渲染图像即第一图像和已显示的渲染图像(或者说上屏图像)，并且可以获取预设视点相机拍摄的上屏图像得到第二图像。处理器可以根据上述第一图像、第二图像和上屏图像进行标定计算，得到预设视点相机到虚拟渲染相机的外参矩阵即上述的标定关系，并将其应用于Unity3D工具渲染显示。当不同用户佩戴时，通过用户交互调整预设的自由度(如xyz方向的移动，yaw、pitch和roll旋转)，直至实现满足特定用户的虚实结合效果，从而获得各个自由 度上的补偿量或偏移量，获得更新后预设视点相机到虚拟渲染相机的外参矩阵即目标标定关系。
参见图5、图6和图7，预设视点相机可以获取第一图像(即apriltag图像)，虚拟渲染相机可以获取第二图像(即apriltag图像)。
图像处理模块，对输入的apriltag图像进行滤波处理，得到滤波图像；然后对滤波图像进行角点检测处理，得到角点；之后，再对角点进行标签匹配，从而得到第一角点或者第二角点。可理解的是，当apriltag图像中的角点的数量大于或等于预设角点数量阈值时即可完成位姿求解，避免黑白棋格检测不全时无法标定的问题。
虚拟3D点生成模块，根据虚拟屏幕的垂直视场角、水平视场角、宽度和/或高度，为虚拟屏幕上的每个角点在xy方向赋予一个尺度，将Z设置为0，从而获取虚拟屏幕上角点的三维坐标数据，即得到上述三维角点。
相对位姿求解模块，其输入包括预设视点相机对应图像处理模块输出的二维角点即第二角点，虚拟渲染相机对应图像处理模块输出的二维角点即第一角点和虚拟3D点生成模块输出的三维角点。这样，相对位姿求解模块可以根据第一角点和三维角点生成虚拟渲染相机与虚拟屏幕的第一相对位姿，以及根据第二角点和三维角点生成预设视点相机与虚拟屏幕的第二相对位姿。
***建模待标定量求解模块，用于根据第一相对位姿和第二相对位姿以及预设的针孔相机模型计算出标定关系Koff。或者***建模待标定量求解模块根据用户的调整操作来调整上述标定关系Koff，得到目标标定关系Koff’。
在本公开实施例提供的一种相机标定方法的基础上，本公开实施例还提供了一种相机标定装置，适用于增强现实设备，参见图8，所述装置包括：
第一图像获取模块81，用于获取第一图像，所述第一图像是指所述增强现实设备中虚拟渲染相机所渲染出的渲染图像；
第二图像获取模块82，用于获取第二图像，所述第二图像是指所述增强现实设备显示所述渲染图像时预设视点相机拍摄所述渲染图像得到的图像；
标定关系获取模块83，用于根据所述第一图像和所述第二图像标定所述预设视 点相机和所述虚拟渲染相机，得到所述预设视点相机和所述虚拟渲染相机之间的标定关系。
可选地，所述标定关系获取模块包括：
第一角点获取子模块，用于对所述第一图像进行处理得到第一角点并对所述第一角点进行变换，得到三维角点；
第二角点获取子模块，用于对所述第二图像进行处理得到第二角点；
相对位移获取子模块，用于根据所述第一角点、所述第二角点和所述三维角点分别确定所述虚拟渲染相机与虚拟屏幕的第一相对位姿以及预设视点相机与虚拟屏幕的第二相对位姿；
标定关系获取子模块，用于根据预设的针孔相机模型、所述第一相对位姿和所述第二相对位姿确定所述预设视点相机和所述虚拟渲染相机之间的标定关系。
可选地，所述第一角点获取子模块包括：
第一滤波图像获取单元，用于对所述第一图像进行滤波处理，得到第一滤波图像；
第一角点获取单元，用于对所述第一滤波图像进行角点检测，得到第一角点。
可选地，所述第二角点获取子模块包括：
第二滤波图像获取单元，用于对所述第二图像进行滤波处理，得到第二滤波图像；
第二角点获取单元，用于对所述第二滤波图像进行角点检测，得到第二角点。
可选地，所述相对位移获取子模块包括：
运动算法获取单元，用于获取预设的点对运动算法；
相对位姿获取单元，用于将所述第一角点和所述三维角点输入到所述点对运动算法，并获取所述点对运动算法输出的所述虚拟渲染相机与虚拟屏幕的第一相对位姿；以及将所述第二角点和所述三维角点输入到所述点对运动算法，并获取所述点对运动算 法输出的所述预设视点相机与虚拟屏幕的第二相对位姿。
可选地，所述标定关系获取子模块，包括：
转换关系获取单元，用于根据所述第一相对位姿和所述第二相对位姿获取所述预设视点相机和所述虚拟渲染相机之间的转换关系；
调整关系获取单元，用于根据所述第一相对位姿和所述第二相对位姿获取所述预设视点相机和所述虚拟渲染相机之间的调整关系；
标定关系获取单元，用于获取所述针孔相机模型的参数矩阵、所述转换关系和所述调整关系的乘积，并将所述乘积作为所述预设视点相机和所述虚拟渲染相机之间的标定关系。
可选地，所述装置还包括：
偏移量获取模块，用于响应于用户的调整操作，获取所述调整操作对应的偏移量；
标定关系调整模块，用于根据所述偏移量调整所述标定关系，得到与所述用户相匹配的目标标定关系。
需要说明的是，本实施例中示出的装置实施例与上述方法实施例的内容相匹配，可以参考上述方法实施例的内容，在此不再赘述。
图9是根据一示例性实施例示出的一种增强现实设备的框图。例如，增强现实设备900可以是智能手机，计算机，数字广播终端，平板设备，医疗设备，健身设备，个人数字助理等。
参照图9，增强现实设备900可以包括以下一个或多个组件：处理组件902，存储器904，电源组件906，多媒体组件908，音频组件910，输入/输出(I/O)的接口912，传感器组件914，通信组件916，图像采集组件918。
处理组件902通常控制增强现实设备900的整体操作，诸如与显示，电话呼叫，数据通信，相机操作和记录操作相关联的操作。处理组件902可以包括一个或多个处理器920来执行计算机程序。此外，处理组件902可以包括一个或多个模块，便于处理组 件902和其他组件之间的交互。例如，处理组件902可以包括多媒体模块，以方便多媒体组件908和处理组件902之间的交互。
存储器904被配置为存储各种类型的数据以支持在增强现实设备900的操作。这些数据的示例包括用于在增强现实设备900上操作的任何应用程序或方法的计算机程序，联系人数据，电话簿数据，消息，图片，视频等。存储器904可以由任何类型的易失性或非易失性存储设备或者它们的组合实现，如静态随机存取存储器(SRAM)，电可擦除可编程只读存储器(EEPROM)，可擦除可编程只读存储器(EPROM)，可编程只读存储器(PROM)，只读存储器(ROM)，磁存储器，快闪存储器，磁盘或光盘。
电源组件906为增强现实设备900的各种组件提供电力。电源组件906可以包括电源管理***，一个或多个电源，及其他与为增强现实设备900生成、管理和分配电力相关联的组件。电源组件906可以包括电源芯片，控制器可以电源芯片通信，从而控制电源芯片导通或者断开开关器件，使电池向主板电路供电或者不供电。
多媒体组件908包括在增强现实设备900和目标对象之间的提供一个输出接口的屏幕。在一些实施例中，屏幕可以包括液晶显示屏(LCD)和触摸面板(TP)。如果屏幕包括触摸面板，屏幕可以被实现为触摸屏，以接收来自目标对象的输入信息。触摸面板包括一个或多个触摸传感器以感测触摸、滑动和触摸面板上的手势。触摸传感器可以不仅感测触摸或滑动动作的边界，而且还检测与触摸或滑动操作相关的持续时间和压力。
音频组件910被配置为输出和/或输入音频文件信息。例如，音频组件910包括一个麦克风(MIC)，当增强现实设备900处于操作模式，如呼叫模式、记录模式和语音识别模式时，麦克风被配置为接收外部音频文件信息。所接收的音频文件信息可以被进一步存储在存储器904或经由通信组件916发送。在一些实施例中，音频组件910还包括一个扬声器，用于输出音频文件信息。
I/O接口912为处理组件902和***接口模块之间提供接口，上述***接口模块可以是键盘，点击轮，按钮等。
传感器组件914包括一个或多个传感器，用于为增强现实设备900提供各个方 面的状态评估。例如，传感器组件914可以检测到增强现实设备900的打开/关闭状态，组件的相对定位，例如组件为增强现实设备900的显示屏和小键盘，传感器组件914还可以检测增强现实设备900或一个组件的位置改变，目标对象与增强现实设备900接触的存在或不存在，增强现实设备900方位或加速/减速和增强现实设备900的温度变化。本示例中，传感器组件914可以包括磁力传感器、陀螺仪和磁场传感器，其中磁场传感器包括以下至少一种：霍尔传感器、薄膜磁致电阻传感器、磁性液体加速度传感器。
通信组件916被配置为便于增强现实设备900和其他设备之间有线或无线方式的通信。增强现实设备900可以接入基于通信标准的无线网络，如WiFi，2G、3G、4G、5G，或它们的组合。在一个示例性实施例中，通信组件916经由广播信道接收来自外部广播管理***的广播信息或广播相关信息。在一个示例性实施例中，通信组件916还包括近场通信(NFC)模块，以促进短程通信。例如，在NFC模块可基于射频识别(RFID)技术，红外数据协会(IrDA)技术，超宽带(UWB)技术，蓝牙(BT)技术和其他技术来实现。
在示例性实施例中，增强现实设备900可以被一个或多个应用专用集成电路(ASIC)、数字信息处理器(DSP)、数字信息处理设备(DSPD)、可编程逻辑器件(PLD)、现场可编程门阵列(FPGA)、控制器、微控制器、微处理器或其他电子元件实现。
在示例性实施例中，还提供了一种增强现实设备，包括：
存储器与处理器；
所述存储器用于存储所述处理器可执行的计算机程序；
所述处理器用于执行所述存储器中的计算机程序，以实现如上述的方法。
在示例性实施例中，还提供了一种非暂态计算机可读存储介质，例如包括指令的存储器904，上述可执行的计算机程序可由处理器执行。其中，可读存储介质可以是ROM、随机存取存储器(RAM)、CD-ROM、磁带、软盘和光数据存储设备等。
本领域技术人员在考虑说明书及实践这里公开的公开后，将容易想到本公开的其它实施方案。本公开旨在涵盖任何变型、用途或者适应性变化，这些变型、用途或者 适应性变化遵循本公开的一般性原理并包括本公开未公开的本技术领域中的公知常识或惯用技术手段。说明书和实施例仅被视为示例性的，本公开的真正范围和精神由下面的权利要求指出。
应当理解的是，本公开并不局限于上面已经描述并在附图中示出的精确结构，并且可以在不脱离其范围进行各种修改和改变。本公开的范围仅由所附的权利要求来限制。
Claims (10)
- 一种相机标定方法，其特征在于，应用于增强现实设备，包括：获取第一图像和第二图像，所述第一图像是指所述增强现实设备中虚拟渲染相机所渲染出的渲染图像；所述第二图像是指所述增强现实设备显示所述渲染图像时预设视点相机拍摄所述渲染图像得到的图像；根据所述第一图像和所述第二图像标定所述预设视点相机和所述虚拟渲染相机，得到所述预设视点相机和所述虚拟渲染相机之间的标定关系。
- 根据权利要求1所述的方法，其特征在于，根据所述第一图像和所述第二图像标定所述预设视点相机和所述虚拟渲染相机，得到所述预设视点相机和所述虚拟渲染相机之间的标定关系，包括：对所述第一图像进行处理得到第一角点并对所述第一角点进行变换，得到三维角点；以及对所述第二图像进行处理得到第二角点；根据所述第一角点、所述第二角点和所述三维角点分别确定所述虚拟渲染相机与虚拟屏幕的第一相对位姿以及预设视点相机与虚拟屏幕的第二相对位姿；根据预设的针孔相机模型、所述第一相对位姿和所述第二相对位姿确定所述预设视点相机和所述虚拟渲染相机之间的标定关系。
- 根据权利要求2所述的方法，其特征在于，对所述第一图像进行处理得到第一角点，包括：对所述第一图像进行滤波处理，得到第一滤波图像；对所述第一滤波图像进行角点检测，得到第一角点。
- 根据权利要求2所述的方法，其特征在于，对所述第二图像进行处理得到第二角点，包括：对所述第二图像进行滤波处理，得到第二滤波图像；对所述第二滤波图像进行角点检测，得到第二角点。
- 根据权利要求2所述的方法，其特征在于，根据所述第一角点、所述第二角点和所述三维角点分别确定所述虚拟渲染相机与虚拟屏幕的第一相对位姿以及预设视点相机与虚拟屏幕的第二相对位姿，包括：获取预设的点对运动算法；将所述第一角点和所述三维角点输入到所述点对运动算法，并获取所述点对运动算法输出的所述虚拟渲染相机与虚拟屏幕的第一相对位姿；以及将所述第二角点和所述三维角点输入到所述点对运动算法，并获取所述点对运动算法输出的所述预设视点相机与虚拟屏幕的第二相对位姿。
- 根据权利要求2所述的方法，其特征在于，根据预设的针孔相机模型、所述第一相对位姿和所述第二相对位姿确定所述预设视点相机和所述虚拟渲染相机之间的标定关系，包括：根据所述第一相对位姿和所述第二相对位姿获取所述预设视点相机和所述虚拟渲染相机之间的转换关系，以及根据所述第一相对位姿和所述第二相对位姿获取所述预设视点相机和所述虚拟渲染相机之间的调整关系；获取所述针孔相机模型的参数矩阵、所述转换关系和所述调整关系的乘积，并将所述乘积作为所述预设视点相机和所述虚拟渲染相机之间的标定关系。
- 根据权利要求1所述的方法，其特征在于，所述方法还包括：响应于用户的调整操作，获取所述调整操作对应的偏移量；根据所述偏移量调整所述标定关系，得到与所述用户相匹配的目标标定关系。
- 一种相机标定装置，其特征在于，应用于增强现实设备，所述装置包括：第一图像获取模块，用于获取第一图像，所述第一图像是指所述增强现实设备中虚拟渲染相机所渲染出的渲染图像；第二图像获取模块，用于获取第二图像，所述第二图像是指所述增强现实设备显示所述渲染图像时预设视点相机拍摄所述渲染图像得到的图像；标定关系获取模块，用于根据所述第一图像和所述第二图像标定所述预设视点相机和所述虚拟渲染相机，得到所述预设视点相机和所述虚拟渲染相机之间的标定关系。
- 一种增强现实设备，其特征在于，包括：预设视点相机；显示模组；存储器与处理器；所述存储器用于存储所述处理器可执行的计算机程序；所述处理器用于执行所述存储器中的计算机程序，以实现如权利要求1～7任一项所 述的方法。
- 一种非暂态计算机可读存储介质，其特征在于，当所述存储介质中的可执行的计算机程序由处理器执行时，能够实现如权利要求1～7任一项所述的方法。
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| PCT/CN2022/133551 WO2024108393A1 (zh) | 2022-11-22 | 2022-11-22 | 相机标定方法和装置、增强现实设备、可读存储介质 |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| PCT/CN2022/133551 WO2024108393A1 (zh) | 2022-11-22 | 2022-11-22 | 相机标定方法和装置、增强现实设备、可读存储介质 |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| WO2024108393A1 true WO2024108393A1 (zh) | 2024-05-30 |
Family
ID=91194932
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| PCT/CN2022/133551 WO2024108393A1 (zh) | 2022-11-22 | 2022-11-22 | 相机标定方法和装置、增强现实设备、可读存储介质 |
Country Status (1)
| Country | Link |
|---|---|
| WO (1) | WO2024108393A1 (zh) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105320271A (zh) * | 2014-07-10 | 2016-02-10 | 精工爱普生株式会社 | 利用直接几何建模的头戴式显示器校准 |
| US20190318546A1 (en) * | 2016-12-28 | 2019-10-17 | Cloudminds (Shenzhen) Robotics Systems Co., Ltd. | Method and apparatus for processing display data |
| CN111164971A (zh) * | 2017-07-31 | 2020-05-15 | 脸谱公司 | 用于3d内容的视差观察器*** |
| CN113902796A (zh) * | 2021-08-31 | 2022-01-07 | 杭州易现先进科技有限公司 | 一种基于图像的头显设备的校准方法和*** |
-
2022
- 2022-11-22 WO PCT/CN2022/133551 patent/WO2024108393A1/zh unknown
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105320271A (zh) * | 2014-07-10 | 2016-02-10 | 精工爱普生株式会社 | 利用直接几何建模的头戴式显示器校准 |
| US20190318546A1 (en) * | 2016-12-28 | 2019-10-17 | Cloudminds (Shenzhen) Robotics Systems Co., Ltd. | Method and apparatus for processing display data |
| CN111164971A (zh) * | 2017-07-31 | 2020-05-15 | 脸谱公司 | 用于3d内容的视差观察器*** |
| CN113902796A (zh) * | 2021-08-31 | 2022-01-07 | 杭州易现先进科技有限公司 | 一种基于图像的头显设备的校准方法和*** |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10078367B2 (en) | Stabilization plane determination based on gaze location | |
| US11308692B2 (en) | Method and device for processing image, and storage medium | |
| WO2020035002A1 (en) | Methods and devices for acquiring 3d face, and computer readable storage media | |
| US8648879B2 (en) | Apparatus and method for tracking augmented reality content | |
| US10133364B2 (en) | Image processing apparatus and method | |
| WO2022037285A1 (zh) | 相机外参标定方法及装置 | |
| WO2018040328A1 (zh) | 测试虚拟现实头显设备软件的方法及装置 | |
| CN109685915B (zh) | 一种图像处理方法、装置及移动终端 | |
| US20120236180A1 (en) | Image adjustment method and electronics system using the same | |
| CN113643356B (zh) | 相机位姿确定、虚拟物体显示方法、装置及电子设备 | |
| US20220375258A1 (en) | Image processing method and apparatus, device and storage medium | |
| KR101690646B1 (ko) | 시 스루 디스플레이를 위한 카메라 구동 장치 및 그 방법 | |
| CN110969658B (zh) | 利用来自多个设备的图像进行定位和标测 | |
| TW202025719A (zh) | 圖像處理方法及裝置、電子設備及儲存介質 | |
| CN112541400B (zh) | 基于视线估计的行为识别方法及装置、电子设备、存储介质 | |
| TW201915663A (zh) | 用於眼球視線校正的影像處理方法及其系統 | |
| WO2023169283A1 (zh) | 双目立体全景图像的生成方法、装置、设备、存储介质和产品 | |
| US9600938B1 (en) | 3D augmented reality with comfortable 3D viewing | |
| CN113870213A (zh) | 图像显示方法、装置、存储介质以及电子设备 | |
| US9536133B2 (en) | Display apparatus and control method for adjusting the eyes of a photographed user | |
| TWI603225B (zh) | 液晶顯示器顯示視角的調整方法和裝置 | |
| CN109934168B (zh) | 人脸图像映射方法及装置 | |
| TWI731430B (zh) | 資訊顯示方法與資訊顯示系統 | |
| WO2024108393A1 (zh) | 相机标定方法和装置、增强现实设备、可读存储介质 | |
| CN115100253A (zh) | 图像对比方法、装置、电子设备及存储介质 |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| 121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 22966092 Country of ref document: EP Kind code of ref document: A1 |