WO2013125098A1 - System and method for computer graphics image processing using augmented reality technology - Google Patents
System and method for computer graphics image processing using augmented reality technology Download PDFInfo
- Publication number
- WO2013125098A1 WO2013125098A1 PCT/JP2012/078175 JP2012078175W WO2013125098A1 WO 2013125098 A1 WO2013125098 A1 WO 2013125098A1 JP 2012078175 W JP2012078175 W JP 2012078175W WO 2013125098 A1 WO2013125098 A1 WO 2013125098A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- image
- computer graphics
- camera
- marker
- virtual camera
- Prior art date
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/006—Mixed reality
Definitions
- the present invention relates to a computer graphics image processing system and method using augmented reality (AR).
- AR augmented reality
- CG computer graphics
- AR augmented reality
- General AR technology basically has the processing contents shown in FIG. That is, in STEP 1 and STEP 2, a scene including the AR marker 101 is captured by the camera 103 such as a web camera or a digital video camera to acquire the camera frame 105. In STEP 3, the position detection of the AR marker image 101 in the camera frame 105 is performed. Spatial image recognition is performed, and in step 4, the CG object 107 previously associated with the AR marker image 101 according to the position, posture, and scale of the AR marker image 101 is synthesized and displayed with the same posture and scale.
- the camera 103 such as a web camera or a digital video camera to acquire the camera frame 105.
- the position detection of the AR marker image 101 in the camera frame 105 is performed. Spatial image recognition is performed, and in step 4, the CG object 107 previously associated with the AR marker image 101 according to the position, posture, and scale of the AR marker image 101 is synthesized and displayed with the same posture and scale.
- pinhole camera model In computer vision that performs such AR processing, it is generally approximated using a pinhole camera model.
- the idea of the pinhole camera model is that all the light reaching the image plane passes through a pinhole that is the focal point of one point and forms an image at a position intersecting the image plane. Such a projection is called a central projection.
- the intersection of the optical axis 111 and the image plane 113 is set as the origin O1, and the x axis and the y axis are taken on the image plane 113 in accordance with the image sensor arrangement direction of the camera 103.
- the coordinate system is called an image coordinate system.
- a coordinate system in which the pinhole O2 is regarded as the center of the camera 103, the direction of the optical axis 111 is the Z axis, and the X axis and the Y axis are parallel to the x axis and the y axis of the image coordinate system is defined as a camera coordinate system. Call.
- digital images that are actually captured are images that are recorded through correction by a lens or a computer.
- the origin of the image, the aspect ratio of the pixel, etc., depending on the mechanical characteristics of the camera 103, lens distortion, image sensor characteristics, etc. It does not match that of the actual (x, y, z) image coordinate system. Therefore, in the digital image, a coordinate system is set in which the coordinate origin is set at the upper left, the right direction is the u axis, and the vertical direction is the v axis, and this is called a digital image coordinate system.
- Such a projective transformation matrix P is given by the camera internal parameter matrix A, the rotation matrix R, and the translation vector t.
- the rotation matrix R is a 3 ⁇ 3 matrix
- t] is a homogeneous coordinate system, and is represented as a 3 ⁇ 4 matrix.
- Such determination of the internal parameter matrix A, rotation matrix R, and translation vector t of the camera 103 is called camera internal parameter estimation or camera calibration.
- the camera calibration pattern examples P1 and P2 as shown in FIG.
- the solution of the equation is obtained from the correlation obtained from a plurality of images, and camera parameters are determined.
- Zang's method is used. This technique is described in Non-Patent Document 1 below.
- the system that detects the position of the AR marker image 101 from the digital image 105 actually captured by the camera 103 shown in FIG. 22 by image recognition using the camera parameters obtained in this way is called an AR analyzer.
- the detected orientation of the AR marker image 101 is drawn with a three-dimensional CG, so that 4 ⁇ used in general three-dimensional computer graphics calculations.
- 4 projection matrices Pa and 4 ⁇ 4 model view matrix Ma are calculated, and an arbitrary point in the three-dimensional space is projectively transformed and displayed with reference to the position of the AR marker image 101 on the digital image 105.
- the projection matrix Pa is defined when the pinhole camera model is defined as the frustum 121 shown in FIG.
- transposed matrix Or its transpose matrix.
- the above notation is used here because the matrix calculation direction may be reversed.
- the upper left vertex of the upper bottom surface SCR-A before the frustum 121 is (l, t, ⁇ n)
- the bottom left vertex of the bottom surface SCR-A is (l, b, -n)
- the top right vertex is (r, t, -n)
- the bottom right vertex is (r, b, -n)
- the top bottom surface is f.
- the projection matrix Pa becomes a fixed value in the imaging system of the AR analyzer
- the model view matrix Ma represents the detection position, orientation, and scale of the AR marker image.
- CG computer graphics
- CG computer graphics
- the reason why the model view matrix Ma cannot be determined is none other than when the AR marker image cannot be recognized.
- the AR marker image 101 is not included in the range that can be recognized by the AR analyzer in the digital image 105 captured by the camera 103 as shown in FIG.
- the AR marker image 101 is blurred in the digital image 105 photographed due to the rapid change of the moving speed and the angle of view due to the performance of the image sensor of the camera.
- a so-called markerless AR technology that does not use a dedicated AR marker has been developed.
- This is an AR technology that detects a feature point from the shape of a real-world object such as a mountain or a human face in a frame photographed by a camera, and performs posture detection and solid identification.
- Even in this markerless AR technology in a system that uses a feature point group as an identification target, when the feature point group is out of the camera frame or when the feature point group is difficult to identify due to a change in the angle of view, the same problem is encountered. Occurs.
- the present invention has been made in order to solve the above-described problems of the prior art that occur when CG is synthesized and displayed using the image recognition AR technology, and restricts the camera position, angle of view, and camera work.
- the purpose is to provide AR technology that is less imposed.
- a camera for detecting the position of an AR marker is always a camera that observes a fixed point, and a virtual camera is defined in the CG space and then the angle of view and position on the virtual camera side are changed.
- one feature of the present invention is that the AR marker is captured, a fixed camera whose position is fixed, a parameter setting unit that stores camera parameters of the fixed camera, and the fixed camera
- the camera parameters stored in the parameter setting unit are used to analyze the AR marker position, posture and scale, and the AR marker
- the corresponding object is placed at a position corresponding to the position of the AR marker image on the image frame on the computer graphics image space based on the analysis result of the AR marker attitude analysis unit.
- An object generated as a computer graphics image of a posture and scale corresponding to the scale And a computer graphics image of the object generated by the object image generation unit when viewed from a virtual camera installed at a predetermined coordinate position on the computer graphics image space.
- a virtual camera observation image generation unit that determines the appearance and generates a virtual camera observation image, and a background image and a computer graphics image of the object viewed from the virtual camera generated by the virtual camera observation image generation unit
- a computer graphics image processing system using AR technology comprising: a computer graphics image composition unit for performing a display; and a display unit for displaying a computer graphics composite image synthesized by the computer graphics image composition unit.
- a computer is used to capture an image frame including an AR marker image captured by a fixed camera, and camera parameters stored in advance in the computer are used to obtain the AR marker image.
- the position and orientation of the AR marker image are determined by analyzing the image frame including the image frame, and the object corresponding to the AR marker is converted into the image frame in the computer graphics image space based on the analysis result of the orientation of the AR marker image.
- a computer graphics image of the posture and scale corresponding to the posture and scale of the AR marker image is generated, and the AR marker image on the object image frame is generated on the AR marker image.
- the computer graphics of the object A visual image when it is viewed from a virtual camera installed at a predetermined coordinate position in the computer graphics image space, and is generated as a virtual camera observation image, from a background image and the virtual camera
- a part of the AR marker is detached from the camera frame, or the AR marker image in the camera frame is small, making it difficult to identify the AR marker image, and the position of the AR marker image cannot be detected correctly. It is possible to provide a computer graphics image processing technology using AR technology that does not cause problems and does not impose restrictions on the position, angle of view, and camera work of the camera.
- FIG. 1 is a functional block diagram of a computer graphics image processing system according to an embodiment of the present invention.
- FIG. 2 is a flowchart of computer graphics image processing executed by the computer graphics image processing system.
- FIG. 3 is a diagram for explaining the principle of computer graphics image processing by the system of the embodiment.
- FIG. 4 is an explanatory diagram showing a relationship between a real camera screen and a virtual camera screen in computer graphics image processing by the system of the above embodiment.
- FIG. 5 is an explanatory diagram of a digital image projected by the AR analyzer in the system of the above embodiment.
- FIG. 6 is an explanatory diagram showing a spatial coordinate arrangement between a projection volume and a CG object in an actual camera photographing system in computer graphics image processing by the system of the above embodiment.
- FIG. 1 is a functional block diagram of a computer graphics image processing system according to an embodiment of the present invention.
- FIG. 2 is a flowchart of computer graphics image processing executed by the computer graphics image processing system.
- FIG. 3 is
- FIG. 8 is a diagram illustrating projection of a CG object on the first screen plane (when t + b ⁇ 0) in the YZ plane of the real camera photographing coordinate system in the computer graphics image processing by the system of the above embodiment.
- FIG. 9 is an explanatory diagram showing a process of projecting a CG object onto the first screen surface in the real camera photographing coordinate system in the computer graphics image processing by the system of the above embodiment.
- FIG. 10 is an explanatory diagram of a state in which a CG object is projected onto the first screen surface in the real camera photographing coordinate system in the computer graphics image processing by the system of the above embodiment.
- FIG. 11 is an explanatory diagram of a state in which a CG object is projected onto the first screen surface in the actual camera photographing coordinate system by the conventional AR analysis technique as viewed from the second screen.
- FIG. 12 is an explanatory diagram of a state in which the CG object is projected onto the first screen surface in the real camera photographing coordinate system in the computer graphics image processing by the system according to the above-described embodiment, as viewed from the second screen.
- FIG. 11 is an explanatory diagram of a state in which a CG object is projected onto the first screen surface in the actual camera photographing coordinate system by the conventional AR analysis technique as viewed from the second screen.
- FIG. 12 is an explanatory diagram of a state in which the CG object is projected onto the first screen surface in the real camera photographing coordinate system in the computer graphics image
- FIG. 13 is an explanatory diagram of a state in which a CG object is displayed at the position of the AR marker image on the first screen surface in the real camera photographing coordinate system in the computer graphics image processing by the system of the above embodiment.
- FIG. 14 shows a state in which the CG object is displayed at the position of the AR marker image on the first screen surface in the real camera photographing coordinate system in the computer graphics image processing by the system of the above embodiment, and the second screen surface of the virtual camera.
- FIG. 15 is an explanatory diagram of image processing when a virtual camera is moved in computer graphics image processing by the system of the embodiment.
- FIG. 16 is an explanatory diagram showing the movement of the AR marker image on the digital image of the real camera when the AR marker is moved in the computer graphics image processing by the system of the embodiment.
- FIG. 17 is an explanatory diagram showing movement of a CG object viewed from a virtual camera when an AR marker is moved in computer graphics image processing by the system of the above embodiment.
- FIG. 18 is a photograph of an image obtained by synthesizing a CG object at the position of the AR marker image with respect to a fixed-point image captured by the AR analyzer of the comparative example.
- FIG. 19 is a photograph of a projective transformation image from a virtual camera observation system on a CG by the computer graphics image processing system according to the first embodiment of the present invention.
- FIG. 20 is an explanatory diagram of a soccer commentary image (CG for moving two corresponding players by moving two AR markers) by the computer graphics image processing system according to the second embodiment of the present invention.
- FIG. 21 is a photograph of a captured image of a scene including a general AR marker.
- FIG. 22 is an explanatory diagram of a conventional AR analysis process.
- FIG. 23 is an explanatory diagram showing a relationship between a camera coordinate system (X, Y, Z) and an image coordinate system (x, y, z) in a pinhole camera model in a general AR analysis process.
- FIG. 24 is an explanatory diagram of a pattern example used for general camera calibration.
- FIG. 25 is an explanatory diagram of the definition of a viewing frustum in a general pinhole camera model.
- FIG. 26 is an explanatory diagram of problems of the conventional AR analysis technique.
- a computer graphics image processing system using AR technology has the configuration shown in FIG. 1, and includes an AR analyzer 1, a computer graphics (CG) rendering unit 2, and a display 3. Further, the camera calibration unit 4, the fixed camera 5 as the real camera CAM-A placed at the fixed position, the offset matrix setting unit 6, and chroma key composition processing are performed on the captured image of the fixed camera 5 when necessary. A chroma key device 7 is provided.
- the AR analyzer 1, the CG rendering unit 2, the camera calibration unit 4, and the offset matrix setting unit 6 are installed by executing necessary software programs installed in one computer system.
- each processing function required for implementation is disassembled and each is described as a specific processing unit.
- the AR analyzer 1 finds an AR marker image from the video of the fixed camera 5 and the storage unit 11 that stores a projection matrix Pa, a view model matrix Ma, camera parameters, and other necessary data, which will be described later, and its position and orientation.
- An AR marker image analysis unit 13 that analyzes the scale and registers the view model matrix Ma in the storage unit 11 and an Mq matrix determination unit 15 that calculates an affine transformation matrix Mq for the analysis result of the AR marker image are provided.
- the CG rendering unit 2 is composed of, for example, a CG graphics card, and stores a digital image of an object to be displayed at the AR marker image position, a background image, and other necessary data, a Pb matrix for setting a Pb matrix
- the setting unit 23, the object posture determining unit 25 that determines the display position, posture, and scale of the object image, and the object posture determining unit 25 determines the object image stored in the storage unit 21 with respect to the captured image of the fixed camera 5.
- a CG image synthesizing unit 27 that synthesizes the position of the AR marker image with the posture and creates a CG synthesized image is provided.
- the CG rendering unit 2 also includes a background image input unit 29 for inputting a background image in order to synthesize an object image on the background image.
- the fixed camera 5 uses a web camera or a video camera capable of digital output of video.
- the chroma key device 7 is used to input the chroma key composite image to the CG rendering unit 2.
- the camera calibration unit 4 calculates camera internal parameters and external parameters by camera calibration of the fixed camera 5 and registers them in the storage unit 11 of the AR analyzer 1.
- the fixed camera 5 fixes its position and angle of view to a fixed state mainly in order to photograph the AR marker 101 clearly.
- the offset matrix setting unit 6 sets an offset matrix Mp, and the data of the matrix Mp set here is registered in the storage unit 21 of the CG rendering unit 2.
- an observation system is defined by the virtual camera CAM-B separately from the photographing system by the fixed camera 5 that is the real camera CAM-A. As shown in FIG.
- an AR marker image MRK1 obtained by projective transformation of an image photographed by the real camera CAM-A (5) by the AR analyzer 1 is affine transformed to the first screen surface SCR-A on the CG space 20
- the AR marker image MRK1 affine transformed to the position corresponding to the first screen surface SCR-A is projectively transformed to the second screen surface SCR-B viewed from the virtual camera VRCAM-B arranged on the same CG space 20. Since the first screen surface SCR-A is arranged on the same CG space 20, it can be observed from the virtual camera VRCAM-B system at a free position and angle of view.
- the first screen surface SCR-A when the first screen surface SCR-A is observed from the virtual camera VRCAM-B, a CG image photographed by the real camera CAM-A and projectively transformed by the AR analyzer 1 is used as the first screen surface SCR-A. Projective transformation can be easily performed on A. However, with this method, only a planar CG image projected onto the screen surface SCR-A can be observed.
- the first screen surface SCR-A that is, a rectangle having the same aspect ratio as that of the projected image of the real camera CAM-A photographing system at a position corresponding to the first screen surface SCR-A.
- a region SQ-A is defined on the VRCAM-B world space 20.
- texture mapping of the digital image obtained by projective transformation by the AR analyzer 1 is performed on the rectangular area SQ-A.
- the observation result from the virtual camera VRCAM-B that is, the projection conversion result to the second screen surface SCR-B is transferred to the second screen surface SCR-B in the rectangular area SQ-A.
- the CG object OBJ1 designed as a three-dimensional shape is only distorted into a plane and projected onto the second screen surface SCR-B.
- the virtual camera VRCAM-B observation system is positioned at the position of the AR marker image MRK1 projected and converted onto the first screen surface SCR-A by the AR analyzer 1.
- the CG object OBJ1 is placed at the angle of view, and this is projected and converted to the second screen SCR-B.
- the CG object OBJ1 that is projected and converted to the second screen SCR-B is an AR marker image that is projected and converted onto the first screen surface SCR-A, which is arranged on the coordinates of the virtual camera VRCAM-B system. Since the projection conversion of the CG object OBJ1 is correctly performed at the position of the MRK1 and is the projection conversion of the virtual camera VRCAM-B system, it is projected onto the second screen surface SCR-B while maintaining the three-dimensional shape.
- STEP 11 The AR marker 101 is created in advance, and the CG of the object OBJ1 corresponding to the AR marker 101 is created and stored.
- STEP 13 In addition, the internal parameter matrix A, the rotation matrix R, and the translation vector t, which are camera parameters, are determined and stored in advance by camera calibration of the fixed camera 5.
- STEP 15 The shooting space of the AR analyzer 1 corresponding to the image shot by the fixed camera 5 is determined and stored. That is, the projection matrix Pa is determined and stored.
- STEP17 A scene in which the AR marker 101 exists is photographed by the fixed camera 5, and a photographed image in which the AR marker image MRK1 is captured is obtained.
- STEP19 The AR marker image MRK1 is found from the photographed digital image.
- STEP 21 The position (depth), orientation (posture), and size (scale) of the AR marker image MRK1 are determined, and the view model matrix Ma is determined and stored.
- STEP23 The appearance of the CG object OBJ1 corresponding to the AR marker image MRK1 on the real camera screen SCR-A is calculated using the matrices Pa and Ma of the storage unit 11.
- STEP 25 Determines the appearance when projected on the virtual camera (second) screen SCR-B with respect to the CG object OBJ1 for which the appearance on the real camera (first) screen SCR-A has been determined.
- STEP 27 A digital image as a background and a CG object OBJ1 on the virtual camera screen are synthesized.
- STEP 29 A composite image of a digital image as a background and the CG object OBJ1 on the virtual camera screen is displayed.
- Ma represents a 4 ⁇ 4 model view matrix in the real camera CAM-A photographing system, and is an affine transformation itself of the spatial coordinates in the coordinates of the real camera CAM-A photographing system. As described above, this is a relative value calculated from the camera parameters of the real camera CAM-A. If the projection matrix Pa is not always multiplied, the AR in the digital image of the real camera CAM-A photographing system A CG object cannot be correctly displayed at the position of the marker image MRK1. However, since the projective transformation by the projection matrix Pa is equivalent to the mapping to the first screen surface SCR-A in the real camera CAM-A system, the projective transformation matrix Pa cannot be applied as it is.
- a frustum shape corresponding to the view volume in the coordinates of the real camera CAM-A imaging system is defined on the coordinates of the virtual camera VRCAM-B observation system and arranged at the position of the AR marker image MRK1.
- An affine transformation is performed to project the coordinates of the CG object OBJ1 onto the position of the AR marker image MRK1 on the first screen surface SCR-A.
- the known parameters are the 4 ⁇ 4 projection matrix Pa defined by the AR analyzer 1 and
- the 4 ⁇ 4 model view matrix Ma in the coordinates of the real camera CAM-A imaging system is determined as follows.
- FIG. 5 shows a CG object OBJ1 projected onto the first screen surface SCR-A by the AR analyzer 1.
- the CG object OBJ1 projected on the first screen surface SCR-A is translated in the spatial coordinate system by the affine transformation by the model view matrix Ma in the spatial coordinates of the real camera CAM-A photographing system.
- This is a mapping of rotation / scaling and projective transformation by the projection matrix Pa. Therefore, arbitrary spatial coordinates representing the CG object OBJ1 placed on the AR marker image MRK1 by affine transformation using the model view matrix Ma.
- the geometric elements constituting the real camera CAM-A system view volume are derived from the projective transformation matrix Pa.
- this matrix Pa The components of this matrix Pa are defined in FIG. 25, Formula 5 and Formula 6, as in the past.
- n 1 is often given.
- the projective transformation matrix Pa includes an optical center deflection component.
- Ma is a projection matrix Pa, that is, a model view that coincides with the AR marker image MRK1 in appearance in the real camera CAM-A view volume determined by the camera parameters of the real camera CAM-A.
- the actually captured digital image is an image that is recorded through correction by a lens or a computer. The origin of the image, the aspect ratio of the pixel, etc. are actually changed depending on the mechanical characteristics of the fixed camera 5, lens distortion, characteristics of the image sensor, and the like.
- camera parameters are estimated as r + l ⁇ 0 or t + b ⁇ 0.
- a scaling parameter considering the projection transformation by the projection matrix Pa is defined as follows.
- Vb is a constant and is the height scale of the first screen surface SCR-A in the virtual camera VRCAM-B observation system.
- the movement amount Tp at the position of the first screen surface SCR-A in consideration of the deflection component of the optical center axis Oax is defined as follows.
- Ax is a constant indicating the aspect ratio in the horizontal direction of the first screen surface SCR-A.
- the digital image of the actual camera CAM-A system is a 16: 9 image, 16/9, 4: 3 image. In this case, the value is 4/3.
- This matrix Mp is set in advance by the offset matrix setting unit 6 and stored in the storage unit 21. The data is variable.
- [Tp] and [Tr] are 4 ⁇ 4 matrix homogeneous coordinate expressions of the respective translation vectors.
- an arbitrary affine transformation matrix for arranging the coordinates of the real camera CAM-A photographing system is set as Mq at the coordinates of the virtual camera VRCAM-B observation system, and the coordinates on the virtual camera VRCAM-B observation system are set.
- a 4 ⁇ 4 matrix projected onto the second screen surface SCR-B is set as Pb.
- an arbitrary space coordinate which is a space coordinate representation of the CG object OBJ1
- the projective transformation mb ′ to the second screen surface SCR-B can be expressed by the following equation.
- the angle of view by the virtual camera VRCAM-B coordinate system arbitrarily set in the CG space on the AR marker image MRK1 observed in the digital image of the real camera CAM-A system. It is possible to projectively convert the CG object OBJ1 to the second screen surface SCR-B of the virtual camera VRCAM-B system while maintaining its three-dimensional shape at the photographing position.
- the real camera CAM-A coordinate system observed from the virtual camera VRCAM-B system is moved and rotated to an arbitrary position. ⁇ Scaling and drawing angle of view can be changed.
- the following operations and effects can be achieved.
- the digital image obtained by the fixed camera 5 in the prior art, as shown in FIG. 11, for example, when the virtual camera VRCAM-B system is arranged beside the real camera CAM-A photographing system, the CG object OBJ1 is only a plane. Because it was not drawn, it was converted to a straight line. On the other hand, in the present embodiment, it is possible to draw the CG object OBJ1 as seen from the side as shown in FIG.
- the virtual camera VRCAM-B can change the angle of view, the problem that the AR marker 101 cannot be recognized when the AR marker 101 protrudes from the frame 105 as shown in FIG. As shown in FIG. 13, when the real camera CAM-A system is set and fixed at a field angle capable of recognizing the AR marker, the virtual camera VRCAM-B can be enlarged and reduced as shown in FIG. . This can be similarly solved in the case where it is difficult to identify the AR marker 101 shown in FIG.
- the blur when the AR marker 101 is moving as shown in FIG. 26C is a problem that the AR analyzer 1 cannot recognize when the AR marker 101 is moving at high speed.
- the AR analyzer 1 can be expressed by changing the position of the virtual camera VRCAM-B as shown in FIG. 15, and in this case, the real camera CAM-A system is fixed.
- the marker analyzer 1 can express high-speed camera work with the virtual camera VRCAM-B system while correctly detecting the position of the AR marker image MRK1.
- FIGS. 16 and 17 show how the movement of the AR marker image MRK1 observed by the AR analyzer 1 is reflected on the movement of the CG object OBJ1 viewed from the virtual camera VRCAM-B system.
- the movement of the AR marker image MRK1 is expressed on a plane on the first screen surface SCR-A, but the second is arranged on the virtual camera VRCAM-B coordinate system. Since the one-screen surface SCR-A can be affine-transformed into an arbitrary plane space, the movement on the plane of the real camera CAM-A imaging system can be converted into the movement on the space in the virtual camera VRCAM-B coordinate system.
- the affine transformation by is nothing but the transformation to the virtual camera VRCAM-B coordinate system, for example,
- FIG. 18 shows a CG image in which a three-dimensional AR marker object is synthesized and displayed at a position of an AR marker image with respect to a fixed-point image captured by the AR analyzer 1 as a comparative example.
- FIG. 19 shows a CG image obtained by performing chroma key processing on a fixed-point image captured by the same real camera CAM-A and performing projective transformation on the chroma key image from the virtual camera CAM-B system as Example 1. Show.
- the AR marker 101 is too small to be resolved and the CG object OBJ1 cannot be displayed at the position of the AR marker image MRK1, but the CG object OBJ1 can be displayed at an arbitrary small angle of view. Can be displayed with the position, posture and scale of the AR marker image MRK1.
- FIG. 20 shows a second embodiment in which a commentary image of a soccer game is generated by chroma key processing.
- two AR markers prepared for soccer commentary are arranged in front of the soccer field board as shown in FIG.
- the CG images of the two soccer players are placed on the AR marker image on the soccer field by chroma key processing and CG image composition processing as shown in FIG.
- a realistic CG image in which the soccer player image moves is represented.
- the soccer field image can be expressed as a video image of the player image on the top as well as the video image viewed from the angle and direction according to the image.
- the fixed camera 5 for detecting the position of the AR marker is the real camera CAM-A that always observes a fixed point. Solves the problems that occur when combining and displaying CG using image recognition AR (AR marker) by defining virtual camera VRCAM-B on the CG space side and changing the angle of view and position on the virtual camera side In addition, it is possible to create and display a CG composite image using an image recognition type AR (AR marker), which is impossible with a conventional system. Therefore, according to the present invention, utilization in the television broadcasting field is facilitated.
- the technical scope of the present invention also includes a program for causing the computer system to perform the above-described series of processing and a recording medium on which the program is recorded.
Landscapes
- Engineering & Computer Science (AREA)
- Computer Graphics (AREA)
- Computer Hardware Design (AREA)
- General Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Processing Or Creating Images (AREA)
Abstract
The present invention pertains to augmented reality (AR) technology in which problems such as a part of an AR marker image exiting the camera frame, an AR marker image being so small as to impede the identification thereof, or the position of an AR marker image being impossible to detect accurately have been eliminated without imposing limits on camerawork or the position or angle of view of the camera. The present invention is characterized by a CG image processing technique using AR in which a camera (5) for detecting the position of an AR marker (101) is set to normally observe a fixed point, a virtual camera is defined in the CG space, and the angle of view and/or position of the virtual camera are changed.
Description
本発明は、拡張現実感(AR)を利用したコンピュータ・グラフィックス画像処理システム及び方法に関する。
The present invention relates to a computer graphics image processing system and method using augmented reality (AR).
近年、さまざまな分野において、画像認識技術による拡張現実感(AR)を取り入れたCG(コンピュータ・グラフィックス)演出が行われるようになってきている。既存の画像認識型AR技術としては、例えば、図21に示すような特定形状と特徴点群を表記したARマーカ101を使用したものがある。
In recent years, CG (computer graphics) production incorporating augmented reality (AR) by image recognition technology has been performed in various fields. As an existing image recognition type AR technology, for example, there is a technique using an AR marker 101 that describes a specific shape and a feature point group as shown in FIG.
一般的なAR技術は、基本的には図22に示す処理内容である。すなわち、STEP1,STEP2でウェブカメラやデジタルビデオカメラのようなカメラ103によってARマーカ101を含むシーンを撮影してカメラフレーム105を取得し、STEP3でカメラフレーム105内のARマーカ像101の位置検出や空間的な画像認識を行い、STEP4でARマーカ像101の位置、姿勢、スケールに応じてその上にあらかじめ対応づけたCGオブジェクト107を同姿勢、スケールにて合成して表示するものである。
General AR technology basically has the processing contents shown in FIG. That is, in STEP 1 and STEP 2, a scene including the AR marker 101 is captured by the camera 103 such as a web camera or a digital video camera to acquire the camera frame 105. In STEP 3, the position detection of the AR marker image 101 in the camera frame 105 is performed. Spatial image recognition is performed, and in step 4, the CG object 107 previously associated with the AR marker image 101 according to the position, posture, and scale of the AR marker image 101 is synthesized and displayed with the same posture and scale.
このようなAR処理を行うコンピュータビジョンでは、一般的にピンホールカメラモデルを使用して近似される。ピンホールカメラモデルにおいての考え方は、画像平面に至る光は全て1点の焦点であるピンホールを通過し、画像平面と交差した位置で像を結ぶと考える。このような射影は中心射影と呼ばれる。
In computer vision that performs such AR processing, it is generally approximated using a pinhole camera model. The idea of the pinhole camera model is that all the light reaching the image plane passes through a pinhole that is the focal point of one point and forms an image at a position intersecting the image plane. Such a projection is called a central projection.
図23に示すように、ピンホールカメラモデルにおいて、光軸111と画像平面113との交点を原点O1とし、カメラ103の撮像素子配置方向に合わせて画像平面113上にx軸とy軸をとる座標系を画像座標系と呼ぶ。また、ピンホールO2をカメラ103の中心とみなし、光軸111の方向をZ軸とし、X軸とY軸を画像座標系のx軸とy軸に平行にとる座標系を、カメラ座標系と呼ぶ。
As shown in FIG. 23, in the pinhole camera model, the intersection of the optical axis 111 and the image plane 113 is set as the origin O1, and the x axis and the y axis are taken on the image plane 113 in accordance with the image sensor arrangement direction of the camera 103. The coordinate system is called an image coordinate system. A coordinate system in which the pinhole O2 is regarded as the center of the camera 103, the direction of the optical axis 111 is the Z axis, and the X axis and the Y axis are parallel to the x axis and the y axis of the image coordinate system is defined as a camera coordinate system. Call.
また、実際に撮影したデジタル画像はレンズやコンピュータによる補正などを通して記録される画像であり、カメラ103の機械的特性やレンズ歪み、撮像素子の特性などにより、画像の原点や画素のアスペクト比などは実際の(x,y,z)画像座標系のそれとは一致しない。そこで、デジタル画像において、座標原点を左上にとり、右方向をu軸、上下方向をv軸とするような座標系を設定し、これをデジタル画像座標系と呼ぶ。
In addition, digital images that are actually captured are images that are recorded through correction by a lens or a computer. The origin of the image, the aspect ratio of the pixel, etc., depending on the mechanical characteristics of the camera 103, lens distortion, image sensor characteristics, etc. It does not match that of the actual (x, y, z) image coordinate system. Therefore, in the digital image, a coordinate system is set in which the coordinate origin is set at the upper left, the right direction is the u axis, and the vertical direction is the v axis, and this is called a digital image coordinate system.
ここで、三次元空間中に任意の世界座標系をとり、その空間中に存在するカメラ座標(X,Y,Z)系のデジタル画像座標(u,v,w)系への中心射影を考えた場合、一般的に3×4の射影変換行列Pを用いて、
Here, an arbitrary world coordinate system is taken in the three-dimensional space, and the central projection of the camera coordinate (X, Y, Z) system existing in the space onto the digital image coordinate (u, v, w) system is considered. In general, using a 3 × 4 projective transformation matrix P,
と表現できる。
Can be expressed.
は画像平面の座標の同次座標表現である。
Is a homogeneous coordinate representation of the coordinates of the image plane.
このような射影変換行列Pは、カメラの内部パラメータ行列Aと回転行列R、及び並進ベクトルtによって、
Such a projective transformation matrix P is given by the camera internal parameter matrix A, the rotation matrix R, and the translation vector t.
と決定できる。回転行列Rは3×3行列、[R|t]は同次座標系であり、3×4行列として表される。
Can be determined. The rotation matrix R is a 3 × 3 matrix, and [R | t] is a homogeneous coordinate system, and is represented as a 3 × 4 matrix.
このような、カメラ103の内部パラメータ行列Aと回転行列R及び並進ベクトルtを求めることを、カメラ内部パラメータ推定又はカメラキャリブレーションという。一般的には、カメラ103の内部パラメータAと外部パラメータ[R|t]を決定するために、図24に示すようなカメラキャリブレーション用のパターン例P1,P2を角度などを変えて撮影し、複数画像から得られる相関関係によって方程式の解を求め、カメラパラメータを決定する。これにはZangの手法を利用する。この手法は、下記の非特許文献1に説明されている。
Such determination of the internal parameter matrix A, rotation matrix R, and translation vector t of the camera 103 is called camera internal parameter estimation or camera calibration. In general, in order to determine the internal parameter A and the external parameter [R | t] of the camera 103, the camera calibration pattern examples P1 and P2 as shown in FIG. The solution of the equation is obtained from the correlation obtained from a plurality of images, and camera parameters are determined. For this, Zang's method is used. This technique is described in Non-Patent Document 1 below.
このようにして得たカメラパラメータを利用し、図22に示した実際にカメラ103で撮影されたデジタル画像105からARマーカ像101の位置を画像認識により検出するシステムをAR解析器と呼ぶ。
The system that detects the position of the AR marker image 101 from the digital image 105 actually captured by the camera 103 shown in FIG. 22 by image recognition using the camera parameters obtained in this way is called an AR analyzer.
通常のAR解析器を利用するコンピュータ・グラフィックス画像処理システムでは、検出したARマーカ像101の姿勢を三次元CGで描画するため、一般的な三次元コンピュータ・グラフィックスの計算で用いられる4×4のプロジェクション行列Paと4×4のモデルビュー行列Maを算出し、デジタル画像105上のARマーカ像101の位置を基準にして三次元空間上の任意の点を射影変換し、表示する。
In a computer graphics image processing system that uses a normal AR analyzer, the detected orientation of the AR marker image 101 is drawn with a three-dimensional CG, so that 4 × used in general three-dimensional computer graphics calculations. 4 projection matrices Pa and 4 × 4 model view matrix Ma are calculated, and an arbitrary point in the three-dimensional space is projectively transformed and displayed with reference to the position of the AR marker image 101 on the digital image 105.
ここで、プロジェクション行列Paは、ピンホールカメラモデルを図25に示す錐台121として定義した場合に、
Here, the projection matrix Pa is defined when the pinhole camera model is defined as the frustum 121 shown in FIG.
と定義し、又はその転置行列と定義できる。転置行列の場合、行列の演算方向を逆にすればよいので、ここでは上記表記を用いる。
Or its transpose matrix. In the case of a transposed matrix, the above notation is used here because the matrix calculation direction may be reversed.
であり、(X,Y,Z)カメラ座標系においてピンホールに相当する原点O2から、錐台121の手前の上底面SCR-Aの左上の頂点が(l,t,-n)、その上底面SCR-Aの左下の頂点が(l,b,-n)、その右上の頂点が(r,t,-n)、その右下の頂点が(r,b,-n)とし、上底面SCR-Aから奥の面(下底面に相当)までの距離をfとしている。
From the origin O2 corresponding to the pinhole in the (X, Y, Z) camera coordinate system, the upper left vertex of the upper bottom surface SCR-A before the frustum 121 is (l, t, −n), The bottom left vertex of the bottom surface SCR-A is (l, b, -n), the top right vertex is (r, t, -n), the bottom right vertex is (r, b, -n), and the top bottom surface The distance from the SCR-A to the inner surface (corresponding to the lower bottom surface) is f.
一般的に、プロジェクション行列Paはカメラキャリブレーションを行えば、AR解析器の撮影系では固定値となり、モデルビュー行列MaがARマーカ像の検出位置、姿勢及びスケールを表わす。
Generally, when camera calibration is performed, the projection matrix Pa becomes a fixed value in the imaging system of the AR analyzer, and the model view matrix Ma represents the detection position, orientation, and scale of the AR marker image.
で表され、求められたm’に対してデジタル画像の解像度に応じたビューポート変換を経てCG(コンピュータ・グラフィックス)が表示される。また、上記射影変換により変換されたm’をビューポート変換した後のデジタル画像座標と、Mと原点を通る直線が平面Z=nと交わる交点の位置は相似関係である。平面Z=nの[l,t]-[r,b]をプロジェクション面ともいう。
CG (computer graphics) is displayed through viewport conversion corresponding to the resolution of the digital image with respect to the determined m ′. Also, the digital image coordinates after the viewport conversion of m ′ converted by the projective conversion and the position of the intersection where the straight line passing through M and the origin intersects the plane Z = n are similar. [L, t]-[r, b] of the plane Z = n is also referred to as a projection surface.
このような一般的なAR解析器においては、モデルビュー行列Maが決定できなければ、ARマーカ像101の検出位置に対応するCGオブジェクトを描画することができない。
In such a general AR analyzer, a CG object corresponding to the detected position of the AR marker image 101 cannot be drawn unless the model view matrix Ma can be determined.
モデルビュー行列Maが決定できない要因は、ARマーカ像が認識できない場合に他ならない。この要因として、図26(a)に示すようにカメラ103で撮影されたデジタル画像105中のAR解析器が認識し得る範囲にARマーカ像101が含まれていない場合、同図(b)に示すように画角が広い若しくはカメラ103から実際のARマーカまでの距離が遠くてARマーカ像101がフレーム中に小さく写り、識別が困難な場合、同図(c)に示すようにカメラ103の移動速度や画角の変化が速くカメラの撮像素子の性能に起因して撮影されたデジタル画像105の中でARマーカ像101がぼやけてしまうといった場合がある。これらのことが、カメラワークに制約を与える。特に、テレビ放送用途では大きな制約となる。
The reason why the model view matrix Ma cannot be determined is none other than when the AR marker image cannot be recognized. As a cause of this, when the AR marker image 101 is not included in the range that can be recognized by the AR analyzer in the digital image 105 captured by the camera 103 as shown in FIG. As shown in FIG. 5C, when the angle of view is wide or the distance from the camera 103 to the actual AR marker is far and the AR marker image 101 appears small in the frame and is difficult to identify. In some cases, the AR marker image 101 is blurred in the digital image 105 photographed due to the rapid change of the moving speed and the angle of view due to the performance of the image sensor of the camera. These limit the camera work. In particular, it is a major limitation in television broadcasting applications.
また、専用のARマーカを使用しない、いわゆるマーカレスのAR技術も開発されている。これは、カメラで撮影されたフレーム内の現実世界のオブジェクト、例えば山や人の顔の形状から特徴点を検出し、姿勢検知と固体識別を行うAR技術である。このマーカレスのAR技術においても、特徴点群を識別対象として利用するシステムでは、特徴点群がカメラフレームから外れる場合や画角の変化等により特徴点群が識別困難な場合には、同様の問題が生じる。
Also, a so-called markerless AR technology that does not use a dedicated AR marker has been developed. This is an AR technology that detects a feature point from the shape of a real-world object such as a mountain or a human face in a frame photographed by a camera, and performs posture detection and solid identification. Even in this markerless AR technology, in a system that uses a feature point group as an identification target, when the feature point group is out of the camera frame or when the feature point group is difficult to identify due to a change in the angle of view, the same problem is encountered. Occurs.
本発明は、画像認識型AR技術を利用してCGを合成表示する際に生じる上記従来技術の問題点を解決するためになされたものであり、カメラの位置や画角、カメラワークに制約を課すことが少ないAR技術を提供することを目的とする。
The present invention has been made in order to solve the above-described problems of the prior art that occur when CG is synthesized and displayed using the image recognition AR technology, and restricts the camera position, angle of view, and camera work. The purpose is to provide AR technology that is less imposed.
本発明のAR技術は、ARマーカの位置検出のためのカメラは常に定点を観測するカメラとしておき、CG空間に仮想カメラを定義した上で当該仮想カメラ側の画角や位置を変化させることを特徴とする。
According to the AR technology of the present invention, a camera for detecting the position of an AR marker is always a camera that observes a fixed point, and a virtual camera is defined in the CG space and then the angle of view and position on the virtual camera side are changed. Features.
より具体的には、本発明の1つの特徴は、ARマーカを撮影する、位置が固定されている固定カメラと、前記固定カメラのカメラパラメータを記憶するパラメータ設定部と、前記固定カメラの撮影した前記ARマーカ像を含む画像フレームに対して、前記パラメータ設定部に記憶されているカメラパラメータを使用し、前記ARマーカの位置、姿勢及びスケールを解析するARマーカ姿勢解析部と、前記ARマーカに対応したオブジェクトを、前記ARマーカ姿勢解析部の解析結果に基づいてコンピュータ・グラフィックス画像空間上における前記画像フレーム上での前記ARマーカ像の位置に対応した位置に、当該ARマーカ像の姿勢及びスケールに対応した姿勢及びスケールのコンピュータ・グラフィックス画像として生成するオブジェクト画像生成部と、前記オブジェクト画像生成部の生成した前記オブジェクトのコンピュータ・グラフィックス画像に対して、前記コンピュータ・グラフィックス画像空間上の所定の座標位置に設置された仮想カメラから見た場合の見え方を割り出し、仮想カメラ観察画像として生成する仮想カメラ観察画像生成部と、背景画像と前記仮想カメラ観察画像生成部の生成する前記仮想カメラから見た前記オブジェクトのコンピュータ・グラフィックス画像とを合成するコンピュータ・グラフィックス画像合成部と、前記コンピュータ・グラフィックス画像合成部の合成したコンピュータ・グラフィックス合成画像を表示する表示部とを備えたAR技術を用いたコンピュータ・グラフィックス画像処理システムにある。
More specifically, one feature of the present invention is that the AR marker is captured, a fixed camera whose position is fixed, a parameter setting unit that stores camera parameters of the fixed camera, and the fixed camera For an image frame including the AR marker image, the camera parameters stored in the parameter setting unit are used to analyze the AR marker position, posture and scale, and the AR marker The corresponding object is placed at a position corresponding to the position of the AR marker image on the image frame on the computer graphics image space based on the analysis result of the AR marker attitude analysis unit. An object generated as a computer graphics image of a posture and scale corresponding to the scale And a computer graphics image of the object generated by the object image generation unit when viewed from a virtual camera installed at a predetermined coordinate position on the computer graphics image space. A virtual camera observation image generation unit that determines the appearance and generates a virtual camera observation image, and a background image and a computer graphics image of the object viewed from the virtual camera generated by the virtual camera observation image generation unit And a computer graphics image processing system using AR technology, comprising: a computer graphics image composition unit for performing a display; and a display unit for displaying a computer graphics composite image synthesized by the computer graphics image composition unit. .
また、本発明の別の特徴は、コンピュータを用いて、固定カメラで撮影したARマーカの画像を含む画像フレームを取り込み、当該コンピュータにあらかじめ記憶されているカメラパラメータを使用し、前記ARマーカ像を含む画像フレームを解析して前記ARマーカ像の位置及び姿勢を割り出し、前記ARマーカに対応したオブジェクトを、前記ARマーカ像の姿勢の解析結果に基づいてコンピュータ・グラフィックス画像空間上における前記画像フレーム上での前記ARマーカ像の位置に対応した位置に、当該ARマーカ像の姿勢及びスケールに対応した姿勢及びスケールのコンピュータ・グラフィックス画像として生成し、前記オブジェクト画像フレーム上の前記ARマーカ像に対応して前記オブジェクトのコンピュータ・グラフィックス画像に対して、前記コンピュータ・グラフィックス画像空間上の所定の座標位置に設置された仮想カメラから見た場合の見え方を割り出し、仮想カメラ観察画像として生成し、背景画像と前記仮想カメラから見た前記オブジェクトのコンピュータ・グラフィックス画像とを合成し、前記合成したコンピュータ・グラフィックス合成画像を表示するAR技術を用いたコンピュータ・グラフィックス画像処理方法にある。
Another feature of the present invention is that a computer is used to capture an image frame including an AR marker image captured by a fixed camera, and camera parameters stored in advance in the computer are used to obtain the AR marker image. The position and orientation of the AR marker image are determined by analyzing the image frame including the image frame, and the object corresponding to the AR marker is converted into the image frame in the computer graphics image space based on the analysis result of the orientation of the AR marker image. At the position corresponding to the position of the AR marker image above, a computer graphics image of the posture and scale corresponding to the posture and scale of the AR marker image is generated, and the AR marker image on the object image frame is generated on the AR marker image. Correspondingly, the computer graphics of the object A visual image when it is viewed from a virtual camera installed at a predetermined coordinate position in the computer graphics image space, and is generated as a virtual camera observation image, from a background image and the virtual camera A computer graphics image processing method using an AR technique for combining a computer graphics image of the viewed object with the computer graphics image and displaying the combined computer graphics image.
本発明によれば、ARマーカの一部がカメラフレームから外れたり、カメラフレーム内のARマーカ像が小さかったりしてその識別が困難になり、正しくARマーカ像の位置を検出することができないといった問題が起きず、カメラの位置や画角、カメラワークに制約を課すことがないAR技術を用いたコンピュータ・グラフィックス画像処理技術を提供できる。
According to the present invention, a part of the AR marker is detached from the camera frame, or the AR marker image in the camera frame is small, making it difficult to identify the AR marker image, and the position of the AR marker image cannot be detected correctly. It is possible to provide a computer graphics image processing technology using AR technology that does not cause problems and does not impose restrictions on the position, angle of view, and camera work of the camera.
以下、本発明の実施の形態を図に基づいて詳説する。本発明の1つの実施の形態のAR技術を利用したコンピュータ・グラフィックス画像処理システムは、図1に示す構成であり、AR解析器1とコンピュータ・グラフィックス(CG)レンダリング部2、表示器3、さらにカメラキャリブレーション部4、固定位置に据えられている実カメラCAM-Aとしての固定カメラ5、オフセット行列設定部6、固定カメラ5の撮影画像に対して必要な場合にクロマキー合成処理を行うクロマキー装置7を備えている。
Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. A computer graphics image processing system using AR technology according to an embodiment of the present invention has the configuration shown in FIG. 1, and includes an AR analyzer 1, a computer graphics (CG) rendering unit 2, and a display 3. Further, the camera calibration unit 4, the fixed camera 5 as the real camera CAM-A placed at the fixed position, the offset matrix setting unit 6, and chroma key composition processing are performed on the captured image of the fixed camera 5 when necessary. A chroma key device 7 is provided.
ここで、AR解析器1、CGレンダリング部2、カメラキャリブレーション部4、オフセット行列設定部6については、これらを1台のコンピュータシステムに対して必要なソフトウェアプログラムをインストールして実行させることにより1台のコンピュータシステムに統合することができるが、以下の説明では、実施に必要な処理機能毎に分解し、それぞれを特定の処理部として説明する。
Here, the AR analyzer 1, the CG rendering unit 2, the camera calibration unit 4, and the offset matrix setting unit 6 are installed by executing necessary software programs installed in one computer system. However, in the following description, each processing function required for implementation is disassembled and each is described as a specific processing unit.
AR解析器1は、後述するプロジェクション行列Pa、ビューモデル行列Ma、カメラパラメータ、その他の必要なデータを記憶する記憶部11、固定カメラ5の映像に対してARマーカ像を見つけ出し、その位置、姿勢、スケールを解析してビューモデル行列Maを記憶部11に登録するARマーカ像解析部13、ARマーカ像の解析結果に対してアフィン変換行列Mqを算出するMq行列決定部15を備えている。
The AR analyzer 1 finds an AR marker image from the video of the fixed camera 5 and the storage unit 11 that stores a projection matrix Pa, a view model matrix Ma, camera parameters, and other necessary data, which will be described later, and its position and orientation. An AR marker image analysis unit 13 that analyzes the scale and registers the view model matrix Ma in the storage unit 11 and an Mq matrix determination unit 15 that calculates an affine transformation matrix Mq for the analysis result of the AR marker image are provided.
CGレンダリング部2は、例えばCG用のグラフィックスカードにより構成され、ARマーカ像位置に表示させるオブジェクトのデジタル画像、背景画像やその他の必要なデータを記憶する記憶部21、Pb行列を設定するPb行列設定部23、オブジェクト画像の表示位置、姿勢、スケールを決定するオブジェクト姿勢決定部25、固定カメラ5の撮影画像に対して記憶部21に記憶させているオブジェクト画像をオブジェクト姿勢決定部25が決定した姿勢にてARマーカ像の位置に合成し、CG合成画像を作成するCG画像合成部27を備えている。CGレンダリング部2は背景画像上にオブジェクト画像を合成する場合のために、背景画像を入力する背景画像入力部29も備えている。
The CG rendering unit 2 is composed of, for example, a CG graphics card, and stores a digital image of an object to be displayed at the AR marker image position, a background image, and other necessary data, a Pb matrix for setting a Pb matrix The setting unit 23, the object posture determining unit 25 that determines the display position, posture, and scale of the object image, and the object posture determining unit 25 determines the object image stored in the storage unit 21 with respect to the captured image of the fixed camera 5. A CG image synthesizing unit 27 that synthesizes the position of the AR marker image with the posture and creates a CG synthesized image is provided. The CG rendering unit 2 also includes a background image input unit 29 for inputting a background image in order to synthesize an object image on the background image.
テレビ放送に利用するシステムとして、固定カメラ5はウェブカメラや映像のデジタル出力が可能なビデオカメラを用いる。固定カメラ5の撮影画像にクロマキー合成する場合はクロマキー装置7を用い、クロマキー合成映像をCGレンダリング部2に入力する。
As a system used for television broadcasting, the fixed camera 5 uses a web camera or a video camera capable of digital output of video. When the chroma key composition is performed on the captured image of the fixed camera 5, the chroma key device 7 is used to input the chroma key composite image to the CG rendering unit 2.
カメラキャリブレーション部4は、固定カメラ5のカメラキャリブレーションによりカメラ内部パラメータ、外部パラメータを演算し、AR解析器1の記憶部11に登録する。
The camera calibration unit 4 calculates camera internal parameters and external parameters by camera calibration of the fixed camera 5 and registers them in the storage unit 11 of the AR analyzer 1.
固定カメラ5は主にARマーカ101を鮮明に撮影するために、その位置や画角を一定の状態に固定している。
The fixed camera 5 fixes its position and angle of view to a fixed state mainly in order to photograph the AR marker 101 clearly.
オフセット行列設定部6はオフセット行列Mpを設定するもので、ここで設定された行列MpのデータはCGレンダリング部2の記憶部21に登録される。
The offset matrix setting unit 6 sets an offset matrix Mp, and the data of the matrix Mp set here is registered in the storage unit 21 of the CG rendering unit 2.
次に、上記構成のAR技術を用いたCGコンピュータ・グラフィックス画像処理システムの動作原理について説明する。本実施の形態のシステムでは、CGレンダリング部2において、実カメラCAM-Aである固定カメラ5による撮影系とは別に、仮想カメラCAM-Bにより観察系を定義する。図3に示すように、実カメラCAM-A(5)で撮影した画像をAR解析器1で射影変換したARマーカ像MRK1をCG空間20上の第1スクリーン面SCR-Aにアフィン変換し、第1スクリーン面SCR-A相当位置にアフィン変換されたARマーカ像MRK1を、同じCG空間20上に配置された仮想カメラVRCAM-Bから見た第2スクリーン面SCR-Bに射影変換する。第1スクリーン面SCR-Aは同じCG空間20上に配置されているので、仮想カメラVRCAM-B系から自由な位置及び画角で観察できる。
Next, the operation principle of the CG computer graphics image processing system using the AR technology with the above configuration will be described. In the system of the present embodiment, in the CG rendering unit 2, an observation system is defined by the virtual camera CAM-B separately from the photographing system by the fixed camera 5 that is the real camera CAM-A. As shown in FIG. 3, an AR marker image MRK1 obtained by projective transformation of an image photographed by the real camera CAM-A (5) by the AR analyzer 1 is affine transformed to the first screen surface SCR-A on the CG space 20, The AR marker image MRK1 affine transformed to the position corresponding to the first screen surface SCR-A is projectively transformed to the second screen surface SCR-B viewed from the virtual camera VRCAM-B arranged on the same CG space 20. Since the first screen surface SCR-A is arranged on the same CG space 20, it can be observed from the virtual camera VRCAM-B system at a free position and angle of view.
ここで従来技術では、仮想カメラVRCAM-Bから第1スクリーン面SCR-Aを観察した場合、実カメラCAM-Aで撮影しAR解析器1で射影変換されたCG画像を第1スクリーン面SCR-Aに容易に射影変換可能である。ところが、この方法では、スクリーン面SCR-Aに射影された平面状態のCG画像しか観察できない。
Here, in the conventional technique, when the first screen surface SCR-A is observed from the virtual camera VRCAM-B, a CG image photographed by the real camera CAM-A and projectively transformed by the AR analyzer 1 is used as the first screen surface SCR-A. Projective transformation can be easily performed on A. However, with this method, only a planar CG image projected onto the screen surface SCR-A can be observed.
そこで、図4に示すように、第1スクリーン面SCR-Aに相当する位置に、第1スクリーン面SCR-A、すなわち、実カメラCAM-A撮影系の射影画像と同じ縦横比をもった矩形領域SQ-Aを、VRCAM-Bワールド空間20上に定義する。そしてこの矩形領域SQ-Aに、AR解析器1で射影変換したデジタル画像をテクスチャマッピングする。しかし、前述のように、この状態では仮想カメラVRCAM-Bからの観察結果、すなわち、第2スクリーン面SCR-Bへの射影変換結果は、矩形領域SQ-Aの第2スクリーン面SCR-Bへの写像しか得られず、立体形状として設計されるCGオブジェクトOBJ1が平面へ歪められて第2スクリーン面SCR-Bに射影されるにすぎない。
Therefore, as shown in FIG. 4, the first screen surface SCR-A, that is, a rectangle having the same aspect ratio as that of the projected image of the real camera CAM-A photographing system at a position corresponding to the first screen surface SCR-A. A region SQ-A is defined on the VRCAM-B world space 20. Then, texture mapping of the digital image obtained by projective transformation by the AR analyzer 1 is performed on the rectangular area SQ-A. However, as described above, in this state, the observation result from the virtual camera VRCAM-B, that is, the projection conversion result to the second screen surface SCR-B is transferred to the second screen surface SCR-B in the rectangular area SQ-A. The CG object OBJ1 designed as a three-dimensional shape is only distorted into a plane and projected onto the second screen surface SCR-B.
そこで、本実施の形態では、この問題を解決するために、AR解析器1によって第1スクリーン面SCR-A上に射影変換されるARマーカ像MRK1の位置に、仮想カメラVRCAM-B観察系の画角でCGオブジェクトOBJ1を置き、これを第2スクリーンSCR-Bに射影変換する。これによると、第2スクリーンSCR-Bに射影変換されるCGオブジェクトOBJ1は、仮想カメラVRCAM-B系の座標上に配置された、第1スクリーン面SCR-A上に射影変換されるARマーカ像MRK1の位置に正しく配置され、かつ、CGオブジェクトOBJ1の射影変換は仮想カメラVRCAM-B系の射影変換であるから、立体形状を保った状態で第2スクリーン面SCR-Bへ投影される。
Therefore, in the present embodiment, in order to solve this problem, the virtual camera VRCAM-B observation system is positioned at the position of the AR marker image MRK1 projected and converted onto the first screen surface SCR-A by the AR analyzer 1. The CG object OBJ1 is placed at the angle of view, and this is projected and converted to the second screen SCR-B. According to this, the CG object OBJ1 that is projected and converted to the second screen SCR-B is an AR marker image that is projected and converted onto the first screen surface SCR-A, which is arranged on the coordinates of the virtual camera VRCAM-B system. Since the projection conversion of the CG object OBJ1 is correctly performed at the position of the MRK1 and is the projection conversion of the virtual camera VRCAM-B system, it is projected onto the second screen surface SCR-B while maintaining the three-dimensional shape.
本実施の形態のシステムの動作に対応する、AR技術を用いたコンピュータ・グラフィックス画像処理方法について、図2のフローチャートを用いて説明する。
A computer graphics image processing method using AR technology corresponding to the operation of the system of the present embodiment will be described with reference to the flowchart of FIG.
STEP11:あらかじめARマーカ101を作成すると共に、ARマーカ101に対応するオブジェクトOBJ1のCGを作成して記憶しておく。
STEP 11: The AR marker 101 is created in advance, and the CG of the object OBJ1 corresponding to the AR marker 101 is created and stored.
STEP13:また、あらかじめ固定カメラ5のカメラキャリブレーションによりカメラパラメータである、内部パラメータ行列Aと回転行列R及び並進ベクトルtを決定して記憶する。
STEP 13: In addition, the internal parameter matrix A, the rotation matrix R, and the translation vector t, which are camera parameters, are determined and stored in advance by camera calibration of the fixed camera 5.
STEP15:固定カメラ5による撮影画像に対応するAR解析器1の撮影空間を決定して記憶する。つまり、プロジェクション行列Paを決定して記憶する。
STEP 15: The shooting space of the AR analyzer 1 corresponding to the image shot by the fixed camera 5 is determined and stored. That is, the projection matrix Pa is determined and stored.
STEP17:ARマーカ101の存在するシーンを固定カメラ5で撮影し、ARマーカ像MRK1が写る撮影画像を得る。
STEP17: A scene in which the AR marker 101 exists is photographed by the fixed camera 5, and a photographed image in which the AR marker image MRK1 is captured is obtained.
STEP19:撮影したデジタル画像からARマーカ像MRK1を見つけ出す。
STEP19: The AR marker image MRK1 is found from the photographed digital image.
STEP21:ARマーカ像MRK1の位置(奥行き)、向き(姿勢)、大きさ(スケール)を割り出し、ビューモデル行列Maを決定し記憶する。
STEP 21: The position (depth), orientation (posture), and size (scale) of the AR marker image MRK1 are determined, and the view model matrix Ma is determined and stored.
STEP23:記憶部11の行列Pa,Maを用いて、ARマーカ像MRK1に対応するCGオブジェクトOBJ1の実カメラスクリーンSCR-A上での見え方を計算する。
STEP23: The appearance of the CG object OBJ1 corresponding to the AR marker image MRK1 on the real camera screen SCR-A is calculated using the matrices Pa and Ma of the storage unit 11.
STEP25:実カメラ(第1)スクリーンSCR-A上での見え方を決定したCGオブジェクトOBJ1に対して、仮想カメラ(第2)スクリーンSCR-B上に投影した場合の見え方を決定する。
STEP 25: Determines the appearance when projected on the virtual camera (second) screen SCR-B with respect to the CG object OBJ1 for which the appearance on the real camera (first) screen SCR-A has been determined.
STEP27:背景となるデジタル画像と仮想カメラスクリーン上でのCGオブジェクトOBJ1とを合成する。
STEP 27: A digital image as a background and a CG object OBJ1 on the virtual camera screen are synthesized.
STEP29:背景となるデジタル画像と仮想カメラスクリーン上でのCGオブジェクトOBJ1との合成画像を表示する。
STEP 29: A composite image of a digital image as a background and the CG object OBJ1 on the virtual camera screen is displayed.
以上のコンピュータ・グラフィックス画像処理について、さらに詳しく説明する。実カメラCAM-A系のデジタル画像の射影変換については、従来技術と同じであり、以下の式で表される。
The above computer graphics image processing will be described in more detail. The projective transformation of the digital image of the real camera CAM-A system is the same as in the prior art and is expressed by the following equation.
ここで、Maは、実カメラCAM-A撮影系における4×4モデルビュー行列を表しており、実カメラCAM-A撮影系の座標における空間座標のアフィン変換そのものである。これは、前に説明したとおり、実カメラCAM-Aのカメラパラメータから算出された相対的な値であり、常にプロジェクション行列Paを乗じなければ、実カメラCAM-A撮影系のデジタル画像中のARマーカ像MRK1の位置に正しくCGオブジェクトを表示することはできない。しかし、プロジェクション行列Paによる射影変換は、実カメラCAM-A系における第1スクリーン面SCR-Aへの写像に等しいことから、射影変換行列Paをそのまま適用することはできない。
Here, Ma represents a 4 × 4 model view matrix in the real camera CAM-A photographing system, and is an affine transformation itself of the spatial coordinates in the coordinates of the real camera CAM-A photographing system. As described above, this is a relative value calculated from the camera parameters of the real camera CAM-A. If the projection matrix Pa is not always multiplied, the AR in the digital image of the real camera CAM-A photographing system A CG object cannot be correctly displayed at the position of the marker image MRK1. However, since the projective transformation by the projection matrix Pa is equivalent to the mapping to the first screen surface SCR-A in the real camera CAM-A system, the projective transformation matrix Pa cannot be applied as it is.
そこで、本実施の形態では、実カメラCAM-A撮影系の座標におけるビューボリュームに相当する錐台形状を仮想カメラVRCAM-B観察系の座標上に定義し、ARマーカ像MRK1の位置に配置するCGオブジェクトOBJ1の座標を第1スクリーン面SCR-AのARマーカ像MRK1の位置へ投影するアフィン変換を行う。
Therefore, in the present embodiment, a frustum shape corresponding to the view volume in the coordinates of the real camera CAM-A imaging system is defined on the coordinates of the virtual camera VRCAM-B observation system and arranged at the position of the AR marker image MRK1. An affine transformation is performed to project the coordinates of the CG object OBJ1 onto the position of the AR marker image MRK1 on the first screen surface SCR-A.
仮想カメラVRCAM-B観察系の座標上に実カメラCAM-Aのビューボリューム状の幾何学領域を定義するにあたって、既知であるパラメータは、AR解析器1によって定義された4×4プロジェクション行列Paと、実カメラCAM-A撮影系の座標での4×4モデルビュー行列Maであり、次のように決定される。
In defining the view volume-like geometric region of the real camera CAM-A on the coordinates of the virtual camera VRCAM-B observation system, the known parameters are the 4 × 4 projection matrix Pa defined by the AR analyzer 1 and The 4 × 4 model view matrix Ma in the coordinates of the real camera CAM-A imaging system is determined as follows.
図5には、AR解析器1によって第1スクリーン面SCR-Aに投影されるCGオブジェクトOBJ1を示している。この第1スクリーン面SCR-Aに投影されるCGオブジェクトOBJ1は、図6に示すとおり、実カメラCAM-A撮影系の空間座標において、モデルビュー行列Maによるアフィン変換による空間座標系での並進・回転・スケーリングと、プロジェクション行列Paによる射影変換の写像である。そこで、モデルビュー行列Maによるアフィン変換を、ARマーカ像MRK1の上に配置するCGオブジェクトOBJ1を表現する任意の空間座標
FIG. 5 shows a CG object OBJ1 projected onto the first screen surface SCR-A by the AR analyzer 1. As shown in FIG. 6, the CG object OBJ1 projected on the first screen surface SCR-A is translated in the spatial coordinate system by the affine transformation by the model view matrix Ma in the spatial coordinates of the real camera CAM-A photographing system. This is a mapping of rotation / scaling and projective transformation by the projection matrix Pa. Therefore, arbitrary spatial coordinates representing the CG object OBJ1 placed on the AR marker image MRK1 by affine transformation using the model view matrix Ma.
に乗じる。
Multiply
は、実カメラCAM-A座標系においてプロジェクション行列Paで射影変換を行えばARマーカ像MRK1の観測位置へ正しく射影される状態にある。
Is in a state of being correctly projected to the observation position of the AR marker image MRK1 if projective transformation is performed with the projection matrix Pa in the real camera CAM-A coordinate system.
次に、実カメラCAM-A系ビューボリュームを構成する幾何学的要素を射影変換行列Paから導く。
Next, the geometric elements constituting the real camera CAM-A system view volume are derived from the projective transformation matrix Pa.
この行列Paの構成要素は、従来同様に図25、数5式、数6式にて定義され、
The components of this matrix Pa are defined in FIG. 25, Formula 5 and Formula 6, as in the past.
であるから、r,l,t,bについての連立方程式は、nが既知であれば容易に解くことが可能である。nが既知であるとき、r,l,t,bは次のように決定できる。
Therefore, simultaneous equations for r, l, t, and b can be easily solved if n is known. When n is known, r, l, t, b can be determined as follows.
通常、実カメラCAM-A系のカメラパラメータを決定する際、nには任意の値を設定するので、既知である。一般的に、n=1を与えることが多い。
Normally, when determining camera parameters for the real camera CAM-A system, an arbitrary value is set for n, which is known. In general, n = 1 is often given.
こうして実カメラCAM-A系のビューボリュームの幾何学的構成要素が得られたので、次に、
Now that the geometric components of the real camera CAM-A view volume have been obtained,
を第1スクリーン面SCR-A上のARマーカ像MRK1の位置へアフィン変換する。この場合には、射影変換行列Paには光学的中心偏向成分が含まれていることを考慮する必要がある。
Is affine transformed to the position of the AR marker image MRK1 on the first screen surface SCR-A. In this case, it is necessary to consider that the projective transformation matrix Pa includes an optical center deflection component.
r+l=0かつt+b=0場合のビューボリュームは、実カメラCAM-A系の光軸中心がスクリーン面Aの中心座標に完全に一致している場合であり、ビューポート変換されたデジタル画像の中心座標に一致することを意味している。
The view volume when r + l = 0 and t + b = 0 is the case where the optical axis center of the real camera CAM-A system completely coincides with the center coordinate of the screen surface A, and the center of the digital image subjected to viewport conversion It means to match the coordinates.
t+b=0場合には、図7に示すように実カメラCAM-A座標系のZ-Y座標平面において、
When t + b = 0, as shown in FIG. 7, in the ZY coordinate plane of the real camera CAM-A coordinate system,
のZ-Y平面における座標の第1スクリーン面SCR-A、すなわちZ=n上への写像m′(z′,y′)とすると、
When the coordinates m ′ (z ′, y ′) on the first screen surface SCR-A of the coordinates in the ZY plane, that is, Z = n, are
として計算できる。しかし、ここで着目すべきは、Maは射影行列Pa、すなわち実カメラCAM-Aのカメラパラメータにより決定された実カメラCAM-Aビューボリューム内での見かけ上、ARマーカ像MRK1に一致するモデルビュー行列である点である。実際に撮影したデジタル画像はレンズや計算機による補正などを通して記録される画像であり、固定カメラ5の機械的特性やレンズ歪み、撮像素子の特性などにより、画像の原点や画素のアスペクト比などは実際の画像座標系のそれと一致することはほとんどなく、カメラキャリブレーションを行った場合、r+l≠0又はt+b≠0としてカメラパラメータが推定される。
Can be calculated as However, it should be noted here that Ma is a projection matrix Pa, that is, a model view that coincides with the AR marker image MRK1 in appearance in the real camera CAM-A view volume determined by the camera parameters of the real camera CAM-A. A point that is a matrix. The actually captured digital image is an image that is recorded through correction by a lens or a computer. The origin of the image, the aspect ratio of the pixel, etc. are actually changed depending on the mechanical characteristics of the fixed camera 5, lens distortion, characteristics of the image sensor, and the like. In the case of camera calibration, camera parameters are estimated as r + l ≠ 0 or t + b ≠ 0.
すなわち、t+b≠0である場合、図8に示すように第1スクリーン面SCR-Aとカメラキャリブレーションを行った結果の光学的中心軸Oaxとの交点をC、画像高さ方向の画角線AG1と第1スクリーン面SCR-Aとの交点をHとした場合に、線分の長さHm′とCm′との比d
That is, when t + b ≠ 0, as shown in FIG. 8, the intersection of the first screen surface SCR-A and the optical center axis Oax as a result of camera calibration is C, and the angle of view in the image height direction. The ratio d between the lengths of the line segments Hm ′ and Cm ′, where H is the intersection of AG1 and the first screen surface SCR-A
は、図8の場合の方が図7の場合よりも小さくなる。
Is smaller in the case of FIG. 8 than in the case of FIG.
を第1スクリーン面SCR-Aの位置にアフィン変換する場合に、この光学的中心軸Oaxによる偏向を考慮しなければ、第1スクリーン面SCR-Aに対応するデジタル画像上に観測される実際のARマーカ像MRK1の位置とずれが生じることを示している。尚、このことはr+l≠0の場合のX-Y平面にも同様のことが生じる。
When affine transformation is performed to the position of the first screen surface SCR-A, if the deflection due to the optical center axis Oax is not taken into consideration, the actual image observed on the digital image corresponding to the first screen surface SCR-A It shows that the AR marker image MRK1 is displaced from the position. This also applies to the XY plane when r + l ≠ 0.
を第1スクリーン面SCR-Aの位置にアフィン変換する場合に、実カメラCAM-A系で撮影されたデジタル画像のARマーカ像MRK1のスケールを表す、射影変換行列Paによる射影変換のスケーリングを考慮する必要がある。
In consideration of the scaling of the projective transformation by the projective transformation matrix Pa, which represents the scale of the AR marker image MRK1 of the digital image taken by the real camera CAM-A system when the image is affine transformed to the position of the first screen surface SCR-A There is a need to.
以上を考慮し、実カメラCAM-A撮影系の座標における任意空間座標
Considering the above, arbitrary space coordinates in the coordinates of the real camera CAM-A imaging system
を第1スクリーン面SCR-Aに対応するデジタル画像中のARマーカ観測位置MRK1へアフィン変換する。まず、
Is affine transformed to the AR marker observation position MRK1 in the digital image corresponding to the first screen surface SCR-A. First,
を第1スクリーン面SCR-Aの位置へ移動する並進ベクトルTrは、モデルビュー行列Maとnから次のように決定できる。
Can be determined from the model view matrices Ma and n as follows.
プロジェクション行列Paによる射影変換を考慮したスケーリングパラメータを、以下のように定義する。
A scaling parameter considering the projection transformation by the projection matrix Pa is defined as follows.
ここで、Vbは定数であり、仮想カメラVRCAM-B観察系における第1スクリーン面SCR-Aの高さスケールである。
Here, Vb is a constant and is the height scale of the first screen surface SCR-A in the virtual camera VRCAM-B observation system.
さらに、光学的中心軸Oaxの偏向成分を考慮した第1スクリーン面SCR-Aの位置での移動量Tpを次のように定義する。
Further, the movement amount Tp at the position of the first screen surface SCR-A in consideration of the deflection component of the optical center axis Oax is defined as follows.
である。ここで、Axは第1スクリーン面SCR-Aの横方向のアスペクト比を表す定数であり、実カメラCAM-A系のデジタル画像が16:9画像の場合には16/9、4:3画像の場合には4/3という値となる。
It is. Here, Ax is a constant indicating the aspect ratio in the horizontal direction of the first screen surface SCR-A. When the digital image of the actual camera CAM-A system is a 16: 9 image, 16/9, 4: 3 image. In this case, the value is 4/3.
以上のパラメータを用い、同次座標表現で実カメラCAM-A系の座標における任意空間座標
Using the above parameters, arbitrary space coordinates in the coordinates of the real camera CAM-A system in homogeneous coordinate representation
を第1スクリーン面SCR-Aに対応するデジタル画像中のARマーカ像MRK1の観測位置へアフィン変換する4×4行列を、次のMpとする。この行列Mpはあらかじめオフセット行列設定部6により設定され、記憶部21に記憶されている。また、そのデータは可変である。
Is a 4 × 4 matrix that affine-transforms to the observation position of the AR marker image MRK1 in the digital image corresponding to the first screen surface SCR-A. This matrix Mp is set in advance by the offset matrix setting unit 6 and stored in the storage unit 21. The data is variable.
ただし、[Tp]、[Tr]は、それぞれの並進ベクトルの4×4行列同次座標表現である。
However, [Tp] and [Tr] are 4 × 4 matrix homogeneous coordinate expressions of the respective translation vectors.
この行列Mpにより、CGオブジェクトOBJ1の空間座標表現である任意空間座標
By this matrix Mp, arbitrary space coordinates that are the space coordinate representation of the CG object OBJ1
が第1スクリーン面SCR-A相当位置へアフィン変換される写像は、図9のようになり、CGオブジェクトOBJ1の原点座標
Is the affine transformation to the position corresponding to the first screen surface SCR-A, as shown in FIG. 9, and the origin coordinates of the CG object OBJ1
は、立体形状を保ったまま第1スクリーン面SCR-Aのデジタル画像中に観測されるARマーカ像MRK1に一致する位置へ変換される。
Is converted to a position that matches the AR marker image MRK1 observed in the digital image on the first screen surface SCR-A while maintaining the three-dimensional shape.
この状態で、仮想カメラVRCAM-B観察系の座標に、実カメラCAM-A撮影系の座標を配置するための任意のアフィン変換行列をMqとおき、仮想カメラVRCAM-B観察系の座標上の第2スクリーン面SCR-Bに射影する4×4行列をPbとおく。このとき、CGオブジェクトOBJ1の空間座標表現である任意空間座標
In this state, an arbitrary affine transformation matrix for arranging the coordinates of the real camera CAM-A photographing system is set as Mq at the coordinates of the virtual camera VRCAM-B observation system, and the coordinates on the virtual camera VRCAM-B observation system are set. A 4 × 4 matrix projected onto the second screen surface SCR-B is set as Pb. At this time, an arbitrary space coordinate which is a space coordinate representation of the CG object OBJ1
の第2スクリーン面SCR-Bへの射影変換mb′は次の式で表すことができる。
The projective transformation mb ′ to the second screen surface SCR-B can be expressed by the following equation.
この射影変換式を使用することにより、実カメラCAM-A系のデジタル画像中に観測されたARマーカ像MRK1上に、CG空間中に任意に設定された仮想カメラVRCAM-B座標系による画角・撮影位置で、CGオブジェクトOBJ1をその立体形状を保ったまま仮想カメラVRCAM-B系の第2スクリーン面SCR-Bに射影変換することが可能となる。
By using this projective transformation formula, the angle of view by the virtual camera VRCAM-B coordinate system arbitrarily set in the CG space on the AR marker image MRK1 observed in the digital image of the real camera CAM-A system. It is possible to projectively convert the CG object OBJ1 to the second screen surface SCR-B of the virtual camera VRCAM-B system while maintaining its three-dimensional shape at the photographing position.
また、行列PbとMqは一般的な3次元コンピュータ・グラフィックス技術における射影変換を表しているので、仮想カメラVRCAM-B系から観察する実カメラCAM-A座標系は任意の位置へ移動・回転・スケーリング及び描画画角の変更が可能である。
Since the matrices Pb and Mq represent projective transformation in general three-dimensional computer graphics technology, the real camera CAM-A coordinate system observed from the virtual camera VRCAM-B system is moved and rotated to an arbitrary position.・ Scaling and drawing angle of view can be changed.
本実施の形態のAR技術を用いたコンピュータ・グラフィックス画像処理システム及びそれが実行する画像処理方法によれば、次のような作用、効果を奏する。固定カメラ5によるデジタル画像に対して、従来技術では、図11に示すように、例えば仮想カメラVRCAM-B系を実カメラCAM-A撮影系の真横に配置した場合、CGオブジェクトOBJ1は平面でしか描画されないため直線に変換されてしまっていた。これに対して、本実施の形態では、図12に示すようにCGオブジェクトOBJ1を真横から見るように描画することが可能である。
According to the computer graphics image processing system using the AR technology of this embodiment and the image processing method executed by the computer graphics image processing system, the following operations and effects can be achieved. With respect to the digital image obtained by the fixed camera 5, in the prior art, as shown in FIG. 11, for example, when the virtual camera VRCAM-B system is arranged beside the real camera CAM-A photographing system, the CG object OBJ1 is only a plane. Because it was not drawn, it was converted to a straight line. On the other hand, in the present embodiment, it is possible to draw the CG object OBJ1 as seen from the side as shown in FIG.
仮想カメラVRCAM-Bは画角の変更も可能であることから、図26(a)に示したようにフレーム105からARマーカ101がはみ出す場合にARマーカ101が認識できなくなる問題点を解消できる。図13に示すように、実カメラCAM-A系はARマーカを認識できる画角に設定し固定しておくと、図14に示すように、仮想カメラVRCAM-Bで拡大縮小表示するようにできる。これは、図26(b)に示したARマーカ101の識別が困難なケースも同様に解決可能である。
Since the virtual camera VRCAM-B can change the angle of view, the problem that the AR marker 101 cannot be recognized when the AR marker 101 protrudes from the frame 105 as shown in FIG. As shown in FIG. 13, when the real camera CAM-A system is set and fixed at a field angle capable of recognizing the AR marker, the virtual camera VRCAM-B can be enlarged and reduced as shown in FIG. . This can be similarly solved in the case where it is difficult to identify the AR marker 101 shown in FIG.
また、図26(c)に示したようなARマーカ101が移動している時のボケは、ARマーカ101自体が高速で動いている場合にはAR解析器1での認識が不可能な問題は残るが、カメラ側が高速に動くケースでは、図15に示すように、仮想カメラVRCAM-Bの位置を変化させることにより表現可能となり、この場合、実カメラCAM-A系は固定してあるのでマーカ解析器1はARマーカ像MRK1の位置を正しく検出しながら、仮想カメラVRCAM-B系で高速なカメラワークを表現することが可能となる。
In addition, the blur when the AR marker 101 is moving as shown in FIG. 26C is a problem that the AR analyzer 1 cannot recognize when the AR marker 101 is moving at high speed. However, in the case where the camera side moves at high speed, it can be expressed by changing the position of the virtual camera VRCAM-B as shown in FIG. 15, and in this case, the real camera CAM-A system is fixed. The marker analyzer 1 can express high-speed camera work with the virtual camera VRCAM-B system while correctly detecting the position of the AR marker image MRK1.
さらに、図16、図17は、AR解析器1により観察されるARマーカ像MRK1の移動を、仮想カメラVRCAM-B系から見たCGオブジェクトOBJ1の移動へ反映された様子を表している。これは、図10からも明らかであるが、ARマーカ像MRK1の移動は第1スクリーン面SCR-A上の平面上で表現されているが、仮想カメラVRCAM-B座標系上に配置された第1スクリーン面SCR-Aは任意の平面空間にアフィン変換可能なため、実カメラCAM-A撮影系の平面上の動きを、仮想カメラVRCAM-B座標系における空間上の動きに変換可能となる。
Further, FIGS. 16 and 17 show how the movement of the AR marker image MRK1 observed by the AR analyzer 1 is reflected on the movement of the CG object OBJ1 viewed from the virtual camera VRCAM-B system. As is clear from FIG. 10, the movement of the AR marker image MRK1 is expressed on a plane on the first screen surface SCR-A, but the second is arranged on the virtual camera VRCAM-B coordinate system. Since the one-screen surface SCR-A can be affine-transformed into an arbitrary plane space, the movement on the plane of the real camera CAM-A imaging system can be converted into the movement on the space in the virtual camera VRCAM-B coordinate system.
によるアフィン変換は、仮想カメラVRCAM-B座標系への変換に他ならず、例えば、
The affine transformation by is nothing but the transformation to the virtual camera VRCAM-B coordinate system, for example,
は第1スクリーン面SCR-A上のARマーカ像MRK1の検出位置の原点となるので、この点を注目するように仮想カメラVRCAM-Bの向きや位置を計算したり画角範囲を調節したりするような制御も可能である。
Is the origin of the detection position of the AR marker image MRK1 on the first screen surface SCR-A, so that the orientation and position of the virtual camera VRCAM-B are calculated and the field angle range is adjusted so as to pay attention to this point. Such control is also possible.
図18は比較例としてAR解析器1による定点撮影画像に対し、ARマーカ像の位置に立体ARマーカオブジェクトを合成表示したCG画像を示している。そして、図19は実施例1として、同じ実カメラCAM-Aによる定点撮影画像に対してクロマキー処理を行い、そのクロマキー映像に対し、仮想カメラCAM-B系から射影変換して得られるCG画像を示している。このように、従来であれば、ARマーカ101が小さすぎて解像できず、ARマーカ像MRK1の位置にCGオブジェクトOBJ1を表示できなかったものが、任意の小さな画角にしてもCGオブジェクトOBJ1をARマーカ像MRK1の位置、姿勢、スケールで表示できていることが理解できる。
FIG. 18 shows a CG image in which a three-dimensional AR marker object is synthesized and displayed at a position of an AR marker image with respect to a fixed-point image captured by the AR analyzer 1 as a comparative example. FIG. 19 shows a CG image obtained by performing chroma key processing on a fixed-point image captured by the same real camera CAM-A and performing projective transformation on the chroma key image from the virtual camera CAM-B system as Example 1. Show. As described above, conventionally, the AR marker 101 is too small to be resolved and the CG object OBJ1 cannot be displayed at the position of the AR marker image MRK1, but the CG object OBJ1 can be displayed at an arbitrary small angle of view. Can be displayed with the position, posture and scale of the AR marker image MRK1.
また、図20はクロマキー処理によりサッカーゲームの解説画像を生成する実施例2を示すものである。同図(a)に示すARマーカに代えて、同図(b)に示すようにサッカー解説用に用意された2枚のARマーカをサッカーフィールドボードの前に配し、登場人物にそのARマーカの一方、あるいは両方を移動させてもらうと、同図(c)に示すようにクロマキー処理とCG画像合成処理によりサッカーフィールド上で二人のサッカープレーヤのCG画像がARマーカ像の上に配置され、ARマートの動きと同様にサッカープレーヤ像が移動するというリアルなCG画像が表現される。また、仮想カメラを上下に移動させて観察系を移動させることによりサッカーフィールド像もその上のプレーヤ像と共にそれに応じた角度、方向から俯瞰した状態の映像として表現できる。http://www.youtube.com/watch?v=W8msVZAlI3k&feature=player_embeddedに動画がアップロードされている。
FIG. 20 shows a second embodiment in which a commentary image of a soccer game is generated by chroma key processing. In place of the AR marker shown in FIG. 6A, two AR markers prepared for soccer commentary are arranged in front of the soccer field board as shown in FIG. When one or both of them are moved, the CG images of the two soccer players are placed on the AR marker image on the soccer field by chroma key processing and CG image composition processing as shown in FIG. Like the movement of the AR mart, a realistic CG image in which the soccer player image moves is represented. In addition, by moving the observation system by moving the virtual camera up and down, the soccer field image can be expressed as a video image of the player image on the top as well as the video image viewed from the angle and direction according to the image. Video uploaded to http://www.youtube.com/watch?v=W8msVZAlI3k&feature=player_embedded
以上のように、本発明のAR技術を用いたコンピュータ・グラフィックス画像処理システム及び方法によれば、ARマーカの位置検出のための固定カメラ5は常に定点を観測する実カメラCAM-Aとし、CG空間側に仮想カメラVRCAM-Bを定義し、仮想カメラ側の画角や位置を変化させることにより、画像認識型AR(ARマーカ)を使用したCGを合成表示する際に生じる問題点を解決し、従来のシステムでは不可能であった、画像認識型AR(ARマーカ)を使用したCG合成画像を作成し表示することができる。それ故、本発明によれば、テレビ放送分野での利用が容易になる。
As described above, according to the computer graphics image processing system and method using the AR technology of the present invention, the fixed camera 5 for detecting the position of the AR marker is the real camera CAM-A that always observes a fixed point. Solves the problems that occur when combining and displaying CG using image recognition AR (AR marker) by defining virtual camera VRCAM-B on the CG space side and changing the angle of view and position on the virtual camera side In addition, it is possible to create and display a CG composite image using an image recognition type AR (AR marker), which is impossible with a conventional system. Therefore, according to the present invention, utilization in the television broadcasting field is facilitated.
尚、本発明は上記一連の処理をコンピュータシステムによって行わせるためのプログラム、またそのプログラムを記録した記録媒体も技術的範囲とする。
The technical scope of the present invention also includes a program for causing the computer system to perform the above-described series of processing and a recording medium on which the program is recorded.
1 AR解析器
2 CGレンダリング部
3 表示器
4 カメラキャリブレーション部
5 固定カメラ
6 オフセット行列設定部
7 クロマキー装置
11 記憶部
13 ARマーカ像解析部
15 アフィン変換行列決定部
21 記憶部
23 行列Pb設定部
25 オブジェクト姿勢決定部
27 CG画像合成部
29 背景画像入力部
101 ARマーカ DESCRIPTION OFSYMBOLS 1 AR analyzer 2 CG rendering part 3 Display 4 Camera calibration part 5 Fixed camera 6 Offset matrix setting part 7 Chroma key apparatus 11 Storage part 13 AR marker image analysis part 15 Affine transformation matrix determination part 21 Storage part 23 Matrix Pb setting part 25 Object posture determination unit 27 CG image composition unit 29 Background image input unit 101 AR marker
2 CGレンダリング部
3 表示器
4 カメラキャリブレーション部
5 固定カメラ
6 オフセット行列設定部
7 クロマキー装置
11 記憶部
13 ARマーカ像解析部
15 アフィン変換行列決定部
21 記憶部
23 行列Pb設定部
25 オブジェクト姿勢決定部
27 CG画像合成部
29 背景画像入力部
101 ARマーカ DESCRIPTION OF
Claims (6)
- ARマーカを撮影する、位置が固定されている固定カメラと、
前記固定カメラのカメラパラメータを記憶するパラメータ設定部と、
前記固定カメラの撮影した前記ARマーカ像を含む画像フレームに対して、前記パラメータ設定部に記憶されているカメラパラメータを使用し、前記ARマーカの位置、姿勢及びスケールを解析するARマーカ姿勢解析部と、
前記ARマーカに対応したオブジェクトを、前記ARマーカ姿勢解析部の解析結果に基づいてコンピュータ・グラフィックス画像空間上における前記画像フレーム上での前記ARマーカ像の位置に対応した位置に、当該ARマーカ像の姿勢及びスケールに対応した姿勢及びスケールのコンピュータ・グラフィックス画像として生成するオブジェクト画像生成部と、
前記オブジェクト画像生成部の生成した前記オブジェクトのコンピュータ・グラフィックス画像に対して、前記コンピュータ・グラフィックス画像空間上の所定の座標位置に設置された仮想カメラから見た場合の見え方を割り出し、仮想カメラ観察画像として生成する仮想カメラ観察画像生成部と、
予め設定されている背景画像と前記仮想カメラ観察画像生成部の生成する前記仮想カメラから見た前記オブジェクトのコンピュータ・グラフィックス画像とを合成するコンピュータ・グラフィックス画像合成部と、
前記コンピュータ・グラフィックス画像合成部の合成したコンピュータ・グラフィックス合成画像を表示する表示部とを備えたことを特徴とするAR技術を用いたコンピュータ・グラフィックス画像処理システム。 A fixed camera for shooting the AR marker, the position being fixed;
A parameter setting unit for storing camera parameters of the fixed camera;
An AR marker posture analysis unit that analyzes the position, posture, and scale of the AR marker using the camera parameters stored in the parameter setting unit for an image frame including the AR marker image captured by the fixed camera. When,
The object corresponding to the AR marker is placed at a position corresponding to the position of the AR marker image on the image frame on the computer graphics image space based on the analysis result of the AR marker posture analysis unit. An object image generation unit that generates a computer graphics image of an attitude and scale corresponding to the attitude and scale of the image;
For the computer graphics image of the object generated by the object image generation unit, the appearance when viewed from a virtual camera installed at a predetermined coordinate position in the computer graphics image space is determined, A virtual camera observation image generation unit that generates a camera observation image;
A computer graphics image synthesizing unit that synthesizes a preset background image and a computer graphics image of the object viewed from the virtual camera generated by the virtual camera observation image generating unit;
A computer graphics image processing system using AR technology, comprising: a display unit for displaying a computer graphics synthesized image synthesized by the computer graphics image synthesizing unit. - 前記仮想カメラは、前記コンピュータ・グラフィックス画像空間上での設置座標が可変であることを特徴とする請求項1に記載のコンピュータ・グラフィックス画像処理システム。 2. The computer graphics image processing system according to claim 1, wherein the installation coordinates of the virtual camera on the computer graphics image space are variable.
- 前記ARマーカは、複数枚を前記固定カメラの画角の範囲内の任意の位置にセット可能にしたことを特徴とする請求項1又は2に記載のコンピュータ・グラフィックス画像処理システム。 3. The computer graphics image processing system according to claim 1, wherein a plurality of the AR markers can be set at arbitrary positions within a range of an angle of view of the fixed camera.
- コンピュータを用いて、
固定カメラで撮影したARマーカの画像を含む画像フレームを取り込み、
当該コンピュータにあらかじめ記憶されているカメラパラメータを使用し、前記ARマーカ像を含む画像フレームを解析して前記ARマーカ像の位置及び姿勢を割り出し、
前記ARマーカに対応したオブジェクトを、前記ARマーカ像の姿勢の解析結果に基づいてコンピュータ・グラフィックス画像空間上における前記画像フレーム上での前記ARマーカ像の位置に対応した位置に、当該ARマーカ像の姿勢及びスケールに対応した姿勢及びスケールのコンピュータ・グラフィックス画像として生成し、
前記オブジェクト画像フレーム上の前記ARマーカ像に対応して前記オブジェクトのコンピュータ・グラフィックス画像に対して、前記コンピュータ・グラフィックス画像空間上の所定の座標位置に設置された仮想カメラから見た場合の見え方を割り出し、仮想カメラ観察画像として生成し、
予め設定されている背景画像と前記仮想カメラから見た前記オブジェクトのコンピュータ・グラフィックス画像とを合成し、
前記合成したコンピュータ・グラフィックス合成画像を表示することを特徴とするAR技術を用いたコンピュータ・グラフィックス画像処理方法。 Using a computer
Capture an image frame containing an AR marker image taken with a fixed camera,
Using the camera parameters stored in advance in the computer, analyzing the image frame including the AR marker image to determine the position and orientation of the AR marker image,
The object corresponding to the AR marker is placed at a position corresponding to the position of the AR marker image on the image frame on the computer graphics image space based on the analysis result of the posture of the AR marker image. Generate a computer graphics image of the posture and scale corresponding to the posture and scale of the image,
When a computer graphics image of the object corresponding to the AR marker image on the object image frame is viewed from a virtual camera installed at a predetermined coordinate position on the computer graphics image space Determine how it looks and generate it as a virtual camera observation image,
Combining a preset background image and a computer graphics image of the object viewed from the virtual camera;
A computer graphics image processing method using AR technology, wherein the synthesized computer graphics composite image is displayed. - 前記仮想カメラは、前記コンピュータ・グラフィックス画像空間上での設置座標を可変とすることを特徴とする請求項4に記載のコンピュータ・グラフィックス画像処理方法。 5. The computer graphics image processing method according to claim 4, wherein the virtual camera has a variable installation coordinate in the computer graphics image space.
- 前記ARマーカは、複数枚を前記固定カメラの画角の範囲内の任意の位置にセットし、前記固定カメラにて同時に撮影することを特徴とする請求項4又は5に記載のコンピュータ・グラフィックス画像処理方法。 6. The computer graphics according to claim 4, wherein a plurality of the AR markers are set at arbitrary positions within a range of an angle of view of the fixed camera, and are simultaneously shot by the fixed camera. Image processing method.
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2012036627A JP5847610B2 (en) | 2012-02-22 | 2012-02-22 | Computer graphics image processing system and method using AR technology |
JP2012-036627 | 2012-02-22 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2013125098A1 true WO2013125098A1 (en) | 2013-08-29 |
Family
ID=49005299
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/JP2012/078175 WO2013125098A1 (en) | 2012-02-22 | 2012-10-31 | System and method for computer graphics image processing using augmented reality technology |
Country Status (3)
Country | Link |
---|---|
JP (1) | JP5847610B2 (en) |
TW (1) | TWI501193B (en) |
WO (1) | WO2013125098A1 (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN115460395A (en) * | 2022-06-24 | 2022-12-09 | 北京电影学院 | Camera registration tracking method based on LED background wall time-sharing multiplexing |
Families Citing this family (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP6585665B2 (en) * | 2017-06-29 | 2019-10-02 | ファナック株式会社 | Virtual object display system |
JP6781201B2 (en) | 2018-06-05 | 2020-11-04 | ファナック株式会社 | Virtual object display system |
CA3130160A1 (en) | 2019-04-04 | 2020-10-08 | Centerline Biomedical, Inc. | Registration of spatial tracking system with augmented reality display |
JP7404137B2 (en) | 2020-04-01 | 2023-12-25 | 株式会社豊田中央研究所 | Facial image processing device and facial image processing program |
JP7404282B2 (en) | 2021-02-10 | 2023-12-25 | 株式会社豊田中央研究所 | Facial model parameter estimation device, facial model parameter estimation method, and facial model parameter estimation program |
Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2000057350A (en) * | 1998-08-10 | 2000-02-25 | Toshiba Corp | Image processor, image processing method and image transmitting method |
JP2011141828A (en) * | 2010-01-08 | 2011-07-21 | Sony Corp | Information processing apparatus, information processing system, and information processing method |
JP2012003598A (en) * | 2010-06-18 | 2012-01-05 | Riso Kagaku Corp | Augmented reality display system |
Family Cites Families (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP5476036B2 (en) * | 2009-04-30 | 2014-04-23 | 国立大学法人大阪大学 | Surgical navigation system using retinal projection type head mounted display device and simulation image superimposing method |
TWI419081B (en) * | 2009-12-29 | 2013-12-11 | Univ Nat Taiwan Science Tech | Method and system for providing augmented reality based on marker tracing, and computer program product thereof |
KR101082285B1 (en) * | 2010-01-29 | 2011-11-09 | 주식회사 팬택 | Terminal and method for providing augmented reality |
JP5573238B2 (en) * | 2010-03-04 | 2014-08-20 | ソニー株式会社 | Information processing apparatus, information processing method and program |
TW201126451A (en) * | 2011-03-29 | 2011-08-01 | Yuan-Hong Li | Augmented-reality system having initial orientation in space and time and method |
-
2012
- 2012-02-22 JP JP2012036627A patent/JP5847610B2/en active Active
- 2012-10-31 WO PCT/JP2012/078175 patent/WO2013125098A1/en active Application Filing
- 2012-11-13 TW TW101142213A patent/TWI501193B/en active
Patent Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2000057350A (en) * | 1998-08-10 | 2000-02-25 | Toshiba Corp | Image processor, image processing method and image transmitting method |
JP2011141828A (en) * | 2010-01-08 | 2011-07-21 | Sony Corp | Information processing apparatus, information processing system, and information processing method |
JP2012003598A (en) * | 2010-06-18 | 2012-01-05 | Riso Kagaku Corp | Augmented reality display system |
Non-Patent Citations (2)
Title |
---|
"Augmented Reality(AR: Kakucho Genjitsu) System AR-NIXUS.M1", November 2011 (2011-11-01), Retrieved from the Internet <URL:http://www.nixus.jp/ar-nixus/images/20111101_ar_01.pdf> [retrieved on 20121204] * |
"Real Time 3DCG System 3D-NIXUS xf", November 2011 (2011-11-01), Retrieved from the Internet <URL:http://www.nixus.jp/3d-nixus/images/201111013dnixus.pdf> [retrieved on 20121204] * |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN115460395A (en) * | 2022-06-24 | 2022-12-09 | 北京电影学院 | Camera registration tracking method based on LED background wall time-sharing multiplexing |
Also Published As
Publication number | Publication date |
---|---|
JP2013171522A (en) | 2013-09-02 |
TW201335884A (en) | 2013-09-01 |
JP5847610B2 (en) | 2016-01-27 |
TWI501193B (en) | 2015-09-21 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
JP5872923B2 (en) | AR image processing apparatus and method | |
CN110809786B (en) | Calibration device, calibration chart, chart pattern generation device, and calibration method | |
EP3057066B1 (en) | Generation of three-dimensional imagery from a two-dimensional image using a depth map | |
WO2013125098A1 (en) | System and method for computer graphics image processing using augmented reality technology | |
US7782320B2 (en) | Information processing method and information processing apparatus | |
US10825249B2 (en) | Method and device for blurring a virtual object in a video | |
Tomioka et al. | Approximated user-perspective rendering in tablet-based augmented reality | |
JP7164968B2 (en) | IMAGE PROCESSING DEVICE, CONTROL METHOD AND PROGRAM OF IMAGE PROCESSING DEVICE | |
US20120319949A1 (en) | Pointing device of augmented reality | |
EP3572916B1 (en) | Apparatus, system, and method for accelerating positional tracking of head-mounted displays | |
JP2004213355A (en) | Information processing method | |
KR20140121529A (en) | Method and apparatus for formating light field image | |
KR20110132260A (en) | Monitor based augmented reality system | |
CN110969706B (en) | Augmented reality device, image processing method, system and storage medium thereof | |
JP6061334B2 (en) | AR system using optical see-through HMD | |
JP2008217593A (en) | Subject area extraction device and subject area extraction program | |
CN114913308A (en) | Camera tracking method, device, equipment and storage medium | |
KR101529820B1 (en) | Method and apparatus for determing position of subject in world coodinate system | |
JPWO2018189880A1 (en) | Information processing apparatus, information processing system, and image processing method | |
Zheng et al. | Pixel-wise closed-loop registration in video-based augmented reality | |
JP2013231607A (en) | Calibration tool display device, calibration tool display method, calibration device, calibration method, calibration system and program | |
JP2008203538A (en) | Image display system | |
US20230306676A1 (en) | Image generation device and image generation method | |
Orghidan et al. | Projector-camera calibration for 3D reconstruction using vanishing points | |
JP2018032991A (en) | Image display unit, image display method and computer program for image display |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 12869107 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 12869107 Country of ref document: EP Kind code of ref document: A1 |