WO2021181988A1 - Système de traitement d'informations, procédé de traitement d'informations, et programme - Google Patents

Système de traitement d'informations, procédé de traitement d'informations, et programme Download PDF

Info

Publication number
WO2021181988A1
WO2021181988A1 PCT/JP2021/004923 JP2021004923W WO2021181988A1 WO 2021181988 A1 WO2021181988 A1 WO 2021181988A1 JP 2021004923 W JP2021004923 W JP 2021004923W WO 2021181988 A1 WO2021181988 A1 WO 2021181988A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
dimensional shape
information
contour
unit
Prior art date
Application number
PCT/JP2021/004923
Other languages
English (en)
Japanese (ja)
Inventor
芳宏 真鍋
光祐 吉富
崇裕 坂本
忠義 村上
健志 後藤
Original Assignee
ソニーグループ株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by ソニーグループ株式会社 filed Critical ソニーグループ株式会社
Publication of WO2021181988A1 publication Critical patent/WO2021181988A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G06T17/10—Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G06T7/55—Depth or shape recovery from multiple images
    • G06T7/564—Depth or shape recovery from multiple images from contours
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/74—Projection arrangements for image reproduction, e.g. using eidophor

Definitions

  • the present invention relates to an information processing system, an information processing method and a program.
  • the projected image is calculated in consideration of the three-dimensional positional relationship between the projector and the screen and the unevenness of the screen.
  • the image calculated to be projected on the screen is projected on the object, and a distorted image is displayed on the object. Therefore, it is considered to project an image while avoiding an object.
  • the techniques of Patent Documents 1 and 2 are known.
  • the non-projection region is set based only on the two-dimensional shape of the object photographed by the camera.
  • the shape of the back part of the object that becomes the blind spot of the camera cannot be grasped from the image of the camera. Therefore, it is difficult to project an image along the contour of the object on the area on the screen that becomes the blind spot of the camera due to the object.
  • this disclosure proposes an information processing system, an information processing method, and a program capable of generating an image along the contour of an object over the entire circumference of the object.
  • an estimation unit that estimates the three-dimensional shape of the object based on an image of the object on the screen, and an estimation unit that estimates the space occupied by the object on the screen based on the three-dimensional shape of the object.
  • An information processing system including an image generation unit that generates an image projected on the screen while avoiding the space is provided. Further, according to the present disclosure, there is provided an information processing method in which the information processing of the information processing system is executed by a computer, and a program for realizing the information processing of the information processing system in the computer.
  • FIG. 1 is a schematic view of the information processing system 1 of the first embodiment.
  • the information processing system 1 includes, for example, a processing device 10, a storage device 20, one or more projectors 30, and a camera 40.
  • the processing device 10 is a device that maps image IMs from one or more projectors 30 on the screen SCR.
  • the screen SCR is, for example, a building or furniture.
  • the processing device 10 detects the object OB inserted on the screen SCR, for example, based on the image taken by the camera 40.
  • the processing device 10 avoids the object OB and generates a video IM projected on the screen SCR.
  • the camera 40 is a camera capable of detecting the depth information of the subject.
  • the camera 40 is installed above the screen SCR, for example.
  • the camera 40 captures the entire screen SCR from one location above the screen SCR.
  • the processing device 10 detects the depth information of the object OB inserted on the screen SCR based on the image taken by the camera 40.
  • a passive stereo method is a method of detecting the depth information of a subject by matching between the images of two cameras.
  • the active stereo method is a method of detecting the depth information of a subject by matching between an image of a pattern projected from a projector and an image taken by a camera.
  • the TOF method is a method of detecting the depth information of a subject by irradiating light from a light source and examining the time difference between reflection by an object and return.
  • One or more projectors 30 are installed above the screen SCR.
  • a plurality of projectors 30 including the first projector 30A and the second projector 30B are installed on the screen SCR.
  • the first projector 30A and the second projector 30B are installed at positions facing each other with the center of the screen SCR, for example.
  • the second projector 30B is installed in the vicinity of the camera 40, for example.
  • the optical axis of the camera 40 and the optical axis of the second projector 30B are substantially the same.
  • the processing device 10 includes, for example, an image acquisition unit 11, a coordinate information extraction unit 12, a rough shape determination unit 13, an estimation unit 14, and a video generation unit 15.
  • the image acquisition unit 11 acquires the data of the image taken by the camera 40.
  • the image acquisition unit 11 outputs the image data to the coordinate information extraction unit 12.
  • the coordinate information extraction unit 12 extracts the coordinate information of the object OB in the three-dimensional space from the image taken by the camera 40.
  • the coordinate information extraction unit 12 converts, for example, the depth information of the object OB into the coordinate information in the three-dimensional space.
  • the coordinate information extraction unit 12 uses the coordinate information of a part or all of the point cloud of the front portion of the object OB that is not the blind spot of the camera 40 as the coordinate information of the object OB among the point clouds constituting the object OB. Extract.
  • the outline determination unit 13 determines the outline of the object OB based on the coordinate information of the object OB. For example, the outline determination unit 13 applies the local features of the object OB extracted from the coordinate information of the object OB to the analysis model 23 to determine the outline of the object OB.
  • the analysis model 23 defines, for example, the correspondence between the local characteristics of the object OB and the outline shape of the object OB.
  • the analytical model 23 defines, for example, information about the cross-sectional shape and area ratio of a plurality of objects OBs having different heights as local features.
  • the outline is defined as a type of simple object such as a sphere, a cylinder, a prism, or a cone.
  • the analysis model 23 defines information on the outlines of a plurality of simple objects.
  • the outline determination unit 13 extracts, for example, information on the ratio of the cross-sectional shapes and areas of a plurality of object OBs having different heights from the coordinate information of the object OBs.
  • the shape of the object OB on the back surface which is the blind spot of the camera 40, cannot be grasped from the image of the camera 40. Therefore, the outline determination unit 13 estimates the cross-sectional shape of the object OB based on the shape of the object OB in the front portion of the camera 40 that is not a blind spot. For example, the outline determination unit 13 sets a bounding box that surrounds a point cloud that constitutes the contour of the cross section.
  • the outline determination unit 13 determines that the cross-sectional shape is a circle. When the coordinates of the point cloud are present at the corners of the bounding box, the outline determination unit 13 determines that the cross-sectional shape is a square. The outline determination unit 13 determines the type of the simple object associated with the extracted information as the outline of the object OB. The outline determination unit 13 outputs information on the outline of the object OB to the estimation unit 14.
  • FIG. 2 is a diagram showing the correspondence between the local features defined by the analysis model 23 and the outline.
  • the first cross section CT1, the second cross section CT2, and the third cross section CT3 are used as a plurality of cross sections that serve as criteria for determining the outline.
  • the first cross section CT1 is, for example, the lowest surface of the object OB (the surface facing the screen SCR).
  • the second cross section CT2 is, for example, the highest surface of the object OB (the surface farthest from the screen SCR).
  • the third cross section CT3 is, for example, a cross section having a height intermediate between the first cross section CT1 and the second cross section CT2.
  • the area of the larger cross section is A
  • the area of the smaller cross section is B
  • C be the area of the third cross section.
  • the general shape of the object OB is determined to be columnar. If the coordinates of the point cloud do not exist at the corners of the bounding box of the cross section having the area A, it is determined to be a cylinder, and if the coordinates of the point cloud exist, it is determined to be a prism.
  • the C / A is larger than the third threshold value, it is determined that the approximate shape of the object OB is a sphere.
  • the B / A is less than the first threshold value and the C / A is less than or equal to the third threshold value, it is determined that the approximate shape of the object OB is a cone. If the coordinates of the point cloud do not exist at the corners of the bounding box of the cross section having the area A, it is determined to be a cone, and if the coordinates of the point cloud exist, it is determined to be a quadrangular pyramid.
  • the estimation unit 14 estimates the three-dimensional shape of the object OB based on the image of the object OB on the screen SCR.
  • the estimation unit 14 generates, for example, a three-dimensional model of a pseudo-object POB that simplifies the object OB based on the outline shape and coordinate information of the object OB.
  • the three-dimensional model of the pseudo-object POB is generated, for example, by combining the information on the outline of the object OB with the dimensional information of the pseudo-object POB (information such as the diameter and height of the cross section of the pseudo-object POB).
  • the dimensional information of the pseudo object POB is estimated based on, for example, the coordinate information of the object OB.
  • the estimation unit 14 estimates the three-dimensional shape of the pseudo-object POB as the three-dimensional shape of the object OB.
  • the image generation unit 15 estimates the space on the screen SCR occupied by the object OB based on the three-dimensional shape of the object OB estimated by the estimation unit 14. The image generation unit 15 avoids this space and generates an image IM projected on the screen SCR.
  • FIG. 3 is a diagram showing the relationship between the pseudo-object POB and the video IM.
  • Video IMs from one or more projectors 30 are projected around the pseudo-object POB.
  • the video IM includes, for example, a first video IM1 projected from the first projector 30A and a second video IM2 projected from the second projector 30B.
  • a region SA that becomes a blind spot of the camera 40 is formed by the object OB.
  • the contour SL of the region SA is a projection of the contour POL of the front portion of the pseudo-object POB on the screen SCR.
  • Region SA extends along axis AX.
  • the axis AX is an axis obtained by projecting the optical axis of the camera 40 onto the screen SCR.
  • the image generation unit 15 generates an image projected on the region SA based on the three-dimensional shape of the pseudo object POB.
  • the generated image is projected onto the area SA from the first projector 30A installed at a position different from that of the camera 40.
  • the first image IM1 and the second image IM2 generated based on the three-dimensional shape of the pseudo object POB are projected on the region NSA on the screen SCR that does not become a blind spot of the camera 40.
  • the first video IM1 and the second video IM2 are generated as images along the contour of the bottom surface of the pseudo-object POB.
  • the pseudo-object POB has a shape that imitates the object OB. Therefore, the image IM along the contour of the object OB is displayed on the screen SCR over the entire circumference of the object OB.
  • the storage device 20 stores the screen model 21, the PJ parameter 22, the analysis model 23, and the program 24.
  • the screen model 21 is a three-dimensional model of the screen SCR.
  • the screen model 21 includes, for example, coordinate information on the surface of the screen SCR on which the image IM is projected.
  • the PJ parameter 22 includes, for example, information such as an internal parameter, an installation position, and a posture at the installation position for each projector 30.
  • the video generation unit 15 uses the screen model 21 and the PJ parameter 22 to generate a video IM of geometrically corrected display content for each projector 30.
  • the image generation unit 15 sets, for example, a region on the screen SCR that does not project the image IM for each projector 30 based on the three-dimensional shape of the object OB estimated by the estimation unit 14 and the PJ parameter 22.
  • Program 24 is a program that causes a computer to execute information processing according to this embodiment.
  • the processing device 10 performs various processes according to the program 24 stored in the storage device 20.
  • the storage device 20 may be used as a work area for temporarily storing the processing result of the processing device 10.
  • the storage device 20 includes any non-transient storage medium such as, for example, a semiconductor storage medium and a magnetic storage medium.
  • the storage device 20 includes, for example, an optical disk, a magneto-optical disk, or a flash memory.
  • the program 24 is stored, for example, in a non-transient storage medium that can be read by a computer.
  • the processing device 10 is, for example, a computer composed of a processor and a memory.
  • the memory of the processing device 10 includes a RAM (Random Access Memory) and a ROM (Read Only Memory).
  • the processing device 10 functions as an image acquisition unit 11, a coordinate information extraction unit 12, a rough shape determination unit 13, an estimation unit 14, and a video generation unit 15.
  • FIG. 4 is a flowchart showing the information processing method of the present embodiment.
  • step S1 the image acquisition unit 11 acquires the image of the screen SCR taken by the camera 40.
  • the image includes depth information of the object OB inserted on the screen SCR.
  • step S2 the coordinate information extraction unit 12 extracts the coordinate information of the object OB in the three-dimensional space based on the depth information of the object OB.
  • the extracted coordinate information is the coordinate information of the front portion of the object OB that does not become a blind spot of the camera 40.
  • the coordinate information of the back surface portion of the object OB, which is the blind spot of the camera 40, is not acquired.
  • the outline determination unit 13 determines the outline of the object OB based on the coordinate information of the object OB.
  • the outline determination unit 13 determines the type of a simple object including the characteristics of the object OB as the outline of the object OB, based on, for example, the ratio of the shapes and areas of a plurality of cross sections at a specific height of the object OB.
  • step S4 the estimation unit 14 generates a three-dimensional model of a pseudo-object POB that simplifies the object OB in the image. For example, the estimation unit 14 combines the outline information determined by the outline determination unit 13 with the dimensional information of the pseudo object POB estimated from the coordinate information of the object OB to generate a three-dimensional model of the pseudo object POB. do.
  • step S5 the estimation unit 14 estimates the three-dimensional shape of the pseudo-object POB as the three-dimensional shape of the object OB.
  • step S6 the image generation unit 15 estimates the space on the screen SCR occupied by the object OB based on the three-dimensional shape of the object OB estimated by the estimation unit 14. The image generation unit 15 avoids this space and generates an image IM projected on the screen SCR.
  • the information processing system 1 has an estimation unit 14 and a video generation unit 15.
  • the estimation unit 14 estimates the three-dimensional shape of the object OB based on the image of the object OB on the screen SCR.
  • the image generation unit 15 estimates the space on the screen SCR occupied by the object OB based on the three-dimensional shape of the object OB, and generates the image IM projected on the screen SCR while avoiding this space.
  • the information processing of the information processing system 1 described above is executed by the computer.
  • the program 24 of the present embodiment makes the computer realize the information processing of the information processing system 1 described above.
  • the non-projection region of the video IM is set based on the three-dimensional shape of the object OB. Therefore, it is possible to generate a video IM along the contour of the object OB over the entire circumference of the object OB.
  • the image generation unit 15 generates, for example, an image IM projected by the object OB on the area on the screen SCR that becomes the blind spot of the camera 40 that captured the image.
  • the image IM along the contour of the object OB is also projected on the area that becomes the blind spot of the camera 40.
  • the information processing system 1 has, for example, a coordinate information extraction unit 12 and a rough shape determination unit 13.
  • the coordinate information extraction unit 12 extracts, for example, the coordinate information of the object OB from the image.
  • the outline determination unit 13 determines the outline of the object OB based on, for example, the coordinate information of the object OB.
  • the estimation unit 14 generates, for example, a three-dimensional model of a pseudo-object POB that simplifies the object OB based on the outline shape and coordinate information of the object OB.
  • the estimation unit 14 estimates, for example, the three-dimensional shape of the pseudo-object POB as the three-dimensional shape of the object OB.
  • the three-dimensional shape of the object OB is estimated by a simple calculation. There may be an error between the three-dimensional shape of the actual object OB and the three-dimensional shape of the pseudo-object POB. However, the image IM is not projected in the space occupied by the object OB. Even if there is an error between the three-dimensional shape of the pseudo-object POB and the three-dimensional shape of the actual object OB, the area where the image IM is not projected only increases or decreases. Therefore, the display quality is not significantly impaired.
  • the three-dimensional model of the pseudo-object POB is generated by combining the dimensional information of the pseudo-object POB with the outline information output from the outline determination unit 13.
  • the method of generating a three-dimensional model of the pseudo-object POB is not limited to this.
  • the estimation unit 14 may generate a three-dimensional model of the pseudo-object POB by connecting the coordinates of a plurality of point clouds of the object OB extracted from the image of the camera 40 with a smooth curve.
  • a plurality of cross sections having different heights of an object OB can be connected by NURBS (Non-Uniform Regional B-Spline) to generate a three-dimensional model of a pseudo object POB.
  • NURBS Non-Uniform Regional B-Spline
  • the outline determination unit 13 determines the outline of the object OB based on the information regarding the ratio of the cross-sectional shapes and areas of the plurality of objects OB having different heights.
  • the method for determining the outline is not limited to this.
  • the outline determination unit 13 can determine the outline of the object OB based on the contour of the object OB captured in the image of the camera 40.
  • 5 and 6 are diagrams showing an example of a method for determining the outline shape in this modified example.
  • FIG. 5 shows an image of the object OB taken by the camera 40.
  • the camera 40 is installed, for example, above the center of the screen SCR.
  • the outline determination unit 13 extracts, for example, the contour OL of the object OB from the two-dimensional shape of the object OB shown in the image.
  • the outline determination unit 13 detects, for example, the contour obtained by projecting the contour OL on the screen SCR along the optical axis of the camera 40 that captured the image as a pseudo contour PC.
  • FIG. 6 is a diagram showing the relationship between the contour OL and the pseudo contour PC.
  • the outline determination unit 13 determines, for example, a columnar body having the pseudo-contour PC as the outline of the lowest surface as the outline of the object OB.
  • the estimation unit 14 generates a three-dimensional model of a columnar body based on the outline shape and coordinate information of the object OB.
  • the three-dimensional model of the pseudo-object POB which is a columnar body, is generated by combining the information on the shape and dimensions of the pseudo-contour PC with the information on the height of the object OB.
  • the height of the object OB is calculated as the difference between the coordinates of the contour LOL of the lowest surface of the object OB and the contour HOL of the highest surface in the height direction.
  • the estimation unit 14 estimates the three-dimensional shape of the pseudo-object POB as the three-dimensional shape of the object OB.
  • the image generation unit 15 estimates the space on the screen SCR occupied by the object OB based on the three-dimensional shape of the object OB estimated by the estimation unit 14. The image generation unit 15 avoids this space and generates an image IM projected on the screen SCR.
  • the pseudo-object POB is a columnar body having a pseudo-contour PC as the lowest surface. Therefore, the video generation unit 15 generates, for example, a video IM along the pseudo-contour PC.
  • the image generation unit 15 estimates, for example, the region of the contour OL where the depth information of the subject changes discontinuously between the inside (object OB side) and the outside (screen SCR side) of the contour OL as the step region LL.
  • the step region LL has a low likelihood as the lowest surface of the object OB. Therefore, the image generation unit 15 estimates that the region of the pseudo contour PC corresponding to the step region LL is the low likelihood contour region LA.
  • the image generation unit 15 blurs the image IM displayed along the low-likelihood contour region LA.
  • a known method such as Gaussian blur is adopted.
  • the image generation unit 15 makes the amount of blur different depending on the likelihood, for example.
  • the image generation unit 15 determines, for example, the likelihood of the low-likelihood contour region LA based on the amount of change in the depth information that occurs in the step region LL corresponding to the low-likelihood contour region LA.
  • the image generation unit 15 determines, for example, that the lower the likelihood contour region LA corresponding to the step region LL in which the amount of change in depth information is large, the smaller the likelihood.
  • the image generation unit 15 blurs the image IM along the low-likelihood contour region LA, which has a small likelihood, as much as possible.
  • the image generation unit 15 estimates, for example, the region of the contour OL where the depth information of the subject continuously changes between the inside and the outside of the contour OL as the non-step region HL.
  • the non-step region HL has a high likelihood as the lowest surface of the object OB. Therefore, the image generation unit 15 estimates that the region of the pseudo contour PC corresponding to the non-step region HL is the high likelihood contour region HA.
  • the image generation unit 15 does not blur the image IM displayed along the high-likelihood contour region HA.
  • FIG. 7 is a diagram showing an example of an image projected along the low-likelihood contour region LA.
  • the image IM is blurred by adding a gradient to the brightness of the dot pattern.
  • the upper part of FIG. 7 shows the positional relationship between the video IM and the pseudo-object POB.
  • the lower part of FIG. 7 shows the positional relationship between the image IM and the object OB.
  • the back part of the object OB facing the low-likelihood contour region LA is the blind spot of the camera.
  • the shape of the object OB on the back surface, which is the blind spot of the camera 40, cannot be grasped from the image of the camera 40.
  • the image IM is projected along the pseudo-contour PC, if a deviation ⁇ occurs between the estimated three-dimensional shape of the object OB and the three-dimensional shape of the actual object OB, the contour of the actual object OB is different.
  • a boundary of the image IM is formed at the position (the position of the pseudo contour PC).
  • the image is blurred, a clear boundary line cannot be formed. Therefore, the display is less likely to cause discomfort. By making the amount of blur different according to the likelihood, the feeling of discomfort is reduced.
  • the camera 40 for detecting the depth information of the subject is provided at one place above the screen SCR, but the number of cameras 40 is not limited to one. A plurality of cameras 40 may be installed above the screen SCR.
  • FIG. 8 is a schematic view of the information processing system 2 of the second embodiment.
  • the points different from the first embodiment in this embodiment are that the class to which the object OB belongs is determined by using the object recognition technique, and the information of one or more object models defined for each class and the coordinates of the object OB. This is a point where the three-dimensional shape of the object OB is estimated by collating with the information.
  • the differences from the first embodiment will be mainly described.
  • the processing device 50 has, for example, a class determination unit 51 and a collation unit 52.
  • the storage device 60 stores, for example, the object model information 61 and the analysis model 62.
  • the analysis model 62 is a machine learning model in which the image of the subject is used as the input layer and the class to which the subject belongs is used as the output layer.
  • the analysis model 62 classifies subjects into classes such as PET bottles, watches, smartphones, chairs, and desks by using general object recognition technology based on deep learning.
  • the object model information 61 defines, for example, the characteristics of one or more object models belonging to each class for each class. For example, in the PET bottle class, a plurality of PET bottles having different three-dimensional shapes are registered as a plurality of object models. In the object model information 61, a plurality of PET bottles and the characteristics of the three-dimensional shape of each PET bottle are defined in association with each other in the PET bottle class.
  • the image acquisition unit 11 outputs the image acquired from the camera 40 to the class determination unit 51 and the coordinate information extraction unit 12.
  • the class determination unit 51 applies the image output from the image acquisition unit 11 to the analysis model 62, and determines the class of the object OB based on the image of the object OB.
  • the class determination unit 51 uses semantic segmentation to separate the target objects OB and classify the objects OB. conduct.
  • the collating unit 52 collates the features of the object OB extracted from the coordinate information of the object OB with the features of one or more object models belonging to the class of the object OB, and determines the object model including the features of the object OB. ..
  • the collation is performed using, for example, known three-dimensional features such as SHOT (Signature of Histograms of Origin States), PFH (Point Fature Histograms), and PPF (Point Pair Histograms).
  • SHOT Signature of Histograms of Origin States
  • PFH Point Fature Histograms
  • PPF Point Pair Histograms
  • the estimation unit 53 estimates the three-dimensional shape of the object OB based on, for example, the feature information of the object model including the feature of the object OB and the coordinate information of the object OB.
  • the three-dimensional shape of the object OB is estimated by combining the dimensional information of the object OB with the shape defined based on the characteristics of the object model, for example.
  • the dimensional information of the object OB is estimated based on, for example, the coordinate information of the object OB.
  • Program 63 is a program that causes a computer to execute information processing according to this embodiment.
  • the processing device 50 performs various processes according to the program 63 stored in the storage device 60.
  • the processing device 50 functions as an image acquisition unit 11, a coordinate information extraction unit 12, a class determination unit 51, a collation unit 52, an estimation unit 53, and a video generation unit 15.
  • FIG. 9 is a flowchart showing the information processing method of the present embodiment.
  • step S11 the image acquisition unit 11 acquires the image of the screen SCR taken by the camera 40.
  • the image includes depth information of the object OB inserted on the screen SCR.
  • step S12 the coordinate information extraction unit 12 extracts the coordinate information of the object OB based on the depth information of the object OB.
  • the coordinate information extraction unit 12 extracts, for example, the coordinate information of the point cloud of the object OB as the coordinate information from the image.
  • step S13 the class determination unit 51 applies the image output from the image acquisition unit 11 to the analysis model 62, and determines the class of the object OB based on the image of the object OB.
  • step S14 the collating unit 52 collates the features of the object OB extracted from the coordinate information of the object OB with the features of one or more object models belonging to the class of the object OB, and creates an object model including the features of the object OB. judge.
  • step S15 the estimation unit 53 estimates the three-dimensional shape of the object OB based on the feature information of the object model including the feature of the object OB and the coordinate information of the object OB.
  • step S16 the image generation unit 15 estimates the space on the screen SCR occupied by the object OB based on the three-dimensional shape of the object OB estimated by the estimation unit 53.
  • the image generation unit 15 avoids this space and generates an image IM projected on the screen SCR.
  • the information processing system 2 has a class determination unit 51 and a collation unit 52.
  • the class determination unit 51 determines the class of the object OB based on the image of the object OB inserted on the screen SCR.
  • the storage device 60 stores the object model information 61 that defines the characteristics of one or more object models belonging to the class for each class.
  • the collation unit 52 collates the features of the object OB extracted from the coordinate information of the object OB with the features of one or more object models belonging to the class of the object OB, and determines the object model including the features of the object OB.
  • the estimation unit 53 estimates the three-dimensional shape of the object OB based on the feature information of the object model including the feature of the object OB and the coordinate information of the object OB.
  • FIG. 10 is a schematic view of the information processing system 3 of the third embodiment.
  • the difference from the first embodiment in this embodiment is that the three-dimensional shape of the object OB is estimated by collating the information of one or more object models with the coordinate information of the object OB.
  • the information processing system 3 is applied to an environment in which the types of objects OB are limited, such as a play facility in which available toys are specified. Therefore, the classification using the object recognition technique as in the second embodiment is not performed.
  • the differences from the first embodiment will be mainly described.
  • the storage device 80 stores, for example, the object model information 81.
  • the object model information 81 defines, for example, the features of one or more object models. For example, when a toy such as a building block is provided in a play facility, the object model information 81 defines one or more toys that can be used and the three-dimensional shape characteristics of each toy in association with each other. NS.
  • the processing device 70 has, for example, a collating unit 71.
  • the collating unit 71 collates the features of the object OB extracted from the coordinate information of the object OB with the features of the one or more object models, and determines one or more object models including the features of the object OB.
  • the estimation unit 72 estimates the three-dimensional shape of the object OB based on, for example, the three-dimensional shape of one or more object models having the characteristics of the object OB.
  • 11 and 12 are diagrams showing an example of the object model OMB defined in the object model information 81.
  • FIG. 11 shows a toy T in which a plurality of cube-shaped unit structure TUs are combined.
  • the amusement facility is provided with a plurality of types of toys T according to the combination of the unit structure TY.
  • the unit structure TU is defined as the object model OBM.
  • the estimation unit 72 estimates, for example, one object model including the features of the object OB as the object model OBM of the unit structure YU.
  • the estimation unit 72 estimates, for example, the three-dimensional shape of the structure (toy T) in which a plurality of unit structures TUs are combined as the three-dimensional shape of the object OB.
  • the number of unit structures YU and the connection structure are estimated based on, for example, the coordinate information of the object OB.
  • FIG. 12 shows a plurality of types of building blocks registered as an object model OBM.
  • five types of building blocks are registered as the object model OBM.
  • the object model OBM1 is a plate-shaped building block having a square plate surface.
  • the object model OBM2 is a columnar building block.
  • the object model OBM3 is a cube-shaped building block.
  • the object model OBM4 is a plate-shaped building block having a rectangular plate surface.
  • the object model OBM5 is a triangular columnar building block.
  • the object model information 81 defines five types of object model OBMs and the features of the three-dimensional shapes of the respective object model OBMs in association with each other.
  • the estimation unit 72 identifies, for example, one or more object model OBMs having the characteristics of the object OB.
  • FIG. 12 for example, an object OB obtained by stacking an object model OBM3, an object model OBM2, and an object model OBM1 from the bottom in this order is shown.
  • the estimation unit 72 determines, for example, the relative position of one or more object model OBMs based on the coordinate information of the object OB.
  • the estimation unit 72 estimates the three-dimensional shape of the object OB based on, for example, the three-dimensional shape and the relative position of each object model OBM.
  • Program 82 is a program that causes a computer to execute information processing according to this embodiment.
  • the processing device 70 performs various processes according to the program 82 stored in the storage device 80.
  • the processing device 70 functions as an image acquisition unit 11, a coordinate information extraction unit 12, a collation unit 71, an estimation unit 72, and a video generation unit 15.
  • FIG. 13 is a flowchart showing the information processing method of the present embodiment.
  • step S21 the image acquisition unit 11 acquires the image of the screen SCR taken by the camera 40.
  • the image includes depth information of the object OB inserted on the screen SCR.
  • step S22 the coordinate information extraction unit 12 extracts the coordinate information of the object OB based on the depth information of the object OB.
  • the coordinate information extraction unit 12 extracts, for example, the coordinate information of the point cloud of the object OB as the coordinate information from the image.
  • step S23 the collating unit 71 collates the features of the object OB extracted from the coordinate information of the object OB with the features of the one or more object model OBM, and determines one or more object model OBM including the features of the object OB. ..
  • step S24 the estimation unit 72 estimates the three-dimensional shape of the object OB based on the three-dimensional shape of one or more object model OBMs having the characteristics of the object OB.
  • step S25 the image generation unit 15 estimates the space on the screen SCR occupied by the object OB based on the three-dimensional shape of the object OB estimated by the estimation unit 72.
  • the image generation unit 15 avoids this space and generates an image IM projected on the screen SCR.
  • the information processing system 3 has a collation unit 71.
  • the storage device 80 stores the object model information 81 that defines the characteristics of one or more object model OBMs.
  • the collation unit 71 collates the features of the object OB extracted from the coordinate information of the object OB with the features of the one or more object model OBM, and determines one or more object model OBM including the features of the object OB.
  • the estimation unit 72 estimates the three-dimensional shape of the object OB based on the three-dimensional shape of one or more object model OBMs having the characteristics of the object OB.
  • the three-dimensional shape of the object OB is accurately estimated based on the object model information 81.
  • the estimation unit 72 estimates one object model OBM including the features of the object OB as the object model OBM of the unit structure TU, and three-dimensionally of the structure (toy T) in which a plurality of unit structure TUs are combined.
  • the shape is estimated to be the three-dimensional shape of the object OB.
  • An estimation unit that estimates the three-dimensional shape of the object based on the image of the object on the screen
  • An image generation unit that estimates the space on the screen occupied by the object based on the three-dimensional shape of the object and generates an image projected on the screen while avoiding the space.
  • Information processing system with (2) The information processing system according to (1), wherein the image generation unit generates an image projected by the object onto an area on the screen that becomes a blind spot of a camera that has taken the image.
  • a coordinate information extraction unit that extracts coordinate information of the object from the image, An outline determination unit that determines the outline of the object based on the coordinate information of the object, Have,
  • the estimation unit generates a three-dimensional model of a pseudo-object that simplifies the object based on the outline shape of the object and the coordinate information, and estimates the three-dimensional shape of the pseudo-object as the three-dimensional shape of the object.
  • the information processing system according to (1) or (2) above.
  • the rough shape determination unit extracts the contour of the object from the two-dimensional shape of the object reflected in the image, and projects the contour onto the screen along the optical axis of the camera that captured the image.
  • the information processing system wherein the obtained contour is detected as a pseudo contour, and a columnar body having the pseudo contour as the contour of the lowest surface is determined as a rough shape of the object.
  • the image contains depth information of the subject and contains The image generation unit estimates that the contour region in which the depth information of the subject changes discontinuously between the inside and the outside of the contour as a step region, and the pseudo contour region corresponding to the step region. Is estimated as a low-likelihood contour region, and the image along the low-likelihood contour region is blurred.
  • the information processing system according to (4).
  • the video generation unit determines the likelihood of the low likelihood contour region based on the amount of change in the depth information that occurs in the step region corresponding to the low likelihood contour region.
  • the image generation unit determines that the lower likelihood contour region corresponding to the step region where the amount of change in the depth information is larger has a smaller likelihood.
  • a class determination unit that determines the class of the object based on the image, A storage device that stores object model information that defines the characteristics of one or more object models belonging to the class for each class.
  • a coordinate information extraction unit that extracts coordinate information of the object from the image,
  • a collating unit that collates the features of the object extracted from the coordinate information of the object with the features of the one or more object models belonging to the class of the object and determines the object model including the features of the object.
  • the information processing system estimates a three-dimensional shape of the object based on information on the characteristics of the object model including the characteristics of the object and coordinate information of the object. .. (8)
  • a storage device that stores object model information that defines the characteristics of one or more object models, and
  • a coordinate information extraction unit that extracts coordinate information of the object from the image
  • a collating unit that collates the features of the object extracted from the coordinate information of the object with the features of the one or more object models and determines one or more object models including the features of the object.
  • the estimation unit estimates the three-dimensional shape of the object based on the three-dimensional shape of one or more object models having the characteristics of the object.
  • the estimation unit estimates one object model including the features of the object as an object model of the unit structure, and sets the three-dimensional shape of the structure in which a plurality of the unit structures are combined as the three-dimensional shape of the object.
  • the information processing system according to (8) above.
  • the three-dimensional shape of the object is estimated based on the image of the object on the screen. Based on the three-dimensional shape of the object, the space on the screen occupied by the object is estimated, and an image projected on the screen is generated while avoiding the space.
  • the three-dimensional shape of the object is estimated based on the image of the object on the screen. Based on the three-dimensional shape of the object, the space on the screen occupied by the object is estimated, and an image projected on the screen is generated while avoiding the space.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Geometry (AREA)
  • Computer Graphics (AREA)
  • Software Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Computer Hardware Design (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Transforming Electric Information Into Light Information (AREA)
  • Image Analysis (AREA)
  • Processing Or Creating Images (AREA)
  • Image Generation (AREA)
  • Controls And Circuits For Display Device (AREA)

Abstract

L'invention concerne un système de traitement d'informations (1) comprenant une unité d'estimation (14) et une unité de génération d'image (15). L'unité d'estimation (14) estime une forme tridimensionnelle d'un objet (OB) sur la base d'une image de l'objet (OB) sur un écran (SCR). L'unité de génération d'image (15) estime un espace occupé par l'objet (OB)) dans l'écran (SCR) sur la base de la forme tridimensionnelle de l'objet (OB), et génère une image (IM) projetée sur l'écran (SCR) en évitant l'espace.
PCT/JP2021/004923 2020-03-12 2021-02-10 Système de traitement d'informations, procédé de traitement d'informations, et programme WO2021181988A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2020042753A JP2021144472A (ja) 2020-03-12 2020-03-12 情報処理システム、情報処理方法およびプログラム
JP2020-042753 2020-03-12

Publications (1)

Publication Number Publication Date
WO2021181988A1 true WO2021181988A1 (fr) 2021-09-16

Family

ID=77671410

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2021/004923 WO2021181988A1 (fr) 2020-03-12 2021-02-10 Système de traitement d'informations, procédé de traitement d'informations, et programme

Country Status (2)

Country Link
JP (1) JP2021144472A (fr)
WO (1) WO2021181988A1 (fr)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2007520945A (ja) * 2004-01-09 2007-07-26 アイマット・インコーポレーテッド プレゼンターの影が投影スクリーンに投影されるのを禁止する方法および装置
JP2008116565A (ja) * 2006-11-01 2008-05-22 Seiko Epson Corp 画像補正装置、プロジェクションシステム、画像補正方法、画像補正プログラム、および記録媒体
JP2015177383A (ja) * 2014-03-17 2015-10-05 カシオ計算機株式会社 投影装置及びプログラム
WO2016125359A1 (fr) * 2015-02-03 2016-08-11 ソニー株式会社 Dispositif de traitement d'informations, procédé de traitement d'informations et programme
WO2019009100A1 (fr) * 2017-07-07 2019-01-10 ソニー株式会社 Dispositif de commande, procédé de commande et programme

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2007520945A (ja) * 2004-01-09 2007-07-26 アイマット・インコーポレーテッド プレゼンターの影が投影スクリーンに投影されるのを禁止する方法および装置
JP2008116565A (ja) * 2006-11-01 2008-05-22 Seiko Epson Corp 画像補正装置、プロジェクションシステム、画像補正方法、画像補正プログラム、および記録媒体
JP2015177383A (ja) * 2014-03-17 2015-10-05 カシオ計算機株式会社 投影装置及びプログラム
WO2016125359A1 (fr) * 2015-02-03 2016-08-11 ソニー株式会社 Dispositif de traitement d'informations, procédé de traitement d'informations et programme
WO2019009100A1 (fr) * 2017-07-07 2019-01-10 ソニー株式会社 Dispositif de commande, procédé de commande et programme

Also Published As

Publication number Publication date
JP2021144472A (ja) 2021-09-24

Similar Documents

Publication Publication Date Title
US9905032B2 (en) Object removal using lidar-based classification
Tóth et al. Automatic LiDAR-camera calibration of extrinsic parameters using a spherical target
US10354129B2 (en) Hand gesture recognition for virtual reality and augmented reality devices
Lysenkov et al. Recognition and pose estimation of rigid transparent objects with a kinect sensor
CN102572505B (zh) 用于校准深度图像传感器的方法和***
US20180189974A1 (en) Machine learning based model localization system
Berger et al. A state of the art report on kinect sensor setups in computer vision
US20170140537A1 (en) System and method for scoring clutter for use in 3d point cloud matching in a vision system
JP5484133B2 (ja) 鏡面反射物体の３ｄ姿勢を推定する方法
US11615547B2 (en) Light field image rendering method and system for creating see-through effects
Taneja et al. Modeling dynamic scenes recorded with freely moving cameras
CN113689503B (zh) 目标对象的姿态检测方法、装置、设备及存储介质
WO2021114776A1 (fr) Procédé de détection d'objet, dispositif de détection d'objet, dispositif terminal, et support
US9129375B1 (en) Pose detection
CN110189343A (zh) 图像标注方法、装置及***
TWI684956B (zh) 物體辨識與追蹤系統及其方法
WO2021181988A1 (fr) Système de traitement d'informations, procédé de traitement d'informations, et programme
Chang et al. 3d pose estimation and segmentation using specular cues
CN110009683B (zh) 基于MaskRCNN的实时平面上物体检测方法
Kumara et al. Real-time 3D human objects rendering based on multiple camera details
Xu et al. Kinect-based easy 3d object reconstruction
Donner et al. Awakening history: Preparing a museum tour guide robot for augmenting exhibits
English et al. Development of a practical 3D automatic target recognition and pose estimation algorithm
Liebelt et al. Precise registration of 3D models to images by swarming particles
Riecický et al. Optical-inertial Synchronization of MoCap Suit with Single Camera Setup for Reliable Position Tracking.

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21767552

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21767552

Country of ref document: EP

Kind code of ref document: A1