WO2023120770A1 - Procédé et appareil d'interaction entre des informations de maillage cognitif générées dans un espace tridimensionnel et des objets virtuels - Google Patents

Procédé et appareil d'interaction entre des informations de maillage cognitif générées dans un espace tridimensionnel et des objets virtuels Download PDF

Info

Publication number
WO2023120770A1
WO2023120770A1 PCT/KR2021/019669 KR2021019669W WO2023120770A1 WO 2023120770 A1 WO2023120770 A1 WO 2023120770A1 KR 2021019669 W KR2021019669 W KR 2021019669W WO 2023120770 A1 WO2023120770 A1 WO 2023120770A1
Authority
WO
WIPO (PCT)
Prior art keywords
information
cognitive
virtual object
space
virtual objects
Prior art date
Application number
PCT/KR2021/019669
Other languages
English (en)
Korean (ko)
Inventor
한상준
Original Assignee
엔센스코리아주식회사
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 엔센스코리아주식회사 filed Critical 엔센스코리아주식회사
Publication of WO2023120770A1 publication Critical patent/WO2023120770A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G06T19/20—Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G06T17/20—Finite element generation, e.g. wire-frame surface description, tesselation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G06T17/20—Finite element generation, e.g. wire-frame surface description, tesselation
    • G06T17/205—Re-meshing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G06T19/003—Navigation within 3D models or images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
    • G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods

Definitions

  • augmented reality technology is a technology field derived from virtual reality technology that synthesizes and overlaps virtual objects on a real space and shows them. It can increase the sense of presence by creating an illusion as if the virtual object actually exists in the real space.
  • a 3D point cloud map is created from a depth image obtained using a depth camera, and an object in a real space where augmented content is to be projected is tracked with the 3D point cloud map, and a display device such as a projector is used to create a virtual image in real space.
  • a display device such as a projector is used to create a virtual image in real space.
  • the position value of a target object in virtual space is calculated from a depth image obtained using a depth camera to obtain a reference position database and There is a way to compare and generate an event execution signal.
  • the fragmentary mesh information generated through the 3D point cloud map obtained in this way does not separate the space and the object, and the area corresponding to the space There is a limit to obtaining cognitive information, such as unknown properties.
  • the above limitation is to provide a user experience with high presence because interaction with real space is ignored in creating content or scenarios in implementing augmented reality, or arbitrary objects can be expressed only according to a pre-planned script. There are significant problems.
  • all of the above examples must be accompanied by a device capable of acquiring depth such as a stereo camera or a depth camera to generate spatial information in a 3D real space. Therefore, there is a problem in that the technology such as the examples described above cannot be implemented in a device that does not have a built-in camera capable of obtaining a depth of field.
  • the present invention implements an augmented reality technology that synthesizes virtual objects in an arbitrary space, and first generates 3D mesh information of the real space for interaction between the virtual object and the real space, and through cognitive domain segmentation.
  • the obtained area information is projected onto a 3D space and used again for area division of the 3D mesh information to generate cognitive mesh information, and then virtual objects use the cognitive mesh information to apply physical characteristics to the location of the object.
  • a real space was recognized based on depth information acquired through an infrared ToF camera, an RGB-D depth camera, a stereo camera, etc. and create depth information by estimating the distance to an object such as a real space or object using the positional relationship between adjacent images, and at this time, among the feature points extracted from each image, the distance to the camera is correctly calculated.
  • We analyze the space by implementing the SLAM (Simultaneous Localization and Mapping) method, which collects the feature points that are determined to be estimated and configures them into a 3D point cloud map.
  • SLAM Simultaneous Localization and Mapping
  • the present invention performs semantic segmentation on the analyzed spatial information and the two-dimensional image at the same acquisition time using a region segmentation method using a convolutional neural network, and as a result, 2 Dimensional cognitive region segmentation information is projected onto the analyzed space, and the meaning of the region is given to cognitive mesh information from the analyzed spatial information.
  • Cognitive mesh information is used to calculate anchor points in 3D space so that virtual objects can more naturally exist in real space when arranging virtual objects in real space. It helps to render a virtual object in a realistic way in a situation where some or all of its shape should not be visible because it is hidden in space.
  • the virtual object provides additional information so that the real space and the virtual objects can interact with each other using cognitive mesh information. For example, if a table in a real space is recognized through cognitive domain segmentation, the virtual object should be able to express its position by fixing it as if it were real on top of the table. Alternatively, if a potted plant in a real space is recognized, the virtual object should be displayed naturally in harmony with the planter's surroundings, and it should be possible to express that the virtual object is hidden behind the pot and partially or entirely invisible.
  • the depth of the real space is extracted, a 3D point cloud map is created, mesh information is created from the generated spatial information, and 2
  • cognitive domain segmentation of a dimensional image, 3D projection and combining for domain division of the mesh information cognitive mesh information divided according to objects or areas is generated, and the gravity direction acquired through the IMU sensor is combined.
  • the present invention relates to an electronic recording medium capable of mounting a camera, an IMU sensor, a display, and an algorithm, which is a method for realizing the augmented reality, in order for virtual objects to interact with each other in a real space, and a method for executing the same. It consists of a device composed of processors.
  • the device is equipped with an algorithm that can implement augmented reality, which includes image acquisition processing, spatial information generation processing, mesh information generation processing, cognitive area division processing, and cognitive mesh information generation processing. generating cognitive mesh information of the real space from the configured image analyzer;
  • An object expression unit that processes a virtual object creation command through a user's input or a pre-designed scenario, calculates the location of a virtual object, and synthesizes the virtual object by combining cognitive mesh information on the acquired image. It is characterized by being composed of composed object expression steps.
  • the present invention as described above is a method for enhancing a sense of presence by creating an illusion that a virtual object physically exists in a real space to a user, in terms of providing augmented reality content as virtual objects can interact with each other in a real space. It has become possible to construct immersive scenarios.
  • a virtual wall clock is hung on a real wall in a certain space, or a virtual chair is placed around a real table, and the virtual chair is hidden by the real table according to the position where it is placed. It can be implemented with easy operation according to the user's purpose even if it is not elaborately designed in advance to make some or all invisible, so it can be used as augmented reality technology in various fields such as construction, architecture, interior, 3D product design, and games. can
  • the application of the present invention is not limited to the above-described utilization examples, and the present invention can be applied to fields other than the above embodiments without changing the essential content of the present invention.
  • FIG. 1 is a flowchart showing the structure of a device composed of a camera, an IMU sensor, and a display constituting the present invention, and each sub-algorithm for implementing the method of the present invention.
  • Figure 2 is a flow chart describing in detail the processing of the image analysis unit.
  • Figure 3 is a flow chart describing in detail the processing process of the object expression unit.
  • FIG. 4 is a structural diagram detailing a convolutional neural network that performs cognitive domain segmentation.
  • 1 is an electronic recording medium capable of storing a camera D001 capable of obtaining a two-dimensional color image, an image analysis unit S100 composed of an algorithm for realizing augmented reality, and an object representation unit S200. And a processor that can execute it, an IMU sensor (D002) for adding physical environment information about acceleration and attitude to cognitive mesh information, and a video that can reproduce a synthesized image of a virtual object on a two-dimensional image of a real space It is a flow chart of a device with one display (D003) and its method.
  • the image analysis unit (S100) of the above flow chart acquires a two-dimensional image at regular intervals through the image acquisition process (S110) for obtaining an image from the camera (D001), and converts it into an acquired image (R001).
  • the acquired image is divided into the two-dimensional image acquired just before and the two-dimensional image currently acquired and stored, and an algorithm that can find feature points of SIFT, SURF, ORB or similar images from these two images, respectively, and create descriptors of the feature points.
  • Extracts feature information using compares the similarity between feature information between two images with different viewpoints, obtains a pair of closest feature points, and uses the geometric relationship between paired feature points to obtain information from the t-1 frame Get the relative position of the t frame.
  • spatial information generation processing of registering continuous frames at regular intervals having a sufficiently large number of feature point pairs as key frames and generating a 3D point cloud map as a cluster of feature points obtained from 3D location information from geometric relationships (S120). ) step.
  • the 3D point cloud map obtains the Euclidean distance of each feature point in the 3D space, and for all feature points, at least two or more nearest pairs among other feature points other than the feature point are obtained, and the three paired feature points are connected into a triangle.
  • a mesh information generating process (S130) for generating mesh information made of connections is executed.
  • the 2D image obtained in the image acquisition process (S110) is subjected to a cognitive domain segmentation process (S140) using the convolutional neural network configured as shown in FIG. 4 to generate 2D cognitive domain segmentation information.
  • the three-dimensional coordinates of each corner of the triangle on the information side are added to the mesh information to divide the real space so that it can be expressed in detail. Make the area have meaning about what kind of object it represents, obtain information about the device's posture and the acceleration of the device's movement obtained from the IMU sensor (D002), and calculate the direction in which gravity acts at this time, A cognitive mesh information generation process (S150) is executed, which is added to the cognitive mesh information (R002).
  • the object expression unit (S200) the user touches an arbitrary position of the display (D003), uses a scenario for object appearance information using a timer or clock, or conditions for interacting with a virtual object , or not limited to the method described in the present invention, by receiving a virtual object creation command by a method of instructing object creation, whether or not a virtual object appears in a temporary storage device, location information, direction information, and state information are created.
  • the virtual object position calculation process (S220) for updating the location information, direction information or state information is executed, and the physical effect or masking of the object is performed to make the virtual object feel as if it is real in the real space by using the cognitive mesh information (R002).
  • Virtual object synthesis processing (S230) is performed to synthesize the scene of interaction with the acquired image (R001) and render it as if a virtual object exists in a real space, without being limited to the method described in the present invention. This is output to the display (D003).

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Software Systems (AREA)
  • Computer Graphics (AREA)
  • Computer Hardware Design (AREA)
  • General Engineering & Computer Science (AREA)
  • Architecture (AREA)
  • Geometry (AREA)
  • Remote Sensing (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Processing Or Creating Images (AREA)

Abstract

Afin de mettre en œuvre des techniques de réalité augmentée qui synthétisent des objets virtuels dans un certain espace, la présente invention concerne un procédé qui crée des éléments d'informations de maillage tridimensionnel segmenté de manière cognitive pour une interaction d'un objet virtuel et met à jour des informations sur une position, une direction et un état de l'objet virtuel de sorte que l'objet virtuel interagit avec un espace réel, ce qui améliore le sens de la présence et de la manipulabilité pour un utilisateur. Plus spécifiquement, des informations d'espace tridimensionnel sont générées en estimant une relation géométrique à l'aide d'images continues obtenues par une caméra, et une segmentation de domaine cognitif est réalisée sur les images de l'espace courant à l'aide d'un réseau neuronal à convolutions, et cela est combiné avec les informations d'espace pour générer des informations de maillage cognitif segmenté. La présente invention concerne un procédé et un appareil pour mettre en œuvre une interaction entre des informations de maillage cognitif générées dans un espace tridimensionnel et des objets virtuels par une mise à jour continuelle d'informations d'emplacement, d'informations de direction et d'informations d'état dans lesquelles les objets virtuels disposés dans l'espace tridimensionnel peuvent physiquement exister, le fait de déterminer si les objets virtuels peuvent réellement exister à l'emplacement courant à l'aide des informations de maillage cognitif pour amener les objets virtuels et les informations de maillage cognitif à interagir les uns avec les autres, et la comparaison de la relation de position entre les objets virtuels et les informations de maillage tridimensionnel pour représenter une occlusion.
PCT/KR2021/019669 2021-12-22 2021-12-22 Procédé et appareil d'interaction entre des informations de maillage cognitif générées dans un espace tridimensionnel et des objets virtuels WO2023120770A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
KR1020210184420A KR20230095197A (ko) 2021-12-22 2021-12-22 3차원 공간에서 생성한 인지적 매쉬 정보와 가상 객체들간의 상호작용 방법 및 그 장치
KR10-2021-0184420 2021-12-22

Publications (1)

Publication Number Publication Date
WO2023120770A1 true WO2023120770A1 (fr) 2023-06-29

Family

ID=86903015

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/KR2021/019669 WO2023120770A1 (fr) 2021-12-22 2021-12-22 Procédé et appareil d'interaction entre des informations de maillage cognitif générées dans un espace tridimensionnel et des objets virtuels

Country Status (2)

Country Link
KR (1) KR20230095197A (fr)
WO (1) WO2023120770A1 (fr)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117475117B (zh) * 2023-12-28 2024-03-08 广州市大湾区虚拟现实研究院 一种基于卷积神经网络的非刚性物体虚实遮挡方法和***

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20170081964A (ko) * 2016-01-05 2017-07-13 한국전자통신연구원 공간 구조 인식을 통한 증강 현실 장치 및 그 방법
KR20200130903A (ko) * 2019-05-13 2020-11-23 주식회사 케이티 증강 현실 공간과 가상 현실 공간을 연동하는 서비스를 제공하는 서버, 방법 및 사용자 단말
KR20210036212A (ko) * 2019-09-25 2021-04-02 주식회사 케이티 증강 현실 서비스를 제공하는 서버, 사용자 단말 및 방법
KR102333682B1 (ko) * 2020-02-21 2021-11-30 전남대학교산학협력단 3차원 공간의 의미적 분할 시스템 및 이를 이용한 3차원 공간의 의미적 분할 방법

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20090009651A (ko) 2007-07-20 2009-01-23 삼성에스디아이 주식회사 포트를 통한 양방향 전력 전달이 가능한 전자기기 및 그동작 방법
KR20180130753A (ko) 2017-05-30 2018-12-10 큐디브릭 주식회사 양자점이 분산된 박막의 제조방법

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20170081964A (ko) * 2016-01-05 2017-07-13 한국전자통신연구원 공간 구조 인식을 통한 증강 현실 장치 및 그 방법
KR20200130903A (ko) * 2019-05-13 2020-11-23 주식회사 케이티 증강 현실 공간과 가상 현실 공간을 연동하는 서비스를 제공하는 서버, 방법 및 사용자 단말
KR20210036212A (ko) * 2019-09-25 2021-04-02 주식회사 케이티 증강 현실 서비스를 제공하는 서버, 사용자 단말 및 방법
KR102333682B1 (ko) * 2020-02-21 2021-11-30 전남대학교산학협력단 3차원 공간의 의미적 분할 시스템 및 이를 이용한 3차원 공간의 의미적 분할 방법

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
FELIX JÄREMO LAWIN, DANELLJAN MARTIN, TOSTEBERG PATRIK, BHAT GOUTAM, FAHAD SHAHBAZ KHAN, FELSBERG MICHAEL: "Deep Projective 3D Semantic Segmentation", ARXIV.ORG, CORNELL UNIVERSITY LIBRARY, ARXIV.ORG, ITHACA, 9 May 2017 (2017-05-09), Ithaca, XP093073947, Retrieved from the Internet <URL:https://arxiv.org/abs/1705.03428v1> [retrieved on 20230816] *

Also Published As

Publication number Publication date
KR20230095197A (ko) 2023-06-29

Similar Documents

Publication Publication Date Title
CN105981076B (zh) 合成增强现实环境的构造
US8933931B2 (en) Distributed asynchronous localization and mapping for augmented reality
Vallino Interactive augmented reality
TWI505709B (zh) 擴增實境場景中決定個體化深度資訊的系統和方法
JP7448566B2 (ja) クロスリアリティシステムにおけるスケーラブル３次元オブジェクト認識
EP2981079A1 (fr) Dispositif et méthode de traitement d&#39;image, et programme
KR101723823B1 (ko) 인터랙티브 공간증강 체험전시를 위한 동적 객체와 가상 객체 간의 인터랙션 구현 장치
CN108028871A (zh) 移动设备上的无标记的多用户多对象增强现实
Jimeno-Morenilla et al. Augmented and virtual reality techniques for footwear
US20160148429A1 (en) Depth and Chroma Information Based Coalescence of Real World and Virtual World Images
KR20170086077A (ko) 증강 현실 장면에서의 드로잉을 위한 깊이 정보의 사용
CN115562499B (zh) 基于智能指环的精准交互控制方法、***及存储介质
JP7150894B2 (ja) Ａｒシーン画像処理方法及び装置、電子機器並びに記憶媒体
KR101723828B1 (ko) 공간 확장이 가능한 공간증강 인터랙티브 엔진이 적용된 체험큐브 시스템
WO2023120770A1 (fr) Procédé et appareil d&#39;interaction entre des informations de maillage cognitif générées dans un espace tridimensionnel et des objets virtuels
CN116097316A (zh) 用于非模态中心预测的对象识别神经网络
KR102148103B1 (ko) 스테레오 카메라를 장착한 드론을 이용한 혼합현실 환경 생성 방법 및 장치
KR102521221B1 (ko) 단말의 싱글 카메라를 이용한 혼합 현실 제작 방법, 장치 및 컴퓨터 프로그램
Park et al. Strategy for Creating AR Applications in Static and Dynamic Environments Using SLAM-andMarker Detector-Based Tracking.
Park et al. AR room: Real-time framework of camera location and interaction for augmented reality services
CN116863107A (zh) 增强现实提供方法、装置以及非暂时性计算机可读介质
Díaz et al. Augmented reality without fiducial markers
Davies et al. Stereoscopic human detection in a natural environment
Treskunov et al. Range Camera for Simple behind Display Interaction.
Liu et al. Human performance capture using multiple handheld kinects

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21969135

Country of ref document: EP

Kind code of ref document: A1