TWI817875B

TWI817875B - Electronic device and method of generating 3d image based on multi-thread

Info

Publication number: TWI817875B
Application number: TW111149809A
Authority: TW
Inventors: 佑和; 譚馳澔; 林士豪
Original assignee: 宏碁股份有限公司
Priority date: 2022-12-23
Filing date: 2022-12-23
Publication date: 2023-10-01

Abstract

An electronic device and a method of generating 3D image based on multi-thread are provided. The method includes: operating a first thread and a second thread by a processor; transmitting an image frame of 3D image to a first buffer in a storage medium by the first thread, wherein the image frame includes a 3D mesh and a texture corresponding to the 3D mesh; in response to the first buffer completing a reception of the image frame, loading the image frame in the first buffer to perform a depth estimation for the image frame by the second thread so as to generate depth information; updating the 3D mesh according to the depth information by the first thread; and generating a side-by-side image according to the updated 3D mesh and the texture by the first thread.

Description

基於多執行緒產生三維影像的電子裝置和方法Electronic device and method for generating three-dimensional images based on multi-threading

本發明是有關於一種影像處理技術，且特別是有關於一種基於多執行緒產生三維影像的電子裝置和方法。The present invention relates to an image processing technology, and in particular to an electronic device and method for generating three-dimensional images based on multiple threads.

隨著機器學習技術的發展，越來越多顯示器開始使用機器學習技術來處理影像以提供給用戶多樣化的觀看體驗。例如，支援三維影像播放技術的顯示器可利用機器學習技術即時地將二維影像轉換為三維影像。然而，應用基於機器學習的影像處理技術可能需要消耗大量的運算時間。這些運算時間可能導致播放中的輸出影像與影像處理的結果不同步，進而造成輸出影像的品質下降。With the development of machine learning technology, more and more displays are beginning to use machine learning technology to process images to provide users with a diverse viewing experience. For example, displays that support 3D image playback technology can use machine learning technology to instantly convert 2D images into 3D images. However, applying machine learning-based image processing technology may consume a lot of computing time. These calculation times may cause the output image during playback to be out of sync with the image processing results, resulting in a decrease in the quality of the output image.

本發明提供一種基於多執行緒產生三維影像的電子裝置和方法，可最佳化圖形資源（graphic resource）和深度預測模型的讀取。The present invention provides an electronic device and method for generating three-dimensional images based on multi-threading, which can optimize the reading of graphic resources and depth prediction models.

本發明的一種基於多執行緒產生三維影像的電子裝置，包含處理器以及儲存媒體。儲存媒體包含第一緩衝器。處理器耦接儲存媒體並且運行第一執行緒和第二執行緒，其中第一執行緒將三維影像的影像幀傳送至第一緩衝器，其中影像幀包含三維網格以及對應於三維網格的紋理；響應於第一緩衝器完成影像幀的接收，第二執行緒讀取第一緩衝器中的影像幀以對影像幀執行深度預測，進而產生深度資訊；第一執行緒根據深度資訊更新三維網格；以及第一執行緒根據經更新的三維網格以及紋理產生並排影像。The present invention is an electronic device that generates three-dimensional images based on multi-threading, including a processor and a storage medium. The storage medium includes a first buffer. The processor is coupled to the storage medium and runs a first execution thread and a second execution thread, wherein the first execution thread transmits an image frame of the three-dimensional image to the first buffer, wherein the image frame includes a three-dimensional grid and a frame corresponding to the three-dimensional grid. Texture; in response to the first buffer completing the reception of the image frame, the second execution thread reads the image frame in the first buffer to perform depth prediction on the image frame, thereby generating depth information; the first execution thread updates the three-dimensional image frame according to the depth information grid; and the first thread generates side-by-side images based on the updated three-dimensional grid and texture.

在本發明的一實施例中，上述的儲存媒體更包含第二緩衝器，其中第二執行緒將深度資訊傳送至第二緩衝器；以及響應於第二緩衝器完成深度資訊的接收，第一執行緒讀取第二緩衝器中的深度資訊。In an embodiment of the present invention, the above-mentioned storage medium further includes a second buffer, wherein the second thread transmits the depth information to the second buffer; and in response to the second buffer completing the reception of the depth information, the first The thread reads the depth information in the second buffer.

在本發明的一實施例中，其中響應於第一緩衝器完成影像幀的接收，處理器傳送對應於影像幀的識別碼至第二執行緒；以及響應於識別碼，第二執行緒讀取第一緩衝器中的影像幀。In an embodiment of the present invention, in response to the first buffer completing the reception of the image frame, the processor transmits an identification code corresponding to the image frame to the second execution thread; and in response to the identification code, the second execution thread reads Image frames in the first buffer.

在本發明的一實施例中，其中響應於第二緩衝器完成深度資訊的接收，處理器傳送對應於影像幀的識別碼至第一執行緒；以及響應於識別碼，第一執行緒讀取第二緩衝器中的深度資訊。In an embodiment of the present invention, in response to the second buffer completing the reception of the depth information, the processor transmits an identification code corresponding to the image frame to the first execution thread; and in response to the identification code, the first execution thread reads Depth information in the second buffer.

在本發明的一實施例中，上述的電子裝置更包含影像擷取裝置。影像擷取裝置耦接處理器，其中第一執行緒經配置以：通過影像擷取裝置執行眼睛追蹤以取得視野；將視野投影至經更新的三維網格以取得感興趣區域；將紋理映射至感興趣區域以產生輸出幀；以及根據輸出幀產生並排影像。In an embodiment of the present invention, the above-mentioned electronic device further includes an image capturing device. The image capture device is coupled to the processor, wherein the first execution thread is configured to: perform eye tracking through the image capture device to obtain the field of view; project the field of view to the updated three-dimensional grid to obtain the region of interest; map the texture to region of interest to generate an output frame; and generate side-by-side images based on the output frame.

在本發明的一實施例中，上述的第一執行緒將紋理映射至經更新的三維網格以產生並排影像。In an embodiment of the present invention, the above-mentioned first thread maps the texture to the updated three-dimensional mesh to generate side-by-side images.

在本發明的一實施例中，上述的電子裝置更包含收發器。收發器耦接處理器，其中第一執行緒對並排影像執行編織操作以產生編織影像，並且通過收發器輸出編織影像。In an embodiment of the present invention, the above-mentioned electronic device further includes a transceiver. The transceiver is coupled to the processor, wherein the first thread performs a knitting operation on the side-by-side images to generate a knitted image, and outputs the knitted image through the transceiver.

在本發明的一實施例中，上述的第二執行緒將影像幀輸入至機器學習模型以執行深度預測。In an embodiment of the present invention, the above-mentioned second thread inputs image frames to the machine learning model to perform depth prediction.

本發明的一種基於多執行緒產生三維影像的方法，包含：由處理器運行第一執行緒和第二執行緒；由第一執行緒將三維影像的影像幀傳送至儲存媒體中的第一緩衝器，其中影像幀包含三維網格以及對應於三維網格的紋理；響應於第一緩衝器完成影像幀的接收，由第二執行緒讀取第一緩衝器中的影像幀以對影像幀執行深度預測，進而產生深度資訊；由第一執行緒根據深度資訊更新三維網格；以及由第一執行緒根據經更新的三維網格以及紋理產生並排影像。The present invention provides a method for generating a three-dimensional image based on multiple threads, including: a processor running a first thread and a second thread; and the first thread transmitting image frames of the three-dimensional image to a first buffer in a storage medium. The image frame includes a three-dimensional grid and a texture corresponding to the three-dimensional grid; in response to the first buffer completing the reception of the image frame, the second execution thread reads the image frame in the first buffer to execute the image frame depth prediction, thereby generating depth information; the first execution thread updates the three-dimensional grid according to the depth information; and the first execution thread generates side-by-side images according to the updated three-dimensional grid and texture.

基於上述，本發明的電子裝置可在一執行緒通過緩衝器完成影像幀或深度資訊的接收後，觸發另一執行緒基於緩衝器中的完整資料進行後續的影像處理之步驟。據此，本發明可改善緩衝器存取速度以及影像處理流程與產生三維影像流程之間的同步性。Based on the above, the electronic device of the present invention can trigger another thread to perform subsequent image processing steps based on the complete data in the buffer after one thread completes receiving the image frame or depth information through the buffer. Accordingly, the present invention can improve the buffer access speed and the synchronization between the image processing process and the three-dimensional image generation process.

為了使本發明之內容可以被更容易明瞭，以下特舉實施例作為本發明確實能夠據以實施的範例。另外，凡可能之處，在圖式及實施方式中使用相同標號的元件/構件/步驟，係代表相同或類似部件。In order to make the content of the present invention easier to understand, the following embodiments are given as examples according to which the present invention can be implemented. In addition, wherever possible, elements/components/steps with the same reference numbers in the drawings and embodiments represent the same or similar parts.

圖1根據本發明的一實施例繪示電子裝置100的示意圖。電子裝置100可基於多執行緒產生三維影像。電子裝置100可包含處理器110、儲存媒體120以及收發器130。在一實施例中，電子裝置100可進一步包含影像擷取裝置140。FIG. 1 is a schematic diagram of an electronic device 100 according to an embodiment of the invention. The electronic device 100 can generate three-dimensional images based on multiple threads. The electronic device 100 may include a processor 110, a storage medium 120, and a transceiver 130. In an embodiment, the electronic device 100 may further include an image capturing device 140 .

處理器110例如是中央處理單元（central processing unit，CPU），或是其他可程式化之一般用途或特殊用途的微控制單元（micro control unit，MCU）、微處理器（microprocessor）、數位信號處理器（digital signal processor，DSP）、可程式化控制器、特殊應用積體電路（application specific integrated circuit，ASIC）、圖形處理器（graphics processing unit，GPU）、影像訊號處理器（image signal processor，ISP）、影像處理單元（image processing unit，IPU）、算數邏輯單元（arithmetic logic unit，ALU）、複雜可程式邏輯裝置（complex programmable logic device，CPLD）、現場可程式化邏輯閘陣列（field programmable gate array，FPGA）或其他類似元件或上述元件的組合。處理器110可耦接至儲存媒體120、收發器130以及影像擷取裝置140，並且存取和執行儲存於儲存媒體120中的多個模組和各種應用程式。處理器110可具有多執行緒能力。在本實施例中，處理器110可運行執行緒111和執行緒112。The processor 110 is, for example, a central processing unit (CPU), or other programmable general-purpose or special-purpose micro control unit (MCU), microprocessor, or digital signal processing unit. Digital signal processor (DSP), programmable controller, application specific integrated circuit (ASIC), graphics processing unit (GPU), image signal processor (ISP) ), image processing unit (IPU), arithmetic logic unit (ALU), complex programmable logic device (CPLD), field programmable gate array (field programmable gate array) , FPGA) or other similar components or a combination of the above components. The processor 110 can be coupled to the storage medium 120, the transceiver 130 and the image capture device 140, and access and execute multiple modules and various applications stored in the storage medium 120. Processor 110 may have multi-threading capabilities. In this embodiment, the processor 110 can run the thread 111 and the thread 112.

儲存媒體120例如是任何型態的固定式或可移動式的隨機存取記憶體（random access memory，RAM）、唯讀記憶體（read-only memory，ROM）、快閃記憶體（flash memory）、硬碟（hard disk drive，HDD）、固態硬碟（solid state drive，SSD）或類似元件或上述元件的組合，而用於儲存可由處理器110執行的多個模組或各種應用程式。在本實施例中，儲存媒體120可儲存包含緩衝器121、緩衝器122、緩衝器管理器123以及深度預測模型124等多個模組，其功能將於後續說明。The storage medium 120 is, for example, any type of fixed or removable random access memory (RAM), read-only memory (ROM), or flash memory. , hard disk drive (HDD), solid state drive (SSD) or similar components or a combination of the above components, used to store multiple modules or various application programs that can be executed by the processor 110 . In this embodiment, the storage medium 120 can store multiple modules including the buffer 121, the buffer 122, the buffer manager 123, and the depth prediction model 124, the functions of which will be described later.

收發器130以無線或有線的方式傳送及接收訊號。收發器130還可以執行例如低噪聲放大、阻抗匹配、混頻、向上或向下頻率轉換、濾波、放大以及類似的操作。The transceiver 130 transmits and receives signals in a wireless or wired manner. Transceiver 130 may also perform, for example, low noise amplification, impedance matching, mixing, up or down frequency conversion, filtering, amplification, and similar operations.

影像擷取裝置140例如包含互補式金屬氧化物半導體（complementary metal oxide semiconductor，CMOS）感測器或感光耦合元件（charge coupled device，CCD）感測器等影像感測器。影像擷取裝置140可用於擷取影像。The image capturing device 140 includes, for example, an image sensor such as a complementary metal oxide semiconductor (CMOS) sensor or a charge coupled device (CCD) sensor. The image capture device 140 can be used to capture images.

圖2根據本發明的一實施例繪示一種基於多執行緒產生三維影像的方法的示意圖，其中所述方法可由如圖1所示的電子裝置100實施。方法可包含由執行緒111執行的任務200以及由執行緒112執行的任務300。FIG. 2 is a schematic diagram of a method for generating three-dimensional images based on multiple threads according to an embodiment of the present invention, where the method can be implemented by the electronic device 100 shown in FIG. 1 . The method may include task 200 performed by thread 111 and task 300 performed by thread 112 .

在步驟S210中，執行緒111可取得三維影像的影像幀。具體來說，三維影像可包含多個影像幀，且每一個影像幀可包含三維網格（3D mesh）和對應於三維網格的紋理（texture）。執行緒111例如可通過收發器130以自外部裝置接收三維影像的影像幀。影像幀的取得可觸發執行緒111開始執行任務200。In step S210, the execution thread 111 may obtain the image frame of the three-dimensional image. Specifically, the three-dimensional image may include multiple image frames, and each image frame may include a three-dimensional mesh (3D mesh) and a texture corresponding to the three-dimensional mesh. The execution thread 111 may, for example, receive image frames of three-dimensional images from an external device through the transceiver 130 . The acquisition of the image frame may trigger the execution thread 111 to start executing the task 200 .

在步驟S220中，若必要，執行緒111可對影像幀執行尺寸調整（resizing）。舉例來說，假設電子裝置100所要產生的輸出影像（或輸出幀）的解析度為1920x1080。執行緒111可根據輸出影像的解析度將影像幀的解析度調整為與輸出影像相同，亦即，將解析度調整為1920x1080。In step S220, if necessary, the thread 111 may perform resizing on the image frame. For example, assume that the resolution of the output image (or output frame) to be generated by the electronic device 100 is 1920x1080. The execution thread 111 may adjust the resolution of the image frame to be the same as the output image according to the resolution of the output image, that is, adjust the resolution to 1920x1080.

在步驟S230中，執行緒111可將影像幀傳送至緩衝器121儲存。緩衝器121可在影像幀的接收完成後，傳送確認訊息至緩衝器管理器123。確認訊息可包含但不限於對應於影像幀的識別碼。舉例來說，確認訊息可包含對應於緩衝器121所接收的影像幀之紋理的紋理識別碼（texture identifier）。In step S230, the execution thread 111 may transmit the image frame to the buffer 121 for storage. The buffer 121 may send a confirmation message to the buffer manager 123 after the reception of the image frame is completed. The confirmation message may include but is not limited to an identification code corresponding to the image frame. For example, the confirmation message may include a texture identifier corresponding to the texture of the image frame received by the buffer 121 .

在步驟S310中，緩衝器管理器123可傳送確認訊息至執行緒112。執行緒112可根據確認訊息讀取儲存在緩衝器121中的影像幀。確認訊息可包含但不限於對應於影像幀的識別碼。舉例來說，確認訊息可包含對應於緩衝器121所接收的影像幀之紋理的紋理識別碼。執行緒112可根據紋理識別碼從緩衝器121中讀取對應於紋理識別碼的影像幀。In step S310, the buffer manager 123 may send a confirmation message to the thread 112. The execution thread 112 can read the image frame stored in the buffer 121 according to the confirmation message. The confirmation message may include but is not limited to an identification code corresponding to the image frame. For example, the acknowledgment message may include a texture identification code corresponding to the texture of the image frame received by the buffer 121 . The execution thread 112 may read the image frame corresponding to the texture identification code from the buffer 121 according to the texture identification code.

在步驟S320中，執行緒112可對影像幀執行預處理。舉例來說，處理器110可調整影像幀的尺寸，以使影像幀符合深度預測模型124的輸入格式。In step S320, the thread 112 may perform preprocessing on the image frame. For example, the processor 110 may adjust the size of the image frame so that the image frame conforms to the input format of the depth prediction model 124 .

在完成影像幀的預處理後，在步驟S330中，執行緒112可對影像幀執行深度預測（depth estimation）以產生深度資訊。具體來說，執行緒112可將影像幀輸入至深度預測模型124，以由深度預測模型124輸出對應於影像幀的深度資訊。深度預測模型124可包含但不限於監督式機器學習模型或非監督式機器學習模型。After completing the pre-processing of the image frame, in step S330, the execution thread 112 may perform depth estimation (depth estimation) on the image frame to generate depth information. Specifically, the execution thread 112 may input the image frame to the depth prediction model 124, so that the depth prediction model 124 outputs depth information corresponding to the image frame. The deep prediction model 124 may include, but is not limited to, a supervised machine learning model or an unsupervised machine learning model.

在步驟S340中，執行緒112可將深度資訊傳送至緩衝器122儲存。緩衝器122可在深度資訊的接收完成後，傳送確認訊息至緩衝器管理器123。確認訊息可包含但不限於對應於影像幀的識別碼。舉例來說，確認訊息可包含對應於緩衝器121所接收的影像幀之紋理的紋理識別碼。In step S340, the execution thread 112 may transmit the depth information to the buffer 122 for storage. The buffer 122 may send a confirmation message to the buffer manager 123 after completion of receiving the depth information. The confirmation message may include but is not limited to an identification code corresponding to the image frame. For example, the acknowledgment message may include a texture identification code corresponding to the texture of the image frame received by the buffer 121 .

在步驟S240中，緩衝器管理器123可傳送確認訊息至執行緒111。確認訊息可包含但不限於對應於影像幀的識別碼（例如：紋理識別碼）。確認訊息用以通知執行緒111緩衝器121與緩衝器122的資料接收已經完成。In step S240, the buffer manager 123 may send a confirmation message to the thread 111. The confirmation message may include but is not limited to an identification code corresponding to the image frame (for example, a texture identification code). The acknowledgment message is used to notify the execution thread 111 that the data reception of the buffer 121 and the buffer 122 has been completed.

在步驟S250中，執行緒111可根據深度資訊更新影像幀的三維網格，進而根據經更新的三維網格以及紋理產生並排（side-by-side，SBS）影像。具體來說，步驟S250可包含步驟S251至步驟S254。在步驟S251中，執行緒111可根據確認訊息讀取儲存在緩衝器122中的深度資訊，並且根據深度資訊更新三維網格。In step S250 , the execution thread 111 may update the three-dimensional grid of the image frame according to the depth information, and then generate a side-by-side (SBS) image based on the updated three-dimensional grid and texture. Specifically, step S250 may include steps S251 to S254. In step S251, the execution thread 111 may read the depth information stored in the buffer 122 according to the confirmation message, and update the three-dimensional grid according to the depth information.

在步驟S252中，執行緒111可根據經更新的三維網格產生感興趣區域（region of interest，ROI）。感興趣區域可與正在觀看三維影像的觀眾的視野（field of view，FOV）有關。具體來說，在步驟S410中，執行緒111可通過影像擷取裝置140執行眼睛追蹤（eye tracking）以取得觀眾的視野。在步驟S252中，執行緒111可將視野投影至經更新的三維網格以取得感興趣區域。換句話說，影像幀的感興趣區域可隨著觀眾的視野變化而變化。In step S252, the execution thread 111 may generate a region of interest (ROI) according to the updated three-dimensional grid. The region of interest may be related to the field of view (FOV) of the viewer who is viewing the three-dimensional image. Specifically, in step S410 , the execution thread 111 may perform eye tracking through the image capture device 140 to obtain the viewer's field of view. In step S252, the thread 111 may project the view to the updated three-dimensional grid to obtain the region of interest. In other words, the region of interest of the image frame may change as the viewer's field of view changes.

在步驟S253中，執行緒111可將與三維網格相對應的紋理映射至與三維網格相對應的感興趣區域，以產生輸出幀。輸出幀即為影像幀經過深度資訊改善後產生的結果。在步驟S254中，執行緒111可根據輸出幀產生並排影像。In step S253, the thread 111 may map the texture corresponding to the three-dimensional grid to the region of interest corresponding to the three-dimensional grid to generate an output frame. The output frame is the result of the image frame being improved by depth information. In step S254, the execution thread 111 may generate side-by-side images according to the output frames.

在步驟S260中，執行緒111可對並排影像執行編織（weaving）操作以產生編織影像，並可通過收發器130輸出編織影像。舉例來說，執行緒111可通過收發器130將編織影像傳送至支援三維影像的顯示器，以通過顯示器來播放編織影像。In step S260 , the execution thread 111 may perform a weaving operation on the side-by-side images to generate a weaving image, and may output the weaving image through the transceiver 130 . For example, the execution thread 111 can transmit the knitting image to a display that supports three-dimensional images through the transceiver 130, so that the knitting image can be played through the display.

圖3根據本發明的一實施例繪示一種基於多執行緒產生三維影像的方法的流程圖，其中所述方法可由如圖1所示的電子裝置100實施。在步驟S510中，由處理器運行第一執行緒和第二執行緒。在步驟S520中，由第一執行緒將三維影像的影像幀傳送至儲存媒體中的第一緩衝器，其中影像幀包含三維網格以及對應於三維網格的紋理。在步驟S530中，響應於第一緩衝器完成影像幀的接收，由第二執行緒讀取第一緩衝器中的影像幀以對影像幀執行深度預測，進而產生深度資訊。在步驟S540中，由第一執行緒根據深度資訊更新三維網格。在步驟S550中，由第一執行緒根據經更新的三維網格以及紋理產生並排影像。FIG. 3 illustrates a flowchart of a method for generating three-dimensional images based on multiple threads according to an embodiment of the present invention, wherein the method can be implemented by the electronic device 100 shown in FIG. 1 . In step S510, the first execution thread and the second execution thread are executed by the processor. In step S520, the first thread transmits the image frame of the three-dimensional image to the first buffer in the storage medium, where the image frame includes the three-dimensional grid and the texture corresponding to the three-dimensional grid. In step S530, in response to the first buffer completing the reception of the image frame, the second thread reads the image frame in the first buffer to perform depth prediction on the image frame, thereby generating depth information. In step S540, the first execution thread updates the three-dimensional grid according to the depth information. In step S550, the first execution thread generates side-by-side images according to the updated three-dimensional mesh and texture.

綜上所述，本發明的電子裝置可利用多執行緒讀取圖形資源、讀取影像處理模型以及產生三維影像。在一執行緒通過緩衝器完成影像幀或深度資訊的接收後，緩衝器管理器可觸發另一執行緒基於緩衝器中的完整資料進行後續的影像處理之步驟。據此，本發明可改善緩衝器存取速度以及影像處理流程與產生三維影像流程之間的同步性。本發明還可最佳化圖形資源和影像處理模型的讀取。In summary, the electronic device of the present invention can utilize multiple threads to read graphics resources, read image processing models, and generate three-dimensional images. After one thread completes receiving the image frame or depth information through the buffer, the buffer manager can trigger another thread to perform subsequent image processing steps based on the complete data in the buffer. Accordingly, the present invention can improve the buffer access speed and the synchronization between the image processing process and the three-dimensional image generation process. The present invention can also optimize the reading of graphics resources and image processing models.

100:電子裝置100: Electronic devices

110:處理器110: Processor

111、112:執行緒111, 112: Execution thread

120:儲存媒體120:Storage media

121、122:緩衝器121, 122: Buffer

123:緩衝器管理器123:Buffer Manager

124:深度預測模型124: Depth prediction model

130:收發器130:Transceiver

140:影像擷取裝置140:Image capture device

200、300:任務200, 300: Mission

S210、S220、S230、S240、S250、S251、S252、S253、S254、S260、S310、S320、S330、S340、S410、S510、S520、S530、S540、S550:步驟S210, S220, S230, S240, S250, S251, S252, S253, S254, S260, S310, S320, S330, S340, S410, S510, S520, S530, S540, S550: Steps

圖1根據本發明的一實施例繪示電子裝置的示意圖。圖2根據本發明的一實施例繪示一種基於多執行緒產生三維影像的方法的示意圖。圖3根據本發明的一實施例繪示一種基於多執行緒產生三維影像的方法的流程圖。 FIG. 1 is a schematic diagram of an electronic device according to an embodiment of the invention. FIG. 2 illustrates a schematic diagram of a method for generating three-dimensional images based on multiple threads according to an embodiment of the present invention. FIG. 3 illustrates a flowchart of a method for generating three-dimensional images based on multiple threads according to an embodiment of the present invention.

S510、S520、S530、S540、S550:步驟 S510, S520, S530, S540, S550: Steps

Claims

一種基於多執行緒產生三維影像的電子裝置，包括：儲存媒體，包括第一緩衝器；以及處理器，耦接所述儲存媒體並且運行第一執行緒和第二執行緒，其中所述第一執行緒將三維影像的影像幀傳送至所述第一緩衝器，其中所述影像幀包括三維網格以及對應於所述三維網格的紋理；響應於所述第一緩衝器完成所述影像幀的接收，所述第二執行緒讀取所述第一緩衝器中的所述影像幀並將所述影像幀輸入至機器學習模型以對所述影像幀執行深度預測，進而產生深度資訊；所述第一執行緒根據所述深度資訊更新所述三維網格；以及所述第一執行緒根據經更新的所述三維網格以及所述紋理產生並排影像。 An electronic device that generates three-dimensional images based on multiple threads, including: a storage medium, including a first buffer; and a processor, coupled to the storage medium and running a first execution thread and a second execution thread, wherein the first execution thread The execution thread transmits an image frame of a three-dimensional image to the first buffer, wherein the image frame includes a three-dimensional grid and a texture corresponding to the three-dimensional grid; in response to the first buffer completing the image frame The second thread reads the image frame in the first buffer and inputs the image frame to the machine learning model to perform depth prediction on the image frame, thereby generating depth information; The first execution thread updates the three-dimensional mesh according to the depth information; and the first execution thread generates side-by-side images according to the updated three-dimensional mesh and the texture.

如請求項1所述的電子裝置，其中所述儲存媒體更包括第二緩衝器，其中所述第二執行緒將所述深度資訊傳送至所述第二緩衝器；以及響應於所述第二緩衝器完成所述深度資訊的接收，所述第一執行緒讀取所述第二緩衝器中的所述深度資訊。 The electronic device of claim 1, wherein the storage medium further includes a second buffer, wherein the second thread transmits the depth information to the second buffer; and in response to the second The buffer completes receiving the depth information, and the first thread reads the depth information in the second buffer.

如請求項1所述的電子裝置，其中響應於所述第一緩衝器完成所述影像幀的所述接收，所述處理器傳送對應於所述影像幀的識別碼至所述第二執行緒；以及響應於所述識別碼，所述第二執行緒讀取所述第一緩衝器中的所述影像幀。 The electronic device of claim 1, wherein in response to the first buffer completing the reception of the image frame, the processor transmits an identification code corresponding to the image frame to the second execution thread. ;as well as In response to the identification code, the second thread reads the image frame in the first buffer.

如請求項2所述的電子裝置，其中響應於所述第二緩衝器完成所述深度資訊的所述接收，所述處理器傳送對應於所述影像幀的識別碼至所述第一執行緒；以及響應於所述識別碼，所述第一執行緒讀取所述第二緩衝器中的所述深度資訊。 The electronic device of claim 2, wherein in response to the second buffer completing the reception of the depth information, the processor transmits an identification code corresponding to the image frame to the first execution thread. ; And in response to the identification code, the first thread reads the depth information in the second buffer.

如請求項1所述的電子裝置，更包括：影像擷取裝置，耦接所述處理器，其中所述第一執行緒經配置以：通過所述影像擷取裝置執行眼睛追蹤以取得視野；將所述視野投影至經更新的所述三維網格以取得感興趣區域；將所述紋理映射至所述感興趣區域以產生輸出幀；以及根據所述輸出幀產生所述並排影像。 The electronic device of claim 1, further comprising: an image capture device coupled to the processor, wherein the first execution thread is configured to: perform eye tracking through the image capture device to obtain the field of view; Project the field of view to the updated three-dimensional grid to obtain a region of interest; map the texture to the region of interest to generate an output frame; and generate the side-by-side image based on the output frame.

如請求項1所述的電子裝置，其中所述第一執行緒將所述紋理映射至經更新的所述三維網格以產生所述並排影像。 The electronic device of claim 1, wherein the first thread maps the texture to the updated three-dimensional grid to generate the side-by-side image.

如請求項1所述的電子裝置，更包括：收發器，耦接所述處理器，其中所述第一執行緒對所述並排影像執行編織操作以產生編織影像，並且通過所述收發器輸出所述編織影像。 The electronic device of claim 1, further comprising: a transceiver coupled to the processor, wherein the first thread performs a knitting operation on the side-by-side images to generate a knitted image, and outputs it through the transceiver The weaving image.

一種基於多執行緒產生三維影像的方法，包括：由處理器運行第一執行緒和第二執行緒；由所述第一執行緒將三維影像的影像幀傳送至儲存媒體中的第一緩衝器，其中所述影像幀包括三維網格以及對應於所述三維網格的紋理；響應於所述第一緩衝器完成所述影像幀的接收，由所述第二執行緒讀取所述第一緩衝器中的所述影像幀並將所述影像幀輸入至機器學習模型以對所述影像幀執行深度預測，進而產生深度資訊；由所述第一執行緒根據所述深度資訊更新所述三維網格；以及由所述第一執行緒根據經更新的所述三維網格以及所述紋理產生並排影像。 A method for generating three-dimensional images based on multi-threading, including: The processor runs a first thread and a second thread; the first thread transmits an image frame of the three-dimensional image to a first buffer in the storage medium, wherein the image frame includes a three-dimensional grid and a corresponding The texture of the three-dimensional grid; in response to the first buffer completing the reception of the image frame, the second thread reads the image frame in the first buffer and transfers the image Frames are input to a machine learning model to perform depth prediction on the image frames to generate depth information; the first execution thread updates the three-dimensional grid according to the depth information; and the first execution thread updates the three-dimensional grid according to the depth information. The updated three-dimensional mesh and texture produce side-by-side images.