TWI817875B - Electronic device and method of generating 3d image based on multi-thread - Google Patents
Electronic device and method of generating 3d image based on multi-thread Download PDFInfo
- Publication number
- TWI817875B TWI817875B TW111149809A TW111149809A TWI817875B TW I817875 B TWI817875 B TW I817875B TW 111149809 A TW111149809 A TW 111149809A TW 111149809 A TW111149809 A TW 111149809A TW I817875 B TWI817875 B TW I817875B
- Authority
- TW
- Taiwan
- Prior art keywords
- thread
- buffer
- image
- image frame
- depth information
- Prior art date
Links
- 238000000034 method Methods 0.000 title claims abstract description 19
- 230000004044 response Effects 0.000 claims abstract description 16
- 238000010801 machine learning Methods 0.000 claims description 9
- 238000009940 knitting Methods 0.000 claims description 4
- 238000009941 weaving Methods 0.000 claims description 4
- 238000012790 confirmation Methods 0.000 description 11
- 238000010586 diagram Methods 0.000 description 4
- 230000003321 amplification Effects 0.000 description 2
- 238000003199 nucleic acid amplification method Methods 0.000 description 2
- 238000007781 pre-processing Methods 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 229910044991 metal oxide Inorganic materials 0.000 description 1
- 150000004706 metal oxides Chemical class 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
Images
Landscapes
- Ultra Sonic Daignosis Equipment (AREA)
- Stereoscopic And Panoramic Photography (AREA)
Abstract
Description
本發明是有關於一種影像處理技術,且特別是有關於一種基於多執行緒產生三維影像的電子裝置和方法。The present invention relates to an image processing technology, and in particular to an electronic device and method for generating three-dimensional images based on multiple threads.
隨著機器學習技術的發展,越來越多顯示器開始使用機器學習技術來處理影像以提供給用戶多樣化的觀看體驗。例如,支援三維影像播放技術的顯示器可利用機器學習技術即時地將二維影像轉換為三維影像。然而,應用基於機器學習的影像處理技術可能需要消耗大量的運算時間。這些運算時間可能導致播放中的輸出影像與影像處理的結果不同步,進而造成輸出影像的品質下降。With the development of machine learning technology, more and more displays are beginning to use machine learning technology to process images to provide users with a diverse viewing experience. For example, displays that support 3D image playback technology can use machine learning technology to instantly convert 2D images into 3D images. However, applying machine learning-based image processing technology may consume a lot of computing time. These calculation times may cause the output image during playback to be out of sync with the image processing results, resulting in a decrease in the quality of the output image.
本發明提供一種基於多執行緒產生三維影像的電子裝置和方法,可最佳化圖形資源(graphic resource)和深度預測模型的讀取。The present invention provides an electronic device and method for generating three-dimensional images based on multi-threading, which can optimize the reading of graphic resources and depth prediction models.
本發明的一種基於多執行緒產生三維影像的電子裝置,包含處理器以及儲存媒體。儲存媒體包含第一緩衝器。處理器耦接儲存媒體並且運行第一執行緒和第二執行緒,其中第一執行緒將三維影像的影像幀傳送至第一緩衝器,其中影像幀包含三維網格以及對應於三維網格的紋理;響應於第一緩衝器完成影像幀的接收,第二執行緒讀取第一緩衝器中的影像幀以對影像幀執行深度預測,進而產生深度資訊;第一執行緒根據深度資訊更新三維網格;以及第一執行緒根據經更新的三維網格以及紋理產生並排影像。The present invention is an electronic device that generates three-dimensional images based on multi-threading, including a processor and a storage medium. The storage medium includes a first buffer. The processor is coupled to the storage medium and runs a first execution thread and a second execution thread, wherein the first execution thread transmits an image frame of the three-dimensional image to the first buffer, wherein the image frame includes a three-dimensional grid and a frame corresponding to the three-dimensional grid. Texture; in response to the first buffer completing the reception of the image frame, the second execution thread reads the image frame in the first buffer to perform depth prediction on the image frame, thereby generating depth information; the first execution thread updates the three-dimensional image frame according to the depth information grid; and the first thread generates side-by-side images based on the updated three-dimensional grid and texture.
在本發明的一實施例中,上述的儲存媒體更包含第二緩衝器,其中第二執行緒將深度資訊傳送至第二緩衝器;以及響應於第二緩衝器完成深度資訊的接收,第一執行緒讀取第二緩衝器中的深度資訊。In an embodiment of the present invention, the above-mentioned storage medium further includes a second buffer, wherein the second thread transmits the depth information to the second buffer; and in response to the second buffer completing the reception of the depth information, the first The thread reads the depth information in the second buffer.
在本發明的一實施例中,其中響應於第一緩衝器完成影像幀的接收,處理器傳送對應於影像幀的識別碼至第二執行緒;以及響應於識別碼,第二執行緒讀取第一緩衝器中的影像幀。In an embodiment of the present invention, in response to the first buffer completing the reception of the image frame, the processor transmits an identification code corresponding to the image frame to the second execution thread; and in response to the identification code, the second execution thread reads Image frames in the first buffer.
在本發明的一實施例中,其中響應於第二緩衝器完成深度資訊的接收,處理器傳送對應於影像幀的識別碼至第一執行緒;以及響應於識別碼,第一執行緒讀取第二緩衝器中的深度資訊。In an embodiment of the present invention, in response to the second buffer completing the reception of the depth information, the processor transmits an identification code corresponding to the image frame to the first execution thread; and in response to the identification code, the first execution thread reads Depth information in the second buffer.
在本發明的一實施例中,上述的電子裝置更包含影像擷取裝置。影像擷取裝置耦接處理器,其中第一執行緒經配置以:通過影像擷取裝置執行眼睛追蹤以取得視野;將視野投影至經更新的三維網格以取得感興趣區域;將紋理映射至感興趣區域以產生輸出幀;以及根據輸出幀產生並排影像。In an embodiment of the present invention, the above-mentioned electronic device further includes an image capturing device. The image capture device is coupled to the processor, wherein the first execution thread is configured to: perform eye tracking through the image capture device to obtain the field of view; project the field of view to the updated three-dimensional grid to obtain the region of interest; map the texture to region of interest to generate an output frame; and generate side-by-side images based on the output frame.
在本發明的一實施例中,上述的第一執行緒將紋理映射至經更新的三維網格以產生並排影像。In an embodiment of the present invention, the above-mentioned first thread maps the texture to the updated three-dimensional mesh to generate side-by-side images.
在本發明的一實施例中,上述的電子裝置更包含收發器。收發器耦接處理器,其中第一執行緒對並排影像執行編織操作以產生編織影像,並且通過收發器輸出編織影像。In an embodiment of the present invention, the above-mentioned electronic device further includes a transceiver. The transceiver is coupled to the processor, wherein the first thread performs a knitting operation on the side-by-side images to generate a knitted image, and outputs the knitted image through the transceiver.
在本發明的一實施例中,上述的第二執行緒將影像幀輸入至機器學習模型以執行深度預測。In an embodiment of the present invention, the above-mentioned second thread inputs image frames to the machine learning model to perform depth prediction.
本發明的一種基於多執行緒產生三維影像的方法,包含:由處理器運行第一執行緒和第二執行緒;由第一執行緒將三維影像的影像幀傳送至儲存媒體中的第一緩衝器,其中影像幀包含三維網格以及對應於三維網格的紋理;響應於第一緩衝器完成影像幀的接收,由第二執行緒讀取第一緩衝器中的影像幀以對影像幀執行深度預測,進而產生深度資訊;由第一執行緒根據深度資訊更新三維網格;以及由第一執行緒根據經更新的三維網格以及紋理產生並排影像。The present invention provides a method for generating a three-dimensional image based on multiple threads, including: a processor running a first thread and a second thread; and the first thread transmitting image frames of the three-dimensional image to a first buffer in a storage medium. The image frame includes a three-dimensional grid and a texture corresponding to the three-dimensional grid; in response to the first buffer completing the reception of the image frame, the second execution thread reads the image frame in the first buffer to execute the image frame depth prediction, thereby generating depth information; the first execution thread updates the three-dimensional grid according to the depth information; and the first execution thread generates side-by-side images according to the updated three-dimensional grid and texture.
基於上述,本發明的電子裝置可在一執行緒通過緩衝器完成影像幀或深度資訊的接收後,觸發另一執行緒基於緩衝器中的完整資料進行後續的影像處理之步驟。據此,本發明可改善緩衝器存取速度以及影像處理流程與產生三維影像流程之間的同步性。Based on the above, the electronic device of the present invention can trigger another thread to perform subsequent image processing steps based on the complete data in the buffer after one thread completes receiving the image frame or depth information through the buffer. Accordingly, the present invention can improve the buffer access speed and the synchronization between the image processing process and the three-dimensional image generation process.
為了使本發明之內容可以被更容易明瞭,以下特舉實施例作為本發明確實能夠據以實施的範例。另外,凡可能之處,在圖式及實施方式中使用相同標號的元件/構件/步驟,係代表相同或類似部件。In order to make the content of the present invention easier to understand, the following embodiments are given as examples according to which the present invention can be implemented. In addition, wherever possible, elements/components/steps with the same reference numbers in the drawings and embodiments represent the same or similar parts.
圖1根據本發明的一實施例繪示電子裝置100的示意圖。電子裝置100可基於多執行緒產生三維影像。電子裝置100可包含處理器110、儲存媒體120以及收發器130。在一實施例中,電子裝置100可進一步包含影像擷取裝置140。FIG. 1 is a schematic diagram of an
處理器110例如是中央處理單元(central processing unit,CPU),或是其他可程式化之一般用途或特殊用途的微控制單元(micro control unit,MCU)、微處理器(microprocessor)、數位信號處理器(digital signal processor,DSP)、可程式化控制器、特殊應用積體電路(application specific integrated circuit,ASIC)、圖形處理器(graphics processing unit,GPU)、影像訊號處理器(image signal processor,ISP)、影像處理單元(image processing unit,IPU)、算數邏輯單元(arithmetic logic unit,ALU)、複雜可程式邏輯裝置(complex programmable logic device,CPLD)、現場可程式化邏輯閘陣列(field programmable gate array,FPGA)或其他類似元件或上述元件的組合。處理器110可耦接至儲存媒體120、收發器130以及影像擷取裝置140,並且存取和執行儲存於儲存媒體120中的多個模組和各種應用程式。處理器110可具有多執行緒能力。在本實施例中,處理器110可運行執行緒111和執行緒112。The
儲存媒體120例如是任何型態的固定式或可移動式的隨機存取記憶體(random access memory,RAM)、唯讀記憶體(read-only memory,ROM)、快閃記憶體(flash memory)、硬碟(hard disk drive,HDD)、固態硬碟(solid state drive,SSD)或類似元件或上述元件的組合,而用於儲存可由處理器110執行的多個模組或各種應用程式。在本實施例中,儲存媒體120可儲存包含緩衝器121、緩衝器122、緩衝器管理器123以及深度預測模型124等多個模組,其功能將於後續說明。The
收發器130以無線或有線的方式傳送及接收訊號。收發器130還可以執行例如低噪聲放大、阻抗匹配、混頻、向上或向下頻率轉換、濾波、放大以及類似的操作。The
影像擷取裝置140例如包含互補式金屬氧化物半導體(complementary metal oxide semiconductor,CMOS)感測器或感光耦合元件(charge coupled device,CCD)感測器等影像感測器。影像擷取裝置140可用於擷取影像。The image capturing
圖2根據本發明的一實施例繪示一種基於多執行緒產生三維影像的方法的示意圖,其中所述方法可由如圖1所示的電子裝置100實施。方法可包含由執行緒111執行的任務200以及由執行緒112執行的任務300。FIG. 2 is a schematic diagram of a method for generating three-dimensional images based on multiple threads according to an embodiment of the present invention, where the method can be implemented by the
在步驟S210中,執行緒111可取得三維影像的影像幀。具體來說,三維影像可包含多個影像幀,且每一個影像幀可包含三維網格(3D mesh)和對應於三維網格的紋理(texture)。執行緒111例如可通過收發器130以自外部裝置接收三維影像的影像幀。影像幀的取得可觸發執行緒111開始執行任務200。In step S210, the
在步驟S220中,若必要,執行緒111可對影像幀執行尺寸調整(resizing)。舉例來說,假設電子裝置100所要產生的輸出影像(或輸出幀)的解析度為1920x1080。執行緒111可根據輸出影像的解析度將影像幀的解析度調整為與輸出影像相同,亦即,將解析度調整為1920x1080。In step S220, if necessary, the
在步驟S230中,執行緒111可將影像幀傳送至緩衝器121儲存。緩衝器121可在影像幀的接收完成後,傳送確認訊息至緩衝器管理器123。確認訊息可包含但不限於對應於影像幀的識別碼。舉例來說,確認訊息可包含對應於緩衝器121所接收的影像幀之紋理的紋理識別碼(texture identifier)。In step S230, the
在步驟S310中,緩衝器管理器123可傳送確認訊息至執行緒112。執行緒112可根據確認訊息讀取儲存在緩衝器121中的影像幀。確認訊息可包含但不限於對應於影像幀的識別碼。舉例來說,確認訊息可包含對應於緩衝器121所接收的影像幀之紋理的紋理識別碼。執行緒112可根據紋理識別碼從緩衝器121中讀取對應於紋理識別碼的影像幀。In step S310, the
在步驟S320中,執行緒112可對影像幀執行預處理。舉例來說,處理器110可調整影像幀的尺寸,以使影像幀符合深度預測模型124的輸入格式。In step S320, the
在完成影像幀的預處理後,在步驟S330中,執行緒112可對影像幀執行深度預測(depth estimation)以產生深度資訊。具體來說,執行緒112可將影像幀輸入至深度預測模型124,以由深度預測模型124輸出對應於影像幀的深度資訊。深度預測模型124可包含但不限於監督式機器學習模型或非監督式機器學習模型。After completing the pre-processing of the image frame, in step S330, the
在步驟S340中,執行緒112可將深度資訊傳送至緩衝器122儲存。緩衝器122可在深度資訊的接收完成後,傳送確認訊息至緩衝器管理器123。確認訊息可包含但不限於對應於影像幀的識別碼。舉例來說,確認訊息可包含對應於緩衝器121所接收的影像幀之紋理的紋理識別碼。In step S340, the
在步驟S240中,緩衝器管理器123可傳送確認訊息至執行緒111。確認訊息可包含但不限於對應於影像幀的識別碼(例如:紋理識別碼)。確認訊息用以通知執行緒111緩衝器121與緩衝器122的資料接收已經完成。In step S240, the
在步驟S250中,執行緒111可根據深度資訊更新影像幀的三維網格,進而根據經更新的三維網格以及紋理產生並排(side-by-side,SBS)影像。具體來說,步驟S250可包含步驟S251至步驟S254。在步驟S251中,執行緒111可根據確認訊息讀取儲存在緩衝器122中的深度資訊,並且根據深度資訊更新三維網格。In step S250 , the
在步驟S252中,執行緒111可根據經更新的三維網格產生感興趣區域(region of interest,ROI)。感興趣區域可與正在觀看三維影像的觀眾的視野(field of view,FOV)有關。具體來說,在步驟S410中,執行緒111可通過影像擷取裝置140執行眼睛追蹤(eye tracking)以取得觀眾的視野。在步驟S252中,執行緒111可將視野投影至經更新的三維網格以取得感興趣區域。換句話說,影像幀的感興趣區域可隨著觀眾的視野變化而變化。In step S252, the
在步驟S253中,執行緒111可將與三維網格相對應的紋理映射至與三維網格相對應的感興趣區域,以產生輸出幀。輸出幀即為影像幀經過深度資訊改善後產生的結果。在步驟S254中,執行緒111可根據輸出幀產生並排影像。In step S253, the
在步驟S260中,執行緒111可對並排影像執行編織(weaving)操作以產生編織影像,並可通過收發器130輸出編織影像。舉例來說,執行緒111可通過收發器130將編織影像傳送至支援三維影像的顯示器,以通過顯示器來播放編織影像。In step S260 , the
圖3根據本發明的一實施例繪示一種基於多執行緒產生三維影像的方法的流程圖,其中所述方法可由如圖1所示的電子裝置100實施。在步驟S510中,由處理器運行第一執行緒和第二執行緒。在步驟S520中,由第一執行緒將三維影像的影像幀傳送至儲存媒體中的第一緩衝器,其中影像幀包含三維網格以及對應於三維網格的紋理。在步驟S530中,響應於第一緩衝器完成影像幀的接收,由第二執行緒讀取第一緩衝器中的影像幀以對影像幀執行深度預測,進而產生深度資訊。在步驟S540中,由第一執行緒根據深度資訊更新三維網格。在步驟S550中,由第一執行緒根據經更新的三維網格以及紋理產生並排影像。FIG. 3 illustrates a flowchart of a method for generating three-dimensional images based on multiple threads according to an embodiment of the present invention, wherein the method can be implemented by the
綜上所述,本發明的電子裝置可利用多執行緒讀取圖形資源、讀取影像處理模型以及產生三維影像。在一執行緒通過緩衝器完成影像幀或深度資訊的接收後,緩衝器管理器可觸發另一執行緒基於緩衝器中的完整資料進行後續的影像處理之步驟。據此,本發明可改善緩衝器存取速度以及影像處理流程與產生三維影像流程之間的同步性。本發明還可最佳化圖形資源和影像處理模型的讀取。In summary, the electronic device of the present invention can utilize multiple threads to read graphics resources, read image processing models, and generate three-dimensional images. After one thread completes receiving the image frame or depth information through the buffer, the buffer manager can trigger another thread to perform subsequent image processing steps based on the complete data in the buffer. Accordingly, the present invention can improve the buffer access speed and the synchronization between the image processing process and the three-dimensional image generation process. The present invention can also optimize the reading of graphics resources and image processing models.
100:電子裝置100: Electronic devices
110:處理器110: Processor
111、112:執行緒111, 112: Execution thread
120:儲存媒體120:Storage media
121、122:緩衝器121, 122: Buffer
123:緩衝器管理器123:Buffer Manager
124:深度預測模型124: Depth prediction model
130:收發器130:Transceiver
140:影像擷取裝置140:Image capture device
200、300:任務200, 300: Mission
S210、S220、S230、S240、S250、S251、S252、S253、S254、S260、S310、S320、S330、S340、S410、S510、S520、S530、S540、S550:步驟S210, S220, S230, S240, S250, S251, S252, S253, S254, S260, S310, S320, S330, S340, S410, S510, S520, S530, S540, S550: Steps
圖1根據本發明的一實施例繪示電子裝置的示意圖。 圖2根據本發明的一實施例繪示一種基於多執行緒產生三維影像的方法的示意圖。 圖3根據本發明的一實施例繪示一種基於多執行緒產生三維影像的方法的流程圖。 FIG. 1 is a schematic diagram of an electronic device according to an embodiment of the invention. FIG. 2 illustrates a schematic diagram of a method for generating three-dimensional images based on multiple threads according to an embodiment of the present invention. FIG. 3 illustrates a flowchart of a method for generating three-dimensional images based on multiple threads according to an embodiment of the present invention.
S510、S520、S530、S540、S550:步驟 S510, S520, S530, S540, S550: Steps
Claims (8)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
TW111149809A TWI817875B (en) | 2022-12-23 | 2022-12-23 | Electronic device and method of generating 3d image based on multi-thread |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
TW111149809A TWI817875B (en) | 2022-12-23 | 2022-12-23 | Electronic device and method of generating 3d image based on multi-thread |
Publications (2)
Publication Number | Publication Date |
---|---|
TWI817875B true TWI817875B (en) | 2023-10-01 |
TW202427360A TW202427360A (en) | 2024-07-01 |
Family
ID=89857802
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
TW111149809A TWI817875B (en) | 2022-12-23 | 2022-12-23 | Electronic device and method of generating 3d image based on multi-thread |
Country Status (1)
Country | Link |
---|---|
TW (1) | TWI817875B (en) |
Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20190362549A1 (en) * | 2018-05-22 | 2019-11-28 | Sick Ag | Visualization of 3d image data |
TWI723048B (en) * | 2015-09-23 | 2021-04-01 | 荷蘭商皇家飛利浦有限公司 | Apparatus and method for generating a triangle mesh for a three dimensional image, and computer program product |
CN115398180A (en) * | 2020-03-26 | 2022-11-25 | 索尼集团公司 | Information processing apparatus, 3D model generation method, and program |
-
2022
- 2022-12-23 TW TW111149809A patent/TWI817875B/en active
Patent Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
TWI723048B (en) * | 2015-09-23 | 2021-04-01 | 荷蘭商皇家飛利浦有限公司 | Apparatus and method for generating a triangle mesh for a three dimensional image, and computer program product |
US20190362549A1 (en) * | 2018-05-22 | 2019-11-28 | Sick Ag | Visualization of 3d image data |
CN115398180A (en) * | 2020-03-26 | 2022-11-25 | 索尼集团公司 | Information processing apparatus, 3D model generation method, and program |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11831980B2 (en) | Electronic device and controlling method of electronic device | |
US20220261960A1 (en) | Super-resolution reconstruction method and related apparatus | |
TWI441091B (en) | Method for performing image signal processing with aid of a graphics processing unit and apparatus for performing image signal processing | |
JP2004213641A (en) | Image processor, image processing method, information processor, information processing system, semiconductor device and computer program | |
KR101214675B1 (en) | Method for processing a image in a handheld device and apparatus for the same | |
JP6207618B2 (en) | Drawing device | |
JP4958499B2 (en) | Image display control device, image display method, and program | |
WO2022237289A1 (en) | Vibration control method and apparatus, mobile terminal, and storage medium | |
EP4393145A1 (en) | Camera initialization for reduced latency | |
TWI817875B (en) | Electronic device and method of generating 3d image based on multi-thread | |
CN102479387A (en) | Method for generating multimedia animation and playing multimedia animation and apparatus thereof | |
TW202427360A (en) | Electronic device and method of generating 3d image based on multi-thread | |
CN110572715B (en) | Method for solving problem of webp blockage during Android TV playing | |
CN112801150A (en) | Method and device for expanding periodic video data | |
CN109925715B (en) | Virtual water area generation method and device and terminal | |
US20140313381A1 (en) | Image pickup apparatus | |
WO2021237736A1 (en) | Image processing method, apparatus and system, and computer-readable storage medium | |
US11471773B2 (en) | Occlusion in mobile client rendered augmented reality environments | |
WO2022133954A1 (en) | Image rendering method, apparatus and system, and computer-readable storage medium | |
TW202427383A (en) | Electronic device and method of depth enhancement for face in three-dimensional image | |
CN115842964A (en) | Image acquisition device, display equipment, image processing method and device | |
JP2014044569A (en) | Electronic apparatus | |
WO2023197284A1 (en) | Saliency-based adaptive color enhancement | |
He | Video Image Processing System Based on Directshow | |
KR20210153497A (en) | Electronic device and controlling method of electronic device |