TWI787672B - Surgical instrument inventory system and surgical instrument inventory method - Google Patents

Surgical instrument inventory system and surgical instrument inventory method Download PDF

Info

Publication number
TWI787672B
TWI787672B TW109140489A TW109140489A TWI787672B TW I787672 B TWI787672 B TW I787672B TW 109140489 A TW109140489 A TW 109140489A TW 109140489 A TW109140489 A TW 109140489A TW I787672 B TWI787672 B TW I787672B
Authority
TW
Taiwan
Prior art keywords
image
surgical instrument
recognition module
inventory
global
Prior art date
Application number
TW109140489A
Other languages
Chinese (zh)
Other versions
TW202221727A (en
Inventor
楊昇宏
潘柏瑋
曾建嘉
Original Assignee
財團法人金屬工業研究發展中心
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 財團法人金屬工業研究發展中心 filed Critical 財團法人金屬工業研究發展中心
Priority to TW109140489A priority Critical patent/TWI787672B/en
Publication of TW202221727A publication Critical patent/TW202221727A/en
Application granted granted Critical
Publication of TWI787672B publication Critical patent/TWI787672B/en

Links

Images

Landscapes

  • Image Analysis (AREA)
  • Surgical Instruments (AREA)

Abstract

A surgical instrument inventory system and a surgical instrument inventory method are provided. The surgical instrument inventory system includes a memory and a processor. The memory stores a global identification module and a local identification module. The processor is coupled to the memory. The processor inputs a surgical instrument image to the global recognition module and the local recognition module. The global recognition module outputs multiple global image features corresponding to the surgical instrument image. The local recognition module outputs multiple local image features corresponding to an end instrument image of the surgical instrument image. The processor determines a surgical instrument type of the surgical instrument image according to the multiple global image features and the multiple local image features.

Description

手術器械盤點系統以及手術器械盤點方法Surgical instrument inventory system and surgical instrument inventory method

本發明是有關於一種盤點技術，且特別是有關於一種手術器械盤點系統以及手術器械盤點方法。 The present invention relates to an inventory technology, and in particular to an inventory system for surgical instruments and a method for inventory of surgical instruments.

目前對於手術進行之前的手術器械準備，醫療人員必須於手術進行之前將對應於本次手術所需要使用的各式手術器械進行準備。醫療人員例如需將多個手術器械收納於手術器械盤或手術器械包中，並且透過人為盤點的方式進行手術器械盤點。換言之，對於不同手術而言，醫療人員必須花費大量時間進行各別手術的手術器械準備與盤點。因此，傳統的手術器械準備與盤點容易發生人為疏失，例如發生放置錯誤的手術器械或遺漏手術器械的情況。有鑑於此，如何可自動且準確地盤點手術器械，以下將提出幾個實施例的解決方案。 Currently, for the preparation of surgical instruments before the operation, medical personnel must prepare various surgical instruments corresponding to the operation before the operation. Medical personnel, for example, need to store a plurality of surgical instruments in a surgical instrument tray or a surgical instrument bag, and perform an inventory of the surgical instruments by manual inventory. In other words, for different operations, medical personnel have to spend a lot of time preparing and inventorying surgical instruments for each operation. Therefore, traditional surgical instrument preparation and inventory are prone to human error, such as misplaced surgical instruments or missing surgical instruments. In view of this, how to automatically and accurately inventory surgical instruments, several solutions of embodiments will be proposed below.

本發明提供一種手術器械盤點系統以及手術器械盤點方法可提供自動盤點手術器械的功能。 The invention provides a surgical instrument inventory system and a surgical instrument inventory method which can provide the function of automatically inventorying surgical instruments.

本發明的手術器械盤點系統包括記憶體以及處理器。記憶體用以儲存全局辨識模組以及局部辨識模組。處理器耦接記憶體，並且用以將手術器械影像輸入至全局辨識模組以及局部辨識模組。全局辨識模組用以輸出對應於手術器械影像的多個全局影像特徵。局部辨識模組用以輸出對應於手術器械影像的末端器械影像的多個局部影像特徵。處理器依據所述多個全局影像特徵以及所述多個局部影像特徵判斷手術器械影像的手術器械類型。 The surgical instrument inventory system of the present invention includes a memory and a processor. The memory is used to store the global identification module and the local identification module. The processor is coupled to the memory and is used for inputting the surgical instrument image to the global recognition module and the local recognition module. The global recognition module is used to output a plurality of global image features corresponding to the surgical instrument image. The local recognition module is used for outputting a plurality of local image features corresponding to the terminal instrument image of the surgical instrument image. The processor determines the surgical instrument type of the surgical instrument image according to the plurality of global image features and the plurality of local image features.

本發明的手術器械盤點方法包括以下步驟：將手術器械影像輸入至全局辨識模組，以產生對應於手術器械影像的多個全局影像特徵；將手術器械影像輸入至局部辨識模組，以產生對應於手術器械影像的末端器械影像的多個局部影像特徵；以及依據所述多個全局影像特徵以及所述多個局部影像特徵判斷手術器械影像的手術器械類型。 The surgical instrument inventory method of the present invention includes the following steps: inputting the surgical instrument image into the global recognition module to generate a plurality of global image features corresponding to the surgical instrument image; inputting the surgical instrument image into the local recognition module to generate corresponding A plurality of local image features of an end instrument image of the surgical instrument image; and determining a surgical instrument type of the surgical instrument image according to the plurality of global image features and the plurality of local image features.

基於上述，本發明的手術器械盤點系統以及手術器械盤點方法可透過影像分析的方式來辨識手術器械類型，以準確地對手術器械進行類別盤點及其數量點收。 Based on the above, the surgical instrument inventory system and the surgical instrument inventory method of the present invention can identify the types of surgical instruments through image analysis, so as to accurately carry out inventory of surgical instruments by category and quantity.

為讓本發明的上述特徵和優點能更明顯易懂，下文特舉實施例，並配合所附圖式作詳細說明如下。 In order to make the above-mentioned features and advantages of the present invention more comprehensible, the following specific embodiments are described in detail together with the accompanying drawings.

100:全局辨識模組 100: Global identification module

110:處理器 110: Processor

120:記憶體 120: memory

121:全局辨識模組 121:Global identification module

122:局部辨識模組 122: Local identification module

122_1:末端辨識模組 122_1: Terminal identification module

122_2:特徵辨識模組 122_2: Feature recognition module

123:部件辨識模組 123: Part identification module

130:攝影機 130: camera

140:投影機 140:Projector

310:虛擬手術器械盤影像 310:Virtual surgical instrument tray image

311~319:手術器械輪廓 311~319: Outline of Surgical Instruments

510:手術器械影像 510: Surgical Instrument Image

511~514:部件影像 511~514: Part image

520:末端器械影像 520: end device image

501_1~501_M:全局影像特徵 501_1~501_M: global image features

502_1~502_N:局部影像特徵 502_1~502_N: Local image features

503_1~503_P:部件影像特徵 503_1~503_P: Part image features

S1:表面 S1: surface

S210~S230、S410~S440:步驟 S210~S230, S410~S440: steps

圖1是本發明的一實施例的手術器械盤點系統的示意圖。 Fig. 1 is a schematic diagram of a surgical instrument inventory system according to an embodiment of the present invention.

圖2是本發明的一實施例的取得手術器械影像的流程圖。 FIG. 2 is a flow chart of obtaining images of surgical instruments according to an embodiment of the present invention.

圖3是本發明的一實施例的虛擬手術器械盤影像的投影示意圖。 FIG. 3 is a schematic diagram of projection of a virtual surgical instrument tray image according to an embodiment of the present invention.

圖4是本發明的一實施例的手術器械盤點方法的流程圖。 Fig. 4 is a flow chart of a surgical instrument inventory method according to an embodiment of the present invention.

圖5是本發明的一實施例的產生對應於手術器械影像的多個影像特徵的示意圖。 FIG. 5 is a schematic diagram of generating a plurality of image features corresponding to surgical instrument images according to an embodiment of the present invention.

為了使本發明之內容可以被更容易明瞭，以下特舉實施例做為本發明確實能夠據以實施的範例。另外，凡可能之處，在圖式及實施方式中使用相同標號的元件/構件/步驟，係代表相同或類似部件。 In order to make the content of the present invention more comprehensible, the following specific examples are given as examples in which the present invention can indeed be implemented. In addition, wherever possible, elements/components/steps using the same reference numerals in the drawings and embodiments represent the same or similar parts.

圖1是本發明的一實施例的手術器械盤點系統的示意圖。參考圖1，手術器械盤點系統100包括處理器110、記憶體120、攝影機130以及投影機140。處理器110耦接記憶體120、攝影機130以及投影機140。在本實施例中，記憶體120可儲存全局辨識模組121、局部辨識模組122以及部件辨識模組123。在本實施例中，手術器械盤點系統100可先利用投影機140於表面投射虛擬手術器械盤影像，其中虛擬手術器械盤影像可包括一個或多個手術器械輪廓，並且所述多個手術器械輪廓可對應於相同或不相同 的手術器械類型。在本實施例中，當使用者將對應的手術器械放置於虛擬手術器械盤影像上之後，手術器械盤點系統100可接著利用攝影機130拍攝所述虛擬手術器械盤影像，以取得盤點影像，其中盤點影像可包括一個或多個手術器械影像。 Fig. 1 is a schematic diagram of a surgical instrument inventory system according to an embodiment of the present invention. Referring to FIG. 1 , the surgical instrument inventory system 100 includes a processor 110 , a memory 120 , a camera 130 and a projector 140 . The processor 110 is coupled to the memory 120 , the camera 130 and the projector 140 . In this embodiment, the memory 120 can store the global identification module 121 , the local identification module 122 and the component identification module 123 . In this embodiment, the surgical instrument inventory system 100 can first use the projector 140 to project a virtual surgical instrument tray image on the surface, wherein the virtual surgical instrument tray image can include one or more surgical instrument outlines, and the plurality of surgical instrument outlines can correspond to the same or different type of surgical instrument. In this embodiment, after the user places the corresponding surgical instruments on the virtual surgical instrument tray image, the surgical instrument inventory system 100 can then use the camera 130 to capture the virtual surgical instrument tray image to obtain an inventory image, wherein the inventory The images may include one or more surgical instrument images.

在本實施例中，處理器110可透過全局辨識模組121、局部辨識模組122以及部件辨識模組123來分析盤點影像中的一個或多個手術器械影像，以對於每一個手術器械影像產生對應的多個影像特徵。處理器110可依據所述多個影像特徵來有效地分別辨識對應的手術器械影像的手術器械類型。並且，當處理器110判斷在同一個盤點影像中的多個手術器械影像的多個手術器械類型後，處理器110可依據被辨識出來的多個手術器械的多個手術器械類型決定對應於盤點影像的盤點資訊。另外，處理器110還可例如計算相同或不相同的手術器械類型的數量，並且記載於所述盤點資訊中。 In this embodiment, the processor 110 can analyze one or more surgical instrument images in the inventory image through the global recognition module 121, the local recognition module 122 and the component recognition module 123, so as to generate Corresponding multiple image features. The processor 110 can effectively identify the types of surgical instruments corresponding to the surgical instrument images according to the plurality of image features. Moreover, after the processor 110 determines the multiple types of surgical instruments in the multiple surgical instrument images in the same inventory image, the processor 110 can determine the corresponding inventory according to the multiple surgical instrument types of the identified multiple surgical instruments. Image inventory information. In addition, the processor 110 can also, for example, calculate the quantity of the same or different types of surgical instruments, and record it in the inventory information.

在本發明的一些實施例中，手術器械盤點系統100可不包括投影機140，而可同樣達成本發明的自動盤點手術器械的功能。對此，在本發明的一些實施例中，使用者可將對應的手術器械放置於實體手術器械盤或由其他投影設備所投影的虛擬手術器械盤影像上，以使手術器械盤點系統100可利用攝影機130進行拍攝所述實體手術器械盤或所述由其他投影設備所投影的虛擬手術器械盤影像而取得盤點影像。或者，在本發明的另一些實施情境中，使用者可將一個或多個手術器械放置於任意平台上，以供 手術器械盤點系統100進行手術器械的辨識與盤點。 In some embodiments of the present invention, the surgical instrument inventory system 100 may not include the projector 140, but can also achieve the function of automatically inventorying surgical instruments of the present invention. For this, in some embodiments of the present invention, the user can place the corresponding surgical instruments on the physical surgical instrument tray or the virtual surgical instrument tray image projected by other projection devices, so that the surgical instrument inventory system 100 can use The camera 130 captures the physical surgical instrument tray or the virtual surgical instrument tray image projected by other projection devices to obtain an inventory image. Or, in other implementation scenarios of the present invention, the user can place one or more surgical instruments on any platform for The surgical instrument inventory system 100 performs identification and inventory of surgical instruments.

在本發明的另一些實施例中，手術器械盤點系統100可不包括攝影機130，而可同樣達成本發明的自動盤點手術器械的功能。對此，在本發明的另一些實施例中，使用者可透過任意的獨立攝影設備來取得所述盤點影像。使用者可將所述盤點影像輸入至手術器械盤點系統100的處理器110來進行如本發明各實施例所提出的相關影像處理與分析運算。 In other embodiments of the present invention, the surgical instrument inventory system 100 may not include the camera 130, but can also achieve the function of automatically inventorying surgical instruments of the present invention. For this, in other embodiments of the present invention, the user can obtain the inventory image through any independent photographic equipment. The user can input the inventory image to the processor 110 of the surgical instrument inventory system 100 to perform related image processing and analysis operations as proposed in various embodiments of the present invention.

在本發明的又一些實施例中，手術器械盤點系統100的記憶體120可僅儲存全局辨識模組121以及局部辨識模組122。對此，在本發明的又一些實施例中，手術器械盤點系統100的處理器110可只利用全局辨識模組121以及局部辨識模組122來分析盤點影像中的一個或多個手術器械影像，以對於每一個手術器械影像產生對應的多個影像特徵。對此，雖然此實施範例的辨識準確度與包括部件辨識模組123的實施範例的辨識準確度不同，但在又一些實施例中(例如是在多個手術器械之間具有手術器械類型的差異度較大的應用情境)的處理器110仍可依據所述多個影像特徵來有效地分別辨識對應的手術器械影像的手術器械類型。 In some other embodiments of the present invention, the memory 120 of the surgical instrument inventory system 100 can only store the global identification module 121 and the local identification module 122 . For this, in some other embodiments of the present invention, the processor 110 of the surgical instrument inventory system 100 can only use the global identification module 121 and the local identification module 122 to analyze one or more surgical instrument images in the inventory images, A plurality of corresponding image features are generated for each surgical instrument image. In this regard, although the recognition accuracy of this embodiment is different from that of the embodiment including the component recognition module 123, in some other embodiments (for example, there are differences in the types of surgical instruments among multiple surgical instruments) The processor 110 can still effectively identify the surgical instrument type of the corresponding surgical instrument image according to the plurality of image features.

在本實施例中，處理器110可包括中央處理器(Central processing unit；CPU)、可程式設計的一般用途或特殊用途的微處理器(Microprocessor)、數位訊號處理器(Digital signal processor；DSP)、可程式設計控制器、專用積體電路(Application specific integrated circuit；ASIC)、圖形處理器(Graphics processing unit， GPU)或其他類似元件或上述元件的組合且可用於實現本發明的相關功能電路。 In this embodiment, the processor 110 may include a central processing unit (Central processing unit; CPU), a programmable general-purpose or special-purpose microprocessor (Microprocessor), a digital signal processor (Digital signal processor; DSP) , programmable controller, application specific integrated circuit (ASIC), graphics processing unit (Graphics processing unit, GPU) or other similar elements or a combination of the above-mentioned elements and can be used to realize the relevant functional circuits of the present invention.

在本實施例中，記憶體120可例如包括隨機存取記憶體(Random-Access Memory；RAM)、唯讀記憶體(Read-Only Memory；ROM)、光碟(Optical disc)、磁碟(Magnetic disk)、硬驅動機(Hard drive)、固態驅動機(Solid-state driv)、快閃驅動機(Flash drive)、安全數位(Security digital；SD)卡、記憶條(Memory stick)、緊密快閃(Compact flash；CF)卡或任何類型的儲存裝置。在本實施例中，記憶體120可用於儲存本發明各實施例所提到的相關模組、相關影像資料及相關參數等，以使處理器110可透過存取記憶體120，以執行相關資料處理及運算。 In this embodiment, the memory 120 may include, for example, random access memory (Random-Access Memory; RAM), read-only memory (Read-Only Memory; ROM), optical disc (Optical disc), magnetic disk (Magnetic disk) ), Hard drive, Solid-state drive, Flash drive, Security digital (SD) card, Memory stick, Compact flash ( Compact flash; CF card or any type of storage device. In this embodiment, the memory 120 can be used to store related modules, related image data, and related parameters mentioned in various embodiments of the present invention, so that the processor 110 can access the memory 120 to execute related data. processing and computing.

圖2是本發明的一實施例的取得手術器械影像的流程圖。圖3是本發明的一實施例的虛擬手術器械盤影像的投影示意圖。參考圖1至圖3，手術器械盤點系統100可執行如圖2的步驟S210~S230來取得手術器械影像，並且手術器械盤點系統100可投射如圖3的虛擬手術器械盤影像310。在步驟S210，手術器械盤點系統100可藉由投影機140於表面S1投射虛擬手術器械盤影像310，其中虛擬手術器械盤影像310包括多個手術器械輪廓311~319。在本實施例中，表面S1可是指手術器械盤或手術器械包，但本發明並不限於此。在本發明的某一些實施情境中，表面S1可以是指任意平台的表面(可為平整面或非平整面)，而本發明並不限制投影位置及投影環境。在本實施例中，使用者可於虛擬 手術器械盤影像310中的手術器械輪廓311~319上擺放對應的多個手術器械。當使用者擺放完成後，使用者可操作手術器械盤點系統100進行取像或由手術器械盤點系統100於預設時間進行自動取像。或者，手術器械盤點系統100可預取像以透過影像分析的方式判斷所得的影像中對應於手術器械輪廓311~319的位置是否皆存在物件，以進行盤點影像的取像操作。 FIG. 2 is a flow chart of obtaining images of surgical instruments according to an embodiment of the present invention. FIG. 3 is a schematic diagram of a projected image of a virtual surgical instrument tray according to an embodiment of the present invention. Referring to FIG. 1 to FIG. 3 , the surgical instrument inventory system 100 can execute steps S210 to S230 as shown in FIG. 2 to obtain surgical instrument images, and the surgical instrument inventory system 100 can project a virtual surgical instrument tray image 310 as shown in FIG. 3 . In step S210 , the surgical instrument inventory system 100 can project a virtual surgical instrument tray image 310 on the surface S1 through the projector 140 , wherein the virtual surgical instrument tray image 310 includes a plurality of surgical instrument outlines 311 - 319 . In this embodiment, the surface S1 may refer to a surgical instrument tray or a surgical instrument bag, but the present invention is not limited thereto. In some implementation scenarios of the present invention, the surface S1 may refer to the surface of any platform (which may be a flat surface or an uneven surface), and the present invention does not limit the projection location and projection environment. In this embodiment, users can use virtual A plurality of corresponding surgical instruments are placed on the surgical instrument outlines 311 - 319 in the surgical instrument tray image 310 . After the user completes the placement, the user can operate the surgical instrument inventory system 100 to take an image, or the surgical instrument inventory system 100 can automatically take an image at a preset time. Alternatively, the surgical instrument inventory system 100 can pre-capture images to determine whether there are objects in the obtained images corresponding to the surgical instrument outlines 311 - 319 through image analysis, so as to perform an image capture operation of the inventory images.

值得注意的是，虛擬手術器械盤影像310可有助於使用者將正確數量且正確的手術器械類型的一個或多個手術器械放置於手術器械盤上一個或多個特定位置上，以使符合當前手術醫療設備規範的要求，或者是可便於手術進行中，所述一個或多個手術器械可被醫療人員正確且便利地拿取與使用。然而，在本發明的一些實施情境中，手術器械盤點系統100亦可不需執行步驟S210，而直接執行以下步驟S220。 Notably, the virtual surgical instrument tray image 310 may assist the user in placing one or more surgical instruments of the correct quantity and type of surgical instrument at one or more specific locations on the surgical instrument tray to conform to It may be a requirement of current surgical medical equipment regulations, or may facilitate surgery, that the one or more surgical instruments be correctly and conveniently held and used by medical personnel. However, in some implementation scenarios of the present invention, the surgical instrument inventory system 100 may also directly perform the following step S220 without performing the step S210.

在步驟S220，手術器械盤點系統100可藉由攝影機130朝表面拍攝虛擬手術器械盤310，以取得盤點影像。在步驟S230，手術器械盤點系統100可從所述盤點影像中對應於所述多個手術器械輪廓的位置來取得所述多個手術器械影像。因此，手術器械盤點系統100可將取得的所述多個手術器械影像的每一個進行如以下實施例的手術器械辨識。另外，需聲明的是，本發明的虛擬手術器械盤的投影影像不限於圖3，並且其虛擬手術器械盤中的手術器械數量以及手術器械類型也不限於圖3。 In step S220 , the surgical instrument inventory system 100 can use the camera 130 to shoot the virtual surgical instrument tray 310 toward the surface to obtain an inventory image. In step S230, the surgical instrument inventory system 100 can obtain the plurality of surgical instrument images from positions corresponding to the outlines of the plurality of surgical instruments in the inventory image. Therefore, the surgical instrument inventory system 100 can perform surgical instrument recognition on each of the acquired surgical instrument images as in the following embodiment. In addition, it should be declared that the projected image of the virtual surgical instrument tray of the present invention is not limited to FIG. 3 , and the number and types of surgical instruments in the virtual surgical instrument tray are not limited to FIG. 3 .

圖4是本發明的一實施例的手術器械盤點方法的流程 圖。圖5是本發明的一實施例的產生對應於手術器械影像的多個影像特徵的示意圖。參考圖1、圖4以及圖5，手術器械盤點系統100可執行如圖4的步驟S410~S440來判斷手術器械的手術器械類型，並且手術器械盤點系統100的處理器110所執行的多個模組的結果可如圖5所示。在步驟S410，手術器械盤點系統100的處理器110可將手術器械影像510輸入至全局辨識模組121，以產生對應於手術器械影像510的多個全局影像特徵501_1~501_M(分別具有不同特徵值)，其中M為正整數。在本實施例中，手術器械影像510可例如是取自於上述圖3實施例的拍攝虛擬手術器械盤影像310的多個手術器械輪廓311~319的其中之一。在本實施例中，全局辨識模組121用以對手術器械影像510進行卷積神經網路運算(Convolutional Neural Network；CNN)，以輸出全局影像特徵501_1~501_M。 Fig. 4 is the flow chart of the surgical instrument inventory method according to an embodiment of the present invention picture. FIG. 5 is a schematic diagram of generating a plurality of image features corresponding to surgical instrument images according to an embodiment of the present invention. Referring to FIG. 1, FIG. 4 and FIG. 5, the surgical instrument inventory system 100 can execute steps S410 to S440 as shown in FIG. The results of the group can be shown in FIG. 5 . In step S410, the processor 110 of the surgical instrument inventory system 100 can input the surgical instrument image 510 to the global recognition module 121 to generate a plurality of global image features 501_1~501_M (with different feature values) corresponding to the surgical instrument image 510 ), where M is a positive integer. In this embodiment, the surgical instrument image 510 may be, for example, one of the plurality of surgical instrument outlines 311 - 319 taken from the virtual surgical instrument tray image 310 captured in the above-mentioned embodiment of FIG. 3 . In this embodiment, the global recognition module 121 is used to perform a convolutional neural network (Convolutional Neural Network; CNN) operation on the surgical instrument image 510 to output global image features 501_1˜501_M.

在步驟S420，手術器械盤點系統100的處理器110可將手術器械影像510輸入至局部辨識模組122，以產生對應於手術器械影像510的末端器械影像520的多個局部影像特徵502_1~502_N(分別具有不同特徵值)，其中N為正整數。在本實施例中，局部辨識模組122可進一步包括末端辨識模組122_1以及特徵辨識模組122_2。局部辨識模組122可將手術器械影像510輸入至末端辨識模組122_1，以使末端辨識模組122_1進行快速卷積神經網路運算(Faster R-CNN)，以從手術器械影像510決定末端器械影像520。末端器械影像520的影像尺寸小於手術器械影像 510。接著，局部辨識模組122可將局部辨識模組122可輸入至特徵辨識模組122_2，以使特徵辨識模組122_2可進行卷積神經網路運算，以輸出對應於末端器械影像520的局部影像特徵502_1~502_N。換言之，由於不同手術器械類型的手術器械可能具有不同的末端器械構造，因此本實施例的手術器械盤點系統100可藉由辨識手術器械的末端器械構造來有助於區別不同的手術器械。 In step S420, the processor 110 of the surgical instrument inventory system 100 can input the surgical instrument image 510 to the local recognition module 122 to generate a plurality of local image features 502_1~502_N( have different eigenvalues), where N is a positive integer. In this embodiment, the partial identification module 122 may further include an end identification module 122_1 and a feature identification module 122_2. The local recognition module 122 can input the surgical instrument image 510 to the end recognition module 122_1, so that the end recognition module 122_1 performs Faster R-CNN to determine the end instrument from the surgical instrument image 510 Image 520. The image size of the terminal instrument image 520 is smaller than the surgical instrument image 510. Next, the local recognition module 122 can input the partial recognition module 122 to the feature recognition module 122_2, so that the feature recognition module 122_2 can perform convolutional neural network calculations to output a partial image corresponding to the terminal device image 520 Features 502_1~502_N. In other words, since surgical instruments of different surgical instrument types may have different terminal instrument configurations, the surgical instrument inventory system 100 of this embodiment can help distinguish different surgical instruments by identifying the terminal instrument configurations of surgical instruments.

在步驟S430，手術器械盤點系統100的處理器110可將手術器械影像510輸入至部件辨識模組123，以產生對應於手術器械影像510的多個部件影像511~514的多個部件影像特徵503_1~503_P(分別具有不同特徵值)，其中P為正整數。在本實施例中，部件辨識模組123用以對手術器械影像510進行快速卷積神經網路運算，以從手術器械影像510中決定部件影像511~514，並且對部件影像511~514分別進行部件分析，以決定對應於部件影像511~514的部件影像特徵503_1~503_P。對此，由於不同手術器械類型的手術器械可能具有不同的部件構造，因此本實施例的手術器械盤點系統100可藉由辨識手術器械的至少一個特定部件構造來有助於區別不同的手術器械。 In step S430, the processor 110 of the surgical instrument inventory system 100 can input the surgical instrument image 510 to the component recognition module 123 to generate a plurality of component image features 503_1 corresponding to the plurality of component images 511-514 of the surgical instrument image 510 ~503_P (respectively with different eigenvalues), where P is a positive integer. In this embodiment, the component recognition module 123 is used to perform fast convolutional neural network calculation on the surgical instrument image 510 to determine the component images 511-514 from the surgical instrument image 510, and perform The component analysis is used to determine the component image features 503_1 - 503_P corresponding to the component images 511 - 514 . In this regard, since surgical instruments of different surgical instrument types may have different component configurations, the surgical instrument inventory system 100 of this embodiment can help distinguish different surgical instruments by identifying at least one specific component configuration of the surgical instruments.

在步驟S440，手術器械盤點系統100的處理器110可依據全局影像特徵501_1~501_M、局部影像特徵502_1~502_N以及部件影像特徵503_1~503_P判斷手術器械影像510的手術器械類型。對此，由於不同手術器械類型的手術器械可能具有不同的整 體(全局)器械影像特徵、末端器械構造及部件構造，因此本實施例的手術器械盤點系統100可藉由取得手術器械的影像辨識結果的多個影像特徵來辨識手術器械的手術器械類型。並且，本實施例的手術器械盤點系統100可例如於記憶體120更儲存器械對照表，其中器械對照表可例如儲存一個或多個手術器械的個別的多個預設全局影像特徵值、局部預設影像特徵值以及部件預設影像特徵值。因此，處理器110可依據全局影像特徵501_1~501_M、局部影像特徵502_1~502_N以及部件影像特徵503_1~503_P來比對器械對照表(與預設特徵值是否一致)，以快速且正確地判斷手術器械影像510的手術器械類型。 In step S440, the processor 110 of the surgical instrument inventory system 100 can determine the surgical instrument type of the surgical instrument image 510 according to the global image features 501_1~501_M, the local image features 502_1~502_N, and the component image features 503_1~503_P. In this regard, since surgical instruments of different surgical instrument types may have different Therefore, the surgical instrument inventory system 100 of this embodiment can identify the type of surgical instrument by obtaining multiple image features of the image recognition result of the surgical instrument. Moreover, the surgical instrument inventory system 100 of the present embodiment can, for example, further store an instrument comparison table in the memory 120, wherein the instrument comparison table can, for example, store a plurality of individual preset global image feature values and local prediction values of one or more surgical instruments. Set the image characteristic value and the default image characteristic value of the component. Therefore, the processor 110 can compare the device comparison table (whether it is consistent with the preset feature value) according to the global image features 501_1~501_M, the local image features 502_1~502_N, and the component image features 503_1~503_P, so as to quickly and correctly judge the operation The surgical instrument type of the instrument image 510 .

並且，在本實施例中，記憶體120更用以儲存盤點對照表。當手術器械盤點系統100例如將拍攝圖3的拍攝虛擬手術器械盤310的對應於手術器械輪廓311~319的位置的多個手術器械影像皆進行如上述步驟S410~S440的辨識操作，而取得對應的多個手術器械類型的資訊後，處理器110可依據所述多個手術器械類型的資訊比對所述盤點對照表，以決定盤點資訊。舉例而言，盤點對照表例如包括記載有剪刀、鉗子以及針筒。對此，若手術器械盤點系統100從同一張盤點影像中可判斷具有剪刀、鉗子以及針筒的手術器械影像，則表示使用者擺放在手術器械盤上的手術器械的數量以及類型為正確。因此，本實施例的手術器械盤點系統100可快速且正確的自動盤點手術器械的功能。 Moreover, in this embodiment, the memory 120 is further used to store an inventory comparison table. When the surgical instrument inventory system 100 takes, for example, multiple surgical instrument images of the virtual surgical instrument tray 310 in FIG. After the information of the multiple types of surgical instruments, the processor 110 can compare the inventory comparison table according to the information of the multiple types of surgical instruments to determine the inventory information. For example, the inventory comparison table includes records of scissors, pliers and syringes. In this regard, if the surgical instrument inventory system 100 can determine the images of surgical instruments including scissors, pliers, and syringes from the same inventory image, it means that the quantity and type of surgical instruments placed on the surgical instrument tray by the user are correct. Therefore, the surgical instrument inventory system 100 of this embodiment can quickly and accurately perform automatic inventory of surgical instruments.

另外，值得注意的是，上述的全局辨識模組121、局部辨 識模組122以及部件辨識模組123可經預先訓練來實現其影像辨識及分析功能。對此，使用者可預先拍攝多張參考手術器械影像(例如100張)，並且將具有影像中所有目標標註框構成的列表資訊(BBOX list)的多張參考手術器械影像輸入至上述的全局辨識模組121、局部辨識模組122以及部件辨識模組123的卷積神經網路運算模型或快速卷積神經網路運算模型，以進行訓練與推論，以使上述的全局辨識模組121、局部辨識模組122以及部件辨識模組123的卷積神經網路運算模型或快速卷積神經網路運算模型可對應輸出多個辨識結果。接著，經由資料轉換(例如將BBOX list轉換為使用者可編輯的Pascal VOC XML格式)之後，使用者可針對所述多個辨識結果的各別的參考手術器械影像的列表資訊(BBOX list)進行修正，再接著將具有修正後的列表資訊的多張參考手術器械影像再次輸入至上述的全局辨識模組121、局部辨識模組122以及部件辨識模組123的卷積神經網路運算模型或快速卷積神經網路運算模型中進行訓練。對此，上述的全局辨識模組121、局部辨識模組122以及部件辨識模組123的卷積神經網路運算模型或快速卷積神經網路運算模型可例如經由多次循環訓練且連續新增多張參考手術器械影像(例如新增1000張)，來完成上述的全局辨識模組121、局部辨識模組122以及部件辨識模組123可準確且有效地辨識全局影像特徵、局部影像特徵及部件影像特徵。 In addition, it is worth noting that the above-mentioned global identification module 121, local identification The recognition module 122 and the component recognition module 123 can be pre-trained to realize their image recognition and analysis functions. In this regard, the user can take multiple reference surgical instrument images (for example, 100) in advance, and input the multiple reference surgical instrument images with list information (BBOX list) composed of all target label boxes in the image to the above-mentioned global recognition The convolutional neural network computing model or the fast convolutional neural network computing model of the module 121, the local identification module 122, and the component identification module 123 are used for training and inference, so that the above-mentioned global identification module 121, local The convolutional neural network computing model or the fast convolutional neural network computing model of the recognition module 122 and the component recognition module 123 can correspondingly output a plurality of recognition results. Then, after data conversion (for example, converting the BBOX list into a Pascal VOC XML format that can be edited by the user), the user can carry out the list information (BBOX list) of the respective reference surgical instrument images of the multiple recognition results. correction, and then re-input multiple reference surgical instrument images with the revised list information into the convolutional neural network computing model or fast Convolutional neural network computing model for training. In this regard, the above-mentioned convolutional neural network operation model or fast convolutional neural network operation model of the global identification module 121, local identification module 122, and component identification module 123 can be continuously added, for example, through multiple rounds of training Multiple reference surgical instrument images (for example, 1000 new ones) are used to complete the above-mentioned global recognition module 121, local recognition module 122 and component recognition module 123, which can accurately and effectively identify global image features, local image features and components image features.

綜上所述，本發明的手術器械盤點系統以及手術器械盤 點方法可透過多視角的影像分析以及神經網路運算的方式來快速且正確地辨識手術器械，並且可透過查找表(Look-up table)來準確且自動地對於多個手術器械進行類型及數量的盤點。 In summary, the surgical instrument inventory system and the surgical instrument tray of the present invention The point method can quickly and accurately identify surgical instruments through multi-view image analysis and neural network calculations, and can accurately and automatically identify the type and quantity of multiple surgical instruments through a look-up table. inventory.

為了使本發明之內容可以被更容易明瞭，以下特舉實施例做為本發明確實能夠據以實施的範例。另外，凡可能之處，在圖式及實施方式中使用相同標號的元件/構件/步驟，係代表相同或類似部件。 In order to make the content of the present invention more comprehensible, the following specific examples are given as examples in which the present invention can indeed be implemented. In addition, wherever possible, elements/components/steps using the same reference numerals in the drawings and embodiments represent the same or similar parts.

100:手術器械盤點系統 100: Surgical instrument inventory system

110:處理器 110: Processor

120:記憶體 120: memory

121:全局辨識模組 121:Global identification module

122:局部辨識模組 122: Local identification module

123:部件辨識模組 123: Part identification module

130:攝影機 130: camera

140:投影機 140:Projector

Claims (8)

一種手術器械盤點系統，包括：一記憶體，用以儲存一全局辨識模組以及一局部辨識模組；以及一處理器，耦接該記憶體，並且用以將一手術器械影像輸入至該全局辨識模組以及該局部辨識模組，其中該全局辨識模組用以輸出對應於該手術器械影像的多個全局影像特徵，並且該局部辨識模組用以輸出對應於該手術器械影像的一末端器械影像的多個局部影像特徵，其中該記憶體更用以儲存一部件辨識模組，並且該處理器更用以將該手術器械影像輸入至該部件辨識模組，其中該部件辨識模組用以輸出對應於該手術器械影像的多個部件影像的多個部件影像特徵，並且該處理器依據該些全局影像特徵、該些局部影像特徵以及該些部件影像特徵判斷該手術器械影像的一手術器械類型，其中該部件辨識模組用以對該手術器械影像進行快速卷積神經網路運算，以從該手術器械影像決定該些部件影像，其中該部件辨識模組還用以對該些部件影像進行部件分析，以決定對應於該些部件影像的該些部件影像特徵。 A surgical instrument inventory system, comprising: a memory for storing a global recognition module and a local recognition module; and a processor coupled to the memory and used for inputting a surgical instrument image into the global The recognition module and the local recognition module, wherein the global recognition module is used to output a plurality of global image features corresponding to the surgical instrument image, and the local recognition module is used to output an end corresponding to the surgical instrument image A plurality of partial image features of an instrument image, wherein the memory is further used to store a component recognition module, and the processor is further used to input the surgical instrument image to the component recognition module, wherein the component recognition module uses To output a plurality of component image features corresponding to the plurality of component images of the surgical instrument image, and the processor determines an operation of the surgical instrument image according to the global image features, the local image features and the component image features Device type, wherein the component recognition module is used to perform fast convolutional neural network operations on the surgical device image to determine the component images from the surgical device image, wherein the component recognition module is also used to identify the components Part analysis is performed on the images to determine features of the part images corresponding to the part images. 如請求項1所述的手術器械盤點系統，其中該全局辨識模組以及該局部辨識模組分別用以對該手術器械影像以及該末端器械影像進行卷積神經網路運算，以輸出該些全局影像特徵以 及該些局部影像特徵，其中該末端器械影像的影像尺寸小於該手術器械影像，其中該局部辨識模組還用以對該手術器械影像進行快速卷積神經網路運算，以從該手術器械影像決定該末端器械影像。 The surgical instrument inventory system as described in claim 1, wherein the global recognition module and the local recognition module are respectively used to perform convolutional neural network operations on the surgical instrument image and the terminal instrument image to output the global image features and the local image features, wherein the image size of the terminal instrument image is smaller than the surgical instrument image, wherein the local recognition module is also used to perform fast convolutional neural network operations on the surgical instrument image to obtain the surgical instrument image Determine the end-device image. 如請求項1所述的手術器械盤點系統，更包括：一投影機，耦接該處理器，並且用以於一表面投射一虛擬手術器械盤影像，其中該虛擬手術器械盤影像包括多個手術器械輪廓；以及一攝影機，耦接該處理器，並且用以朝該表面拍攝該虛擬手術器械盤，以取得一盤點影像，其中該盤點影像包括多個手術器械影像，其中該處理器從該盤點影像中對應於該些手術器械輪廓的位置來取得該些手術器械影像，並且該處理器分別辨識該些手術器械影像的該些手術器械類型，其中該記憶體更用以儲存一盤點對照表，並且該處理器依據該些手術器械類型比對該盤點對照表，以決定該盤點資訊。 The surgical instrument inventory system according to claim 1, further comprising: a projector, coupled to the processor, and used to project a virtual surgical instrument tray image on a surface, wherein the virtual surgical instrument tray image includes a plurality of surgical instruments an instrument outline; and a camera, coupled to the processor, and used to photograph the virtual surgical instrument tray toward the surface to obtain an inventory image, wherein the inventory image includes a plurality of surgical instrument images, wherein the processor obtains an inventory image from the inventory The images of the surgical instruments are acquired at positions corresponding to the outlines of the surgical instruments in the image, and the processor identifies the types of the surgical instruments in the images of the surgical instruments respectively, wherein the memory is further used to store an inventory comparison table, And the processor compares the inventory comparison table according to the surgical instrument types to determine the inventory information. 如請求項1所述的手術器械盤點系統，其中該記憶體更用以儲存一器械對照表，並且該處理器依據該些全局影像特徵、該些局部影像特徵以及該些部件影像特徵比對該器械對照表，以判斷該手術器械影像的該手術器械類型。 The surgical instrument inventory system as described in claim 1, wherein the memory is further used to store an instrument comparison table, and the processor compares the global image features, the local image features, and the component image features The instrument comparison table is used to determine the type of the surgical instrument in the surgical instrument image. 一種手術器械盤點方法，包括：將一手術器械影像輸入至一全局辨識模組，以產生對應於該 手術器械影像的多個全局影像特徵；將該手術器械影像輸入至一局部辨識模組，以產生對應於該手術器械影像的一末端器械影像的多個局部影像特徵；將該手術器械影像輸入至一部件辨識模組，以產生對應於該手術器械影像的多個部件影像的多個部件影像特徵；以及依據該些全局影像特徵、該些局部影像特徵以及該些部件影像特徵判斷該手術器械影像的一手術器械類型，其中該部件辨識模組用以對該手術器械影像進行快速卷積神經網路運算，以從該手術器械影像決定該些部件影像，其中該部件辨識模組還用以對該些部件影像進行部件分析，以決定對應於該些部件影像的該些部件影像特徵。 A surgical instrument inventory method, comprising: inputting a surgical instrument image to a global recognition module to generate A plurality of global image features of the surgical instrument image; input the surgical instrument image to a local recognition module to generate a plurality of local image features corresponding to an end instrument image of the surgical instrument image; input the surgical instrument image to A part recognition module to generate a plurality of component image features corresponding to the plurality of component images of the surgical instrument image; and judge the surgical instrument image according to the global image features, the local image features and the component image features A type of surgical instrument, wherein the component recognition module is used to perform fast convolutional neural network operations on the surgical instrument image to determine the component images from the surgical instrument image, wherein the component recognition module is also used to The component images are subjected to component analysis to determine features of the component images corresponding to the component images. 如請求項5所述的手術器械盤點方法，其中該全局辨識模組以及該局部辨識模組分別用以對該手術器械影像以及該末端器械影像進行卷積神經網路運算，以輸出該些全局影像特徵以及該些局部影像特徵，其中該末端器械影像的影像尺寸小於該手術器械影像，其中該局部辨識模組還用以對該手術器械影像進行快速卷積神經網路運算，以從該手術器械影像決定該末端器械影像。 The surgical instrument inventory method as described in claim 5, wherein the global recognition module and the local recognition module are respectively used to perform convolutional neural network operations on the surgical instrument image and the terminal instrument image to output the global Image features and the local image features, wherein the image size of the terminal instrument image is smaller than the surgical instrument image, wherein the local recognition module is also used to perform fast convolutional neural network calculations on the surgical instrument image to obtain from the surgical instrument The instrument image determines the end instrument image. 如請求項5所述的手術器械盤點方法，更包括：藉由一投影機於一表面投射一虛擬手術器械盤影像，其中該虛擬手術器械盤影像包括多個手術器械輪廓；藉由一攝影機朝該表面拍攝該虛擬手術器械盤，以取得一盤 點影像，其中該盤點影像包括多個手術器械影像；從該盤點影像中對應於該些手術器械輪廓的位置來取得該些手術器械影像；以及分別辨識該些手術器械影像的該些手術器械類型，並且依據該些手術器械類型比對一盤點對照表，以決定對應於該盤點影像的一盤點資訊。 The surgical instrument inventory method as described in claim 5, further comprising: using a projector to project a virtual surgical instrument tray image on a surface, wherein the virtual surgical instrument tray image includes a plurality of surgical instrument outlines; The surface photographs the virtual surgical instrument tray to obtain a tray A point image, wherein the inventory image includes a plurality of surgical instrument images; obtain the surgical instrument images from positions corresponding to the contours of the surgical instrument in the inventory image; and identify the surgical instrument types of the surgical instrument images respectively , and compare with an inventory comparison table according to the surgical instrument types to determine an inventory information corresponding to the inventory image. 如請求項5所述的手術器械盤點方法，其中判斷該手術器械影像的該手術器械類型的步驟包括：依據該些全局影像特徵、該些全局影像特徵以及該些部件影像特徵比對一器械對照表，以判斷該手術器械影像的該手術器械類型。 The surgical instrument inventory method as described in claim 5, wherein the step of judging the surgical instrument type of the surgical instrument image includes: comparing an instrument comparison based on the global image features, the global image features, and the component image features table to determine the surgical instrument type of the surgical instrument image.
TW109140489A 2020-11-19 2020-11-19 Surgical instrument inventory system and surgical instrument inventory method TWI787672B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
TW109140489A TWI787672B (en) 2020-11-19 2020-11-19 Surgical instrument inventory system and surgical instrument inventory method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
TW109140489A TWI787672B (en) 2020-11-19 2020-11-19 Surgical instrument inventory system and surgical instrument inventory method

Publications (2)

Publication Number Publication Date
TW202221727A TW202221727A (en) 2022-06-01
TWI787672B true TWI787672B (en) 2022-12-21

Family

ID=83062245

Family Applications (1)

Application Number Title Priority Date Filing Date
TW109140489A TWI787672B (en) 2020-11-19 2020-11-19 Surgical instrument inventory system and surgical instrument inventory method

Country Status (1)

Country Link
TW (1) TWI787672B (en)

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
TW201641071A (en) * 2015-05-20 2016-12-01 國立交通大學 Method and system for recognizing multiple instruments during minimally invasive surgery
US20200237450A1 (en) * 2017-04-28 2020-07-30 Medtronic Navigation, Inc. Automatic Identification Of Instruments

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
TW201641071A (en) * 2015-05-20 2016-12-01 國立交通大學 Method and system for recognizing multiple instruments during minimally invasive surgery
US20200237450A1 (en) * 2017-04-28 2020-07-30 Medtronic Navigation, Inc. Automatic Identification Of Instruments

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
期刊 Lavado, Diana Martins. Sorting Surgical Tools from a Clustered Tray-Object Detection and Occlusion Reasoning. Diss. non non Universidade de Coimbra 2018 全文 *

Also Published As

Publication number Publication date
TW202221727A (en) 2022-06-01

Similar Documents

Publication Publication Date Title
CN108710847B (en) Scene recognition method and device and electronic equipment
CN105164700B (en) Detecting objects in visual data using a probabilistic model
WO2018221599A1 (en) Surgical instrument detection system and computer program
CN109711472B (en) Training data generation method and device
JP2022083768A (en) Surgical instrument inspection system and surgical instrument inspection method
US20080240518A1 (en) Apparatus and method of image capture
US11461997B2 (en) Matching method and apparatus, electronic device, computer-readable storage medium, and computer program
JP2009157767A (en) Face image recognition apparatus, face image recognition method, face image recognition program, and recording medium recording this program
CN109284700B (en) Method, storage medium, device and system for detecting multiple faces in image
CN112906494B (en) Face capturing method and device, electronic equipment and storage medium
CN109815823B (en) Data processing method and related product
CN111626105A (en) Attitude estimation method and device and electronic equipment
CN108875506B (en) Face shape point tracking method, device and system and storage medium
US20220164576A1 (en) Surgical instrument inventory system and surgical instrument inventory method
CN110991357A (en) Answer matching method and device and electronic equipment
TWI787672B (en) Surgical instrument inventory system and surgical instrument inventory method
CN108764040A (en) A kind of image detecting method, terminal and computer storage media
JP2008123360A (en) Device, method, and program for extracting/determining human body specific area
JP2018124798A (en) Image search device and image search program
JP7187377B2 (en) OBJECT INFORMATION REGISTRATION DEVICE AND OBJECT INFORMATION REGISTRATION METHOD
JP2015203680A (en) Information processing device, method, and program
JP2008210140A (en) Information extraction method, registration device, collation device, and program
JP2000331019A (en) Method and device for indexing aspect image and recording medium with aspect image indexing program recorded
CN113343948A (en) Satisfaction degree analysis method and system and computer storage medium thereof
JP6175904B2 (en) Verification target extraction system, verification target extraction method, verification target extraction program