CN115376676A

CN115376676A - Surgical instrument adjustment method, surgical system, and computer-readable storage medium

Info

Publication number: CN115376676A
Application number: CN202211093760.2A
Authority: CN
Inventors: 不公告发明人
Original assignee: Shanghai Microport Medbot Group Co Ltd
Current assignee: Shanghai Microport Medbot Group Co Ltd
Priority date: 2022-09-08
Filing date: 2022-09-08
Publication date: 2022-11-22

Abstract

The present specification provides a method of adjusting a surgical instrument, a surgical system, and a computer-readable storage medium. Based on the method, when the user operates the target surgical instrument to perform the surgical operation, firstly, detecting whether preset surgical instrument adjusting conditions are met or not at present; under the condition that the preset surgical instrument adjusting condition is met at present, acquiring the current pose data of the target surgical instrument and a target three-dimensional grid map carrying the occupation probability in the current time period; generating an adjusting path of the target surgical instrument according to the current pose data of the target surgical instrument and the target three-dimensional grid map of the current time period; based on the adjustment path, the target surgical instrument is moved to within a preset range of the endoscope field of view. An adjustment path with higher safety can be efficiently and accurately generated by generating and utilizing the target three-dimensional grid map of the current time period; and based on this the path is adjusted so that the path is adjusted, the targeted surgical instrument can be safely moved back within the preset range of the endoscope's field of view.

Description

The adjusting method of the surgical instrument surgical system and computer-readable storage medium

Technical Field

The present disclosure relates to medical devices, and more particularly, to a method for adjusting a surgical device, a surgical system, and a computer-readable storage medium.

Background

During a surgical operation performed on a patient by a surgeon using a device such as a surgeon console to manipulate a surgical instrument, it is often the case that the surgical instrument is moved out of the field of view of the camera. At this point, the surgeon user is typically required to move the surgical instrument back into the field of view of the camera in order to continue with the surgical procedure normally.

However, based on the existing method, most of the doctor users need to move the surgical instrument back to the visual field range of the camera depending on personal experience, errors are easy to occur in the moving process, the surgical instrument may collide with the tissue and organ of the patient, the patient is injured, and the safety of the surgical operation process is affected.

In view of the above problems, no effective solution has been proposed.

Disclosure of Invention

The present specification provides a method for adjusting a surgical instrument, a surgical system, and a computer-readable storage medium, which can automatically generate an adjustment path with high safety efficiently and accurately by generating and using a target three-dimensional grid map of a current time period; then, based on the adjusting path, the target surgical instrument can be safely moved back to the preset range of the endoscope visual field, the operation risk when moving surgical instruments in the operation process is effectively reduced, and the operation experience of a user is improved.

The present specification provides a method of adjusting a surgical instrument, comprising: detecting whether or not current satisfy preset the surgical instrument adjustment condition of (1); in the case where it is determined that the preset surgical instrument adjustment condition is currently satisfied, acquiring current pose data of a target surgical instrument and a target three-dimensional grid map of a current time period; the target three-dimensional grid map of the current time period also carries occupation probability; generating an adjusting path of the target surgical instrument according to the current pose data of the target surgical instrument and the target three-dimensional grid map of the current time period; based on the adjusted path of the target surgical instrument, the target surgical instrument is moved to be within a preset range of the endoscope field of view.

The present specification also provides a surgical system comprising at least: a display and an adjusting device of the surgical instrument; the adjusting device of the surgical instrument is used for executing and realizing the steps of the adjusting method of the surgical instrument under the condition that the current condition of meeting the preset adjusting condition of the surgical instrument is determined, and obtaining the adjusting path of the target surgical instrument; the display is used to show the user the adjusted path of the target surgical instrument, such that the targeted surgical instrument is moved within a predetermined range of the endoscope field of view.

The present specification also provides a computer readable storage medium having stored thereon computer instructions which, when executed, perform the steps of: detecting whether preset surgical instrument adjusting conditions are met or not at present; acquiring current pose data of a target surgical instrument and a target three-dimensional grid map of a current time period under the condition that the current condition of the preset surgical instrument is met; the target three-dimensional grid map of the current time period also carries occupation probability; generating an adjusting path of the target surgical instrument according to the current pose data of the target surgical instrument and the target three-dimensional grid map of the current time period; based on the target operation the path of adjustment of the instrument is such that, such that the targeted surgical instrument is moved within a preset range of the endoscope's field of view.

Based on the adjustment method of the surgical instrument, the surgical system and the computer-readable storage medium provided by the specification, when a user operates a target surgical instrument to perform surgical operation, whether preset surgical instrument adjustment conditions are met or not can be detected at present; under the condition that the preset surgical instrument adjustment condition is met at present, acquiring current pose data of a target surgical instrument and a target three-dimensional grid map carrying a current time period of occupation probability; generating an adjusting path of the target surgical instrument according to the current pose data of the target surgical instrument and the target three-dimensional grid map of the current time period; the target surgical instrument may then be moved back within the preset range of the endoscope field of view based on the adjustment path. The method can efficiently generate and utilize the target three-dimensional grid map of the current time period an adjusting path with high safety and good reference is accurately and automatically generated; based on the adjustment path, the target surgical instrument can be safely moved back to the preset range of the endoscope visual field, the target surgical instrument is prevented from colliding with the tissue and organ of a patient in the moving process, the operation risk when the surgical instrument is moved in the operation process is effectively reduced, and the operation experience of a user is improved.

In specific implementation, the image data of the current time period can be acquired in real time, and the pre-trained preset target detection model is utilized to process the current representative image in the image data of the current time period, whether the target surgical instrument is located in a preset range of an endoscope experiment or not is rapidly detected and judged, and whether preset surgical instrument adjusting conditions are met or not can be automatically determined, so that operation on the user side can be simplified, and operation experience of a user is further improved.

The image data of the current time period can be processed by utilizing a preset segmentation model, so that segmented image data can be obtained; then, the preset SLAM algorithm model is utilized to comprehensively process the image data of the current time period and the segmented image data to carry out three-dimensional scene modeling, the target three-dimensional grid map with better effect and higher precision for the current time period is constructed, and then an adjustment path with higher safety and better reference can be generated based on the target three-dimensional grid map.

Drawings

In order to more clearly illustrate the embodiments of the present description, the drawings needed for the embodiments will be briefly described below, the drawings in the following description are only some of the embodiments described in the present description, and other drawings may be obtained by those skilled in the art without inventive efforts.

FIG. 1 is provided as an example of the present specification a flow diagram of a method of adjusting a surgical instrument;

FIG. 2 is a schematic diagram of an embodiment of a medical system, in one example scenario;

FIG. 3 is a diagram of the structure of a physician's console in a medical system constituting one embodiment, in one example scenario;

FIG. 4 is a diagram of the structure of a patient table in a medical system constituting one embodiment, in one example scenario;

fig. 5 is a schematic diagram of a network structure of an initial detection model used when the method for adjusting a surgical instrument provided in the embodiment of the present specification is applied, in an example scenario;

FIG. 6 is a schematic diagram of an embodiment of labeling a sample image when the method for adjusting a surgical instrument provided in the embodiments of the present description is applied in an example scenario;

fig. 7 is a schematic diagram of an embodiment of constructing a three-dimensional grid map of a target in a current time period when the adjustment method of a surgical instrument provided by an embodiment of the present specification is applied, in a scenario example;

fig. 8 is a schematic diagram of an embodiment of a preset segmentation model used in applying the method for adjusting a surgical instrument provided in the embodiments of the present specification;

fig. 9 is a schematic diagram of an embodiment of generating an initial three-dimensional grid map of a current time period when the adjustment method of the surgical instrument provided by the embodiment of the present specification is applied, in an example scenario;

fig. 10 is a schematic diagram of an embodiment of calculating a movement distance in a manhattan distance mode by applying the adjustment method of the surgical instrument provided by the embodiment of the present specification in an example scenario;

fig. 11 is a schematic diagram of an embodiment of calculating a movement distance by using a euclidean distance mode by applying the adjusting method for a surgical instrument provided in the embodiment of the present specification in a scenario example;

FIG. 12 is a schematic diagram of the structural components of a physician's console provided in one embodiment of the present disclosure;

fig. 13 is a schematic structural component view of an adjusting device of a surgical instrument according to an embodiment of the present disclosure.

Detailed Description

In order to make those skilled in the art better understand the technical solutions in the present specification, the technical solutions in the embodiments of the present specification will be clearly and completely described below with reference to the drawings in the embodiments of the present specification, and it is obvious that the described embodiments are only a part of the embodiments of the present specification, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given in the present specification without inventive step, are intended to fall within the scope of the present disclosure.

Referring to fig. 1, an embodiment of the present disclosure provides a method for adjusting a surgical instrument, where the method may be implemented by the following steps:

s101: detecting whether preset adjusting conditions of the surgical instruments are met or not at present;

s102: under the condition that the preset surgical instrument adjusting condition is met at present, acquiring current pose data of a target surgical instrument and a target three-dimensional grid map of the current time period; the target three-dimensional grid map of the current time period also carries occupation probability;

s103: generating an adjusting path of the target surgical instrument according to the current pose data of the target surgical instrument and the target three-dimensional grid map of the current time period;

s104: based on the adjusted path of the target surgical instrument, the target surgical instrument is moved to be within a preset range of the endoscope field of view.

In some embodiments, the method for adjusting the surgical instrument may be specifically applied to a side of a doctor console in a medical system. Referring to fig. 2 and 3, the physician's console faces a user (e.g., a physician user). A doctor user can operate a patient operating table through the doctor control console to perform operation on the patient.

It should be noted that, in particular, the method for adjusting a surgical instrument may be applied to other medical devices such as a surgical robot according to a specific application scenario and a processing requirement, in addition to a doctor console. The present specification is not limited thereto.

Specifically, referring to FIG. 3, the physician's console can include at least a main operating end and a display. The main operation end is associated with a slave mechanical arm of an operating table of a patient and is used for controlling equipment such as a surgical instrument or an endoscope and the like mounted on the slave mechanical arm to execute corresponding operations. The display is used for displaying image data in the operation process to a doctor user. In addition, the doctor console may be provided with an armrest for support, a pedal panel for switching the operation of the slave arm, and the like.

Referring to fig. 2, the physician's console may be connected to the patient table in the medical system by wire or wirelessly. Wherein, the patient table faces the side of the patient to be operated on.

Specifically, referring to fig. 4, at least a plurality of slave arms are arranged on the patient operating table. For example, the slave robot arms 1, 2, 3, 4, and 5. In addition to this, the present invention is, the patient operating table can also comprise a base, a top plate and other structures.

Further, the robot arm can be used to mount thereon related equipment such as a surgical instrument and an endoscope, which are required to be used in a surgery. For example, a surgical instrument can be mounted on the robot arm 1, and an endoscope can be mounted on the robot arm 5.

The endoscope may specifically include a binocular endoscope and the like. The surgical instrument may specifically include any one of the following: duckbill graspers, ratcheted graspers, powerful duckbill graspers, and the like.

In this specification, the endoscope is mainly used to acquire image data during a surgical procedure. In specific implementation, according to specific conditions and processing requirements, other devices such as a camera and a camera can be used to collect image data in the operation process instead of an endoscope.

In addition, as shown in fig. 2, the medical system may further include an image platform, a surgical instrument table, an anesthesia trolley, and other devices.

Specifically, during a surgical procedure, a user may manipulate an endoscope mounted on the robotic arm through the main manipulation end of the physician's console into the surgical environment (e.g., in the abdominal cavity of the patient) collects image data during the procedure in real time or at regular times and presents the collected image data to the user via the display.

The user can operate and control the surgical instrument mounted on the slave mechanical arm to execute specific surgical operation through the main operation end in the visual field range of the endoscope according to the image data displayed by the display.

Referring to fig. 2, the patient table may also be connected to an imaging platform. The image platform at least comprises a display and an image processing device. The image platform is also connected with an endoscope of a slave mechanical arm mounted on an operating table of a patient. And then the image data in the operation process can be collected through the endoscope, the image data is correspondingly processed by the image processing device, and the processed image data is displayed to a doctor user or an assistant of the doctor user through the display. In addition, the processed image data can be transmitted to the doctor console and displayed to the doctor user through a display of the doctor console.

Further, the medical system may further include a surgical instrument table, a patient bed, and an anesthesia trolley. Wherein, the surgical instrument table is used for placing surgical instruments and other devices required by the operation; the anesthesia trolley is used for realizing anesthesia and physical sign monitoring in the operation process.

Based on the adjusting method of the surgical instrument provided by the specification, when a user starts to start the doctor console, the slave mechanical arm of the patient operating table can be controlled firstly, so that the target surgical instrument is positioned in the preset range of the endoscope visual field; and unifying the position coordinates of the target surgical instrument with the position coordinates of the endoscope.

When a user uses the doctor console to control a target surgical instrument, which is mounted on an operating table of a patient and is driven by a mechanical arm, to perform surgical operation, the doctor console can collect the image data in real time or at regular time and detect and judge whether preset surgical machine adjustment conditions are met or not at present according to the image data. For example, it is detected whether the current target surgical instrument moves out of the preset range of the endoscope field of view, which in turn results in that the user cannot clearly observe the position and posture of the target surgical instrument through the image data displayed on the display. Meanwhile, in the process that a user uses the doctor console to perform surgery operation, the doctor console can also perform three-dimensional scene modeling according to image data acquired in real time or at regular time, and update the used three-dimensional grid map in time.

In the case where it is determined that the preset surgical instrument adjustment condition is currently satisfied, interaction may be performed with the surgeon user first. Under the condition of confirmation of a doctor user, the current pose data of the target surgical instrument and the target three-dimensional grid map of the current time period can be obtained by utilizing the corresponding algorithm model. The target three-dimensional grid map of the current time period also carries the occupation probability of the current time period of each grid point. The occupancy probability is used to characterize the probability value that the current time segment of the grid point is occupied by a tissue organ or the like, resulting in the surgical instrument being unable to move through.

Then, the doctor console can generate an adjustment path of the target surgical instrument with higher safety through path planning based on the target three-dimensional grid map of the current time period according to the current pose data of the target surgical instrument.

Further, the doctor console may present the adjustment path to the user via a display; meanwhile, corresponding guide prompt voice can be played through a voice player arranged on a doctor console, or corresponding guide prompt sentences can be displayed through a display, so that a doctor user can be guided to control the target surgical instrument to move back to the preset range of the visual field of the endoscope through the main operation end according to the adjustment path.

After the target surgical instrument is moved back to the preset range of the endoscope visual field, the endoscope can acquire the image data containing the clear image of the target surgical instrument again, and the image data is displayed to the user through the display. The user can continue to operate the target surgical instrument through the main operation end according to the image data collected by the display to perform surgical operation.

In some embodiments, the detecting whether the preset adjustment condition of the surgical instrument is currently met may include the following steps:

s1: acquiring image data of a current time period; the image data of the current time period comprises multi-frame images;

s2: screening out a current representative image from the image data of the current time period;

s3: detecting the current representative image by using a preset target detection model to obtain a target detection result;

s4: according to the result of the detection of the target, detecting whether a target surgical instrument is currently located within a preset range of the endoscope field of view;

s5: and under the condition that the target surgical instrument is determined not to be located in the preset range of the endoscope visual field at present, determining that the preset surgical instrument adjusting condition is met at present.

In specific implementation, the slave mechanical arm with the endoscope (or shooting equipment such as a camera and a camera) can be controlled to extend into the surgical environment, and the endoscope is controlled to acquire video data of the surgical process in the surgical environment in real time or at regular intervals of fixed time periods and to serve as image data of the current time period.

The video data of the current time period may specifically include multiple frames of images arranged in succession. Each frame of image corresponds to an acquisition time point in the current time period.

In some embodiments, the filtering out the current representative image from the image data of the current time period may include: respectively carrying out edge contour feature detection on multiple frames of images contained in the image data of the current time period to obtain edge contour feature detection results of the multiple frames of images; and screening out an image with edge contour characteristics meeting the requirements from the multi-frame images according to the edge contour characteristic detection results of the multi-frame images as a current representative image.

In specific implementation, edge pixel points of each frame of image in a plurality of frames of images can be extracted first; respectively carrying out edge contour feature detection on each frame image according to edge pixel points of each frame image so as to determine the definition of the edge contour of each frame image as a corresponding edge contour feature detection result; and further according to the detection result of the edge profile characteristic, and screening out an image with relatively highest edge contour definition from the multiple frames of images as a current representative image with satisfactory edge contour characteristics.

Furthermore, whether preset surgical instrument adjustment conditions are met or not can be detected by only processing the frame of image of the current representative image subsequently, and multiple frames of images contained in the image data of the current time period do not need to be processed, so that the data processing amount can be effectively reduced, and the detection efficiency is improved.

In some embodiments, the screening out the current representative image from the image data of the current time period may further include: determining the position coordinates of the surgical instrument and the endoscope corresponding to each frame of image in the multi-frame image; determining the moving trend of the target surgical instrument relative to the endoscope experiment according to the position coordinates of the surgical instrument and the position coordinates of the endoscope corresponding to each frame of image; and then according to the moving trend, screening out a frame image which is representative in the current time period and has relatively high edge contour definition from the multi-frame images as the current representative image.

In some embodiments, the preset target detection model may be specifically understood as a pre-trained algorithm model that is capable of detecting and identifying at least a surgical instrument from an image. The preset target detection model may be specifically an algorithm model based on yolo V5.

In specific implementation, the current representative image may be input as a model, input into a preset target detection model, and run the model to obtain a corresponding target detection result. According to the target detection result, whether a target surgical instrument exists in the current representative image is determined; under the condition that the target surgical instrument does not exist in the current representative image, the current condition that a preset surgical instrument adjusting condition is met can be determined; in the case where it is determined that the target surgical instrument is present in the current representative image, it may be further determined whether the target surgical instrument is currently located within a preset range of the endoscope field of view. In the event that it is determined that the target surgical instrument is not currently within the preset range of the endoscope field of view, it may be determined that the preset surgical instrument adjustment condition is currently satisfied.

The preset range of the endoscope visual field may be a circular area range in which a center pixel point in the current representative image is a circle center and an appointed length is a radius. Of course, the above-listed predetermined ranges of the field of view of the endoscope are merely illustrative. In specific implementation, according to specific situations and precision requirements, other types of area ranges in which the surgical instrument can be clearly identified in the endoscope field of view may be used as the preset range of the endoscope field of view.

Before specific implementation, a preset target detection model can be obtained by training in the following way:

s1: acquiring a sample image, and labeling the sample image to obtain a labeled sample image;

s2 the method comprises the following steps: constructing an initial detection model based on yolo V5;

s3 the method comprises the following steps: and training the initial detection model by using the marked sample image to obtain a preset target detection model meeting the requirements.

The yolo V5 is understood to be a target detection algorithm with relatively higher speed and higher accuracy than yolo V4.

In particular, referring to FIG. 5, an initial test model may be constructed based on yolo V5. Where Conv denotes convolution operation, NMS denotes non-maximum suppression, FC denotes full connectivity, relu denotes nonlinear mapping.

In particular, and with reference to FIG. 6, an image containing a tissue organ and/or surgical instrument may be acquired as a sample image. During specific marking, the tissue organ and/or the surgical instrument in the image can be identified by using the rectangular frame, and whether the tissue organ or the surgical instrument is framed by the rectangular frame is identified at the peripheral position of the rectangular frame.

In some embodiments, upon determining that the target surgical instrument is not currently within the preset range of the endoscope field of view, when it is determined that the preset surgical instrument adjustment condition is currently satisfied, the physician console may first interact with the user to determine whether the target surgical instrument needs to be adjusted.

In particular, the physician's console may present to the user, via the display, a prompt indicating whether the target surgical instrument is currently out of view, please indicate that an adjustment is needed. And after receiving a confirmation instruction fed back by the user aiming at the prompt information, the doctor console can trigger subsequent data processing. In addition, the doctor console can receive a position point designated by the user as a target position of the target surgical instrument while receiving a confirmation instruction fed back by the user.

In some embodiments, the detecting whether the current adjustment condition is met may further include: detecting whether an adjusting instruction about a target surgical instrument actively initiated by a user is received; and under the condition that the adjustment instruction initiated by the user is determined to be received, determining that a preset adjustment condition is met.

In some embodiments, the obtaining of the current pose data of the target surgical instrument may include the following steps:

s1: acquiring current state data of a target slave mechanical arm and current observation data of the target slave mechanical arm determined based on an endoscope; wherein the target is carried with the target surgical machine from the mechanical arm;

s2: determining current pose data of the target slave mechanical arm according to the current state data and the current observation data of the target slave mechanical arm;

s3: and determining the current pose data of the target surgical instrument according to the current pose data of the target slave mechanical arm.

The current pose data of the target surgical instrument may specifically include current position coordinates and pose data of the target surgical instrument.

The current state data of the target slave mechanical arm may specifically include encoder parameters of the target slave mechanical arm. The observation data may be specifically determined based on the position coordinates of the endoscope and the target surgical instrument in the image data of the current time zone.

In some embodiments, the determining, according to the current state data and the current observation data of the target slave mechanical arm, the current pose data of the target slave mechanical arm may include: and performing fusion processing on the current state data and the current observation data of the target slave mechanical arm by using an extended Kalman filtering algorithm to obtain the current pose data of the target slave mechanical arm.

In some embodiments, the acquiring current pose data of the target surgical instrument may further include, in specific implementation: acquiring image data of the previous time period; and determining the current pose data of the target surgical instrument according to the image data of the previous time period, the image data of the current time period and the initial three-dimensional grid map of the current time period.

In some embodiments, the acquiring current pose data of the target surgical instrument may further include, in specific implementation: determining first pose data of the target surgical instrument according to the image data of the previous time period, the image data of the current time period and the initial three-dimensional grid map of the current time period; and determining the current pose data of the target surgical instrument according to the first pose data of the target surgical instrument and the current pose data of the target slave mechanical arm.

In some embodiments, referring to fig. 7, the obtaining of the target three-dimensional grid map of the current time period may include the following steps:

s1: processing the image data of the current time period by using a preset segmentation model to obtain segmented image data;

s2: constructing an initial three-dimensional grid map of the current time period according to the image data of the current time period and the segmented image data;

s3: determining the occupation probability of each grid point in the current time period according to the initial three-dimensional grid map in the current time period;

s4: and mapping the occupation probability of each grid point in the current time period to the initial three-dimensional grid map in the current time period to obtain the target three-dimensional grid map in the current time period.

The preset segmentation model may be specifically understood as a pre-trained network model that can recognize an image area type of an image area included in an input image through image semantics, and can perform image segmentation on the input image according to the recognized image area type.

The image region type may specifically include at least one of the following: surgical instrument regions, tissue organ regions, passable regions (or background regions), etc.

In specific implementation, referring to fig. 8, a plurality of images included in the image data of the current time period may be respectively input to a preset segmentation model for processing, the preset segmentation model is run, and segmented images corresponding to image regions of different image region types in each image are output as the segmented image data. The divided video data includes a plurality of divided images.

Specifically, referring to fig. 8, the segmented image data includes a plurality of segmented images; the segmented image may specifically comprise image regions segmented and marked with image region types.

Before specific implementation, the preset segmentation model can be obtained by training in the following way:

s1: acquiring sample video data; the sample video data comprises a plurality of sample images arranged according to a time sequence;

s2: marking out image areas in each sample image and image area types of the image areas to obtain marked sample video data;

s3: and training an initial segmentation model by using the marked sample video data to obtain a preset segmentation model meeting the requirements.

In the specific labeling, the image areas of different image area types can be labeled by using different gray values according to corresponding labeling rules.

During specific training, an initial segmentation model based on SegNet can be constructed. The SegNet may specifically refer to a deep full-convolution neural network structure for semantic segmentation, and the core of the SegNet is composed of an encoder network, a corresponding decoder network, and a pixel-level classification layer. Accordingly, referring to fig. 8, the preset segmentation model at least includes: pooling indices (posing indices), and convolutional encoders and decoders (convolutional encoder-decoders), among others.

In some embodiments, the constructing an initial three-dimensional grid map of the current time period according to the image data of the current time period and the segmented image data may include, in specific implementation: and based on a preset SLAM algorithm model, performing three-dimensional scene modeling by processing the image data of the current time period and the segmented image data to obtain an initial three-dimensional grid map of the current time period.

The SLAM algorithm may specifically be an algorithm that gradually builds a map in an unknown environment, and determines its own position according to the map, thereby further positioning. Specifically, for example, algorithms such as Gmiping, karto, hector, cartographer, and the like are used.

In some embodiments, the preset SLAM algorithm model may include at least: trace threads (e.g., tracking), local mapping threads (e.g., local mapping), loop optimization threads (e.g., loop & map merging), etc.

In some embodiments, referring to fig. 9, in the above-mentioned three-dimensional scene modeling by processing the image data of the current time period and the segmented image data based on the preset SLAM algorithm model to obtain the initial three-dimensional grid map of the current time period, when implemented, the following contents may be included:

s1: extracting by utilizing a tracking thread, and screening out an attention image in the current time period through tracking matching according to the image data in the current time period and the image characteristics in the segmented image data;

s2: filtering repeated images in the concerned images in the current time period and repeated key points in the images by using a local map construction thread; constructing a local map of the current time period based on the processed attention image;

s3: and fusing the local map of the current time period and the initial three-dimensional grid map of the previous time period by utilizing a loop optimization thread to obtain the initial three-dimensional grid map of the current time period.

Specifically, when the tracking thread is used for processing, image features (for example, edge pixels) of each image in the image data of the current time period and image features of each image in the segmented image data of the current time period may be extracted first. Specifically, the orb operator may be used to extract the image features.

Secondly, carrying out pose initialization and repositioning by using image characteristics of the image acquired at the same time point and the segmented image, carrying out characteristic matching on the image characteristics of the image acquired at the time point and the segmented image and the image characteristics of the image acquired at the adjacent time point and the segmented image, and carrying out local map information tracking on the image acquired at the adjacent time point and the segmented image; and screening out a plurality of key frame images and images obtained after key frame segmentation as the concerned images of the current time period according to the tracking result, and providing the concerned images to a local map construction thread for further processing.

When the local map is specifically utilized to construct thread processing, the concerned images in the current time period can be added firstly; detecting whether repeated key points exist in the concerned image in the current time period or not, and rejecting the repeated key points; then, local Bundle Adjustment (local Bundle Adjustment) is carried out on the concerned image in the current time period after the repeated key points are removed, and the image with the repetition is deleted in the optimization process, so that the concerned image after processing is obtained; and then constructing a local map of the current time period based on the processed attention image, and providing the local map of the current time period for a loop optimization thread for further processing.

When the loop optimization thread is used for processing, firstly, the concerned image in the current time period is matched with the concerned image in the previous time period to obtain a corresponding matching result; according to the matching result, performing data fusion on the local map of the current time period and the initial three-dimensional grid map of the previous time period to obtain a fused three-dimensional grid map; and then carrying out global optimization on the fused three-dimensional grid map, and outputting the optimized three-dimensional grid map as an initial three-dimensional grid map of the current time period.

In some embodiments, the determining, according to the initial three-dimensional grid map of the current time period, an occupation probability of each grid point of the current time period may include, in specific implementation: and according to the initial three-dimensional grid map of the current time period and the occupation probability of each grid point of the previous time period, calculating the binary Bayesian filtering confidence of each grid point in the initial three-dimensional grid map of the current time period to obtain the occupation probability of each grid point of the current time period.

Wherein, the occupation probability (which can be denoted as l) of each grid point of the last time period _t-1 ) Can be determined according to the target three-dimensional grid map of the last time period. The initial occupation probability of each grid point may be uniformly set to 0.5 at the time of initialization.

In specific implementation, an observed value (which may be denoted as z) of each grid point in the current time period may be determined according to the initial three-dimensional grid map in the current time period _t ) (ii) a And determining a reference probability value according to the observed value. Specifically, the image area type of the image area where the grid point is located may be determined according to the gray value of the image area where each grid point is located; and determining the observation value of the grid point in the current time period according to the type of the image area. For example, the type of the region of the image region where the grid point x is located is a passable region, and accordingly, the observation value may be 0. For another example, the region type of the image region where the grid point x is located is a tissue and organ region, and accordingly, the observation value may be 1. Further, the corresponding reference probability value can be determined according to the observation value.

Then, the confidence of the binary bayesian filter of each grid point in the current time period may be calculated according to the reference probability value of each grid point in the current time period and the occupation probability of each grid point in the previous time period, as the occupation probability of each grid point in the current time period.

Specifically, the binary bayesian filtering confidence of the grid points can be calculated according to the following formula:

wherein l _t Occupation probability of grid point x for the current time segment, l _t-1 Is the probability of occupation of grid point x in the last time period, x is the number of grid points, p (x) is the initial probability of occupation of grid point x (0.5), z _t An observed value of a grid point x of the current time period; p (x | z) _t ) Is the reference probability value of grid point x of the current time segment.

In some embodiments, after obtaining the occupation probability of each grid point in the current time period, when the method is implemented, the method may further include: and checking the occupation probability of each grid point in the current time period by using the check threshold range of the image area to which each grid point in the current time period belongs.

The checking threshold range of the image area is determined according to the image area type of the image area. For example, for a tissue organ region, the corresponding verification threshold range may be: [0.4,1]; for a passable region, the corresponding check threshold range may be [0,0.6]. In specific implementation, the calibration threshold ranges corresponding to different image area types can be determined by clustering different image area types according to the sample image.

Specifically, whether the numerical value of the occupation probability of each grid point in the current time period belongs to the corresponding check threshold range or not can be detected in a traversal mode; when the numerical value of the occupation probability of the grid point is determined to be in the corresponding check threshold range, the occupation probability of the grid point can be judged to be accurate and can be reserved; in contrast, when it is determined that the numerical value of the occupation probability of a grid point does not fall within the corresponding check threshold range, it may be determined that there may be an error in the occupation probability of the grid point, and the occupation probability may be recalculated only for the grid point.

Based on the embodiment, the occupation probability of each grid point in the three-dimensional grid points can be updated in each time period, and errors caused by changes of the occupation conditions of the grid points due to neglect of the fact that the patient turns over the body or an external force touches and moves the body of the patient when a path is planned in the follow-up process are effectively avoided.

In some embodiments, the mapping the occupation probability of each grid point to the initial three-dimensional grid map of the current time period may include: and binding the occupation probability of each grid point with the position coordinates of each grid point, and then storing to obtain the target three-dimensional grid map of the current time period.

In some embodiments, the generating an adjustment path of the target surgical instrument according to the current pose data of the target surgical instrument and the target three-dimensional grid map of the current time period may include the following steps:

s1: determining a path starting point according to the current pose data of the target surgical instrument; determining a proper position point as a path terminal point;

s2: and generating a movement path meeting the requirement as an adjusting path of the target surgical instrument according to the path starting point, the path end point and the target three-dimensional grid map of the current time period based on a preset search algorithm.

In specific implementation, when the path starting point is determined, the position point indicated by the corresponding current position coordinate can be determined as the path starting point according to the current pose data of the target surgical instrument.

When the path end point is determined, the corresponding path end point can be determined according to the target position appointed by the user in the process of interaction between the doctor console and the user; or selecting a central point of the current endoscope visual field or a position point which is closest to the current position coordinate of the target surgical instrument in the preset range of the current endoscope visual field as a path terminal point according to a default rule.

In some embodiments, the predetermined search algorithm may specifically include a-Star (a ×) algorithm. The a-Star algorithm may be a search algorithm that is more effective and direct in solving the shortest path in a static road network, and based on the search algorithm, the closer the estimated distance value is to the actual distance value, the faster the search speed is.

In some embodiments, based on a preset search algorithm, a moving path meeting requirements is generated according to a path starting point, a path ending point and a target three-dimensional grid map of a current time period, and in specific implementation, the following contents may be included: determining the current mobile node in the moving path according to the following modes:

s1: traversing the current candidate node set to determine a first distance cost between each current candidate node in the current candidate node set and the starting point of the path and a second distance cost between each current candidate node in the current candidate node set and the ending point of the path;

s2: and screening out the candidate nodes meeting the requirements from the current candidate node set according to the first distance cost and the second distance cost of each current candidate node, and taking the candidate nodes as the current mobile nodes.

In specific implementation, during initialization, a plurality of possible grid points to be traversed can be screened out as candidate nodes according to a target three-dimensional grid map to construct a candidate node set which can be marked as open _ set; meanwhile, an empty set can be constructed as an eliminated node set and can be marked as close _ set for storing traversed candidate nodes.

In addition, comprehensive cost function data for traversing the candidate nodes and screening the mobile nodes meeting the requirements can be constructed according to a preset search algorithm.

Specifically, the corresponding comprehensive cost function can be constructed according to the following equation: f (n) = g (n) + h (n).

Wherein f (n) represents the composite cost of the candidate node n, n represents the number of the candidate node, g (n) represents the first distance cost of the candidate node n, and h (n) represents the second distance cost of the candidate node n.

Further, the first distance cost may be specifically determined according to a moving distance between the candidate node n and the path starting point, and an occupation probability of the candidate node n. The second distance cost may be specifically determined according to a moving distance between the candidate node n and the path end point, and an occupation probability of the candidate node n. In specific implementation, the larger the moving distance is, the larger the occupation probability is, and the larger the corresponding distance cost is; conversely, the smaller the moving distance and the smaller the occupation probability, the smaller the corresponding distance cost.

Specifically, at the beginning, calculation may be started from the starting point of the path, and the comprehensive cost of the starting point of the path is determined to be 0; and determines the path starting point as the current mobile node.

Then, when any one current mobile node is determined, traversing the current candidate node set according to the previous mobile node to find a plurality of matched candidate nodes as the current candidate nodes; then, by utilizing a cost synthesis function, respectively calculating a first distance cost between each current candidate node and the path starting point and a second distance cost between each current candidate node and the path terminal point, and calculating the synthesis cost of each current candidate node; and screening out the candidate node with the minimum comprehensive cost from the current candidate nodes as the current mobile node. And deleting other candidate nodes except the current mobile node in the current candidate nodes from the candidate node set, and updating the candidate node set.

After determining the current mobile node in the above manner, it may also be detected whether the current mobile node is a path end point. Specifically, the distance difference between two location points may be calculated according to the current location coordinates of the mobile node and the location coordinates of the path end point, and it may be detected whether the distance difference is less than or equal to a preset distance difference threshold.

When the distance difference is determined to be smaller than or equal to the preset distance difference threshold, the current mobile node can be determined as the path end point, and then the path start point and other mobile nodes can be combined and connected in sequence to obtain the mobile path meeting the requirement.

When it is determined that the distance difference is greater than the preset distance threshold, the above process may be repeated, and the determination of the next mobile node may be continued until the distance difference between the determined mobile node and the path end point is less than or equal to the preset distance difference threshold.

In some embodiments, when the moving distance required to be used for calculating the first distance cost and the second distance cost is specifically calculated, the moving mode of the target surgical instrument can be determined and distinguished; and calculating the moving distance by adopting the matched distance mode according to the moving mode of the target surgical instrument.

Specifically, when it is determined that the movement mode of the target surgical instrument supports only the movement in the up-down, left-right, and left directions, the manhattan distance mode may be selected to calculate the movement distance between two position points. As can be seen in fig. 10. In this way, the advantage of Manhattan distance can be utilized to perform relatively wider search in the target three-dimensional grid map.

When the movement mode of the target surgical instrument is determined to support movement in any direction, the Euclidean distance mode can be selected to calculate the movement distance between two position points. As can be seen in fig. 11. Thus, the advantage that the Euclidean distance is suitable for large calculation amount can be utilized, the whole calculation amount is reduced, and the whole processing efficiency is improved.

In some embodiments, the adjusting the path based on the target surgical instrument to move the target surgical instrument to the preset range of the endoscope field of view may include the following steps: and displaying the adjustment path of the target surgical instrument to the user, and guiding the user to move the target surgical instrument to a preset range of the endoscope visual field according to the adjustment path of the target surgical instrument.

When the display is carried out specifically, a transparent or semitransparent superposition layer which is matched with the image data to be displayed and contains the adjusting path of the target surgical instrument can be generated according to the image data to be displayed currently acquired by the endoscope and the target three-dimensional grid map of the current time period; superposing the superposed layer and the image data to be displayed to obtain superposed image data containing an adjusting path; and the superposed video data can be displayed to a user through a display.

In addition, when the display is specifically carried out, a superposed layer with stereoscopic impression can be generated according to the AR technology; and the superposed layer is superposed in the image data to be displayed and then displayed to the user, so that the user can obtain relatively better interactive experience.

When the endoscope is guided, the voice player arranged on the doctor console can be used for playing related prompt audios, or related prompt sentences can be played on the display, so that a user can be guided to control the main operation end to move the target surgical instrument back to the preset range of the endoscope visual field according to the displayed moving path.

In some embodiments, the adjusting the path based on the target surgical instrument to move the target surgical instrument to a preset range of the endoscope visual field may further include: receiving a reply instruction initiated by a user aiming at the adjustment path; and responding to the return instruction, and automatically controlling the target to move the target surgical instrument to the preset visual field of the endoscope from the mechanical arm according to the adjusting path.

As can be seen from the above, based on the adjustment method of the surgical instrument provided in the embodiment of the present specification, when a user operates a target surgical instrument to perform a surgical operation, it is first detected whether a preset adjustment condition of the surgical instrument is currently satisfied; under the condition that the preset surgical instrument adjusting condition is met at present, acquiring the current pose data of the target surgical instrument and a target three-dimensional grid map carrying the occupation probability in the current time period; generating an adjusting path of the target surgical instrument according to the current pose data of the target surgical instrument and the target three-dimensional grid map of the current time period; and moving the target surgical instrument to a preset range of the endoscope visual field based on the adjusting path. An adjustment path with higher safety can be efficiently and accurately generated by generating and utilizing the target three-dimensional grid map of the current time period; and based on the adjustment path, the target surgical instrument can be safely moved back to the preset range of the endoscope visual field, the operation risk when the surgical instrument is moved in the operation process is effectively reduced, and the operation experience of a user is improved. During specific implementation, whether the target surgical instrument is currently located in a preset range of an endoscope experiment or not is detected and determined by processing the image data of the current time period through acquiring the image data of the current time period in real time and utilizing a pre-trained preset target detection model, so that whether preset surgical instrument adjustment conditions are met or not is automatically determined, the operation of a user side can be simplified, and the operation experience of the user is further improved. The image data of the current time period can be processed by utilizing a preset segmentation model, so that segmented image data can be obtained; and then, the preset SLAM algorithm model is utilized to comprehensively process the image data of the current time period and the segmented image data to carry out three-dimensional scene modeling, a target three-dimensional grid map with better effect and higher precision for the current time period is constructed, and an adjusting path with higher safety can be generated based on the target three-dimensional grid map.

Embodiments of the present disclosure also provide a surgical system (e.g., an abdominal cavity endoscope surgical system), as shown in fig. 12, including at least: a display 1201 and an adjustment device 1202 for a surgical instrument; wherein the adjusting device 1202 of the surgical instrument is configured to execute and implement the following steps if it is determined that the preset adjustment condition of the surgical instrument is currently met: acquiring current pose data of a target surgical instrument and a target three-dimensional grid map of a current time period; the target three-dimensional grid map of the current time period also carries occupation probability; generating an adjusting path of the target surgical instrument according to the current pose data of the target surgical instrument and the target three-dimensional grid map of the current time period to obtain the adjusting path of the target surgical instrument; the display 1201 is used to show the user the adjusted path of the target surgical instrument to move the target surgical instrument within a preset range of the endoscope field of view. The display 1201 and the adjustment device 1202 of the surgical instrument are connected by an internal cable.

An embodiment of the present specification further provides an electronic device, where the electronic device includes a network communication port, a processor, and a memory, and the foregoing structures are connected by an internal cable, so that each structure may perform specific data interaction.

The network communication port may be specifically configured to detect whether a preset adjustment condition of the surgical instrument is currently met.

The processor may be specifically configured to, in a case where it is determined that a preset surgical instrument adjustment condition is currently satisfied, acquire current pose data of a target surgical instrument and a target three-dimensional grid map of a current time period; the target three-dimensional grid map of the current time period also carries occupation probability; generating an adjusting path of the target surgical instrument according to the current pose data of the target surgical instrument and the target three-dimensional grid map of the current time period; based on the adjusted path of the target surgical instrument, the target surgical instrument is moved to be within a preset range of the endoscope field of view.

The memory may be specifically configured to store a corresponding instruction program.

In this embodiment, the network communication port may be a virtual port bound to different communication protocols, so that different data can be sent or received. For example, the network communication port may be a port responsible for web data communication, a port responsible for FTP data communication, or a port responsible for mail data communication. In addition, the network communication port can also be a communication interface or a communication chip of an entity. For example, it may be a wireless mobile network communication chip, such as GSM, CDMA, etc.; it can also be a Wifi chip; it may also be a bluetooth chip.

In this embodiment, the processor may be implemented in any suitable manner. For example, the processor may take the form of, for example, a microprocessor or processor and a computer-readable medium that stores computer-readable program code (e.g., software or firmware) executable by the (micro) processor, logic gates, switches, an Application Specific Integrated Circuit (ASIC), a programmable logic controller and embedded microcontroller, and so forth. The description is not intended to be limiting.

In this embodiment, the memory may include multiple layers, and in a digital system, the memory may be any memory as long as binary data can be stored; in an integrated circuit, a circuit without a physical form and with a storage function is also called a memory, such as a RAM, a FIFO and the like; in the system, the storage device in physical form is also called a memory, such as a memory bank, a TF card and the like.

The embodiments of the present specification further provide a computer-readable storage medium based on the above adjusting method for a surgical instrument, where the computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed, the computer program instructions implement: detecting whether preset adjusting conditions of the surgical instruments are met or not at present; under the condition that the preset surgical instrument adjusting condition is met at present, acquiring current pose data of a target surgical instrument and a target three-dimensional grid map of the current time period; the target three-dimensional grid map of the current time period also carries occupation probability; generating an adjusting path of the target surgical instrument according to the current pose data of the target surgical instrument and the target three-dimensional grid map of the current time period; based on the adjusted path of the target surgical instrument, the target surgical instrument is moved to within a preset range of an endoscope field of view.

In this embodiment, the storage medium includes, but is not limited to, a Random Access Memory (RAM), a Read-Only Memory (ROM), a Cache (Cache), a Hard Disk (HDD), or a Memory Card (Memory Card). The memory may be used to store computer program instructions. The network communication unit may be an interface for performing network connection communication, which is set in accordance with a standard prescribed by a communication protocol.

In this embodiment, the functions and effects specifically realized by the program instructions stored in the computer-readable storage medium can be explained in comparison with other embodiments, and are not described herein again.

Referring to fig. 13, in terms of software, the embodiment of the present specification further provides an adjusting device for a surgical instrument, which may specifically include the following structural modules:

the detection module 1301 may be specifically configured to detect whether a preset surgical instrument adjustment condition is currently met;

the obtaining module 1302 may be specifically configured to, in a case that it is determined that a preset surgical instrument adjustment condition is currently met, obtain current pose data of a target surgical instrument and a target three-dimensional grid map of a current time period; the target three-dimensional grid map of the current time period also carries occupation probability;

the generating module 1303 is specifically configured to generate an adjustment path of the target surgical instrument according to the current pose data of the target surgical instrument and the target three-dimensional grid map in the current time period;

the processing module 1304 may be specifically configured to move the target surgical instrument to a preset range of the endoscope field of view based on the adjusted path of the target surgical instrument.

In some embodiments, when the detection module 1301 is implemented, it may detect whether a preset surgical instrument adjustment condition is currently met according to the following manner: acquiring image data of the current time period; the image data of the current time period comprises a plurality of frames of images; screening out a current representative image from the image data of the current time period; detecting the current representative image by using a preset target detection model to obtain a target detection result; detecting whether the target surgical instrument is currently positioned in a preset range of the endoscope visual field or not according to a target detection result; and under the condition that the target surgical instrument is determined not to be located within the preset range of the endoscope visual field at present, determining that the preset surgical instrument adjustment condition is met at present.

In some embodiments, when the detection module 1301 is implemented, the current representative image may be screened from the image data in the current time period in the following manner: respectively carrying out edge contour feature detection on multiple frames of images contained in the image data of the current time period to obtain edge contour feature detection results of the multiple frames of images; and screening out an image with the edge contour characteristics meeting the requirements from the multi-frame images as a current representative image according to the edge contour characteristic detection result of the multi-frame images.

In some embodiments, the obtaining module 1302 may obtain the current pose data of the target surgical instrument according to the following manners: acquiring current state data of a target slave mechanical arm and current observation data of the target slave mechanical arm determined based on an endoscope; wherein the target is carried with the target surgical machine from the mechanical arm; determining the current pose data of the target slave mechanical arm according to the current state data and the current observation data of the target slave mechanical arm; and determining the current pose data of the target surgical instrument according to the current pose data of the target slave mechanical arm.

In some embodiments, when the obtaining module 1302 is implemented specifically, the current pose data of the target slave mechanical arm may be determined according to the current state data and the current observation data of the target slave mechanical arm in the following manner: and performing fusion processing on the current state data and the current observation data of the target slave mechanical arm by using an extended Kalman filtering algorithm to obtain the current pose data of the target slave mechanical arm.

In some embodiments, when the obtaining module 1302 is implemented specifically, the target three-dimensional grid map of the current time period may be obtained as follows: processing the image data of the current time period by using a preset segmentation model to obtain segmented image data; constructing an initial three-dimensional grid map of the current time period according to the image data of the current time period and the segmented image data; determining the occupation probability of each grid point in the current time period according to the initial three-dimensional grid map in the current time period; and mapping the occupation probability of each grid point in the current time period to the initial three-dimensional grid map in the current time period to obtain the target three-dimensional grid map in the current time period.

In some embodiments, the segmented imagery data may specifically include a plurality of segmented images; the segmented image comprises an image region segmented and marked with an image region type; the image region type includes at least one of: surgical instrument regions, tissue organ regions, passthrough regions, and the like.

In some embodiments, when the obtaining module 1302 is implemented specifically, the initial three-dimensional grid map of the current time period may be constructed according to the image data of the current time period and the segmented image data in the following manner: and based on a preset SLAM algorithm model, performing three-dimensional scene modeling by processing the image data of the current time period and the segmented image data to obtain an initial three-dimensional grid map of the current time period.

In some embodiments, the preset SLAM algorithm model may include at least: the method comprises a tracking thread, a local map construction thread and a loop optimization thread.

In some embodiments, when the obtaining module 1302 is implemented specifically, the three-dimensional scene modeling may be performed by processing the image data of the current time period and the segmented image data based on a preset SLAM algorithm model in the following manner to obtain an initial three-dimensional grid map of the current time period: extracting by using a tracking thread, and screening out a concerned image in the current time period through tracking matching according to the image data in the current time period and the image characteristics in the segmented image data; filtering repeated images in the concerned images in the current time period and repeated key points in the images by using a local map construction thread; constructing and obtaining a local map of the current time period based on the processed attention image; and fusing the local map of the current time period and the initial three-dimensional grid map of the previous time period by using a loop optimization thread to obtain the initial three-dimensional grid map of the current time period.

In some embodiments, when the obtaining module 1302 is implemented specifically, the occupation probability of each grid point in the current time period may be determined according to the initial three-dimensional grid map in the current time period in the following manner: and according to the initial three-dimensional grid map of the current time period and the occupation probability of each grid point of the previous time period, calculating the binary Bayesian filtering confidence of each grid point in the initial three-dimensional grid map of the current time period to obtain the occupation probability of each grid point of the current time period.

In some embodiments, after obtaining the occupation probabilities of the grid points in the current time period, the obtaining module 1302 may be further configured to check the occupation probabilities of the grid points in the current time period by using a check threshold range of an image region to which the grid points in the current time period belong; the checking threshold range of the image area is determined according to the image area type of the image area.

In some embodiments, when the generating module 1303 is implemented, the adjustment path of the target surgical instrument may be generated according to the current pose data of the target surgical instrument and the three-dimensional grid map of the target at the current time period in the following manner: determining a path starting point according to the current pose data of the target surgical instrument; determining a proper position point as a path terminal point; and based on a preset search algorithm, generating a movement path meeting the requirement according to the path starting point, the path end point and the target three-dimensional grid map of the current time period, and using the movement path as an adjustment path of the target surgical instrument.

In some embodiments, when the generating module 1303 is implemented specifically, the moving path meeting the requirement may be generated according to the path starting point, the path ending point, and the target three-dimensional grid map in the current time period based on a preset search algorithm in the following manner: determining the current mobile node in the moving path according to the following modes: traversing the current candidate node set to determine a first distance cost between each current candidate node in the current candidate node set and the starting point of the path and a second distance cost between each current candidate node in the current candidate node set and the ending point of the path; and screening out candidate nodes meeting the requirements from the current candidate node set according to the first distance cost and the second distance cost of each current candidate node, and taking the candidate nodes as the current mobile nodes.

In some embodiments, the processing module 1304 may be implemented to move the target surgical instrument within the preset range of the endoscope field of view based on the adjusted path of the target surgical instrument as follows: and displaying the adjustment path of the target surgical instrument to the user, and guiding the user to move the target surgical instrument to a preset range of the endoscope visual field according to the adjustment path of the target surgical instrument.

It should be noted that, the units, devices, modules, and the like described in the foregoing embodiments may be specifically implemented by a computer chip or an entity, or implemented by a product with certain functions. For convenience of description, the above devices are described as being divided into various modules by functions, which are described separately. It is to be understood that, in implementing the present specification, functions of each module may be implemented in one or more pieces of software and/or hardware, or a module that implements the same function may be implemented by a combination of a plurality of sub-modules or sub-units, or the like. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units is only one type of logical functional division, and other divisions may be realized in practice, for example, multiple units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.

As can be seen from the above, based on the adjusting device for a surgical instrument provided in the embodiment of the present specification, an adjustment path with high safety can be efficiently and accurately generated by generating and using a target three-dimensional grid map of a current time period; and based on the adjustment path, the target surgical instrument can be safely moved back within the preset range of the endoscope field of view. Effectively reduces the operation risk when moving the surgical instrument in the operation process and improves the operation experience of the user.

Although the present specification provides method steps as described in the examples or flowcharts, additional or fewer steps may be included based on conventional or non-inventive approaches. The order of steps recited in the embodiments is merely one manner of performing the steps in a multitude of sequences, and does not represent a unique order of performance. When implemented in practice, an apparatus or client product may execute sequentially or in parallel (e.g., in a parallel processor or multithreaded processing environment, or even in a distributed data processing environment) in accordance with the embodiments or methods depicted in the figures. The terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, the presence of additional identical or equivalent elements in processes, methods, articles, or apparatus that include the recited elements is not excluded. The terms first, second, etc. are used to denote names, but not to denote any particular order.

Those skilled in the art will also appreciate that, in addition to implementing the controller in purely computer readable program code means, the same functionality can be implemented by logically programming method steps such that the controller is in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers and the like. Such a controller may therefore be considered as a hardware component, and the means included therein for performing the various functions may also be considered as a structure within the hardware component. Or even means for performing the functions may be conceived to be both a software module implementing the method and a structure within a hardware component.

This description may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, classes, etc. that perform particular tasks or implement particular abstract data types. The specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer-readable storage media including memory storage devices.

From the above description of the embodiments, it is clear to those skilled in the art that the present specification can be implemented by software plus necessary general hardware platform. Based on such understanding, the technical solutions in this specification may be essentially embodied in the form of a software product, which may be stored in a storage medium, such as a ROM/RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which may be a personal computer, a mobile terminal, a server, or a network device, etc.) to execute the methods described in the embodiments or some parts of the embodiments in this specification.

The embodiments in the present specification are described in a progressive manner, and the same or similar parts in the embodiments are referred to each other, and each embodiment focuses on differences from other embodiments. The description is operational with numerous general purpose or special purpose computing system environments or configurations. For example: personal computers, server computers, hand-held or portable devices, tablet-type devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like.

While the specification has been described with examples, those skilled in the art will appreciate that there are numerous variations and permutations of the specification without departing from the spirit of the specification, and it is intended that the appended claims encompass such variations and modifications without departing from the spirit of the specification.

Claims

1. A method of adjusting a surgical instrument, comprising:

detecting whether preset adjusting conditions of the surgical instruments are met or not at present;

acquiring current pose data of a target surgical instrument and a target three-dimensional grid map of a current time period under the condition that the current condition of the preset surgical instrument is met; the target three-dimensional grid map of the current time period also carries occupation probability;

generating an adjusting path of the target surgical instrument according to the current pose data of the target surgical instrument and the target three-dimensional grid map of the current time period;

based on the adjusted path of the target surgical instrument, the target surgical instrument is moved to be within a preset range of the endoscope field of view.

2. The method for adjusting a surgical instrument according to claim 1, wherein detecting whether a preset surgical instrument adjustment condition is currently satisfied comprises:

acquiring image data of the current time period; the image data of the current time period comprises multi-frame images;

screening out a current representative image from the image data of the current time period;

detecting the current representative image by using a preset target detection model to obtain a target detection result;

detecting whether the target surgical instrument is currently positioned in a preset range of the endoscope visual field or not according to a target detection result;

and under the condition that the target surgical instrument is determined not to be located in the preset range of the endoscope visual field at present, determining that the preset surgical instrument adjusting condition is met at present.

3. The method of adjusting a surgical instrument according to claim 2, wherein the step of screening out a current representative image from the image data of the current time period includes:

respectively carrying out edge contour feature detection on multiple frames of images contained in the image data of the current time period to obtain edge contour feature detection results of the multiple frames of images;

and screening out an image with the edge contour characteristics meeting the requirements from the multi-frame images as a current representative image according to the edge contour characteristic detection result of the multi-frame images.

4. The method for adjusting a surgical instrument according to claim 1, wherein acquiring the current pose data of the target surgical instrument includes:

acquiring current state data of a target slave mechanical arm and current observation data of the target slave mechanical arm determined based on an endoscope; wherein the target is carried with the target surgical machine from the mechanical arm;

determining the current pose data of the target slave mechanical arm according to the current state data and the current observation data of the target slave mechanical arm;

and determining the current pose data of the target surgical instrument according to the current pose data of the target slave mechanical arm.

5. The method for adjusting a surgical instrument according to claim 4, wherein determining the current pose data of the target slave manipulator based on the current state data and the current observation data of the target slave manipulator comprises:

and performing fusion processing on the current state data and the current observation data of the target slave mechanical arm by using an extended Kalman filtering algorithm to obtain the current pose data of the target slave mechanical arm.

6. The method for adjusting surgical instruments according to claim 2, wherein acquiring the target three-dimensional grid map of the current time period includes:

processing the image data of the current time period by using a preset segmentation model to obtain segmented image data;

constructing an initial three-dimensional grid map of the current time period according to the image data of the current time period and the segmented image data;

determining the occupation probability of each grid point in the current time period according to the initial three-dimensional grid map in the current time period;

and mapping the occupation probability of each grid point in the current time period to the initial three-dimensional grid map in the current time period to obtain the target three-dimensional grid map in the current time period.

7. The method of adjusting a surgical instrument according to claim 6, wherein the segmented image data includes a plurality of segmented images; the segmented image comprises an image region segmented and marked with an image region type; the image region type includes at least one of: surgical instrument area, tissue organ area, passable area.

8. The method for adjusting surgical instruments according to claim 6, wherein constructing an initial three-dimensional mesh map of the current time period based on the image data of the current time period and the segmented image data includes:

based on a preset SLAM algorithm model, performing three-dimensional scene modeling by processing image data of the current time period and segmented image data to obtain an initial three-dimensional grid map of the current time period; wherein the preset SLAM algorithm model at least comprises: the method comprises a tracking thread, a local map construction thread and a loop optimization thread.

9. The method of claim 6, wherein determining the occupation probability of each grid point in the current time period according to the initial three-dimensional grid map in the current time period comprises:

and according to the initial three-dimensional grid map of the current time period and the occupation probability of each grid point of the previous time period, calculating the binary Bayesian filtering confidence of each grid point in the initial three-dimensional grid map of the current time period to obtain the occupation probability of each grid point of the current time period.

10. The method of adjusting a surgical instrument according to claim 9, wherein after obtaining the occupation probabilities of the respective grid points for the current time period, the method further comprises:

checking the occupation probability of each grid point in the current time period by using the check threshold range of the image area to which each grid point in the current time period belongs; the checking threshold range of the image area is determined according to the image area type of the image area.

11. The method for adjusting a surgical instrument according to claim 1, wherein generating an adjustment path of the target surgical instrument according to the current pose data of the target surgical instrument and the target three-dimensional grid map of the current time period includes:

determining a path starting point according to the current pose data of the target surgical instrument; determining a proper position point as a path terminal point;

and based on a preset search algorithm, generating a movement path meeting the requirement according to the path starting point, the path end point and the target three-dimensional grid map of the current time period, and using the movement path as an adjustment path of the target surgical instrument.

12. The method for adjusting surgical instruments according to claim 11, wherein generating a movement path according to the path starting point, the path ending point and the target three-dimensional grid map of the current time period based on a preset search algorithm comprises:

determining the current mobile node in the moving path according to the following modes:

traversing the current candidate node set to determine a first distance cost between each current candidate node in the current candidate node set and the starting point of the path and a second distance cost between each current candidate node in the current candidate node set and the ending point of the path;

and screening out candidate nodes meeting the requirements from the current candidate node set according to the first distance cost and the second distance cost of each current candidate node, and taking the candidate nodes as the current mobile nodes.

13. The method of adjusting a surgical instrument according to claim 1, wherein moving the target surgical instrument to be within a preset range of an endoscope field of view based on the adjustment path of the target surgical instrument comprises:

and displaying the adjustment path of the target surgical instrument to the user, and guiding the user to move the target surgical instrument to a preset range of the endoscope visual field according to the adjustment path of the target surgical instrument.

14. Surgical system, characterized in that it comprises at least: a display and an adjusting device of the surgical instrument; the adjusting device of the surgical instrument is used for executing and realizing the steps of the method according to any one of claims 1 to 13 under the condition that the preset surgical instrument adjusting condition is determined to be currently met, and obtaining an adjusting path of the target surgical instrument; the display is used for showing the adjustment path of the target surgical instrument to a user so that the target surgical instrument moves to be within a preset range of the endoscope visual field.

15. A computer readable storage medium having stored thereon computer instructions which, when executed by a processor, carry out the steps of the method of any one of claims 1 to 13.