CN117017439A - Digital twin skin puncture needle auxiliary system - Google Patents

Abstract

The invention provides a digital twin skin puncture needle auxiliary system, which belongs to the technical field of medical appliances and comprises a puncture needle positioning cap, a plurality of positioning mark patches, a high-definition binocular camera and a digital twin terminal; wherein, puncture positioning cap is used for detachable to be connected with skin pjncture needle, including net box and pjncture needle cap, the fixed center department that sets up in net box bottom surface of pjncture needle cap, pjncture needle cap is used for detachable to install on the top of skin pjncture needle handle, net box is square or cuboid structure, the surface of net box is provided with alternate color's square grid, the top surface of net box is liquid crystal display, net box is hollow structure, five faces outside the net box top surface all adopt semitransparent material to make, net box inside is provided with the battery, polychrome LED lamp, singlechip mainboard and wireless communicator, can some doctors lack the puncture experience, solve the technical problem that leads to the puncture inaccurate.

Description

Digital twin skin puncture needle auxiliary system

Technical Field

The invention belongs to the technical field of medical appliances, and particularly relates to a digital twin skin puncture needle auxiliary system.

Background

Currently, some dermatological detection software typically uses image processing and machine learning algorithms to automatically identify skin lesions. The algorithm can analyze the input skin photos, check the characteristics of the colors, the sizes, the shapes and the like of the skin photos, and compare the skin photos with a pre-written lesion database so as to obtain potential disease types matched with the skin photos; however, many skin diseases cannot be analyzed from the surface, and when a dermatologist is diagnosed, a puncture needle is required to be adopted for puncturing in many cases, the puncture position, the puncture depth and the puncture needle inclination are mostly carried out by depending on the experience of the doctor, and the problem that the new doctor is not experienced enough to cause inaccurate puncture often occurs.

The digital twin is to fully utilize data such as a physical model, sensor update, operation history and the like, integrate simulation processes of multiple disciplines, multiple physical quantities, multiple scales and multiple probabilities, and complete mapping in a virtual space, thereby reflecting the full life cycle process of corresponding entity equipment. Digital twinning is a beyond-the-reality concept that can be seen as a digital mapping system of one or more important, mutually dependent equipment systems. Digital twinning is a universally adapted theoretical technology system, can be applied in a plurality of fields, and has more application in the fields of product design, product manufacturing, medical analysis, engineering construction and the like. The most deep application in China is in the engineering construction field, the highest attention is paid, and the hottest research is in the intelligent manufacturing field. In the current prior art, there is no research on dermatological puncture guidance using digital twinning technology.

Disclosure of Invention

In view of the above, the invention provides a digital twin skin puncture needle auxiliary system, which can solve the technical problems that when a dermatologist diagnoses, the puncture needle is needed to be adopted for puncturing in many times, the puncturing position, the depth and the inclination of the puncture needle are mostly carried out by depending on the experience of the doctor, and inaccurate puncturing caused by the insufficient experience of a new doctor is often caused.

The invention is realized in the following way:

the invention provides a digital twin skin puncture needle auxiliary system, which comprises a puncture needle positioning cap, a plurality of positioning mark patches, a high-definition binocular camera and a digital twin terminal;

the puncture positioning cap is used for being detachably connected with a skin puncture needle and comprises a grid box and a puncture needle cap, the puncture needle cap is fixedly arranged at the center of the bottom surface of the grid box, the puncture needle cap is used for being detachably arranged at the top end of a skin puncture needle handle, the grid box is of a cube or cuboid structure, square grids with alternate colors are arranged on the surface of the grid box, the top surface of the grid box is a liquid crystal display screen, the grid box is of a hollow structure, five surfaces except the top surface of the grid box are made of semitransparent materials, a battery, a multicolor LED lamp, a singlechip mainboard and a wireless communicator are arranged in the grid box, a singlechip chip is arranged on the singlechip mainboard, the battery is electrically connected with the singlechip mainboard and supplies power, the singlechip mainboard is electrically connected with the multicolor LED lamp, the wireless communicator and the liquid crystal display screen to provide control and power supply, the wireless communicator is used for communicating with the digital twin terminal, and the liquid crystal display screen is used for displaying the guidance direction during puncture;

The positioning mark patch is used for being applied to the surface of the skin of a patient to carry out auxiliary positioning marks, and is a round patch which can be applied to the skin, the surfaces of the patches are of different colors, the diameter of the patch is 2-3 mm, and the thickness of the patch is 2-3 mm;

the digital twin terminal is used for assisting puncture, and specifically comprises:

the affected part model generation module is used for generating a three-dimensional model of the affected part of the skin of the current patient;

the puncture three-dimensional position determining module is used for determining puncture positioning data according to the data of the current patient, wherein the puncture positioning data comprise a puncture position, a puncture depth and a puncture gradient;

the high-definition binocular camera is used for acquiring the position of the grid box in real time, wherein the high-definition binocular camera is a binocular camera with resolution of 4K;

the auxiliary puncture module is used for carrying out digital twin guiding on the skin of the current patient according to the puncture three-dimensional position.

The digital twin skin puncture needle auxiliary system provided by the invention has the following technical effects: the square grids with alternate colors are arranged on the surface of the grid box, so that the grid can be utilized to correct images, and as the size of the square grid is known, operations such as correcting distortion and the like can be realized according to the size of the square grid as a reference in the digital twin processing process; the small patches with different colors are adopted, so that more accurate calibration can be realized by using different colors; when a doctor performs puncture, the guiding direction during the puncture is displayed by the liquid crystal display, the doctor does not need to try to observe the digital twin digital scene, and the liquid crystal display on the grid box is directly used, and as the grid box and the puncture needle are fixed together, the puncture needle is held in the hand by the doctor, the observation is more visual, and great convenience is brought to the puncture operation.

On the basis of the technical scheme, the digital twin skin puncture needle auxiliary system can be further improved as follows:

the affected part model generating module is used for executing the following steps:

establishing a partial human body surface three-dimensional model containing the affected skin area of a patient as a basic three-dimensional model;

acquiring high-definition images of affected areas of the skin at different angles as an affected area image set;

fusing the image of the affected part to the surface of the basic three-dimensional model to form a three-dimensional model of the affected part;

wherein the affected area of skin is an area containing at least 1 6 even number of positioning mark patches applied by a doctor on the peripheral edge of the affected area of skin of a patient; the high definition image is a 4K resolution image.

The method for establishing the partial human body surface three-dimensional model containing the skin affected area of the patient as the basic three-dimensional model comprises the steps of acquiring a point cloud model of a three-dimensional laser scanner for scanning the partial human body surface containing the skin affected area of the patient, and then carrying out three-dimensional modeling on the point cloud model to form the basic three-dimensional model.

In the basic three-dimensional model obtained by adopting the scheme, because the positioning mark patch has the height, the positions corresponding to the mark patches in the basic three-dimensional model are all the corresponding high-protruding grids, and when the affected part images are fused to the surface of the basic three-dimensional model, the reference positioning can be performed according to the mark patches, so that the fusion efficiency is improved.

Further, the step of fusing the affected part image to the surface of the basic three-dimensional model to form the affected part three-dimensional model specifically comprises the following steps:

preprocessing the image set of the affected part, including correcting distortion, cutting edges and denoising, so as to obtain a first image set;

acquiring points corresponding to all positioning mark patches in the basic three-dimensional model as reference points to form a reference point set;

dividing the basic three-dimensional model into a plurality of quadrilateral grids according to the reference points to form a quadrilateral grid set;

cutting each image in the first image set according to each quadrilateral grid in the quadrilateral grid set to obtain a plurality of image slices to form an image slice set;

selecting an image slice with highest similarity with the quadrilateral grids as a mapping slice for each quadrilateral grid in the quadrilateral grid set;

and pasting all the mapping slices to the corresponding positions of the basic three-dimensional model to form the three-dimensional model of the affected part.

Further, the step of dividing the basic three-dimensional model into a plurality of quadrilateral grids according to the reference points specifically includes:

establishing an externally connected ball: acquiring the minimum outer sphere of the basic three-dimensional model, taking the center of the outer sphere as an origin, and establishing rays from the origin to each reference point on the basic model, wherein the connecting line of the rays and the minimum outer sphere is an external connection point corresponding to the reference point;

Belman projection: carrying out ink card support projection on all the external joints to obtain a projection point set consisting of a plurality of projection points;

establishing a plane coordinate system: taking the geometric centers of all the projection points as an origin, and taking a straight line where a connecting line from the origin to any one projection point is positioned as an x-axis, so as to establish a plane coordinate system;

ordering the point sets: according to the coordinates of the projection points, the projection points are arranged in a clockwise or anticlockwise direction;

constructing a convex hull: processing the projection point set by using a convex hull algorithm to obtain a convex hull of the projection point set, wherein the convex hull is a convex polygon formed by points;

finding out adjacent connecting lines: connecting points on the convex hull with their adjacent points;

eliminating illegal edges: deleting edges formed by non-adjacent points in the convex hull, and removing or replacing connecting lines of the non-adjacent points with connecting lines between adjacent points which are closer to each other;

obtaining a projection circle: after the illegal edges are removed, a projection circle meeting the conditions can be obtained, and the projection points which are not on the projection circle are set as abandoned projection points;

reference point screening: deleting the reference points corresponding to the abandoned projection points in the reference point set;

reference point number: taking two adjacent nearest reference points as a first initial reference point and a second initial reference point, wherein the first initial reference point is in the anticlockwise direction of the second initial reference point, the numbers of the first initial reference point and the second initial reference point are 0, carrying out +1 numbering on each reference point one by one along the clockwise direction by taking the first initial reference point as a starting point, and synchronously, carrying out-1 numbering on each reference point one by one along the anticlockwise direction by taking the second initial reference point as a starting point until each reference point has a number, and stopping numbering; if a certain reference point does not exist, the reference point with the number sum of the reference point corresponding to the certain reference point is 0, and the reference point is deleted;

Building a quadrilateral grid: the two points added together to 0 are connected in a straight line, and the resulting straight line divides the plane of the basic three-dimensional model into a plurality of quadrilateral grids.

Further, the puncture three-dimensional position determining module is configured to perform the following steps:

acquiring a historical skin disease puncture record database which comprises a plurality of groups of skin affected part images and puncture positioning data of historical patients;

establishing a training sample according to the historical skin disease puncture record database, wherein the training sample comprises a training set and a verification set, and the proportion of the training set to the verification set is 6:4, a step of; the input of the training sample training comprises a plurality of groups of skin affected part images of historical patients, and the output of the training is puncture positioning data;

constructing a puncture positioning model prototype by using a convolutional neural network, and training the puncture positioning model prototype by using a training set to obtain the puncture positioning model prototype;

adopting a verification set to optimize the puncture positioning model;

obtaining an image of the affected part of the skin of the current skin patient, inputting the optimized puncture positioning model prototype for calculation, and obtaining puncture positioning data of the current skin patient.

The convolutional neural network comprises 1 input layer, 1 flat layer, 1 full connection layer and 1 output layer;

The input layer is used for inputting a plurality of groups of skin affected part images of the historical patients of the training samples;

the diaphysis layer is used for historic patient's skin affected part image, specifically:

preprocessing the skin affected part image of the historical patient, including correcting distortion, cutting edges and denoising, so as to obtain a preprocessed image;

carrying out gray scale processing on the preprocessed image to obtain a gray scale image;

filtering the gray level image to obtain a filtered image;

performing edge detection on the filtered image to obtain an edge detection image containing a plurality of outlines;

establishing a plane coordinate system by taking the upper left point of the edge detection image as an origin, taking the right as a horizontal axis direction and taking the downward as a vertical axis direction, and representing each contour on the edge detection image by adopting pixel coordinates to obtain a first contour set;

extracting the length of each contour in the first contour set and the coordinates of pixels on the contour to form high-dimensional sparse data;

flattening the high-dimensional sparse data by using a flat layer;

inputting flattened data into a fully-connected network with the node number of 256 and the activation function of Relu of 4 layers for feature extraction to obtain a vector representing the calculated puncture positioning data of a historical patient, and marking the vector as a first vector;

Comparing the first vector with a second vector formed by expected puncture positioning data, and if the similarity of the first vector and the second vector is less than 75%, adopting a random gradient descent optimization algorithm to perform counter-propagation network training, and updating the parameters of a backbone layer;

wherein, the second vector is expressed as [ puncture position, puncture depth, puncture gradient ];

the network skeleton used for feature extraction of the convolutional neural network is a DenseNet121 network, wherein the DenseNet121 network is formed by sequentially stacking 1 DenseBlock containing 6 convolutional layers, 1 DenseBlock containing 12 convolutional layers, 1 DenseBlock containing 24 convolutional layers and 1 DenseBlock containing 16 convolutional layers, and the input of each convolutional layer in the DenseBlock is the output of all the preceding convolutional layers.

Adopting a verification set to optimize a puncture positioning model, and specifically comprising the following steps:

inputting the skin affected part images of the historical patients in the verification set into a puncture positioning model, and outputting a third vector;

if the similarity of the third vector and the fourth vector formed by the actual puncture positioning data in the verification set is less than 80%, adopting a random gradient descent optimization algorithm to perform counter propagation network training, updating the parameters of the backbone layer, and further training the puncture positioning model by taking the union of the training set and the verification set as the training set.

Further, the input of the training sample of the puncture positioning model further comprises the age and sex of the historical patient, and the image of the affected part of the skin of the historical patient is manually marked with the focus position; meanwhile, in the step of acquiring the image of the affected part of the skin of the current skin patient, inputting the optimized puncture positioning model prototype for calculation to obtain the puncture positioning data of the current skin patient, the input of the optimized puncture positioning model prototype for calculation also comprises the age and sex of the current patient, and the image of the affected part of the skin of the current patient is calibrated by manpower.

Further, the auxiliary puncture module is configured to perform the following steps:

establishing a skin puncture digital twin scene, wherein the physical scene is an affected part of the skin of a patient, and the digital scene is a three-dimensional model of the affected part;

constructing a puncture needle with a puncture needle positioning cap in a digital twin digital scene;

marking the puncture data determined by the puncture three-dimensional position determining module in a digital twin scene;

guiding the puncture needle positioning in the physical scene and updating the position of the puncture needle in the digital scene in real time.

The method for marking the puncture data determined by the puncture three-dimensional position determining module in the digital twin scene is to display the puncture data by adopting green light.

Further, the step of constructing the puncture needle with the grid box in the digital twin digital scene specifically comprises the following steps:

acquiring an image of the grid box;

performing image correction and positioning according to square grids of the grid boxes with alternate colors;

obtaining the positioning of the whole puncture needle according to the image and the positioning of the grid box;

in a digital scenario, a twin puncture needle is constructed.

Further, the step of guiding the positioning of the puncture needle in the physical scene specifically comprises the following steps:

acquiring puncture positions in the puncture needle head and the current patient in the digital scene, corresponding to target points, and determining the moving direction of the puncture needle head as a target direction according to the relative positions of the puncture needle head and the target points;

the target direction is sent to the single chip microcomputer chip through the wireless communicator, and the single chip microcomputer chip controls the liquid crystal display to display the target direction in a mode that a starting point is an arrow in the center of the liquid crystal display;

Acquiring the axial direction of a puncture needle and the axial direction of puncture gradient in the puncture data of the current patient in a digital scene, determining the direction of axial change of the puncture needle according to the included angle of the axial direction of the puncture needle and the axial direction of puncture gradient in the puncture data of the current patient, and taking the axial direction of the puncture gradient in the puncture data of the current patient as an offset target direction;

the offset target direction is sent to the single chip microcomputer chip through the wireless communicator, and the single chip microcomputer chip controls the liquid crystal display screen to display the target direction in a mode of an arrow parallel to the periphery of the liquid crystal display screen;

the target direction and the offset target direction are updated in real time according to the position of the puncture needle head.

Further, the step of guiding the positioning of the puncture needle in the physical scene further includes:

when the puncture needle head punctures the skin of an affected part of a patient, if the puncture depth is smaller than the puncture depth in the current puncture data of the patient, an LED control instruction of LED rapid stroboscopic is sent out; if the puncture depth is greater than or equal to the puncture depth in the puncture data of the current patient, an LED common flicker control instruction is sent out; if the puncture inclination is smaller than the puncture inclination in the puncture data of the current patient, an LED control instruction for displaying green by an LED is sent out; if the puncture inclination is greater than or equal to the puncture inclination in the puncture data of the current patient, sending an LED control instruction for displaying red by an LED; the LED control instruction is sent to the singlechip chip through the wireless communicator, and the singlechip chip controls the multicolor LED lamp to display; wherein, the abrupt strobe refers to the flicker with a flicker interval of 0.2s, and the normal flicker refers to the flicker with a flicker interval of 0.5 s.

Compared with the prior art, the digital twin skin puncture needle auxiliary system provided by the invention has the beneficial effects that: the square grids with alternate colors are arranged on the surface of the grid box, so that the grid can be utilized to correct images, and as the size of the square grid is known, operations such as correcting distortion and the like can be realized according to the size of the square grid as a reference in the digital twin processing process; the small patches with different colors are adopted, so that more accurate calibration can be realized by using different colors; the puncture positioning model is constructed and trained by utilizing the convolutional neural network, the puncture position, the puncture depth and the puncture gradient of the current patient can be calculated by utilizing the puncture positioning model, when a doctor performs puncture, the guiding direction during puncture is displayed by utilizing the liquid crystal display screen, the doctor does not need to try to observe a digital twin digital scene, and the liquid crystal display screen on the grid box is directly needed, and as the grid box and the puncture needle are fixed together, the puncture needle is held in the hand of the doctor, the observation is more visual, and great convenience is brought to the puncture operation; the device can solve the technical problems that when a dermatologist diagnoses, the dermatologist needs to puncture by adopting a puncture needle in many times, the puncture position, the puncture depth and the puncture needle inclination are mostly carried out by depending on the experience of the dermatologist, and inaccurate puncture caused by insufficient experience of a new doctor often occurs.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings that are needed in the description of the embodiments of the present invention will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 is a schematic diagram of a digital twin skin puncturing needle auxiliary device in accordance with the present invention;

FIG. 2 is a schematic illustration of the connection of a digital twin skin puncturing needle auxiliary device and a skin puncturing needle provided by the present invention;

FIG. 3 is a diagram of electrical connections within a grid box of a digital twin skin penetrating needle provided by the present invention;

in the drawings, the list of components represented by the various numbers is as follows:

10. a grid box; 11. a battery; 12. multicolor LED lamps; 13. a singlechip motherboard; 14. a wireless communicator; 20. a lancet cap; 31. a handle; 32. a puncture head.

Detailed Description

For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments. All other embodiments, based on the embodiments of the invention, which are apparent to those of ordinary skill in the art without inventive faculty, are intended to be within the scope of the invention.

Thus, the following detailed description of the embodiments of the invention, as presented in the figures, is not intended to limit the scope of the invention, as claimed, but is merely representative of selected embodiments of the invention. All other embodiments, based on the embodiments of the invention, which are apparent to those of ordinary skill in the art without inventive faculty, are intended to be within the scope of the invention.

It should be noted that: like reference numerals and letters denote like items in the following figures, and thus once an item is defined in one figure, no further definition or explanation thereof is necessary in the following figures.

In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", etc. indicate orientations or positional relationships based on the orientations or positional relationships shown in the drawings are merely for convenience in describing the present invention and simplifying the description, and do not indicate or imply that the apparatus or elements referred to must have a specific orientation, be configured and operated in a specific orientation, and thus should not be construed as limiting the present invention.

Furthermore, the terms "first," "second," and the like, are used for descriptive purposes only and are not to be construed as indicating or implying a relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defining "a first" or "a second" may explicitly or implicitly include one or more such feature. In the description of the present invention, the meaning of "a plurality" is two or more, unless explicitly defined otherwise.

1-2, an embodiment of a digital twin skin puncture needle auxiliary system provided by the invention comprises a puncture needle positioning cap, a plurality of positioning mark patches, a high-definition binocular camera and a digital twin terminal;

the puncture positioning cap is used for being detachably connected with a skin puncture needle and comprises a grid box 10 and a puncture needle cap 20, the puncture needle cap 20 is fixedly arranged at the center of the bottom surface of the grid box 10, the puncture needle cap 20 is detachably arranged at the top end of a skin puncture needle handle 31, the grid box 10 is of a cube or cuboid structure, square grids with alternate colors are arranged on the surface of the grid box 10, the top surface of the grid box 10 is a liquid crystal display screen, the grid box 10 is of a hollow structure, five surfaces outside the top surface of the grid box 10 are made of semitransparent materials, a battery 11, a multicolor LED lamp 12, a singlechip mainboard 13 and a wireless communicator 14 are arranged in the grid box 10, a singlechip chip is arranged on the singlechip mainboard 13, the battery 11 is electrically connected with the singlechip mainboard 13 and supplies power, the singlechip mainboard 13 is electrically connected with the multicolor LED lamp 12, the wireless communicator 14 and the liquid crystal display screen to provide control and power supply, the wireless communicator 14 is used for communicating with a digital twin terminal, and the liquid crystal display screen is used for displaying the guiding direction during puncture; the single chip microcomputer chip adopts an 80C51 chip, the liquid crystal display screen adopts a monochromatic liquid crystal display screen, and the wireless communicator adopts a WIFI module; wherein, the puncture positioning cap is used for detachable mode including grafting, screw connection etc. with skin pjncture needle connection.

The positioning mark patch is used for being attached to the skin surface of a patient to carry out auxiliary positioning marks, and is a round patch which can be attached to the skin, the surfaces of the patches are different in color, the diameter of the patch is 2-3 mm, and the thickness of the patch is 2-3 mm;

the digital twin terminal is used for assisting puncture, and specifically comprises:

the affected part model generation module is used for generating a three-dimensional model of the affected part of the skin of the current patient;

the puncture three-dimensional position determining module is used for determining puncture positioning data according to the data of the current patient, wherein the puncture positioning data comprise a puncture position, a puncture depth and a puncture gradient;

the high-definition binocular camera is used for acquiring the position of the grid box 10 in real time, wherein the high-definition binocular camera is a binocular camera with resolution of 4K;

the auxiliary puncture module is used for carrying out digital twin guiding on the skin of the current patient according to the puncture three-dimensional position.

The digital twin is to fully utilize data such as a physical model, sensor update, operation history and the like, integrate simulation processes of multiple disciplines, multiple physical quantities, multiple scales and multiple probabilities, and complete mapping in a virtual space, thereby reflecting the full life cycle process of corresponding entity equipment. Digital twinning is a beyond-the-reality concept that can be seen as a digital mapping system of one or more important, mutually dependent equipment systems.

In the above technical solution, the affected part model generating module is configured to execute the following steps:

establishing a partial human body surface three-dimensional model containing the affected skin area of a patient as a basic three-dimensional model;

acquiring high-definition images of affected areas of the skin at different angles as an affected area image set;

fusing the image of the affected part to the surface of the basic three-dimensional model to form a three-dimensional model of the affected part;

wherein the affected area of skin is an area containing at least 16 even number of positioning mark patches applied by a doctor on the peripheral edge of the affected area of skin of a patient; the high definition image is a 4K resolution image.

The method for establishing the partial human body surface three-dimensional model containing the skin affected area of the patient as the basic three-dimensional model comprises the steps of acquiring a point cloud model of a three-dimensional laser scanner for scanning the partial human body surface containing the skin affected area of the patient, and then carrying out three-dimensional modeling on the point cloud model to form the basic three-dimensional model.

The number of the positioning mark patches applied to the peripheral edge of the affected part of the skin of the patient is at least 16, the positioning mark patches are uniformly distributed around the actual affected part of the skin of the patient, the positioning mark patches are preferably arranged next to each other to surround the affected part of the skin, and the adjacent positioning mark patches have different colors.

Further, in the above technical solution, the step of fusing the affected part image to the surface of the basic three-dimensional model to form the affected part three-dimensional model specifically includes:

preprocessing the image set of the affected part, including correcting distortion, cutting edges and denoising, so as to obtain a first image set;

acquiring points corresponding to all positioning mark patches in the basic three-dimensional model as reference points to form a reference point set;

dividing the basic three-dimensional model into a plurality of quadrilateral grids according to the reference points to form a quadrilateral grid set;

cutting each image in the first image set according to each quadrilateral grid in the quadrilateral grid set to obtain a plurality of image slices to form an image slice set;

selecting an image slice with highest similarity with the quadrilateral grids as a mapping slice for each quadrilateral grid in the quadrilateral grid set;

and (5) attaching all the mapping slices to the corresponding positions of the basic three-dimensional model to form the three-dimensional model of the affected part.

Further, in the above technical solution, the step of dividing the basic three-dimensional model into a plurality of quadrilateral grids according to the reference points specifically includes:

establishing an externally connected ball: acquiring the minimum outer sphere of the basic three-dimensional model, taking the center of the outer sphere as an origin, and establishing rays from the origin to each reference point on the basic model, wherein the connecting line of the rays and the minimum outer sphere is an external connection point corresponding to the reference point;

Belman projection: carrying out ink card support projection on all the external joints to obtain a projection point set consisting of a plurality of projection points;

establishing a plane coordinate system: taking the geometric centers of all the projection points as an origin, and taking a straight line where a connecting line from the origin to any one projection point is positioned as an x-axis, so as to establish a plane coordinate system;

ordering the point sets: according to the coordinates of the projection points, the projection points are arranged in a clockwise or anticlockwise direction;

constructing a convex hull: processing the projection point set by using a convex hull algorithm to obtain a convex hull of the projection point set, wherein the convex hull is a convex polygon formed by points;

finding out adjacent connecting lines: connecting points on the convex hull with their adjacent points;

eliminating illegal edges: deleting edges formed by non-adjacent points in the convex hull, and removing or replacing connecting lines of the non-adjacent points with connecting lines between adjacent points which are closer to each other;

obtaining a projection circle: after the illegal edges are removed, a projection circle meeting the conditions can be obtained, and the projection points which are not on the projection circle are set as abandoned projection points;

reference point screening: deleting the reference points corresponding to the abandoned projection points in the reference point set;

reference point number: taking two adjacent nearest reference points as a first initial reference point and a second initial reference point, wherein the first initial reference point is in the anticlockwise direction of the second initial reference point, the numbers of the first initial reference point and the second initial reference point are 0, carrying out +1 numbering on each reference point one by one along the clockwise direction by taking the first initial reference point as a starting point, and synchronously, carrying out-1 numbering on each reference point one by one along the anticlockwise direction by taking the second initial reference point as a starting point until each reference point has a number, and stopping numbering; if a certain reference point does not exist, the reference point with the number sum of the reference point corresponding to the certain reference point is 0, and the reference point is deleted;

Building a quadrilateral grid: the two points added together to 0 are connected in a straight line, and the resulting straight line divides the plane of the basic three-dimensional model into a plurality of quadrilateral grids.

Among them, the mercator projection, i.e., the positive-axis equiangular cylindrical projection, was created by the netherlands plotter mercator (g.mercator) in 1569. A cylinder consistent with the direction of the earth axis is supposed to be cut or cut on the earth, the longitude and latitude net is projected onto the cylinder surface according to the equiangular condition, and the projection is obtained after the cylinder surface is spread into a plane. The mercator projection is the earliest and most commonly used of the tangent and tangent cylindrical projections.

Convex Hull (Convex Hull) is a concept in computational geometry (graphics).

In a real vector space V, for a given set X, the intersection S of all convex sets containing X is referred to as the convex hull of X. The convex hull of X may be constructed with convex combinations of all points (X1, … Xn) within X.

In two-dimensional Euclidean space, a convex hull can be thought of as a band that just wraps all points.

In terms of imprecision, a convex hull is a convex polygon formed by connecting points of the outermost layers, given a set of points on a two-dimensional plane, and can contain all the points in the set of points.

Further, in the above technical solution, the puncture three-dimensional position determining module is configured to execute the following steps:

acquiring a historical skin disease puncture record database which comprises a plurality of groups of skin affected part images and puncture positioning data of historical patients;

establishing a training sample according to a historical skin disease puncture record database, wherein the training sample comprises a training set and a verification set, and the ratio of the training set to the verification set is 6:4, a step of; the input of training sample training comprises a plurality of groups of skin affected part images of historical patients, and the output of training is puncture positioning data;

constructing a puncture positioning model prototype by using a convolutional neural network, and training the puncture positioning model prototype by using a training set to obtain the puncture positioning model prototype;

adopting a verification set to optimize the puncture positioning model;

obtaining an image of the affected part of the skin of the current skin patient, inputting the optimized puncture positioning model prototype for calculation, and obtaining puncture positioning data of the current skin patient.

The convolutional neural network comprises 1 input layer, 1 flat layer, 1 full connection layer and 1 output layer;

the input layer is used for inputting a plurality of groups of skin affected part images of the historical patients of the training samples;

The diaphysis is used for historic patient's skin affected part image, specifically:

preprocessing the skin affected part image of the historical patient, including correcting distortion, cutting edges and denoising, so as to obtain a preprocessed image;

carrying out gray scale treatment on the preprocessed image to obtain a gray scale image;

filtering the gray level image to obtain a filtered image;

performing edge detection on the filtered image to obtain an edge detection image containing a plurality of outlines;

establishing a plane coordinate system by taking the upper left point of the edge detection image as an origin, taking the right as a transverse axis direction and taking the downward as a longitudinal axis direction, and representing each contour on the edge detection image by adopting pixel coordinates to obtain a first contour set;

extracting the length of each contour in the first contour set and the coordinates of pixels on the contour to form high-dimensional sparse data;

flattening the high-dimensional sparse data by using a flat layer;

inputting flattened data into a fully-connected network with the node number of 256 and the activation function of Relu of 4 layers for feature extraction to obtain a vector representing the calculated puncture positioning data of a historical patient, and marking the vector as a first vector;

comparing the first vector with a second vector formed by expected puncture positioning data, and if the similarity of the first vector and the second vector is less than 75%, adopting a random gradient descent optimization algorithm to perform counter-propagation network training, and updating the parameters of a backbone layer;

Wherein, the second vector is expressed as [ puncture position, puncture depth, puncture gradient ];

the network skeleton used for feature extraction of the convolutional neural network is a DenseNet121 network, the DenseNet121 network is formed by sequentially stacking 1 DenseBlock containing 6 convolutional layers, 1 DenseBlock containing 12 convolutional layers, 1 DenseBlock containing 24 convolutional layers and 1 DenseBlock containing 16 convolutional layers, and the input of each convolutional layer in the DenseBlock is the output of all the preceding convolutional layers.

Adopting a verification set to optimize a puncture positioning model, and specifically comprising the following steps:

inputting the skin affected part images of the historical patients in the verification set into a puncture positioning model, and outputting a third vector;

if the similarity of the third vector and the fourth vector formed by the actual puncture positioning data in the verification set is less than 80%, adopting a random gradient descent optimization algorithm to perform counter propagation network training, updating the parameters of the backbone layer, and further training the puncture positioning model by taking the union of the training set and the verification set as the training set.

Furthermore, in the above technical scheme, the input of the training sample of the puncture positioning model further comprises the age and sex of the historical patient, and the image of the affected part of the skin of the historical patient is manually calibrated to the focus position; meanwhile, in the step of acquiring the image of the affected part of the skin of the current skin patient, inputting the optimized puncture positioning model prototype for calculation to obtain the puncture positioning data of the current skin patient, the input of the optimized puncture positioning model prototype for calculation also comprises the age and sex of the current patient, and the image of the affected part of the skin of the current patient is calibrated by manpower.

Further, in the above technical solution, the auxiliary puncture module is configured to perform the following steps:

establishing a skin puncture digital twin scene, wherein the physical scene is an affected part of the skin of a patient, and the digital scene is a three-dimensional model of the affected part;

constructing a puncture needle with a puncture needle positioning cap in a digital twin digital scene;

marking the puncture data determined by the puncture three-dimensional position determining module in a digital twin scene;

guiding the puncture needle positioning in the physical scene and updating the position of the puncture needle in the digital scene in real time.

The method for marking the puncture data determined by the puncture three-dimensional position determining module in the digital twin scene is to display the puncture data by adopting green light electricity.

Further, in the above technical solution, the step of constructing the puncture needle with the grid box 10 installed in the digital scene of digital twinning specifically includes:

acquiring an image of the grid box 10;

performing image correction and positioning according to square grids of alternate colors of the grid box 10;

the positioning of the whole puncture needle is obtained according to the image and the positioning of the grid box 10;

in a digital scenario, a twin puncture needle is constructed.

Further, in the above technical solution, the step of guiding the positioning of the puncture needle in the physical scene specifically includes:

Acquiring puncture positions in puncture needle heads and current puncture data of a patient in a digital scene to correspond to target points, and determining the moving direction of the puncture needle heads as a target direction according to the relative positions of the puncture needle heads and the target points;

the target direction is sent to a singlechip chip through a wireless communicator 14, and the singlechip chip controls a liquid crystal display screen to display the target direction in a mode of adopting an arrow with a starting point being the center of the liquid crystal display screen;

acquiring the axial direction of a puncture needle and the axial direction of puncture gradient in the puncture data of the current patient in a digital scene, determining the axial direction change direction of the puncture needle according to the included angle of the axial direction of the puncture needle and the axial direction of the puncture gradient in the puncture data of the current patient, and taking the axial direction of the puncture gradient in the puncture data of the current patient as an offset target direction;

the offset target direction is sent to a singlechip chip through a wireless communicator 14, and the singlechip chip controls a liquid crystal display screen to display the target direction in a mode of an arrow parallel to the periphery of the liquid crystal display screen;

wherein, the target direction and the offset target direction are updated in real time according to the position of the needle head of the puncture needle.

Further, in the above technical solution, the step of guiding the positioning of the puncture needle in the physical scene further includes:

When the puncture needle head punctures the skin of an affected part of a patient, if the puncture depth is smaller than the puncture depth in the current puncture data of the patient, an LED control instruction of LED rapid stroboscopic is sent out; if the puncture depth is greater than or equal to the puncture depth in the puncture data of the current patient, an LED common flicker control instruction is sent out; if the puncture inclination is smaller than the puncture inclination in the puncture data of the current patient, an LED control instruction for displaying green by the LED is sent out; if the puncture inclination is greater than or equal to the puncture inclination in the puncture data of the current patient, an LED control instruction for displaying red by an LED is sent out; the LED control instruction is sent to the singlechip chip through the wireless communicator 14, and the singlechip chip controls the multicolor LED lamp 12 to display; wherein, the rapid strobe means a flicker with a flicker interval of 0.2s, and the normal flicker means a flicker with a flicker interval of 0.5 s.

Claims (10)

1. The digital twin skin puncture needle auxiliary system is characterized by comprising a puncture needle positioning cap, a plurality of positioning mark patches, a high-definition binocular camera and a digital twin terminal;

the puncture positioning cap is used for being detachably connected with a skin puncture needle and comprises a grid box and a puncture needle cap, the puncture needle cap is fixedly arranged at the center of the bottom surface of the grid box, the puncture needle cap is used for being detachably arranged at the top end of a skin puncture needle handle, the grid box is of a cube or cuboid structure, square grids with alternate colors are arranged on the surface of the grid box, the top surface of the grid box is a liquid crystal display screen, the grid box is of a hollow structure, five surfaces except the top surface of the grid box are made of semitransparent materials, a battery, a multicolor LED lamp, a singlechip mainboard and a wireless communicator are arranged in the grid box, a singlechip chip is arranged on the singlechip mainboard, the battery is electrically connected with the singlechip mainboard and supplies power, the singlechip mainboard is electrically connected with the multicolor LED lamp, the wireless communicator and the liquid crystal display screen to provide control and power supply, the wireless communicator is used for communicating with the digital twin terminal, and the liquid crystal display screen is used for displaying the guidance direction during puncture;

The positioning mark patch is used for being applied to the surface of the skin of a patient to carry out auxiliary positioning marks, and is a round patch which can be applied to the skin, the surfaces of the patches are of different colors, the diameter of the patch is 2-3 mm, and the thickness of the patch is 2-3 mm;

the digital twin terminal is used for assisting puncture, and specifically comprises:

the affected part model generation module is used for generating a three-dimensional model of the affected part of the skin of the current patient;

the puncture three-dimensional position determining module is used for determining puncture positioning data according to the data of the current patient, wherein the puncture positioning data comprise a puncture position, a puncture depth and a puncture gradient;

the high-definition binocular camera is used for acquiring the position of the grid box in real time, wherein the high-definition binocular camera is a binocular camera with resolution of 4K;

the auxiliary puncture module is used for carrying out digital twin guiding on the skin of the current patient according to the puncture three-dimensional position.

2. A digital twin skin penetration needle assistance system as claimed in claim 1, wherein the lesion model generation module is adapted to perform the steps of:

establishing a partial human body surface three-dimensional model containing the affected skin area of a patient as a basic three-dimensional model;

Acquiring high-definition images of affected areas of the skin at different angles as an affected area image set;

fusing the image of the affected part to the surface of the basic three-dimensional model to form a three-dimensional model of the affected part;

wherein the affected area of skin is an area containing at least 16 even number of positioning mark patches applied by a doctor on the peripheral edge of the affected area of skin of a patient; the high definition image is a 4K resolution image.

3. A digital twin skin penetration needle assistance system as defined in claim 2, wherein said step of fusing the lesion image to the surface of the substantially three-dimensional model to form a three-dimensional model of the lesion comprises:

preprocessing the image set of the affected part, including correcting distortion, cutting edges and denoising, so as to obtain a first image set;

acquiring points corresponding to all positioning mark patches in the basic three-dimensional model as reference points to form a reference point set;

dividing the basic three-dimensional model into a plurality of quadrilateral grids according to the reference points to form a quadrilateral grid set;

cutting each image in the first image set according to each quadrilateral grid in the quadrilateral grid set to obtain a plurality of image slices to form an image slice set;

selecting an image slice with highest similarity with the quadrilateral grids as a mapping slice for each quadrilateral grid in the quadrilateral grid set;

And pasting all the mapping slices to the corresponding positions of the basic three-dimensional model to form the three-dimensional model of the affected part.

4. A digital twin skin penetration needle assistance system according to claim 3, wherein the step of dividing the basic three-dimensional model into a plurality of quadrilateral grids according to reference points comprises:

establishing an externally connected ball: acquiring the minimum outer sphere of the basic three-dimensional model, taking the center of the outer sphere as an origin, and establishing rays from the origin to each reference point on the basic model, wherein the connecting line of the rays and the minimum outer sphere is an external connection point corresponding to the reference point;

belman projection: carrying out ink card support projection on all the external joints to obtain a projection point set consisting of a plurality of projection points;

establishing a plane coordinate system: taking the geometric centers of all the projection points as an origin, and taking a straight line where a connecting line from the origin to any one projection point is positioned as an x-axis, so as to establish a plane coordinate system;

ordering the point sets: according to the coordinates of the projection points, the projection points are arranged in a clockwise or anticlockwise direction;

constructing a convex hull: processing the projection point set by using a convex hull algorithm to obtain a convex hull of the projection point set, wherein the convex hull is a convex polygon formed by points;

finding out adjacent connecting lines: connecting points on the convex hull with their adjacent points;

Eliminating illegal edges: deleting edges formed by non-adjacent points in the convex hull, and removing or replacing connecting lines of the non-adjacent points with connecting lines between adjacent points which are closer to each other;

obtaining a projection circle: after the illegal edges are removed, a projection circle meeting the conditions can be obtained, and the projection points which are not on the projection circle are set as abandoned projection points;

reference point screening: deleting the reference points corresponding to the abandoned projection points in the reference point set;

reference point number: taking two adjacent nearest reference points as a first initial reference point and a second initial reference point, wherein the first initial reference point is in the anticlockwise direction of the second initial reference point, the numbers of the first initial reference point and the second initial reference point are 0, carrying out +1 numbering on each reference point one by one along the clockwise direction by taking the first initial reference point as a starting point, and synchronously, carrying out-1 numbering on each reference point one by one along the anticlockwise direction by taking the second initial reference point as a starting point until each reference point has a number, and stopping numbering; if a certain reference point does not exist, the reference point with the number sum of the reference point corresponding to the certain reference point is 0, and the reference point is deleted;

building a quadrilateral grid: the two points added together to 0 are connected in a straight line, and the resulting straight line divides the plane of the basic three-dimensional model into a plurality of quadrilateral grids.

5. A digital twin skin penetration needle assistance system as defined in claim 2, wherein the penetration three-dimensional location determination module is configured to perform the steps of:

acquiring a historical skin disease puncture record database which comprises a plurality of groups of skin affected part images and puncture positioning data of historical patients;

establishing a training sample according to the historical skin disease puncture record database, wherein the training sample comprises a training set and a verification set, and the proportion of the training set to the verification set is 6:4, a step of; the input of the training sample training comprises a plurality of groups of skin affected part images of historical patients, and the output of the training is puncture positioning data;

constructing a puncture positioning model prototype by using a convolutional neural network, and training the puncture positioning model prototype by using a training set to obtain the puncture positioning model prototype;

adopting a verification set to optimize the puncture positioning model;

obtaining an image of the affected part of the skin of the current skin patient, inputting the optimized puncture positioning model prototype for calculation, and obtaining puncture positioning data of the current skin patient.

6. The digital twin skin penetration needle assistance system of claim 5, wherein the input of training samples for the penetration localization model further comprises age, gender of the historic patient, and the images of the skin lesions of the historic patient have been manually calibrated for lesion location; meanwhile, in the step of acquiring the image of the affected part of the skin of the current skin patient, inputting the optimized puncture positioning model prototype for calculation to obtain the puncture positioning data of the current skin patient, the input of the optimized puncture positioning model prototype for calculation also comprises the age and sex of the current patient, and the image of the affected part of the skin of the current patient is calibrated by manpower.

7. The digital twin skin penetration needle assist system of claim 5 wherein the assist penetration module is configured to perform the steps of:

establishing a skin puncture digital twin scene, wherein the physical scene is an affected part of the skin of a patient, and the digital scene is a three-dimensional model of the affected part;

constructing a puncture needle with a puncture needle positioning cap in a digital twin digital scene;

marking the puncture data determined by the puncture three-dimensional position determining module in a digital twin scene;

guiding the puncture needle positioning in the physical scene and updating the position of the puncture needle in the digital scene in real time.

8. A digital twin skin penetration needle assistance system as defined in claim 7, wherein said step of constructing a penetration needle with said grid box mounted therein in a digital scene of digital twin, comprises:

acquiring an image of the grid box;

performing image correction and positioning according to square grids of the grid boxes with alternate colors;

obtaining the positioning of the whole puncture needle according to the image and the positioning of the grid box;

in a digital scenario, a twin puncture needle is constructed.

9. The digital twin skin penetration needle assistance system of claim 8, wherein the step of guiding the needle positioning in the physical scene is:

Acquiring puncture positions in the puncture needle head and the current patient in the digital scene, corresponding to target points, and determining the moving direction of the puncture needle head as a target direction according to the relative positions of the puncture needle head and the target points;

the target direction is sent to the single chip microcomputer chip through the wireless communicator, and the single chip microcomputer chip controls the liquid crystal display to display the target direction in a mode that a starting point is an arrow in the center of the liquid crystal display;

acquiring the axial direction of a puncture needle and the axial direction of puncture gradient in the puncture data of the current patient in a digital scene, determining the direction of axial change of the puncture needle according to the included angle of the axial direction of the puncture needle and the axial direction of puncture gradient in the puncture data of the current patient, and taking the axial direction of the puncture gradient in the puncture data of the current patient as an offset target direction;

the offset target direction is sent to the single chip microcomputer chip through the wireless communicator, and the single chip microcomputer chip controls the liquid crystal display screen to display the target direction in a mode of an arrow parallel to the periphery of the liquid crystal display screen;

the target direction and the offset target direction are updated in real time according to the position of the puncture needle head.

10. The digital twin skin penetration needle assistance system of claim 9, wherein the step of guiding the needle positioning in a physical scene further comprises:

When the puncture needle head punctures the skin of an affected part of a patient, if the puncture depth is smaller than the puncture depth in the current puncture data of the patient, an LED control instruction of LED rapid stroboscopic is sent out; if the puncture depth is greater than or equal to the puncture depth in the puncture data of the current patient, an LED common flicker control instruction is sent out; if the puncture inclination is smaller than the puncture inclination in the puncture data of the current patient, an LED control instruction for displaying green by an LED is sent out; if the puncture inclination is greater than or equal to the puncture inclination in the puncture data of the current patient, sending an LED control instruction for displaying red by an LED; the LED control instruction is sent to the singlechip chip through the wireless communicator, and the singlechip chip controls the multicolor LED lamp to display; wherein, the abrupt strobe refers to the flicker with a flicker interval of 0.2s, and the normal flicker refers to the flicker with a flicker interval of 0.5 s.