CN115147469A

CN115147469A - Registration method, device, equipment and storage medium

Info

Publication number: CN115147469A
Application number: CN202210507609.2A
Authority: CN
Inventors: 付威福; 李嘉麟; 陈颖; 刘永; 汪铖杰
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2022-05-10
Filing date: 2022-05-10
Publication date: 2022-10-04

Abstract

The application provides a registration method, a registration device, registration equipment and a storage medium, wherein the method comprises the following steps: performing semantic segmentation on an image to be matched to obtain an interest area of the image to be matched, determining a first sub-image where the interest area in the image to be matched is located, determining a second sub-image of a target template image corresponding to the image to be matched in template image data corresponding to a target, and obtaining a transformation matrix when the image to be matched is registered to the target template image according to the first sub-image and the second sub-image. The region of interest can be accurately segmented through semantic segmentation, and therefore the accuracy of registration can be improved when the region of interest based on accurate segmentation is registered. In addition, according to the embodiment of the application, the first sub-image where the interest area in the image to be matched is located and the second sub-image where the interest area in the target template image is located are registered, the registration process is simple, the registration complexity is further reduced, and the registration efficiency is improved.

Description

Registration method, device, equipment and storage medium

Technical Field

The embodiment of the application relates to the technical field of computers, in particular to a registration method, a registration device, registration equipment and a storage medium.

Background

During the assembly or quality inspection process of the industrial part, the image of the industrial part acquired at the current position needs to be registered with the template drawing so as to obtain a transformation matrix when the image of the industrial part is registered to the template drawing. When the industrial part image is transformed according to the transformation matrix, the part in the industrial part image can be aligned with the part in the template drawing, and the assembly or quality inspection of the industrial part can be further assisted.

The current registration method is to extract feature points of a part in an image to be matched, and then align the feature points of the part in the image to be matched with the feature points of the part in a template map to obtain a transformation matrix of the image to be matched. However, the current registration method cannot accurately extract the feature points of the parts in the image to be matched, so that the registration accuracy is low.

Disclosure of Invention

The application provides a registration method, a registration device and a registration medium, so as to improve the registration accuracy of a target.

In a first aspect, the present application provides a registration method, comprising:

performing semantic segmentation on an image to be matched to obtain an interested area of the image to be matched, wherein the interested area is an area where a target is located;

determining a first sub-image in which the region of interest in the image to be matched is located;

determining a second sub-image of the target template drawing corresponding to the image to be matched in the template drawing data corresponding to the target, wherein the second sub-image is an image area where the interested area of the target template drawing is located;

and obtaining a transformation matrix when the image to be matched is registered to the target template drawing according to the first sub-image and the second sub-image.

In a second aspect, there is provided a registration apparatus comprising:

the segmentation unit is used for performing semantic segmentation on the image to be matched to obtain an interested area of the image to be matched, wherein the interested area is an area where a target is located;

the determining unit is used for determining a first sub-image in which the region of interest in the image to be matched is located;

the searching unit is used for determining a second sub-image of the target template drawing corresponding to the image to be matched in the template drawing data corresponding to the target, wherein the second sub-image is an image area where the interested area of the target template drawing is located;

and the registration unit is used for obtaining a transformation matrix when the image to be matched is registered to the target template graph according to the first sub-image and the second sub-image.

In a third aspect, an electronic device is provided that includes a processor and a memory. The memory is configured to store a computer program, and the processor is configured to call and execute the computer program stored in the memory to perform the method in the first aspect or each implementation manner thereof.

In a fourth aspect, a chip is provided for implementing the method in any one of the first to second aspects or implementations thereof. Specifically, the chip includes: a processor, configured to call and run a computer program from a memory, so that a device on which the chip is installed performs the method according to any one of the above first aspects or the implementation manners thereof.

In a fifth aspect, a computer-readable storage medium is provided for storing a computer program, the computer program causing a computer to perform the method of any one of the above aspects or implementations thereof.

A sixth aspect provides a computer program product comprising computer program instructions for causing a computer to perform the method of any of the above aspects or implementations thereof.

In a seventh aspect, a computer program is provided, which, when run on a computer, causes the computer to perform the method of any one of the above first aspects or implementations thereof.

In summary, according to the method and the device, the interested area of the image to be matched is obtained by performing semantic segmentation on the image to be matched, then, a first sub-image where the interested area in the image to be matched is located is determined, a second sub-image of a target template image corresponding to the image to be matched is determined in template image data corresponding to a target, and then a transformation matrix when the image to be matched is registered to the target template image is obtained according to the first sub-image and the second sub-image. According to the method and the device, the region of interest can be accurately segmented through semantic segmentation, and therefore the accuracy of registration can be improved when the region of interest based on accurate segmentation is registered. In addition, according to the embodiment of the application, the first sub-image where the interest area in the image to be matched is located and the second sub-image where the interest area in the target template image is located are registered, the registration process is simple, the registration complexity is further reduced, and the registration efficiency is improved.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.

Fig. 1 is a schematic view of an application scenario according to an embodiment of the present application;

FIG. 2 is a schematic diagram of a region of interest, a null region, and an interference region according to an embodiment of the present application;

fig. 3 is a schematic flowchart of a registration method according to an embodiment of the present application;

fig. 4 is a schematic diagram of a region of interest obtained by a semantic segmentation method according to an embodiment of the present application;

FIG. 5A is a schematic diagram of feature points obtained by a threshold segmentation method;

FIG. 5B is a schematic diagram of feature points obtained in accordance with an embodiment of the present application;

fig. 6 is a schematic flowchart of a registration method according to an embodiment of the present application;

fig. 7A and 7B are schematic diagrams illustrating a pre-and post-registration comparison provided by an embodiment of the present application;

fig. 8 is a schematic block diagram of a registration apparatus provided in an embodiment of the present application;

fig. 9 is a schematic block diagram of an electronic device provided in an embodiment of the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application.

It should be understood that, in the present embodiment, "B corresponding to a" means that B is associated with a. In one implementation, B may be determined from a. It should also be understood that determining B from a does not mean determining B from a alone, but may be determined from a and/or other information.

In the description of the present application, "plurality" means two or more than two unless otherwise specified.

In addition, in order to facilitate clear description of technical solutions of the embodiments of the present application, in the embodiments of the present application, terms such as "first" and "second" are used to distinguish the same items or similar items having substantially the same functions and actions. Those skilled in the art will appreciate that the terms "first," "second," etc. do not denote any order or quantity, nor do the terms "first," "second," etc. denote any order or importance.

In order to facilitate understanding of the embodiments of the present application, the related concepts related to the embodiments of the present application are first briefly described as follows:

artificial Intelligence (AI) is a theory, method, technique and application system that uses a digital computer or a machine controlled by a digital computer to simulate, extend and expand human Intelligence, perceive the environment, acquire knowledge and use the knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technique of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a manner similar to human intelligence. Artificial intelligence is the research of the design principle and the realization method of various intelligent machines, so that the machines have the functions of perception, reasoning and decision making.

The artificial intelligence technology is a comprehensive subject and relates to the field of extensive technology, namely the technology of a hardware level and the technology of a software level. The artificial intelligence infrastructure generally includes technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technologies, operation/interaction systems, mechatronics, and the like. The artificial intelligence software technology mainly comprises a computer vision technology, a voice processing technology, a natural language processing technology, machine learning/deep learning and the like.

Machine Learning (ML) is a multi-domain cross discipline, and relates to a plurality of disciplines such as probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and the like. The method specially studies how a computer simulates or realizes the learning behavior of human beings so as to acquire new knowledge or skills and reorganize the existing knowledge structure to continuously improve the performance of the computer. Machine learning is the core of artificial intelligence, is the fundamental approach for computers to have intelligence, and is applied to all fields of artificial intelligence. Machine learning and deep learning generally include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and formal education learning.

With the research and progress of artificial intelligence technology, the artificial intelligence technology is developed and researched and applied in a plurality of fields, for example, common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, unmanned aerial vehicles, robots, smart medical care, smart customer service, smart industry and other fields are applied, and more important values are played.

Fig. 1 is a schematic view of an application scenario related to an embodiment of the present application, and includes a terminal device 101 and a server 102.

The terminal device 101 may refer to a device for performing target (e.g., industrial part) registration. The terminal device 101 of the embodiment of the present application may include, but is not limited to: PCs (Personal computers), PDAs (tablet computers), cell phones, wearable smart devices, and the like. The device is often configured with a display device, which may also be a display, a display screen, a touch screen, etc., and a touch screen, a touch panel, etc., and the display device may be used to display the registration result of the target, etc.

The server 102 may be one or more. When the number of the servers 102 is multiple, at least two servers exist for providing different services, and/or at least two servers exist for providing the same service, for example, the same service is provided in a load balancing manner, which is not limited in the embodiment of the present application. The server 102 may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server providing basic cloud computing services such as a cloud service, a cloud database, cloud computing, a cloud function, cloud storage, a Network service, cloud communication, a middleware service, a domain name service, a security service, a CDN (Content Delivery Network), a big data and artificial intelligence platform, and the like. The server 102 may also become a node of the blockchain.

The terminal device 101 and the server 102 may be directly or indirectly connected through wired communication or wireless communication, and the application is not limited herein.

In some embodiments, the server 102 of the embodiment of the present application may implement training of the network model related to the embodiment of the present application, for example, training of the semantic segmentation model, and storing the trained semantic segmentation model. Optionally, the feature extraction network and/or the matching network may be trained, and the trained feature extraction network and/or matching network may be stored. The feature extraction network is used for extracting the feature points of interest in the first sub-image and the second sub-image. The matching network is used for matching the first sub-image with the second sub-image to obtain a transformation matrix of the first sub-image and the second value image.

In some embodiments, a registration platform is installed and operated in the terminal device 101 of the embodiment of the present application, and the registration platform may perform the registration method of the embodiment of the present application. For example, a user performs image acquisition on a target (e.g., an industrial part) at a current position to obtain an image to be matched. The image to be matched is then uploaded to the registration stage.

In some embodiments, the registration method in the embodiments of the present application may be completed by the registration platform, for example, the registration platform loads the network model in the server 102, and the registration method in the embodiments of the present application is implemented to obtain a final registration result, and display the obtained registration result.

In some embodiments, the registration method of the embodiment of the present application may be completed by the server 102, for example, the registration platform sends the image to be matched to the server, and the server executes the registration method of the embodiment of the present application to obtain the registration result, and sends the registration result to the registration platform for displaying.

In some embodiments, the registration method of the embodiment of the present application may be performed by the terminal device 101 and the server 102 together, for example, the server 102 performs operations related to a network model, and the terminal device 101 performs operations other than the network model. Illustratively, the server 102 performs semantic segmentation on an image region where a target is located in an image to be matched by using a semantic segmentation model to obtain an area of interest of the image to be matched, the terminal device 101 determines a first sub-image where the area of interest of the image to be matched is located in, determines a second sub-image of a target template image corresponding to the image to be matched in template image data corresponding to the target, and obtains a transformation matrix when the image to be matched is registered to the target template image according to the first sub-image and the second sub-image.

It should be noted that, the application scenarios of the embodiment of the present application include, but are not limited to, that shown in fig. 1.

The embodiment of the application can be applied to any field needing registration, for example, the technical scheme of the application can be applied to cloud industrial quality inspection projects and can also be applied to a front registration module for automatic mechanical arm grabbing. For example, the image to be matched and the template drawing data at the current view angle are subjected to correlation calculation to obtain the pose change of the part at the current position relative to the part at the template position, so that the positioning of a sensitive area and a shielding area in a quality inspection module is assisted to reduce the over-killing and missing rate of quality inspection, or a mechanical arm is assisted to grasp the part at the current position.

The current registration method is to extract feature points of parts in images to be matched, and align the feature points of the parts in the images to be matched with the feature points of the parts in the template drawing to obtain a transformation matrix of the images to be matched. However, the current registration method cannot accurately extract the feature points of the parts in the image to be matched, so that the registration accuracy is low.

In order to solve the technical problem, in the embodiment of the application, a region of interest of an image to be matched is obtained by performing semantic segmentation on the image to be matched, then, a first sub-image where the region of interest in the image to be matched is located is determined, a second sub-image of a target template image corresponding to the image to be matched is determined in template image data corresponding to a target, and then a transformation matrix when the image to be matched is registered to the target template image is obtained according to the first sub-image and the second sub-image. In the embodiment of the application, the region of interest can be accurately segmented through semantic segmentation, so that the accuracy of registration can be improved when the region of interest based on accurate segmentation is registered. In addition, according to the embodiment of the application, the first sub-image where the interest area in the image to be matched is located and the second sub-image where the interest area in the target template image is located are registered, the registration process is simple, the registration complexity is further reduced, and the registration efficiency is improved.

The technical solutions of the embodiments of the present application are described in detail below with reference to some embodiments. The following several embodiments may be combined with each other and may not be described in detail for the same or similar concepts or processes in some embodiments.

First, a process of constructing template map data according to an embodiment of the present application will be described.

In the embodiments of the present application, a region of interest (ROI), a null region, and an interference region are referred to.

The region of interest is a region to be processed, which is delineated from a processed image in a manner of a square frame, a circle, an ellipse, an irregular polygon and the like in machine vision and image processing, and is called as a region of interest. In the embodiment of the present application, the region of interest is a region where an object is located, for example, a region where an industrial part is located, and there is a consistency, which can be used for registration.

Invalid area: the area in the image where the object is not located.

Interference area: the image belongs to the area where the target is located, but the consistency between the targets is not existed, so that the image can not be used for the registered area.

In the process of product quality inspection and the like, a plurality of images of different points are taken for an object, for example, an industrial part, the images of different points of the object are selected as standard images, and the group of images is called a template image.

In some embodiments, the template map data of the embodiments of the present application includes template maps of the target at different sites, and further includes a region of interest of each template map. That is to say, in the embodiment of the present application, an area of interest where an object in a template map is located is labeled to obtain an area of interest of the template map, specifically, template maps at different points are labeled respectively to obtain an area of interest of each template map in the template maps at different points, and an invalid area or an interference area is outside the area of interest. As shown in fig. 2, the region of interest is a black area, the interference area is a grid area and the null area is a white area.

The specific manner of labeling the region of interest of the template map is not limited in the embodiment of the present application.

In some embodiments, the region of interest of the template map is labeled manually.

In some embodiments, the template map data of embodiments of the present application further includes sub-images of the template map where the region of interest is located. That is to say, after the region of interest of the template map is labeled, the region in which the region of interest of the template map is located is also cut to obtain a sub-image after cutting, and for example, the minimum circumscribed rectangular region of the region of interest of the template map is cut to obtain a sub-image after cutting. In some embodiments, to facilitate alignment, the cropped sub-image is further scaled to scale the longest edge of the cropped sub-image to a preset value, so as to obtain a second sub-image of the template map, where the second sub-image may be understood as an image area where the region of interest of the template map is located. The embodiment of the present application does not limit the specific size of the preset value, for example, w.

In some embodiments, the template drawing data of the embodiment of the present application further includes an interesting feature point corresponding to the second sub-image of the template drawing, and at this time, the embodiment of the present application further includes extraction of the interesting feature point when the template drawing data is constructed. Specifically, for a second subimage of each template map in the template maps at different positions, masking other image regions except for the region of interest in the second subimage, for example, assigning the pixel values of the other image regions except for the region of interest in the second subimage to a first numerical value (e.g., 0), so as to obtain a masked second subimage. And then, extracting feature points of the shielded second sub-image to obtain interesting feature points of the second sub-image, wherein the extracted interesting feature points comprise descriptors of the interesting feature points besides specific position coordinates and identification information, and the descriptors can be understood as feature vectors of the interesting feature points. The embodiment of the application does not limit the specific way of extracting the feature point of interest, for example, the second sub-image obtained by shielding the interference region and the invalid region is input into a pre-trained feature extraction network to extract the feature point, so as to obtain the feature point of interest of the second sub-image. And finally storing the interesting characteristic points of the second sub-image in the template drawing data.

In this way, in the subsequent registration process, the template maps of different point locations, the region of interest of each template map, the second sub-images of each template map, and the feature point of interest of the second sub-images of each template map can be directly obtained from the stored template map data.

The embodiment of the present application does not limit the specific storage location of the template map data.

In some embodiments, the template map data is stored in a cloud storage device, and when the registration method is performed by the registration device in the embodiments of the application, the template map data can be read from the cloud storage device.

In some embodiments, the template map data is stored locally, so that when the registration apparatus of the embodiment of the present application performs the registration method, the template map data can be directly read locally.

On the basis of the above detailed description of the construction process of the template map data, the following describes the registration method provided by the embodiment of the present application.

Fig. 3 is a flowchart illustrating a registration method according to an embodiment of the present application.

The execution subject of the embodiment of the application is a device with a registration function, such as a registration device. In some embodiments, the registration device may be a server. In some embodiments, the registration apparatus may be a terminal device. In some embodiments, the registration apparatus may be a system of a server and a terminal device. Both the server and the terminal device can be understood as electronic devices, and therefore, for convenience of description, the following description will take the execution subject as an example of the electronic device.

As shown in fig. 3, the method of the embodiment of the present application includes:

s301, performing semantic segmentation on the image to be matched to obtain the region of interest of the image to be matched.

As can be seen from the above, the registration is to align the target in the image to be registered with the target in the template map, and the region of interest is a region where the target is located, for example, where the industrial part is located, and there is a consistency that can be used for the registration. Therefore, in the embodiment of the application, when registering, firstly, the region of interest of the image to be matched is determined.

In some embodiments, a threshold method may be used to obtain the region of interest of the image to be matched. However, the threshold segmentation method cannot accurately determine the region where the target is located, and further cannot determine an appropriate threshold for threshold segmentation, so that the segmentation effect is poor, and the threshold segmentation method is easily affected by exposure of a camera, cannot effectively determine the threshold boundary between the invalid region and the industrial part region, and cannot accurately segment the region of interest.

In order to solve the technical problem, in the embodiment of the application, a semantic segmentation method is adopted to segment the region of interest in the image to be matched, so as to obtain the region of interest of the image to be matched.

The embodiment of the present application does not limit the specific manner of performing semantic segmentation on the image to be matched in S301 to obtain the region of interest of the image to be matched.

In a possible implementation manner, a semantic segmentation model is used to perform semantic segmentation on an image to be matched to obtain an interested area of the image to be matched.

That is to say, in the embodiment of the application, before unsupervised online registration is performed on an image to be matched of a target, for example, an industrial part, at a certain point, a ROI semantic segmentation model is first used to perform semantic segmentation on the image to be matched, and a region of interest of the image to be matched is confirmed. For example, as shown in fig. 4, it can be found that the ROI semantic segmentation model can well locate the region of interest in the image, and particularly effectively distinguish the region of interest from the interference region, so that the registration accuracy can be improved when subsequent registration is performed based on the accurately segmented region of interest.

The semantic segmentation model of the embodiment of the application is obtained by training an image with a marked region of interest, for example, the region of interest in the image is manually marked, and the semantic segmentation model is trained by using the marked image, so that the trained semantic segmentation model can accurately segment the region of interest in the image.

In some embodiments, in order to reduce the training workload of the semantic segmentation model, the semantic segmentation model of the embodiments of the present application may be trained based on the labeled template map of the region of interest. Therefore, when the template graph data is constructed, the interested regions in the template graph are labeled, so that training of the semantic segmentation model can be realized by using the template graph with labeled interested regions without independently constructing training data when the semantic segmentation model is trained, the training workload of the semantic segmentation model is further reduced, and the training speed of the semantic segmentation model is increased.

In some embodiments, in order to ensure that the semantic segmentation model is sufficiently trained, in the embodiments of the present application, before the semantic segmentation model is trained by using the template map labeled in the region of interest, the number of points of the template map labeled in the region of interest is first detected. For example, when the number of points of the template graph is sufficient, for example, the number of points > =200, the semantic segmentation model may be trained by directly using the previously labeled template graph. If the number of the points of the template graph is less, the semantic segmentation model can be trained by referring to the template graph marking mode and marking a part of data of other points, so that the trained semantic segmentation model can accurately segment the region of interest of the images of different points.

The embodiment of the present application does not limit the specific network structure of the semantic segmentation model.

In a possible implementation manner, the semantic segmentation model in the embodiment of the present application may adopt a SegFormer semantic segmentation model based on deep learning.

S302, determining a first sub-image in which the interest area in the image to be matched is located.

Because the point location of the target in the image to be matched is different from the point location of the target in the template map, and the target is in an irregular shape, the shape of the region of interest obtained by segmentation according to the method is irregular, and the calculation for matching the irregular image is complex. In order to reduce the matching complexity and improve the matching efficiency, the region of interest obtained by segmentation in the previous step is processed in the step, and a first sub-image where the region of interest in the image to be matched is located is determined.

The embodiment of the present application does not limit the specific shape of the first sub-image. For example, any circumscribed regular pattern of the region of interest in the image to be matched may be used, for example, a circumscribed circle, a circumscribed ellipse, a circumscribed polygon, and the like of the region of interest.

In some embodiments, in order to reduce the invalid region and the interference region in the first sub-image, in the embodiments of the present application, the minimum circumscribed rectangular region of the region of interest in the image to be matched is cropped to obtain a cropped sub-image. Further, in order to facilitate alignment with the second sub-image in the template map, the cropped sub-image is scaled so that the longest edge of the cropped sub-image is scaled to a preset value, and the first sub-image is obtained.

The embodiment of the present application does not limit the specific size of the preset value, for example, w.

That is to say, in the embodiment of the present application, the method for determining the first sub-image in which the region of interest in the image to be matched is located is the same as the method for determining the second sub-image in which the region of interest in the template map is located. For example, in the embodiment of the present application, the minimum circumscribed rectangular area of the region of interest in the template map is clipped to obtain the clipped sub-image, and the clipped sub-image is scaled so as to scale the longest edge of the clipped sub-image to a preset value to obtain the second sub-image. And similarly, cutting the minimum circumscribed rectangular area of the interest area in the image to be matched to obtain a cut sub-image, and zooming the cut sub-image to enable the longest edge of the cut sub-image to be zoomed to a preset value to obtain a first sub-image. The shapes of the first sub-image and the second sub-image obtained in this way are the same, for example, both are rectangular, and the longest edges of the first sub-image and the second sub-image are both w, so that the matching efficiency of the first sub-image and the second sub-image can be improved during subsequent matching.

According to the method, after the first sub-image where the region of interest in the image to be matched is located is determined, the following step S303 is executed to obtain the second sub-image in the template data.

S303, determining a second sub-image of the target template drawing corresponding to the image to be matched in the template drawing data corresponding to the target.

And the second sub-image is an image area where the interested area of the target template picture is located.

It should be noted that there is no strict sequence between the execution of the foregoing S303 and the execution of the foregoing S301 and S302, for example, the foregoing S303 may be executed before the foregoing S301, or executed after the foregoing S301, or executed in synchronization with the foregoing S301, or executed between the foregoing S301 and S302, or executed in synchronization with the foregoing S302, which is not limited in this embodiment of the application.

In view of the above, different targets correspond to different template graph data, and the template graph data includes template graphs of targets at different points. In this way, when the image to be matched is registered with the template map, in the embodiment of the present application, firstly, one template map needs to be selected from the template maps at different points included in the template map data corresponding to the target as the target image, so as to register the image to be matched with the target image.

In the embodiment of the present application, a specific manner of determining the target template graph from the template graph data corresponding to the target is not limited.

In some embodiments, an arbitrary one of the template drawings in the template drawing data is determined as the target template drawing.

In some embodiments, the template map corresponding to the point location of the image to be matched in the template map data is determined as the target template map.

In some embodiments, if the template map data does not include a template map that completely coincides with the point locations of the images to be matched, the template map closest to the point locations of the images to be matched in the template map data may be determined as the target template map.

According to the above steps, after the target template drawing is obtained from the template drawing data, a second sub-image of the target template drawing can be obtained from the template drawing data.

S304, obtaining a transformation matrix when the image to be matched is registered to the target template picture according to the first sub-image and the second sub-image.

According to the steps, after the first sub-image of the image to be matched and the second sub-image of the target template drawing are determined, the transformation matrix when the image to be matched is registered to the target template drawing can be obtained according to the first sub-image and the second sub-image.

In the embodiment of the present application, the method for obtaining the transformation matrix when the image to be matched is registered to the target template map according to the first sub-image and the second sub-image includes, but is not limited to, the following:

in the first mode, feature points of the first sub-image and the second sub-image are extracted, and matching is performed based on the feature points of the first sub-image and the second sub-image to obtain a transformation matrix between the first sub-image and the second sub-image. Then, according to the transformation matrix between the first sub-image and the second sub-image, the transformation matrix when the image to be matched is registered to the target template drawing is determined, for example, the transformation matrix between the first sub-image and the second sub-image is determined as the transformation matrix when the image to be matched is registered to the target template drawing.

In a second mode, the step S304 includes the following steps S304-a to S304-C:

S304-A, matching the first sub-image with the second sub-image to obtain a first matching value.

In the second mode, the first sub-image and the second sub-image can be matched to obtain a first matching value, the transformation matrix between the first sub-image and the second sub-image is determined according to the first matching value, and the transformation matrix when the image to be matched is registered to the target template graph is determined according to the transformation matrix between the first sub-image and the second sub-image.

The embodiment of the present application does not limit the specific manner of determining the first matching value of the first sub-image and the second sub-image in S304-a.

In a possible implementation manner, the first sub-image and the second sub-image are subjected to matching of an intersection ratio IoU to obtain a first matching value.

Illustratively, the first matching value of the first sub-image and the second sub-image is determined according to the following formula (1):

IoU＝(A∩B)/(A∪B) (1)

wherein, ioU is a first matching value, a is a first sub-image, B is the first sub-image, u represents a union operation, and n represents an intersection operation. The larger the IoU is, the higher the coincidence degree of the first sub-image and the second sub-image is, and the higher the matching degree is.

And S304-B, determining a transformation matrix between the first sub-image and the second sub-image according to the first matching value.

According to the method, after the first matching degree between the first sub-image and the second sub-image is determined, the transformation matrix between the first sub-image and the second sub-image can be determined according to the first matching value. For example, if the first matching degree is relatively high, for example, the first matching value is greater than or equal to the preset threshold, it indicates that the first sub-image and the second sub-image are relatively matched, that is, the registration of the first sub-image and the second sub-image can be successfully achieved directly through image registration, and the steps of mask masking, feature extraction, and matching and the like are not required. If the first matching value is smaller than the preset threshold, it indicates that the first sub-image and the second sub-image cannot be successfully matched through image registration, and at this time, matching of the feature points needs to be performed.

As can be seen from the above, the determining, in S304-B, the transformation matrix between the first sub-image and the second sub-image according to the first matching value specifically includes the following cases 1 and 2:

in case 1, if the first matching value is greater than or equal to the preset threshold, the transformation matrix between the first sub-image and the second sub-image is determined as the identity matrix.

In this case 1, if the first matching value between the first sub-image and the second sub-image is greater than or equal to the preset threshold, it indicates that the matching between the first sub-image and the second sub-image can be achieved by image registration. That is to say, in the embodiment of the present application, the first sub-image of the image to be matched is matched with the second sub-image of the target template map, and the unit matrix can be directly determined as the transformation matrix of the first sub-image and the second sub-image without performing feature point matching, so that the registration efficiency of the first sub-image and the second sub-image is improved, and the registration complexity of the first sub-image and the second sub-image is reduced.

In case 2, if the first matching value between the first sub-image and the second sub-image is smaller than the preset threshold, it indicates that the matching between the first sub-image and the second sub-image cannot be realized through image calibration, and at this time, the following steps S304-B1 to S304-B3 need to be performed:

S304-B1, if the first matching value is smaller than the preset threshold value, masking other image areas except the region of interest in the first sub-image to obtain a masked first sub-image.

Since the first sub-image may include an interference area and an invalid area in addition to the region of interest. At this time, in order to reduce the interference of the interference region and the invalid region on the region of interest, the embodiment of the present application masks the other image regions in the first sub-image except the region of interest. The method comprises the steps of masking an interference area and an invalid area of a first sub-image to obtain a masked first sub-image, wherein only an interested area in the masked first sub-image is visible, so that accurate extraction of feature points of the interested area can be realized during subsequent feature point extraction.

The embodiment of the present application does not limit the specific manner of masking the other image regions except the region of interest in the first sub-image in S304-B1.

In a possible implementation manner, the pixel values of the other image regions except the region of interest in the first sub-image are assigned to the first numerical value, so as to obtain the masked first sub-image.

The embodiment of the present application does not limit the specific value of the first numerical value, for example, the first numerical value is 0, that is, the embodiment of the present application may set the pixel values in the interference region and the invalid region in the first sub-image, except for the region of interest, to be 0, so as to shield other image regions in the first sub-image, except for the region of interest, and obtain the shielded first sub-image.

That is to say, in the embodiment of the present application, the interest region in the first sub-image is equivalent to a mask, the RGB values of the interest region in the first sub-image obtained in the previous step are retained, and the pixel values outside the interest region, which are the invalid region and the interference region, are assigned as 0 values, so as to obtain the first sub-image after being masked.

S304-B2, extracting the feature points of the shielded first sub-image to obtain the interested feature points of the first sub-image.

In the embodiment of the application, because the interference area and the invalid area in the first sub-image are shielded, when the feature point of the shielded first sub-image is extracted, the feature point of the interest area in the first sub-image can be accurately extracted. For convenience of description, in the embodiment of the present application, the extracted feature points of the region of interest are regarded as feature points of interest. Therefore, the accuracy of registration can be improved when the subsequent registration is carried out based on the accurately extracted interested feature points.

For example, to further illustrate the benefit of the feature point extraction method provided in the embodiment of the present application, the feature point extraction method in the embodiment of the present application is compared with the feature point extraction method in the threshold segmentation method. Fig. 5A and 5B show a comparison result, where 5A is a schematic diagram of feature points obtained by a threshold segmentation method, and fig. 5B is a schematic diagram of feature points obtained in an embodiment of the present application. As can be seen from fig. 5B, in the embodiment of the present application, after the interference region and the invalid region in the first sub-image are shielded, feature point extraction is performed, and most of extracted feature points are concentrated around the region of interest. As shown in fig. 5A, the feature points extracted by using the threshold segmentation method are easily concentrated on the interference region, because there is more texture information in the interference region than in the region of interest, although the texture information is abundant, the textures of different parts in the interference region are different, and the registration cannot be performed by using the textures, so that most of the feature points on the interference region belong to invalid feature points.

In the embodiment of the present application, the specific manner of extracting the feature point of the shielded first sub-image in S304-B2 to obtain the feature point of interest of the first sub-image is not limited.

In a possible implementation manner, feature point extraction is performed on the shielded first sub-image through a feature extraction network, so as to obtain an interesting feature point of the first sub-image. For example, the masked first sub-image is input into a feature extraction network, and the feature extraction network performs multi-layer feature extraction to output the feature point of interest of the first sub-image.

The embodiment of the present application does not limit the specific network structure of the feature extraction network, and may be a neural network model for arbitrarily implementing feature point extraction.

According to the above steps, after the feature point of interest in the masked first sub-image is extracted, the following steps S304-B3 are performed.

S304-B3, matching the interesting characteristic points of the first sub-image with the interesting characteristic points of the second sub-image to obtain a transformation matrix between the first sub-image and the second sub-image.

In the embodiment of the present application, the feature point of interest of the second sub-image may be directly obtained from the template map data, or may be obtained by performing feature extraction on the second sub-image.

The embodiment of the present application does not limit the specific manner of extracting the feature point of interest of the second sub-image, for example, the feature extraction network is used to extract the feature point of interest of the second sub-image, so as to obtain the feature point of interest of the second sub-image.

In some embodiments, in order to further improve the accuracy of the registration, the embodiment of the present application extracts the feature point of interest of the second sub-image by the same method as the above-mentioned extraction of the feature point of interest of the first sub-image. For example, masking other regions except the region of interest in the second sub-image to obtain a masked second sub-image, and inputting the masked second sub-image into the feature extraction network to perform feature point extraction to obtain the feature point of interest of the second sub-image.

According to the method, after the interesting feature point of the first sub-image of the image to be matched and the interesting feature point of the second sub-image of the target template picture are obtained, the interesting feature point of the first sub-image is matched with the interesting feature point of the second sub-image, and the transformation matrix between the first sub-image and the second sub-image is obtained.

The embodiment of the present application does not limit the specific manner of obtaining the transformation matrix between the first sub-image and the second sub-image by matching the feature of interest of the first sub-image with the feature of interest of the second sub-image. For example, any matching method is used to match the feature point of interest of the first sub-image with the feature point of interest of the second sub-image, and the specific manner of obtaining the transformation matrix between the first sub-image and the second sub-image is not limited.

In a possible implementation manner, the feature point of interest of the first sub-image and the feature point of interest of the second sub-image may be matched through a pre-trained matching network, so as to obtain a transformation matrix between the first sub-image and the second sub-image.

The embodiment of the present application does not limit the specific network structure of the matching network.

In the embodiment of the application, the interference area and the invalid area in the first sub-image are shielded, and the feature point of the shielded first sub-image is extracted, so that the extracted feature point is concentrated in the region of interest, and therefore, when the feature point is matched with the feature point of the second sub-image, the transformation matrix between the first sub-image and the second sub-image can be accurately obtained.

S304-C, determining a transformation matrix of the image to be matched according to the transformation matrix between the first sub-image and the second sub-image.

In some embodiments, if the coordinate system of the image to be matched is the same as the coordinate system of the original image, the transformation matrix between the first sub-image and the second sub-image may be determined as the transformation matrix of the image to be matched.

In some embodiments, if the coordinate system of the image to be matched is different from the coordinate system of the original image, the embodiment of the present application further includes matrix transformation, that is, after the transformation matrix between the first sub-image and the second sub-image is obtained, the transformation matrix needs to be back-projected into the coordinate system of the original image. Based on this, S304-C includes the following steps of S304-C1 and S304-C2:

S304-C1, obtaining the scale and the offset of the image to be matched and the scale and the offset of the target template graph.

Suppose that the image to be matched has a scale of

And

the offset of the image to be matched is

And

assume that the target template graph has dimensions of

And

the target template graph has an offset of

And

S304-C2, according to the scale and the offset of the image to be matched and the scale and the offset of the target template graph, projecting the transformation matrix between the first sub-image and the second sub-image into the coordinate system of the original image to obtain the transformation matrix of the image to be matched.

Illustratively, a transformation matrix of the image to be matched is determined according to the following formula (2):

wherein the content of the first and second substances,

and

in order to be a scale of the target template map,

and

for the offset of the target template map,

and

is the scale of the image to be matched,

and

is the offset of the image to be matched. M is a transformation matrix between the first sub-image and the second sub-image, M _final Is a transformation matrix of the image to be matched.

In one example, referring to case 1 above, if the first matching value is greater than or equal to the preset threshold, the transformation matrix between the first sub-image and the second sub-image is determined as an identity matrix, that is, the transformation matrix is determined as an identity matrix

At this time, a transformation matrix of the image to be matched is determined according to the following formula (3):

in another example, referring to the above case 2, if the first matching value is smaller than the preset threshold, masking other image regions except the region of interest in the first sub-image to obtain a masked first sub-image, performing feature point extraction on the masked first sub-image to obtain a feature point of interest in the first sub-image, matching the feature point of interest in the first sub-image with the feature point of interest in the second sub-image to obtain a transformation matrix M1 between the first sub-image and the second sub-image, and at this time, determining the transformation matrix of the image to be matched according to the following formula (4):

in the embodiment of the application, according to the steps, a transformation matrix of the image to be matched can be determined, and the transformation matrix can be understood as the pose change of the target in the image to be matched relative to the target in the template image. Therefore, according to the transformation matrix, the positioning of a sensitive area and a shielding area in the quality inspection module can be assisted so as to reduce the over-killing and missing inspection rate of quality inspection, or the mechanical arm is assisted to grab the target at the current position.

According to the registration method provided by the embodiment of the application, the interesting area of the image to be matched is obtained by performing semantic segmentation on the image to be matched, then, a first sub-image where the interesting area in the image to be matched is located is determined, a second sub-image of a target template image corresponding to the image to be matched is determined in template image data corresponding to a target, and then a transformation matrix when the image to be matched is registered to the target template image is obtained according to the first sub-image and the second sub-image. According to the method and the device, the region of interest can be accurately segmented through semantic segmentation, and therefore the accuracy of registration can be improved when the region of interest based on accurate segmentation is registered. In addition, according to the embodiment of the application, the first sub-image where the interest area in the image to be matched is located and the second sub-image where the interest area in the target template image is located are registered, the registration process is simple, the registration complexity is further reduced, and the registration efficiency is improved.

The foregoing describes a registration method provided in an embodiment of the present application. On this basis, a registration method provided by another embodiment of the present application is described below with reference to fig. 6. The registration method shown in fig. 6 can be understood as a specific registration method of the present application.

Fig. 6 is a flowchart illustrating a registration method according to an embodiment of the present application. As shown in fig. 6, a registration method according to an embodiment of the present application includes:

s401, performing semantic segmentation on the image to be matched to obtain an interested area of the image to be matched.

Wherein the region of interest is the region where the target is located.

For example, a semantic segmentation model is used to perform semantic segmentation on the image to be matched, so as to obtain the region of interest of the image to be matched.

The specific implementation process of S401 may refer to the specific description of S301, which is not described herein again.

S402, determining a first sub-image where the interest area in the image to be matched is located.

For example, in the image to be matched, the minimum circumscribed rectangular region of the region of interest is cropped to obtain a cropped sub-image.

The specific implementation process of S402 may refer to the specific description of S302, which is not described herein again.

S403, determining a second sub-image of the target template image corresponding to the image to be matched in the template image data corresponding to the target.

And the second sub-image is an image area where the region of interest of the target template graph is located.

The specific implementation process of S403 may refer to the specific description of S303, which is not described herein again.

S404, matching the first sub-image with the second sub-image to obtain a first matching value.

For example, the first sub-image and the second sub-image are subjected to cross-over matching compared with the IOU, and a first matching value is obtained.

The specific implementation process of S404 may refer to the specific description of S303, which is not described herein again.

S405, judging whether the first matching value is larger than a preset threshold value.

If the first matching value is greater than or equal to the preset threshold, it indicates that the registration of the first sub-image and the second sub-image can be successfully achieved through image registration, and at this time, the following steps S406 and S407 are performed.

If the first matching value is smaller than the preset threshold, it indicates that the registration of the first sub-image and the second sub-image cannot be achieved through image registration, and at this time, the following steps S408 to S411 are performed.

S406, if the first matching value is larger than or equal to the preset threshold value, determining a transformation matrix between the first sub-image and the second sub-image as an identity matrix.

And S407, determining a transformation matrix of the image to be matched according to the scale and the offset of the image to be matched, the unit matrix and the scale and the offset of the target template picture.

In the embodiment of the present application, if the first matching value is greater than or equal to the preset threshold, it indicates that the registration of the first sub-image and the second sub-image can be successfully achieved through image registration, and at this time, the transformation matrix between the first sub-image and the second sub-image is determined as the unit matrix, and the transformation matrix of the image to be matched is determined according to the scale and the offset of the image to be matched, the unit matrix, and the product of the scale and the offset of the target template map.

For example, a transformation matrix of the image to be matched is determined according to the above formula (3).

And S408, if the first matching value is smaller than the preset threshold, masking other image areas except the region of interest in the first sub-image to obtain a masked first sub-image.

For example, the pixel values of the image regions other than the region of interest in the first sub-image are assigned to the first numerical value, so as to obtain the masked first sub-image.

The specific implementation process of S408 may refer to the specific description of S304-B1, which is not described herein again.

And S409, extracting the feature points of the shielded first sub-image to obtain the interested feature points of the first sub-image.

For example, feature point extraction is performed on the masked first sub-image through a feature extraction network, so as to obtain an interesting feature point of the first sub-image.

S410, matching the interesting characteristic point of the first sub-image with the interesting characteristic point of the second sub-image to obtain a transformation matrix between the first sub-image and the second sub-image.

For example, the feature point of interest of the first sub-image is matched with the feature point of interest of the second sub-image through a matching network, so as to obtain a transformation matrix between the first sub-image and the second sub-image.

S411, determining the transformation matrix of the image to be matched according to the scale and the offset of the image to be matched, the transformation matrix between the first sub-image and the second sub-image and the scale and the offset of the target template graph.

In the embodiment of the present application, if the first matching value is smaller than the preset threshold, it is indicated that the registration of the first sub-image and the second sub-image cannot be achieved through image registration, and at this time, the accurate registration of the first sub-image and the second sub-image is achieved through a registration method of the feature point. In addition, in order to further improve the accuracy of registration, the other regions except the region of interest in the first sub-image are shielded, the feature points of the first sub-image after shielding are extracted, the feature points of interest of the first sub-image can be accurately extracted, and therefore when the feature points of interest of the first sub-image and the feature points of interest of the second sub-image are matched based on the accurately extracted feature points of interest, the accuracy of registration of the first sub-image and the second sub-image can be further improved, and further the transformation matrix of the image to be matched is accurately determined.

Further, the registration method according to the embodiment of the present application is applied to the industrial field, and the registration visualization result is shown in fig. 7A and 7B, in each of fig. 7A and 7B, the left side in the figure is before registration, and the right side in the figure is after registration performed by using the method of the present application, where dark gray is the position of the part in the template map, and white is the position of the part in the image to be registered. As shown in fig. 7A and 7B, no matter whether the current point position image of the industrial part contains an interference region, accurate registration can be achieved by using the registration method provided by the embodiment of the present application, but the registration method based on threshold segmentation and local feature points cannot achieve accurate registration when the current point position image contains the interference region.

According to the method and the device, the region of interest can be accurately segmented through semantic segmentation, and therefore the accuracy of registration can be improved when the region of interest based on accurate segmentation is registered. In addition, according to the embodiment of the application, the first sub-image and the second sub-image are matched to obtain a first matching value, and then whether the first matching value is larger than a preset threshold value or not is judged. In case 1, if the first matching value is greater than or equal to the preset threshold, determining the transformation matrix between the first sub-image and the second sub-image as the identity matrix, and determining the transformation matrix of the image to be matched according to the scale and offset of the image to be matched, the identity matrix, and the scale and offset of the target template map, that is, in case 1, the registration of the first sub-image and the second sub-image can be realized through image registration, and the registration process is simple and the registration efficiency is high. In case 2, if the first matching value is smaller than the preset threshold, masking other image regions except the region of interest in the first sub-image to obtain a masked first sub-image, where the masked first sub-image only includes the region of interest, so that feature point extraction is performed on the basis of the masked first sub-image, and the feature point of interest in the first sub-image can be accurately obtained, and then matching is performed on the basis of the feature point of interest in the accurately extracted first sub-image and the feature point of interest in the second sub-image, so that the accuracy of determining the transformation matrix between the first sub-image and the second sub-image can be effectively improved, and further, on the basis of the accurately determined transformation matrix between the first sub-image and the second sub-image, the accurate determination of the transformation matrix of the image to be matched can be realized, and the accuracy of registration of the image to be matched is improved.

Method embodiments of the present application are described in detail above with reference to fig. 2 to 6, and apparatus embodiments of the present application are described in detail below with reference to fig. 8 to 9.

Fig. 8 is a schematic block diagram of a registration apparatus provided in an embodiment of the present application. The apparatus 10 may be or be part of an electronic device.

As shown in fig. 8, the registration device 10 may include:

the segmentation unit 11 is configured to perform semantic segmentation on an image to be matched to obtain an interested region of the image to be matched, where the interested region is a region where a target is located;

a determining unit 12, configured to determine a first sub-image in which the region of interest in the image to be matched is located;

the searching unit 13 is configured to determine, in the template map data corresponding to the target, a second sub-image of the target template map corresponding to the image to be matched, where the second sub-image is an image area where an area of interest of the target template map is located;

and the registration unit 14 is configured to obtain a transformation matrix when the image to be matched is registered to the target template map according to the first sub-image and the second sub-image.

In some embodiments, the registration unit 14 is specifically configured to match the first sub-image and the second sub-image to obtain a first matching value; determining a transformation matrix between the first sub-image and the second sub-image according to the first matching value; and determining a transformation matrix of the image to be matched according to the transformation matrix between the first sub-image and the second sub-image.

In some embodiments, the registration unit 14 is specifically configured to perform intersection matching on the first sub-image and the second sub-image, and obtain the first matching value.

In some embodiments, the registration unit 14 is specifically configured to determine, if the first matching value is greater than or equal to a preset threshold, that the transformation matrix between the first sub-image and the second sub-image is an identity matrix.

In some embodiments, the registration unit 14 is specifically configured to mask and shield other image regions except the region of interest in the first sub-image to obtain a shielded first sub-image if the first matching value is smaller than a preset threshold; extracting feature points of the shielded first sub-image to obtain interesting feature points of the first sub-image; and matching the interesting characteristic point of the first sub-image with the interesting characteristic point of the second sub-image to obtain a transformation matrix between the first sub-image and the second sub-image.

In some embodiments, the registration unit 14 is specifically configured to assign the pixel values of the image regions of the first sub-image except the region of interest to a first numerical value, so as to obtain the masked first sub-image.

In some embodiments, the registration unit 14 is specifically configured to obtain a scale and an offset of the image to be matched and a scale and an offset of the target template map; and projecting the transformation matrix between the first sub-image and the second sub-image into a coordinate system of an original image according to the scale and the offset of the image to be matched and the scale and the offset of the target template graph to obtain the transformation matrix of the image to be matched.

In some embodiments, the template map data corresponding to the target includes template maps of different sites of the target and second sub-images of the template maps of the different sites, and the search unit 13 is specifically configured to determine the template map corresponding to the point location of the image to be matched in the template map data as the target template map; and obtaining a second sub-image of the target template drawing from the template drawing data.

In some embodiments, the determining unit 12 is specifically configured to crop the minimum circumscribed rectangular region of the region of interest in the image to be matched, so as to obtain a cropped sub-image; and zooming the cropped sub-image to zoom the longest edge of the cropped sub-image to a preset value to obtain the first sub-image.

In some embodiments, the template map data corresponding to the target further includes a feature point of interest corresponding to a second subimage of each template map in the template maps at different positions, and the registration unit 14 is further configured to mask, for the second subimage of each template map in the template maps at different positions, other image areas except for the region of interest in the second subimage, to obtain a masked second subimage; extracting feature points of the shielded second sub-image to obtain interesting feature points of the second sub-image; storing the specific points of interest of the second sub-image in the template map data corresponding to the target.

In some embodiments, the segmentation unit 11 is specifically configured to perform semantic segmentation on the image to be matched by using a semantic segmentation model, so as to obtain an interesting region of the image to be matched.

Optionally, the semantic segmentation model is trained based on a template map with labeled regions of interest.

It is to be understood that the apparatus embodiments and the method embodiments may correspond to each other and similar descriptions may be made with reference to the method embodiments. To avoid repetition, the description is omitted here. Specifically, the apparatus shown in fig. 8 may perform the embodiment of the method, and the foregoing and other operations and/or functions of each module in the apparatus are respectively for implementing the embodiment of the method, and are not described herein again for brevity.

The apparatus of an embodiment of the present application is described above in terms of functional modules in conjunction with the following figures. It should be understood that the functional modules may be implemented by hardware, by instructions in software, or by a combination of hardware and software modules. Specifically, the steps of the method embodiments in the present application may be implemented by integrated logic circuits of hardware in a processor and/or instructions in the form of software, and the steps of the method disclosed in conjunction with the embodiments in the present application may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. Alternatively, the software modules may be located in a random access memory, flash memory, read only memory, programmable read only memory, electrically erasable programmable memory, registers, or other storage medium known in the art. The storage medium is located in a memory, and a processor reads information in the memory and completes the steps in the above method embodiments in combination with hardware thereof.

Fig. 9 is a schematic block diagram of an electronic device provided in an embodiment of the present application, and the electronic device is configured to execute the above method embodiment.

As shown in fig. 9, the electronic device 30 may include:

a memory 31 and a processor 32, the memory 31 being arranged to store a computer program 33 and to transfer the program code 33 to the processor 32. In other words, the processor 32 may call and run the computer program 33 from the memory 31 to implement the method in the embodiment of the present application.

For example, the processor 32 may be adapted to perform the above-mentioned method steps according to instructions in the computer program 33.

In some embodiments of the present application, the processor 32 may include, but is not limited to:

general purpose processors, digital Signal Processors (DSPs), application Specific Integrated Circuits (ASICs), field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components, and the like.

In some embodiments of the present application, the memory 31 includes, but is not limited to:

volatile memory and/or non-volatile memory. The non-volatile Memory may be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically Erasable PROM (EEPROM), or a flash Memory. Volatile Memory can be Random Access Memory (RAM), which acts as external cache Memory. By way of example, and not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), synchronous Dynamic Random Access Memory (SDRAM), double Data Rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), SLDRAM (Synchronous link DRAM), and Direct Rambus RAM (DR RAM).

In some embodiments of the present application, the computer program 33 may be divided into one or more modules, which are stored in the memory 31 and executed by the processor 32 to perform the method of recording pages provided herein. The one or more modules may be a series of computer program instruction segments capable of performing certain functions, which are used to describe the execution of the computer program 33 in the electronic device.

As shown in fig. 9, the electronic device 30 may further include:

a transceiver 34, the transceiver 34 being connectable to the processor 32 or the memory 31.

The processor 32 may control the transceiver 34 to communicate with other devices, and in particular, may transmit information or data to or receive information or data transmitted by other devices. The transceiver 34 may include a transmitter and a receiver. The transceiver 34 may further include one or more antennas.

It should be understood that the various components in the electronic device 30 are connected by a bus system that includes a power bus, a control bus, and a status signal bus in addition to a data bus.

According to an aspect of the present application, there is provided a computer storage medium having a computer program stored thereon, which, when executed by a computer, enables the computer to perform the method of the above-described method embodiments.

Embodiments of the present application also provide a computer program product containing instructions, which when executed by a computer, cause the computer to perform the method of the above method embodiments.

According to another aspect of the application, a computer program product or computer program is provided, comprising computer instructions stored in a computer readable storage medium. The processor of the electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device executes the method of the above method embodiment.

In other words, when implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. The procedures or functions described in accordance with the embodiments of the present application occur, in whole or in part, when the computer program instructions are loaded and executed on a computer. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored on a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, from one website, computer, server, or data center to another website, computer, server, or data center via wire (e.g., coaxial cable, fiber optic, digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that includes one or more of the available media. The usable medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a Digital Video Disk (DVD)), or a semiconductor medium (e.g., a Solid State Disk (SSD)), among others.

Those of ordinary skill in the art will appreciate that the various illustrative modules and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the module is only one logical functional division, and other divisions may be realized in practice, for example, a plurality of modules or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or modules, and may be in an electrical, mechanical or other form.

Modules described as separate parts may or may not be physically separate, and parts shown as modules may or may not be physical modules, may be located in one place, or may be distributed on a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. For example, functional modules in the embodiments of the present application may be integrated into one processing module, or each of the modules may exist alone physically, or two or more modules are integrated into one module.

The above description is only for the specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present application, and shall be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A registration method, comprising:

2. The method according to claim 1, wherein obtaining a transformation matrix when the image to be matched is registered to the target template map according to the first sub-image and the second sub-image comprises:

matching the first sub-image with the second sub-image to obtain a first matching value;

determining a transformation matrix between the first sub-image and the second sub-image according to the first matching value;

and determining a transformation matrix of the image to be matched according to the transformation matrix between the first sub-image and the second sub-image.

3. The method of claim 2, wherein matching the first sub-image with the second sub-image to obtain a first matching value comprises:

and matching the first sub-image and the second sub-image by using an intersection ratio IOU (input output Unit) to obtain the first matching value.

4. The method of claim 2, wherein determining a transformation matrix between the first sub-image and the second sub-image according to the first match value comprises:

if the first matching value is greater than or equal to a predetermined threshold, determining a transformation matrix between the first sub-image and the second sub-image as an identity matrix.

5. The method of claim 2, wherein determining a transformation matrix between the first sub-image and the second sub-image according to the first match value comprises:

if the first matching value is smaller than a preset threshold value, masking other image areas except the region of interest in the first sub-image to obtain a masked first sub-image;

extracting feature points of the shielded first sub-image to obtain interesting feature points of the first sub-image;

and matching the interesting characteristic point of the first sub-image with the interesting characteristic point of the second sub-image to obtain a transformation matrix between the first sub-image and the second sub-image.

6. The method according to claim 5, wherein the masking the other image regions except the region of interest in the first sub-image to obtain the masked first sub-image comprises:

and assigning the pixel values of other image areas except the interesting area in the first sub-image to be a first numerical value to obtain the shielded first sub-image.

7. The method according to any of claims 2-6, wherein determining the transformation matrix of the image to be matched according to the transformation matrix between the first sub-image and the second sub-image comprises:

obtaining the scale and the offset of the image to be matched and the scale and the offset of the target template graph;

and projecting the transformation matrix between the first sub-image and the second sub-image into a coordinate system of an original image according to the scale and the offset of the image to be matched and the scale and the offset of the target template graph to obtain the transformation matrix of the image to be matched.

8. The method according to any one of claims 1 to 6, wherein the determining a first sub-image of the image to be matched in which the region of interest is located comprises:

in the image to be matched, the minimum circumscribed rectangular area of the region of interest is cut to obtain a cut sub-image;

and zooming the cropped sub-image to zoom the longest edge of the cropped sub-image to a preset value to obtain the first sub-image.

9. The method according to any one of claims 1 to 6, wherein the template drawing data corresponding to the object includes template drawings of different positions of the object and second sub-images of the template drawings of the different positions, and the determining of the second sub-images of the template drawings of the object corresponding to the image to be matched in the template drawing data corresponding to the object includes:

determining the template drawing corresponding to the point position of the image to be matched in the template drawing data as the target template drawing;

and obtaining a second sub-image of the target template drawing from the template drawing data.

10. The method according to claim 9, wherein the template map data corresponding to the target further includes the feature point of interest corresponding to the second sub-image of each template map in the template maps of the different sites, and the method further includes:

for a second subimage of each template map in the template maps of different sites, masking and shielding other image areas except the region of interest in the second subimage to obtain a shielded second subimage;

extracting feature points of the shielded second sub-image to obtain interesting feature points of the second sub-image;

storing the specific point of interest of the second sub-image in the template map data corresponding to the target.

11. The method according to any one of claims 1 to 6, wherein the performing semantic segmentation on the image to be matched to obtain the region of interest of the image to be matched comprises:

and performing semantic segmentation on the image to be matched by using a semantic segmentation model to obtain the region of interest of the image to be matched.

12. The method of claim 11, wherein the semantic segmentation model is trained based on a template map labeled in a region of interest.

13. A registration apparatus, comprising:

and the registration unit is used for obtaining a transformation matrix when the image to be matched is registered to the target template drawing according to the first sub-image and the second sub-image.

14. An electronic device comprising a processor and a memory;

the memory for storing a computer program;

the processor for executing the computer program to implement the method of any one of the preceding claims 1 to 13.

15. A computer-readable storage medium for storing a computer program which causes a computer to perform the method of any one of claims 1 to 13.