CN117094996B - Linear detection system based on FPGA and LSD algorithm and control method thereof - Google Patents
Linear detection system based on FPGA and LSD algorithm and control method thereof Download PDFInfo
- Publication number
- CN117094996B CN117094996B CN202311346829.2A CN202311346829A CN117094996B CN 117094996 B CN117094996 B CN 117094996B CN 202311346829 A CN202311346829 A CN 202311346829A CN 117094996 B CN117094996 B CN 117094996B
- Authority
- CN
- China
- Prior art keywords
- unit
- module
- image
- gradient
- scanning unit
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 58
- 238000001514 detection method Methods 0.000 title claims abstract description 42
- 238000007781 pre-processing Methods 0.000 claims abstract description 31
- 238000012545 processing Methods 0.000 claims abstract description 31
- 238000006243 chemical reaction Methods 0.000 claims abstract description 25
- 238000004364 calculation method Methods 0.000 claims abstract description 13
- 238000012216 screening Methods 0.000 claims abstract description 6
- 238000001914 filtration Methods 0.000 claims description 12
- 238000000605 extraction Methods 0.000 claims description 10
- 238000011161 development Methods 0.000 claims description 9
- 230000008569 process Effects 0.000 description 16
- 238000012163 sequencing technique Methods 0.000 description 8
- 238000003708 edge detection Methods 0.000 description 6
- 230000006872 improvement Effects 0.000 description 6
- 230000007547 defect Effects 0.000 description 5
- 238000010586 diagram Methods 0.000 description 5
- 230000000694 effects Effects 0.000 description 4
- 238000009499 grossing Methods 0.000 description 3
- 238000003707 image sharpening Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 241000238097 Callinectes sapidus Species 0.000 description 1
- 230000009471 action Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000006073 displacement reaction Methods 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000000717 retained effect Effects 0.000 description 1
- 230000035945 sensitivity Effects 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/13—Edge detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20024—Filtering details
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20172—Image enhancement details
- G06T2207/20192—Edge enhancement; Edge preservation
Landscapes
- Engineering & Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Quality & Reliability (AREA)
- Image Analysis (AREA)
Abstract
The invention provides a linear detection system based on an FPGA and an LSD algorithm and a control method thereof, wherein the system comprises an image preprocessing module, an edge recognition module, a gradient calculation module, a region growing module and a rectangular approximation module which are connected in sequence; the image preprocessing module is used for preprocessing an image to be detected and comprises a gray level conversion sub-module and a Gao Silv wave sub-module which are connected in sequence; the edge recognition module is used for carrying out edge recognition by utilizing a Sobel operator; the gradient calculating module is used for calculating the gradient amplitude and gradient direction of the image pixel points; the region growing module is used for screening seed points according to a preset rule and carrying out region growing of 4 neighborhood; the rectangle approximation module is used for calculating the starting point coordinates, the end point coordinates and the line width of each straight line and outputting all the detected straight lines; the invention improves the LSD algorithm, and the processing speed and the accuracy of the linear detection system reach higher level.
Description
Technical Field
The invention relates to the technical field of linear detection and FPGA, in particular to a linear detection system based on FPGA and LSD algorithm and a control method thereof.
Background
The LSD (Line Segment Detector) algorithm is a common pixel-level straight line detection algorithm, has the advantages that straight line segments in an image can be extracted within linear time complexity, a plurality of straight lines in the same image can be extracted, and no strict requirements are imposed on the position relationship, the length and the like among the straight lines, so that the application range is wider, and the image straight line detection algorithm is a reliable image straight line detection algorithm.
FPGA (Field-Programmable Gate Array), a Field programmable gate array, is a product of further development on the basis of programmable devices such as PAL, GAL, CPLD. The FPGA is used as an information processing means commonly used in the modern electronic information industry, and has the following remarkable advantages: 1) The FPGA is flexible in design, belongs to a chip structure with reconfigurable hardware, and is internally provided with a plurality of programmable input/output unit pins and triggers; 2) The FPGA is one of devices with the shortest development period, the lowest application risk and the lowest development cost in a special circuit, and is convenient to use; 3) Parallel computing, and corresponding pipelines can be built in the FPGA chip according to the number of data packet steps, so that data parallelism and pipeline parallelism are realized.
As shown in fig. 1, the flow of the conventional LSD straight line detection algorithm is shown, wherein the calculation mainly comprises gaussian downsampling, calculating the angle and the module length of a vector, gradient sequencing and self-growing judgment, and the algorithm can be realized by using a pipeline, so that the speed of the algorithm is further improved, and the real-time performance is higher; in terms of accuracy, the LSD algorithm can control the number of false detections through adjustment of the threshold value, so that the false detection meets the accuracy requirement. In addition, since each step of the algorithm can be realized by addition and displacement operation, the algorithm is quite suitable for being realized by an FPGA.
The prior art discloses a linear feature extraction system based on an FPGA, which comprises a Gaussian filter unit, a gradient modulus value and horizontal angle calculation unit, a gradient pseudo-sequencing unit, a region growing unit and a rectangular approximation unit; the hardware system in the prior art only realizes the traditional LSD algorithm based on the FPGA, and still uses the traditional gradient sorting method and the 8 neighborhood region growing method, the suitability of the LSD algorithm is not improved aiming at the FPGA hardware architecture, the algorithm performance is poor, the hardware realization speed is low, and the resource consumption is overlarge.
Disclosure of Invention
The invention provides a linear detection system based on an FPGA and an LSD algorithm and a control method thereof, which are used for overcoming the defects of slower hardware realization speed and overlarge resource consumption caused by the fact that the suitability of the LSD algorithm is not improved aiming at an FPGA hardware architecture in the prior art, and improving the LSD algorithm, wherein the processing speed and the accuracy of the linear detection reach higher levels.
In order to solve the technical problems, the technical scheme of the invention is as follows:
a linear detection system based on FPGA and LSD algorithm comprises an image preprocessing module, an edge recognition module, a gradient calculation module, a region growing module and a rectangular approximation module which are connected in sequence;
the image preprocessing module is used for preprocessing an image to be detected and comprises a gray level conversion sub-module and a Gao Silv wave sub-module which are connected in sequence;
the edge recognition module is used for carrying out edge recognition by utilizing a Sobel operator;
the gradient calculation module is used for calculating the gradient amplitude and gradient direction of the image pixel points;
the region growing module is used for screening seed points according to preset rules and performing region growing of 4 neighborhood;
the rectangle approximation module is used for calculating the starting point coordinates, the end point coordinates and the line width of each straight line and outputting all the detected straight lines.
Preferably, the gray conversion submodule includes: an R channel multiplication unit, a G channel multiplication unit, a B channel multiplication unit, a first addition unit and a first right shift unit are arranged in parallel;
the outputs of the R channel multiplication unit, the G channel multiplication unit and the B channel multiplication unit are connected with the input of a first adding unit, and the output of the first adding unit is connected with a first right shifting unit;
the multiplication factors of the R channel multiplication unit, the G channel multiplication unit and the B channel multiplication unit are 76, 150 and 29 respectively;
the right shift number of the first right shift unit is 8 bits.
Preferably, the gaussian filtering submodule adopts a first sliding window of 3×3 to scan the image to be detected after being processed by the gray level conversion submodule;
the first sliding window includes: the first scanning unit, the second scanning unit, the third scanning unit, the second adding unit and the second right shifting unit are arranged in parallel;
the output of the first scanning unit, the second scanning unit and the third scanning unit are respectively connected with the input of the second adding unit, and the output of the second adding unit is connected with the second right shifting unit;
the first scanning unit and the second scanning unit have the same structure and both comprise: the output of the two-time multiplier 1, the four-time multiplier and the two-time multiplier 2 is connected with the input of the third adding unit, and the output of the third adding unit is used as the output of the first scanning unit or the second scanning unit;
the third scanning unit includes: the output of the first multiplier 1, the output of the second multiplier 3 and the output of the first multiplier 2 are all connected with the input of the fourth summing unit, and the output of the fourth summing unit is used as the output of the third scanning unit.
Preferably, the scanning processing range of the first sliding window is all areas except the outermost edge pixel point in the image to be detected after being processed by the gray level conversion sub-module.
Preferably, the edge recognition module performs edge recognition by using a second sliding window of 3×3 based on a Sobel operator;
the second sliding window includes: a fourth scanning unit, a fifth scanning unit, a sixth scanning unit and a seventh scanning unit which are arranged in parallel, and a first subtracter, a second subtracter, a first MUX unit, a second MUX unit, a fifth adding unit and a third MUX unit;
the structures of the fourth scanning unit, the fifth scanning unit, the sixth scanning unit and the seventh scanning unit are the same as those of the third scanning unit;
the output of the fourth scanning unit and the output of the fifth scanning unit are respectively connected with the first subtracter, and the output of the sixth scanning unit and the output of the seventh scanning unit are respectively connected with the second subtracter; the output of the first subtracter is sequentially connected with the first MUX unit and the fifth adding unit, and the output of the second subtracter is sequentially connected with the second MUX unit and the fifth adding unit;
the input of the fifth adding unit further comprises a preset threshold value, and the output of the fifth adding unit is connected with the third MUX unit.
Preferably, the FPGA is embodied as a PGL22G development board.
The invention also provides a control method of the linear detection system based on the FPGA and the LSD algorithm, which is based on the linear detection system based on the FPGA and the LSD algorithm and comprises the following steps:
s1: the image preprocessing module is used for preprocessing an image to be detected, and specifically comprises the following steps:
the gray level conversion submodule carries out gray level processing on an image to be detected, acquires a gray level image to be detected, inputs the gray level image to be detected into the Gao Silv wave submodule for Gaussian filtering processing, completes preprocessing, and acquires a preprocessed image;
s2: the edge recognition module performs edge sharpening on the preprocessed image, and performs edge extraction on the image subjected to the edge sharpening based on a Sobel operator to obtain an edge recognition result;
s3: the gradient computing module computes the gradient amplitude and the gradient direction of each pixel point in the corresponding image according to the edge recognition result; the region growing module stores all pixel points with gradient amplitude values larger than a preset value and gradient directions being non-vertical directions as seed points of corresponding images;
s4: in the region growing module, starting from each seed point, carrying out passive region growing of 4 neighborhood to obtain a region growing result;
s5: and the rectangular approximation module calculates the starting point coordinates, the end point coordinates and the line width of each line according to the region growing result, outputs all detected lines and completes line detection.
Preferably, the Sobel operator in the step S2 is specifically:
the Sobel operator comprises a convolution kernel in the horizontal direction and a convolution kernel in the vertical direction;
the convolution kernel in the horizontal direction isThe convolution kernel in vertical direction is +.>。
Preferably, in the step S3, the specific method for calculating the gradient magnitude and the gradient direction of each pixel point in the corresponding image according to the edge recognition result is as follows:
acquiring coordinates of each pixel point in the corresponding image according to the edge recognition resultAnd its grey value +.>Calculating horizontal gradient vector of each pixel point according to the following formula>And vertical gradient vector>：
According to the horizontal gradient vector of each pixel point in the corresponding imageAnd vertical gradient vector>And calculating the gradient amplitude and the gradient direction of each pixel point by using a CORDIC iterative algorithm.
Preferably, in the step S4, the passive region growing is performed from each seed point, and at least 2 straight lines are grown each time.
Compared with the prior art, the technical scheme of the invention has the beneficial effects that:
the invention provides a linear detection system based on an FPGA and an LSD algorithm and a control method thereof, wherein the system comprises an image preprocessing module, an edge recognition module, a gradient calculation module, a region growing module and a rectangular approximation module which are connected in sequence; the image preprocessing module is used for preprocessing an image to be detected and comprises a gray level conversion sub-module and a Gao Silv wave sub-module which are connected in sequence; the edge recognition module is used for carrying out edge recognition by utilizing a Sobel operator; the gradient calculating module is used for calculating the gradient amplitude and gradient direction of the image pixel points; the region growing module is used for screening seed points according to a preset rule and carrying out region growing of 4 neighborhood; the rectangle approximation module is used for calculating the starting point coordinates, the end point coordinates and the line width of each straight line and outputting all the detected straight lines;
the invention improves the LSD straight line detection algorithm facing the FPGA, firstly, adds in the image preprocessing process which is not included in the original LSD algorithm, and solves the defect that the detection result of the LSD algorithm is easily influenced by noise points on the premise of not influencing the subsequent processing effect; secondly, creatively replacing the gradient amplitude sequencing by a Sobel operator edge detection and binarization processing mode in the LSD algorithm identification process, and effectively solving the problems of too long time consumption and too large resource consumption of the original algorithm traversal sequencing; in addition, the invention modifies 8 neighborhood active growth of region growth into 4 neighborhood passive growth, changes the original single growth straight line into multiple single growth straight lines, solves the defect that FPGA can not realize active growth, and achieves higher processing speed and accuracy of system straight line detection.
Drawings
Fig. 1 is a flowchart of a conventional LSD algorithm in the background art.
Fig. 2 is a block diagram of a linear detection system based on FPGA and LSD algorithm provided in embodiment 1.
Fig. 3 is a schematic diagram of a gray scale conversion submodule according to embodiment 1.
Fig. 4 is a first sliding window structure diagram of the gaussian filter sub-module provided in embodiment 1.
Fig. 5 is a second sliding window structure diagram of the edge recognition module provided in embodiment 1.
Fig. 6 is a schematic diagram of the 4-neighborhood passive growth provided in example 1.
Fig. 7 is a flow chart of the region growing module provided in embodiment 1.
Fig. 8 is a flowchart of a control method of the linear detection system based on FPGA and LSD algorithm provided in embodiment 2.
Detailed Description
The drawings are for illustrative purposes only and are not to be construed as limiting the present application;
for the purpose of better illustrating the embodiments, certain elements of the drawings may be omitted, enlarged or reduced and do not represent the actual product dimensions;
it will be appreciated by those skilled in the art that certain well-known structures in the drawings and descriptions thereof may be omitted.
The technical scheme of the invention is further described below with reference to the accompanying drawings and examples.
Example 1
As shown in fig. 2, the embodiment provides a linear detection system based on an FPGA and an LSD algorithm, which includes an image preprocessing module, an edge recognition module, a gradient calculation module, a region growing module and a rectangular approximation module that are sequentially connected;
the image preprocessing module is used for preprocessing an image to be detected and comprises a gray level conversion sub-module and a Gao Silv wave sub-module which are connected in sequence;
the edge recognition module is used for carrying out edge recognition by utilizing a Sobel operator;
the gradient calculation module is used for calculating the gradient amplitude and gradient direction of the image pixel points;
the region growing module is used for screening seed points according to preset rules and performing region growing of 4 neighborhood;
the rectangular approximation module is used for calculating the starting point coordinates, the end point coordinates and the line width of each straight line and outputting all detected straight lines;
as shown in fig. 3, the gray-scale conversion submodule includes: an R channel multiplication unit, a G channel multiplication unit, a B channel multiplication unit, a first addition unit and a first right shift unit are arranged in parallel;
the outputs of the R channel multiplication unit, the G channel multiplication unit and the B channel multiplication unit are connected with the input of a first adding unit, and the output of the first adding unit is connected with a first right shifting unit;
the multiplication factors of the R channel multiplication unit, the G channel multiplication unit and the B channel multiplication unit are 76, 150 and 29 respectively;
the right shift number of the first right shift unit is 8 bits;
the Gaussian filtering submodule adopts a first sliding window of 3 multiplied by 3 to scan the image to be detected after being processed by the gray level conversion submodule;
as shown in fig. 4, the first sliding window includes: the first scanning unit, the second scanning unit, the third scanning unit, the second adding unit and the second right shifting unit are arranged in parallel;
the output of the first scanning unit, the second scanning unit and the third scanning unit are respectively connected with the input of the second adding unit, and the output of the second adding unit is connected with the second right shifting unit;
the first scanning unit and the second scanning unit have the same structure and both comprise: the output of the two-time multiplier 1, the four-time multiplier and the two-time multiplier 2 is connected with the input of the third adding unit, and the output of the third adding unit is used as the output of the first scanning unit or the second scanning unit;
the third scanning unit includes: the device comprises a one-time multiplier 1, a two-time multiplier 3 and a one-time multiplier 2 which are arranged in parallel, and a fourth summation unit, wherein the outputs of the one-time multiplier 1, the two-time multiplier 3 and the one-time multiplier 2 are all connected with the input of the fourth summation unit, and the output of the fourth summation unit is used as the output of the third scanning unit;
the scanning processing range of the first sliding window is all areas except the outermost edge pixel point in the image to be detected processed by the gray level conversion sub-module;
the edge recognition module performs edge recognition by adopting a second sliding window of 3 multiplied by 3 based on a Sobel operator;
as shown in fig. 5, the second sliding window includes: a fourth scanning unit, a fifth scanning unit, a sixth scanning unit and a seventh scanning unit which are arranged in parallel, and a first subtracter, a second subtracter, a first MUX unit, a second MUX unit, a fifth adding unit and a third MUX unit;
the structures of the fourth scanning unit, the fifth scanning unit, the sixth scanning unit and the seventh scanning unit are the same as those of the third scanning unit;
the output of the fourth scanning unit and the output of the fifth scanning unit are respectively connected with the first subtracter, and the output of the sixth scanning unit and the output of the seventh scanning unit are respectively connected with the second subtracter; the output of the first subtracter is sequentially connected with the first MUX unit and the fifth adding unit, and the output of the second subtracter is sequentially connected with the second MUX unit and the fifth adding unit;
the input of the fifth adding unit further comprises a preset threshold value, and the output of the fifth adding unit is connected with the third MUX unit;
the FPGA is specifically a PGL22G development board.
In a specific implementation process, the hardware development platform adopted by the system in the embodiment is a PGL22G development board, and the PGL22G development board is a logo series FPGA high-performance core board developed by purple light co-creation company, has the characteristics of high speed, high bandwidth, high capacity and the like, and is suitable for high-speed data communication, video image processing, high-speed data acquisition and the like;
the system comprises an image preprocessing module, an edge recognition module, a gradient calculation module, a region growing module and a rectangular approximation module which are connected in sequence;
the image preprocessing module is used for preprocessing an image to be detected and comprises a gray level conversion sub-module and a Gao Silv wave sub-module which are connected in sequence;
in general, the images collected by the image collecting device are all RGB color images, the color RGB images have different formats, for example, RGB 565 format, and the data of the red, green and blue channels are respectively 5 bits, 6 bits and 5 bits, so that the color values contained in the images may be about 2 16 The data processing capability requirement of the system is extremely high by directly processing the data, so that the data volume required to be processed by the system is reduced under the condition that the subsequent processing is not influenced, and the color RGB image is converted into a gray image, so that the data volume of each pixel point is reduced from 16 bits to 8 bits, the data processing pressure of the system is greatly reduced, and the processing speed of the subsequent system is increased;
the weighted average method is the most commonly used gray scale conversion method, and the method performs weighted average on the values of three channels of RGB according to different sensitivity degrees of human eye cells to red, green and blue, because human eyes are most sensitive to green and least sensitive to blue, the weight of a green point is the highest in conversion, and the weight of a blue point is the lowest, and the specific conversion method is as follows:
wherein,for pixel coordinates, +.>Gray value converted for pixel gray scale, < >>、/>、/>R, G, B three-channel data of the pixel points respectively;
since all data is calculated in binary form in FPGA, shifting is equivalent to expanding or contracting the corresponding number by a factor of 2; because the gray scale is 0-255, the decimal in the above formula is multiplied by 256 to form a whole, and the final result is divided by 256 to obtain the calculated result, so that the following formula is obtained:
therefore, in order to improve the output real-time performance, the module is implemented by adopting combinational logic, as shown in fig. 3;
in the characteristic extraction process of the digital image, the interference of noise is unavoidable, and the algorithm is greatly influenced in the processing process, and the characteristic is more prominent in the system, because the LSD algorithm is a greedy algorithm based on pixel points, the influence of noise points is very obvious, and therefore, the image needs to be smoothed before subsequent processing; the smoothing method adopted in this embodiment is gaussian filtering; the Gaussian filtering submodule scans the image to be detected processed by the gray level conversion submodule by adopting a first sliding window of 3 multiplied by 3, the structure of the first sliding window is shown in a graph shown in fig. 4, x (a, b) in the graph represents gray level values of pixel points of a row and a column, a is {1,2,3}, b is {1,2,3};
since the 3×3 template cannot operate the points located at the edges of the image, when the edge points of the image are taken as the calculation center points, there are no calculation objects in a row or a column of the template, so that the center points of the template cannot be smoothly operated, in order to solve the problem, the gaussian filter submodule designed in this embodiment outputs the edge points of the original image directly as the edge points of the processed image, that is, the action range of the gaussian template is the area of the original image except the outermost pixel points;
edge detection is usually performed according to the gray level distribution of an image, because gray level values near the edge often have a step phenomenon, and an edge detection algorithm is also performed based on the feature; the edge recognition module in this embodiment is configured to perform edge recognition by using a Sobel operator, and perform edge recognition by using a second sliding window with a structure shown in fig. 5, where the second sliding window is 3×3;
the hardware implementation of the Sobel operator is similar to that of gaussian filtering, a 3×3 template is adopted, and the result of the Sobel algorithm indicates whether the pixel point is judged to be a point forming an edge, so that binarization processing is required to be performed on the calculation result of the Sobel template, the preset threshold value in the embodiment is 100, the Sobel operator is compared with the template after the Sobel template acts in the module, if the threshold value is met, the value is set to be 1, if the threshold value is not met, the value is set to be 0, when the value is set to be 1, the value is set to be an edge point, and when the value is set to be 0, the value is set to be not an edge point;
the gradient calculating module is used for calculating the gradient amplitude and gradient direction of the image pixel points;
after the LSD algorithm finishes the conversion from the gray level image to the gradient image, the region can be increased according to the data of the front-stage module, the gradient sorting step is modified in the improvement process of the algorithm, all pixels in the image do not need to be traversed, all pixels in the image are directly scanned from the first pixel of the image from left to right and from top to bottom, all pixels meeting the requirements are used as seed points, and sorting is started at the same time;
in the gradient amplitude sequencing process, the meaning of starting to grow from the point with larger amplitude is that the algorithm is enabled to more accurately position the point possibly positioned at the edge of the image, because the amplitude of the gradient reflects the difference value between the gray value of the point and the gray value of the nearby point, the point with the most similar edge is extracted after sequencing as a seed point, and thus the straight line in the image can be found out more; the Sobel template has the function of extracting points on edges, so that the extracting function of the Sobel module just replaces the sorting function in the original LSD algorithm, and the related step of gradient amplitude sorting is replaced by Sobel edge detection, which is effective optimization on the LSD algorithm on the FPGA;
the whole frame image is grown by only one time, which is the maximum improvement compared with the original LSD algorithm, because the traditional 8 neighborhood active growth seed points are difficult to realize on the FPGA, the embodiment adopts a method of 4 neighborhoods for passive growth, the structure is as shown in figure 6, the point 5 is the current operating point, the fundamental difference of the active growth is whether the adjacent points are classified into the linear supporting domain of the operating point or the linear supporting domain of the adjacent points, and the FPGA adopts the latter, namely the operating points are classified into the linear supporting domain of four surrounding points according to actual conditions;
the first step of region growing is to judge whether the Sobel operation result of the point is effective, if so, starting to perform growing operation on the current point, if not, directly skipping the point, and ending the operation of the current point; because a passive growth method is adopted, the comparison with other four points is needed in the growth process, and because one point can only be classified into one supporting domain, when the situation that two points belonging to different supporting domains exist in four adjacent points, the judgment needs to be made as to which pixel point the point is classified, in the embodiment, the four adjacent points are uniformly arranged in a hardware-implemented growth module, and the growth priorities are respectively upper left, upper right and left; after determining the priority of the growing sequence, a situation needs to be considered, namely when all four adjacent points are not included in any supporting domain, the operating point needs to be added into a new supporting domain, which is the first point of the supporting domain, meanwhile, the supporting domain count needs to be increased by one, and the flow of region growing is shown in fig. 7;
the rectangle approximation module is used for calculating the starting point coordinates, the end point coordinates and the line width of each straight line and outputting all the detected straight lines;
in this embodiment, a comparison experiment of multiple platforms is performed, and the improved LSD algorithm is used to process images with 1024×768 sizes of the same frame in different platform execution time pairs as shown in table 1;
table 1 comparison of execution time of improved LSD algorithm at different platforms
As can be seen from table 2, the speed of the improved LSD algorithm is improved by about 45 times compared with the speed of the hardware platform based on the FPGA and the speed of the hardware platform based on the FPGA is improved by about 360 times compared with the speed of the hardware platform based on the FPGA and the speed of the hardware platform based on the original LSD algorithm is improved by about 2 times compared with the speed of the hardware platform based on the FPGA, and the comparison result fully proves that the hardware system based on the improved LSD algorithm provided by the embodiment has a great improvement in the running speed;
the system improves the LSD algorithm, and the processing speed and accuracy of the linear detection reach higher level.
Example 2
As shown in fig. 8, the present embodiment provides a control method of a linear detection system based on FPGA and LSD algorithm, and the linear detection system based on FPGA and LSD algorithm in embodiment 1 includes the following steps:
s1: the image preprocessing module is used for preprocessing an image to be detected, and specifically comprises the following steps:
the gray level conversion submodule carries out gray level processing on an image to be detected, acquires a gray level image to be detected, inputs the gray level image to be detected into the Gao Silv wave submodule for Gaussian filtering processing, completes preprocessing, and acquires a preprocessed image;
s2: the edge recognition module performs edge sharpening on the preprocessed image, and performs edge extraction on the image subjected to the edge sharpening based on a Sobel operator to obtain an edge recognition result;
s3: the gradient computing module computes the gradient amplitude and the gradient direction of each pixel point in the corresponding image according to the edge recognition result; the region growing module stores all pixel points with gradient amplitude values larger than a preset value and gradient directions being non-vertical directions as seed points of corresponding images;
s4: in the region growing module, starting from each seed point, carrying out passive region growing of 4 neighborhood to obtain a region growing result;
s5: the rectangular approximation module calculates the starting point coordinates, the end point coordinates and the line width of each line according to the region growing result, and outputs all detected lines to finish line detection;
the Sobel operator in the step S2 specifically includes:
the Sobel operator comprises a convolution kernel in the horizontal direction and a convolution kernel in the vertical direction;
the convolution kernel in the horizontal direction isThe convolution kernel in vertical direction is +.>；
The specific method for calculating the gradient amplitude and gradient direction of each pixel point in the corresponding image according to the edge recognition result in the step S3 is as follows:
acquiring coordinates of each pixel point in the corresponding image according to the edge recognition resultAnd its grey value +.>Calculating horizontal gradient vector of each pixel point according to the following formula>And vertical gradient vector>：
According to the horizontal gradient vector of each pixel point in the corresponding imageAnd vertical gradient vector>Calculating the gradient amplitude and gradient direction of each pixel point by using a CORDIC iterative algorithm;
in the step S4, at least 2 straight lines are grown each time when the passive region growth is performed from each seed point.
In the specific implementation process, firstly, a gray level conversion sub-module carries out gray level processing on an image to be detected, a gray level image to be detected is obtained, the gray level image to be detected is input into a Gao Silv wave sub-module for Gaussian filtering processing, preprocessing is completed, a preprocessed image is obtained, and then an edge recognition module carries out edge sharpening processing on the preprocessed image;
the core principle of the LSD algorithm is a greedy algorithm self-growing process, namely the self-growing process is performed when the gradient amplitude and the angle meet the threshold judgment conditions, the identified straight line is a straight line detected in the process of completing the growing process or a seed point meeting the conditions, but the original LSD algorithm is excessively sensitive to noise points, which can lead to breakage in the straight line identification, if the noise points appear near the edges of the image, the original pixel amplitude meeting the conditions is mistakenly considered to be not meeting the conditions, and then the straight line is interrupted, so that a large number of interrupted straight lines and mistakenly detected straight lines can appear after the LSD algorithm detection without preprocessing, and the extraction precision of the straight line is affected;
in this embodiment, the gaussian filtering has the effect of smoothing noise, the image edge sharpening can make the image edge clearer, the gaussian filtering focuses on smoothing noise, and the image sharpening focuses on enhancing the edge; in addition, because the straight line in the image is often positioned at the edge of the image, the image sharpening can lead the straight line detected by the LSD algorithm to be more accurate and clearer;
before the original LSD algorithm performs region growing, seed points in the image are needed to be screened firstly to serve as a starting point of growing, and the screening of the seed points is to select points with larger gradient amplitude in the image, so that the original LSD algorithm adopts a method of fully sorting gradient values of pixel points of the whole frame of image, and after sorting is completed, the point with the largest gradient amplitude is used as the seed point to perform growing operation, and a first straight line in the image is extracted; then taking the point with the largest gradient amplitude value in the rest pixel points as a seed point, performing growth operation, and extracting a second straight line in the image; and so on until all straight lines in the image are extracted;
according to the method, an original LSD algorithm is improved, a straight line is extracted, an edge recognition module performs edge sharpening on a preprocessed image, and edge extraction is performed on the image subjected to the edge sharpening based on a Sobel operator, so that an edge recognition result is obtained; the convolution kernel of the Sobel operator in the horizontal direction isThe convolution kernel in vertical direction is +.>；
Compared with a full-order method in an original LSD algorithm, the method for carrying out edge identification by adopting the Sobel operator has obvious advantages in terms of time complexity, is particularly characterized in that the time required by the improved method is stable and is not influenced by specific conditions of images, the consumed time is obviously less than that of the original method, and only one clock period is required for processing each pixel point; in terms of space complexity, the improved method saves 99.8% of buffer space and effectively reduces the waste of computing resources;
the gradient computing module computes the gradient amplitude and gradient direction of each pixel point in the corresponding image according to the edge recognition result;
according to each ofAcquiring coordinates of each pixel point in a corresponding frame image according to an edge recognition result of the frame imageAnd its grey value +.>Calculating horizontal gradient vector of each pixel point according to the following formula>Vertical gradient vector：
According to the horizontal gradient vector of each pixel point in each frame imageAnd vertical gradient vector>Calculating the gradient amplitude and gradient direction of each pixel point by using a CORDIC iterative algorithm;
the region growing module stores all pixel points with gradient amplitude larger than 100 and gradient direction being non-vertical direction as seed points of the corresponding image;
in the region growing module, starting from each seed point, carrying out passive region growing of 4 neighborhood to obtain a region growing result; the rectangular approximation module calculates the starting point coordinates, the end point coordinates and the line width of each line according to the region growing result, and outputs all detected lines to finish line detection;
the 8-neighborhood self-growing method of the original LSD algorithm is an active growing method which comprises the steps of firstly determining seed points, then judging whether surrounding points meet the conditions, wherein the method requires that the information of pixel points can be read at will, and the method cannot be realized on an FPGA, so that the algorithm of which the algorithm is 4-neighborhood is required to be modified according to the characteristics of a video data stream of the FPGA;
however, the result of direct detection by adopting the 4 neighborhood method is not good, a large number of intermittent straight lines can be detected in the image, because the most obvious change in principle from the 8 neighborhood to the 4 neighborhood is half less than the point to be judged by self-growing, the point contained in the 4 neighborhood is only one pixel point on the upper row and the left side of each pixel point, and the growing mode makes each pixel point more prone to be in the straight line supporting domain of the upper row, namely after the improvement of the method to the 4 neighborhood growth, the LSD algorithm is more sensitive to the straight line in the vertical direction, because the point of the 4 neighborhood on the left side and the right side of the operating point only contains the upper left point and the right left point, and the testing capability in the horizontal direction can be degraded; therefore, the method in this embodiment needs to optimize the 4-neighborhood growing method, making it more sensitive to straight line growth in the vertical direction, and ignoring straight lines on the horizontal line; on the one hand, the threshold value of thresholding in the image edge detection in the step S2 is reduced, and the angle difference tolerance value tau in the growing process is improved; on the other hand, when the seed points are selected in step S3, only the seed points whose gradient direction is a non-vertical direction (i.e., a straight line other than a horizontal line) are retained;
in addition, at least 2 lines of growth each time the passive region growth is performed starting from each seed point; in the original LSD algorithm, only one straight line can be increased at a time, but a plurality of straight lines cannot be increased at the same time, so that a large amount of time is still required to be consumed, and the embodiment selects to increase the plurality of straight lines at the same time, so that the running time of the LSD algorithm is greatly reduced under the condition that the extraction effect is not influenced;
the time complexity comparison of the improved LSD algorithm and the original LSD algorithm in this embodiment is shown in table 2, where n represents the number of pixels in the image;
table 2 time complexity comparison of modified LSD algorithm with original LSD algorithm
As can be seen from table 2, the real-time performance of the improved LSD algorithm is significantly better than that of the original LSD algorithm, and the requirement of linear extraction on the real-time performance can be satisfied;
the method is directed to the improvement of the LSD straight line detection algorithm by the FPGA, firstly, an image preprocessing process which is not included in the original LSD algorithm is added, and the defect that the detection result of the LSD algorithm is easily influenced by noise points is overcome on the premise that the subsequent processing effect is not influenced; secondly, creatively replacing the gradient amplitude sequencing by a Sobel operator edge detection and binarization processing mode in the LSD algorithm identification process, and effectively solving the problems of too long time consumption and too large resource consumption of the original algorithm traversal sequencing; in addition, the method modifies 8 neighborhood active growth of region growth into 4 neighborhood passive growth, changes an original single growth straight line into a plurality of single growth straight lines, solves the defect that FPGA can not realize active growth, and achieves higher processing speed and accuracy of straight line extraction.
The same or similar reference numerals correspond to the same or similar components;
the terms describing the positional relationship in the drawings are merely illustrative, and are not to be construed as limiting the present application;
it is to be understood that the above examples of the present invention are provided by way of illustration only and not by way of limitation of the embodiments of the present invention. Other variations or modifications of the above teachings will be apparent to those of ordinary skill in the art. It is not necessary here nor is it exhaustive of all embodiments. Any modification, equivalent replacement, improvement, etc. which come within the spirit and principles of the invention are desired to be protected by the following claims.
Claims (7)
1. The linear detection system based on the FPGA and the LSD algorithm is characterized by comprising an image preprocessing module, an edge recognition module, a gradient calculation module, a region growing module and a rectangular approximation module which are connected in sequence;
the image preprocessing module is used for preprocessing an image to be detected and comprises a gray level conversion sub-module and a Gao Silv wave sub-module which are connected in sequence;
the Gaussian filtering submodule adopts a first sliding window of 3 multiplied by 3 to scan the image to be detected after being processed by the gray level conversion submodule;
the first sliding window includes: the first scanning unit, the second scanning unit, the third scanning unit, the second adding unit and the second right shifting unit are arranged in parallel;
the output of the first scanning unit, the second scanning unit and the third scanning unit are respectively connected with the input of the second adding unit, and the output of the second adding unit is connected with the second right shifting unit;
the first scanning unit and the second scanning unit have the same structure and both comprise: the output of the two-time multiplier 1, the four-time multiplier and the two-time multiplier 2 is connected with the input of the third adding unit, and the output of the third adding unit is used as the output of the first scanning unit or the second scanning unit;
the third scanning unit includes: the device comprises a one-time multiplier 1, a two-time multiplier 3 and a one-time multiplier 2 which are arranged in parallel, and a fourth summation unit, wherein the outputs of the one-time multiplier 1, the two-time multiplier 3 and the one-time multiplier 2 are all connected with the input of the fourth summation unit, and the output of the fourth summation unit is used as the output of the third scanning unit;
the scanning processing range of the first sliding window is all areas except the outermost edge pixel point in the image to be detected processed by the gray level conversion sub-module;
the edge recognition module is used for carrying out edge recognition by utilizing a Sobel operator;
the edge recognition module performs edge recognition by adopting a second sliding window of 3 multiplied by 3 based on a Sobel operator;
the second sliding window includes: a fourth scanning unit, a fifth scanning unit, a sixth scanning unit and a seventh scanning unit which are arranged in parallel, and a first subtracter, a second subtracter, a first MUX unit, a second MUX unit, a fifth adding unit and a third MUX unit;
the structures of the fourth scanning unit, the fifth scanning unit, the sixth scanning unit and the seventh scanning unit are the same as those of the third scanning unit;
the output of the fourth scanning unit and the output of the fifth scanning unit are respectively connected with the first subtracter, and the output of the sixth scanning unit and the output of the seventh scanning unit are respectively connected with the second subtracter; the output of the first subtracter is sequentially connected with the first MUX unit and the fifth adding unit, and the output of the second subtracter is sequentially connected with the second MUX unit and the fifth adding unit;
the input of the fifth adding unit further comprises a preset threshold value, and the output of the fifth adding unit is connected with the third MUX unit;
the gradient calculation module is used for calculating the gradient amplitude and gradient direction of the image pixel points;
the region growing module is used for screening seed points according to preset rules and performing region growing of 4 neighborhood;
the rectangle approximation module is used for calculating the starting point coordinates, the end point coordinates and the line width of each straight line and outputting all the detected straight lines.
2. The line detection system based on FPGA and LSD algorithms of claim 1, wherein said gray scale conversion submodule comprises: an R channel multiplication unit, a G channel multiplication unit, a B channel multiplication unit, a first addition unit and a first right shift unit are arranged in parallel;
the outputs of the R channel multiplication unit, the G channel multiplication unit and the B channel multiplication unit are connected with the input of a first adding unit, and the output of the first adding unit is connected with a first right shifting unit;
the multiplication factors of the R channel multiplication unit, the G channel multiplication unit and the B channel multiplication unit are 76, 150 and 29 respectively;
the right shift number of the first right shift unit is 8 bits.
3. The line detection system based on FPGA and LSD algorithm of claim 1, wherein the FPGA is embodied as a PGL22G development board.
4. A method for controlling a linear detection system based on an FPGA and an LSD algorithm, based on the linear detection system based on the FPGA and the LSD algorithm described in any one of claims 1 to 3, comprising the steps of:
s1: the image preprocessing module is used for preprocessing an image to be detected, and specifically comprises the following steps:
the gray level conversion submodule carries out gray level processing on an image to be detected, acquires a gray level image to be detected, inputs the gray level image to be detected into the Gao Silv wave submodule for Gaussian filtering processing, completes preprocessing, and acquires a preprocessed image;
s2: the edge recognition module performs edge sharpening on the preprocessed image, and performs edge extraction on the image subjected to the edge sharpening based on a Sobel operator to obtain an edge recognition result;
s3: the gradient computing module computes the gradient amplitude and the gradient direction of each pixel point in the corresponding image according to the edge recognition result; the region growing module stores all pixel points with gradient amplitude values larger than a preset value and gradient directions being non-vertical directions as seed points of corresponding images;
s4: in the region growing module, starting from each seed point, carrying out passive region growing of 4 neighborhood to obtain a region growing result;
s5: and the rectangular approximation module calculates the starting point coordinates, the end point coordinates and the line width of each line according to the region growing result, outputs all detected lines and completes line detection.
5. The method for controlling a linear detection system based on FPGA and LSD algorithm according to claim 4, wherein the Sobel operator in step S2 is specifically:
the Sobel operator comprises a convolution kernel in the horizontal direction and a convolution kernel in the vertical direction;
the convolution kernel in the horizontal direction isThe convolution kernel in vertical direction is +.>。
6. The method for controlling a linear detection system based on FPGA and LSD algorithm according to claim 5, wherein the specific method for calculating the gradient magnitude and gradient direction of each pixel point in the corresponding image according to the edge recognition result in step S3 is as follows:
acquiring coordinates of each pixel point in the corresponding image according to the edge recognition resultAnd its grey value +.>Calculating horizontal gradient vector of each pixel point according to the following formula>And vertical gradient vector>：
According to the horizontal gradient vector of each pixel point in the corresponding imageAnd vertical gradient vector>And calculating the gradient amplitude and the gradient direction of each pixel point by using a CORDIC iterative algorithm.
7. The method according to claim 6, wherein in step S4, at least 2 lines are grown each time when the passive region growing is performed from each seed point.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311346829.2A CN117094996B (en) | 2023-10-18 | 2023-10-18 | Linear detection system based on FPGA and LSD algorithm and control method thereof |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311346829.2A CN117094996B (en) | 2023-10-18 | 2023-10-18 | Linear detection system based on FPGA and LSD algorithm and control method thereof |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117094996A CN117094996A (en) | 2023-11-21 |
| CN117094996B true CN117094996B (en) | 2024-02-23 |
Family
ID=88780586
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311346829.2A Active CN117094996B (en) | 2023-10-18 | 2023-10-18 | Linear detection system based on FPGA and LSD algorithm and control method thereof |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117094996B (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102324017A (en) * | 2011-06-09 | 2012-01-18 | 中国人民解放军国防科学技术大学 | FPGA (Field Programmable Gate Array)-based lane line detection method |
| CN106504182A (en) * | 2016-11-02 | 2017-03-15 | 山东正晨科技股份有限公司 | A kind of extraction of straight line system based on FPGA |
| CN109816639A (en) * | 2019-01-05 | 2019-05-28 | 西安交通大学 | A kind of real-time line segment detection system and method based on LSD algorithm |
| CN109978878A (en) * | 2019-04-04 | 2019-07-05 | 厦门商集网络科技有限责任公司 | Color image line segment detecting method and its system based on LSD |
| CN111047615A (en) * | 2019-12-09 | 2020-04-21 | Oppo广东移动通信有限公司 | Image-based line detection method and device and electronic equipment |
| CN113450376A (en) * | 2021-06-14 | 2021-09-28 | 石河子大学 | Cotton plant edge detection method based on FPGA |
| CN114529459A (en) * | 2022-04-25 | 2022-05-24 | 东莞市兆丰精密仪器有限公司 | Method, system and medium for enhancing image edge |
-
2023
- 2023-10-18 CN CN202311346829.2A patent/CN117094996B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102324017A (en) * | 2011-06-09 | 2012-01-18 | 中国人民解放军国防科学技术大学 | FPGA (Field Programmable Gate Array)-based lane line detection method |
| CN106504182A (en) * | 2016-11-02 | 2017-03-15 | 山东正晨科技股份有限公司 | A kind of extraction of straight line system based on FPGA |
| CN109816639A (en) * | 2019-01-05 | 2019-05-28 | 西安交通大学 | A kind of real-time line segment detection system and method based on LSD algorithm |
| CN109978878A (en) * | 2019-04-04 | 2019-07-05 | 厦门商集网络科技有限责任公司 | Color image line segment detecting method and its system based on LSD |
| CN111047615A (en) * | 2019-12-09 | 2020-04-21 | Oppo广东移动通信有限公司 | Image-based line detection method and device and electronic equipment |
| CN113450376A (en) * | 2021-06-14 | 2021-09-28 | 石河子大学 | Cotton plant edge detection method based on FPGA |
| CN114529459A (en) * | 2022-04-25 | 2022-05-24 | 东莞市兆丰精密仪器有限公司 | Method, system and medium for enhancing image edge |
Non-Patent Citations (5)
| Title |
|---|
| FPGA Implementation of Resistor Network for Fast Segment Line Detector;Ali Abdallah et al;《2017 29th International Conference on Microelectronics (ICM)》;第1-4页 * |
| FPGA-Based Linear Detection Algorithm of an Underground Inspection Robot;Chuanwei Zhang et al;《Algorithms》;第1-16页 * |
| 基于 FPGA 改进的 Canny 实时边缘检测***;汪睿琪 等;《无线通信技术》(第3期);第32-37页 * |
| 基于 FPGA的 Sobel算子的边缘响应算法实现;赵亮 等;《航空兵器》(第6期);第32-35页 * |
| 基于FPGA的实时彩色图像边缘检测算法的实;陈虎 等;《液晶与显示》;第30卷(第1期);第1-11页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117094996A (en) | 2023-11-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107507173B (en) | No-reference definition evaluation method and system for full-slice image | |
| CN112365494B (en) | Ore material image segmentation method based on deep learning prediction edge | |
| CN110008954B (en) | Complex background text image extraction method and system based on multi-threshold fusion | |
| CN109242853B (en) | PCB defect intelligent detection method based on image processing | |
| JP3549720B2 (en) | Image processing device | |
| US20110123090A1 (en) | Method and device for determining a contour and a center of an object | |
| US7411699B2 (en) | Method and apparatus to enhance digital image quality | |
| CN114820594B (en) | Method for detecting edge sealing defect of plate based on image, related equipment and storage medium | |
| CN114494259A (en) | Cloth defect detection method based on artificial intelligence | |
| CN111223083B (en) | Construction method, system, device and medium of surface scratch detection neural network | |
| CN112614062A (en) | Bacterial colony counting method and device and computer storage medium | |
| WO2023070593A1 (en) | Line width measurement method and apparatus, computing processing device, computer program, and computer readable medium | |
| CN110738106A (en) | optical remote sensing image ship detection method based on FPGA | |
| JP3741672B2 (en) | Image feature learning type defect detection method, defect detection apparatus, and defect detection program | |
| CN115631117A (en) | Image enhancement method, device, detection system and storage medium for defect detection | |
| CN112991283A (en) | Flexible IC substrate line width detection method based on super-pixels, medium and equipment | |
| CN105701491A (en) | Method for making fixed-format document image template and application thereof | |
| CN113609984A (en) | Pointer instrument reading identification method and device and electronic equipment | |
| CN117274113B (en) | Broken silicon wafer cleaning effect visual detection method based on image enhancement | |
| CN116363140B (en) | Method, system and device for detecting defects of medium borosilicate glass and storage medium | |
| CN113785181A (en) | OLED screen point defect judgment method and device, storage medium and electronic equipment | |
| CN115272362A (en) | Method and device for segmenting effective area of digital pathology full-field image | |
| US20240086661A1 (en) | Method and apparatus for processing graphic symbol and computer-readable storage medium | |
| CN117094996B (en) | Linear detection system based on FPGA and LSD algorithm and control method thereof | |
| CN115131336B (en) | Dark line defect detection method under four-color background picture of display screen |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20240123 Address after: Room 501/502/503/504, Building B1, Suikai Science and Technology Park, No. 136 Kaiyuan Avenue, Yunpu Street, Huangpu District, Guangzhou City, Guangdong Province, 510535 Applicant after: Guangzhou Dawan District Virtual Reality Research Institute Country or region after: China Applicant after: Shaoguan Museum (Shaoguan Institute of Cultural Relics and Archaeology) Address before: Room 501/502/503/504, Building B1, Suikai Science and Technology Park, No. 136 Kaiyuan Avenue, Yunpu Street, Huangpu District, Guangzhou City, Guangdong Province, 510535 Applicant before: Guangzhou Dawan District Virtual Reality Research Institute Country or region before: China |
|
| TA01 | Transfer of patent application right | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |