CN117934882A - Element identification method and system based on image feature matching - Google Patents
Element identification method and system based on image feature matching Download PDFInfo
- Publication number
- CN117934882A CN117934882A CN202410158845.7A CN202410158845A CN117934882A CN 117934882 A CN117934882 A CN 117934882A CN 202410158845 A CN202410158845 A CN 202410158845A CN 117934882 A CN117934882 A CN 117934882A
- Authority
- CN
- China
- Prior art keywords
- image
- detected
- control element
- screen background
- background image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 72
- 238000012360 testing method Methods 0.000 claims abstract description 74
- 238000010586 diagram Methods 0.000 claims description 27
- 238000003860 storage Methods 0.000 claims description 7
- 230000002159 abnormal effect Effects 0.000 claims description 4
- 239000011159 matrix material Substances 0.000 claims description 4
- 230000009466 transformation Effects 0.000 claims description 4
- 238000004364 calculation method Methods 0.000 claims description 2
- 238000009826 distribution Methods 0.000 claims description 2
- 230000000694 effects Effects 0.000 claims description 2
- 238000012216 screening Methods 0.000 claims description 2
- 238000005457 optimization Methods 0.000 claims 1
- 230000006870 function Effects 0.000 description 29
- 238000012545 processing Methods 0.000 description 7
- 238000004088 simulation Methods 0.000 description 5
- 230000000903 blocking effect Effects 0.000 description 4
- 238000011161 development Methods 0.000 description 4
- 238000012544 monitoring process Methods 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 238000012790 confirmation Methods 0.000 description 3
- 230000007547 defect Effects 0.000 description 3
- 230000000007 visual effect Effects 0.000 description 3
- 239000008186 active pharmaceutical agent Substances 0.000 description 2
- 230000004622 sleep time Effects 0.000 description 2
- 239000013598 vector Substances 0.000 description 2
- 230000009471 action Effects 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000005520 cutting process Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000005008 domestic process Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000008676 import Effects 0.000 description 1
- 238000012905 input function Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000008520 organization Effects 0.000 description 1
- 238000002360 preparation method Methods 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
- G06V10/75—Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
- G06V10/757—Matching configurations of points or features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
- G06V10/761—Proximity, similarity or dissimilarity measures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/771—Feature selection, e.g. selecting representative features from a multi-dimensional feature space
Landscapes
- Engineering & Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Artificial Intelligence (AREA)
- Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- Databases & Information Systems (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Software Systems (AREA)
- Multimedia (AREA)
- Image Analysis (AREA)
Abstract
The invention relates to an element identification method and system based on image feature matching, wherein the method comprises the following steps: s1: acquiring a current screen background image and a control element image to be tested; s2: graying the element image of the control to be detected and the screen background image to obtain a gray image, constructing a scale space based on the gray image, and searching for characteristic points in the scale space to obtain a characteristic point set; s3: binary coding is carried out on the feature points, euclidean distance between the feature points and the feature points is calculated, and a matched feature point set is obtained; s4: optimizing the matched feature point set, and removing incorrect matched feature point pairs; s5: judging whether the control element to be detected exists in the screen background image according to the optimized matching feature point pairs, and if so, calculating the relative coordinate position of the control element to be detected in the screen background image. The method acquires the relative coordinate position of the control to be tested through image feature matching, thereby realizing the automatic execution of the test steps.
Description
Technical Field
The invention relates to the field of software automated testing, in particular to an element identification method and system based on image feature matching.
Background
Along with the advancement of the domestic process, the application field of the domestic operating system is gradually increased, and the stability and the usability of compatible adapting software on the operating system are also the continuous concerns of users. The automatic test can discover the defect problem in the software in time, and can ensure the quality of the software and improve the overall efficiency in the test process.
How to identify the measured element is the primary problem and key technology for all GUI automation test tasks. Unlike Windows systems, most software applications of domestic operating systems adopt a reverse adaptation mode, and the conventional control element identification method based on development frameworks and APIs of the operating systems is not suitable for automatic testing on the domestic operating systems, so that other identification modes are required to be adopted for effective element control positioning during control element identification. And applying the element control positioning mode to an automatic test platform to complete the software automatic test of the PC end on the domestic operating system.
The invention relates to a system and a method for acquiring GUI element attribute information on a domestic kylin operating system, in particular to a system and a method for acquiring the GUI element attribute information on a domestic kylin operating system, which can meet the requirement of domestic operating system users on acquisition of the GUI application control attribute information by only calling a function interface of a dynamic link library in a control name mode of a query control, and has high identification accuracy and high identification efficiency. However, the method is only suitable for automatic testing of the graphical user interface application based on the Qt development framework, can not meet control identification of the application developed by adopting other graphical interface development frameworks, and has a small application range.
The invention discloses an image matching method, which comprises the steps of firstly carrying out global template matching on a template image in a source image to find out an optimal matching area, then calculating characteristic points and characteristic vectors of the template image and the source image, calculating visual similarity between the optimal matching area and the template image according to the characteristic points and the characteristic vectors, judging that the optimal matching area is not matched with the template image if the visual similarity is zero, and obtaining a characteristic matching point pair of the template image and the source image if the visual similarity is not zero, and calculating positioning coordinates of the optimal matching image according to the characteristic matching point pair. The method has two defects: firstly, the method adopts a global template matching method to search the best matching area, and under the slight change of different screen resolutions or background screens, the error rate of the global template matching method is higher, and the problem that the initial best matching area is searched for errors, so that the subsequent identification is wrong can occur. Second, if there are multiple images similar to the template image in the source image, a problem of positioning error occurs.
The invention discloses a method, a device and a system for identifying a GUI element, which comprises a method for generating GUI element identification information, analyzing a displayed GUI to obtain attribute information of a target GUI element, and finally generating the identification information of the target GUI element. The method uses Xpath information of GUI elements to identify control elements to be tested, for example, a mode of acquiring a Return button in a dialog box is as follows: "/element [ @ text=form 1 ']/child/element [ @ text=form 2' ]/child/element [ @ text= 'Return' ]. The method realizes the GUI element attribute information organization method and clearly displays the GUI element attribute. The method has two defects: first: the method relies on the GUI element acquisition interface of the operating system, and lacks a related query interface in the domestic operating system, so that the method cannot be suitable for the application scene of the domestic operating system. Second,: the attribute acquisition mode is complicated, the attribute information of the elements needs to be queried in a grading way, and the problem of positioning errors is easy to occur.
Disclosure of Invention
In order to solve the technical problems, the invention provides an element identification method and system based on image feature matching.
The technical scheme of the invention is as follows: an element identification method based on image feature matching comprises the following steps:
Step S1: analyzing a control element to be tested in the operation step of the test case, acquiring a current screen background image and a control element image to be tested, and respectively recording the storage positions of the control element image to be tested and the screen background image;
Step S2: graying the control element image to be detected and the screen background image to obtain a gray image, constructing a scale space based on the gray image, searching for characteristic points in each layer of the scale space, and comparing adjacent layers to obtain a local maximum value to obtain a characteristic point set of the control element image to be detected and the screen background image;
Step S3: binary coding is carried out on the characteristic points of the control element gray level diagram to be detected and the screen background gray level diagram, and Euclidean distance between the characteristic points is calculated, so that a matched characteristic point pair set is obtained;
Step S4: optimizing the matched characteristic point pair set, removing incorrect matched characteristic point pairs, and obtaining optimized matched characteristic point pairs;
Step S5: judging whether the control element to be detected exists in the screen background image according to the optimized matching feature point pairs, and if so, calculating the relative coordinate position of the control element to be detected in the screen background image.
Compared with the prior art, the invention has the following advantages:
1. The invention discloses an element identification method based on image feature matching, which adopts a bilinear interpolation method to simulate the resolution of an actual screen, adopts Gaussian blur to simulate the blur degree of the same application icon on different screen resolutions, thereby constructing a scale space, ensuring that feature points of images can be identified under various screen sizes, and completing image matching.
2. According to the invention, the result of the image feature matching is optimized, the feature points are selected in a screen blocking mode, the feature points are fitted into an optimal matching model, the image matching result closest to the image is obtained, and the accuracy of element identification and test result judgment is improved.
3. The method provided by the invention can accurately identify the elements under different screen resolutions, has strong portability of the automatic test platform, can be suitable for software with different development architectures and scenes in which the software element API cannot be acquired, has wide identification range for element control, has high identification accuracy, and can meet the automatic test of most software on domestic operating systems.
Drawings
FIG. 1 is a flowchart of an element identification method based on image feature matching in an embodiment of the invention;
FIG. 2 is a schematic view of a background image of a screen according to an embodiment of the present invention;
FIG. 3 is a schematic diagram 1 of an element image of a control to be tested in an embodiment of the present invention;
FIG. 4 is a schematic diagram of bilinear interpolation in an embodiment of the invention;
FIG. 5 is a schematic diagram of obtaining local maxima pixels in an embodiment of the present invention;
FIG. 6 is a schematic diagram 2 of an element image of a control to be tested in an embodiment of the present invention;
FIG. 7 is a schematic diagram of pixel values of a schematic diagram 2 of a control element to be tested in an embodiment of the present invention;
FIG. 8 is a Gaussian kernel blurred image diagram of FIG. 2 using 3*3 for a control element diagram to be tested in an embodiment of the invention;
FIG. 9 is a schematic diagram of a Gaussian kernel blurred image of FIG. 2 using 5*5 for a control element to be tested in an embodiment of the invention;
FIG. 10 is a block diagram of a system for element identification based on image feature matching;
FIG. 11 is a schematic diagram of a main window UI interface provided in an embodiment of the invention;
fig. 12 is a schematic diagram of a parameter setting window UI interface provided in an embodiment of the present invention.
Detailed Description
The invention provides an element identification method based on image feature matching, which is used for acquiring the relative coordinate position of a control to be tested through image feature matching, so that the automatic execution of a test step is realized.
The present invention will be further described in detail below with reference to the accompanying drawings by way of specific embodiments in order to make the objects, technical solutions and advantages of the present invention more apparent.
Example 1
As shown in fig. 1, the element identification method based on image feature matching provided by the embodiment of the invention includes the following steps:
Step S1: analyzing a control element to be tested in the operation step of the test case, acquiring a current screen background image and a control element image to be tested, and respectively recording the storage positions of the control element image to be tested and the screen background image;
Step S2: graying the element image of the control to be detected and the screen background image to obtain a gray image, constructing a scale space based on the gray image, searching for characteristic points in each layer of the scale space, and comparing adjacent layers to obtain a local maximum value to obtain a characteristic point set of the element image of the control to be detected and the screen background image;
step S3: binary coding is carried out on the gray level diagram of the control element to be detected and the characteristic points of the screen background, and Euclidean distance between the gray level diagram and the characteristic points of the screen background is calculated, so that a matched characteristic point pair set is obtained;
step S4: optimizing the matched feature point set, removing incorrect matched feature point pairs, and obtaining optimized matched feature point pairs;
step S5: judging whether the control element to be detected exists in the screen background image according to the optimized matching feature point pairs, and if so, calculating the relative coordinate position of the control element to be detected in the screen background image.
In one embodiment, step S1 described above: analyzing a control element to be tested in the operation step of the test case, obtaining a current screen background image and a control element image to be tested, and respectively recording the storage positions of the control element image to be tested and the screen background image, wherein the method specifically comprises the following steps:
Step S11: capturing a screen background image of a control element to be tested in a full screen by calling a screenshot function, and storing the screen background image under the same directory as the test script;
And calling a screenshot function to intercept the background image of the screen in a full screen mode, setting a computer screen as a solid background image before automatic testing in order to reduce interference of other pixel points in the background image of the screen, and only reserving recycle bin, a computer and tested application icons in the background image of the desktop. After the screenshot is completed, placing the screen background image under the same catalog as the test script, and directly covering the original image by adopting the same name for reducing the occupation of the image space;
Step S12: intercepting a control element image to be detected according to the characteristics of the control element to be detected in a screen background image, and storing the control element image to be detected in a control element image list to be detected, wherein the three conditions are as follows:
a) The control element to be detected is unique and easy to identify in the screen background image, and a rectangular image containing the control element to be detected is intercepted to be used as a control element image to be detected;
b) The control element to be detected is unique in the screen background image but not easy to identify, and the elements which are easy to identify around the control element to be detected are intercepted together to be used as the control element image to be detected;
c) And the control element to be detected is not unique in the screen background image, and the elements which are easy to identify around the control element to be detected are intercepted together to be used as the control element image to be detected.
And saving the intercepted control element image to be tested under the control element folder to be tested.
Fig. 2 is a schematic diagram of a background image of a screen, and fig. 3 is a WPS document application icon of the control element to be tested in fig. 2. The embodiment of the invention automatically intercepts the current screen background image in a full screen mode by using a screenshot function. For the control element to be tested, the icon of the WPS document needs to be manually intercepted.
After the element image and the background image of the screen of the control to be detected are obtained, feature point detection is needed to be carried out on the two images, so that preparation is made for subsequent determination of the position of the element of the control to be detected in the current screen.
In one embodiment, step S2 above: graying the control element image to be detected and the screen background image to obtain a gray image, constructing a scale space based on the gray image, searching feature points in each layer of the scale space, and comparing adjacent layers to obtain a local maximum value to obtain a feature point set of the control element image to be detected and the screen background image, wherein the method specifically comprises the following steps:
Step S21: extracting R, G and B three-color components of each control element image to be detected and a screen background image respectively, setting a weight for each component, multiplying the pixel values of the three components by the corresponding weights respectively, multiplying 255 and rounding after adding, and finally merging the three components together to obtain a control element gray level image to be detected and a screen background gray level image respectively;
Step S22: respectively constructing a scale space for a control element gray level diagram to be detected and a screen background gray level diagram by adopting a bilinear interpolation method and a Gaussian blur method to obtain the control element scale space to be detected and the screen background scale space to simulate the display effect of the control element to be detected under different screen resolutions, wherein each screen resolution is used as a layer of the scale space;
The embodiment of the invention uses a bilinear interpolation method to scale the original intercepted image to the size of a designated image, simulates different screen resolutions in display, such as 1920 x 1080, 2560 x 1440, 2736 x 1824, 4096 x 2160 and the like, and uses a Gaussian blur method to simulate the image blur degree under different screen resolutions. Each screen resolution is a layer of the scale space.
The specific implementation of bilinear interpolation is as follows:
The bilinear interpolation method calculates the pixel value of the target pixel point by performing linear interpolation once in both the lateral and longitudinal directions, as shown in fig. 4:
First, the target pixel P (x, y) finds its four nearest points, and is assumed to be (0, 0), (0, 1), (1, 0), (1, 1), and its corresponding pixel values are f (0, 0), f (0, 1), f (1, 0), and f (1, 1). Firstly, carrying out linear interpolation on (0, 1) and (1, 1) in the transverse direction (X direction) to obtain a pixel value of a pixel point p1, and carrying out interpolation on (0, 0) and (1, 0) by using the same method to obtain a pixel value of a pixel point p 2:
f(p1)＝f(0,0)*x+f(1,0)*(1-x)
f(p2)＝f(0,1)*x+f(1,1)*(1-x)
After obtaining the pixel values of the P1 and P2 points, the pixel value of the P point is then obtained through the P1 and P2 points in the longitudinal direction (Y direction), and a linear interpolation method is still used:
f(P)＝f(p1)*y+f(p2)*(1-y)＝(f(0,0)*x+f(1,0)*(1-x))*y+(f(0,1)*x+f(1,1)*(1-x))*(1-y)
the bilinear interpolation method uses the distance between the pixel values of two adjacent points in the X and Y directions and the target point as a weighted mean value.
The specific implementation of the Gaussian blur method is as follows:
a gaussian kernel is created containing a series of different weights. And convolving the Gaussian kernel with each pixel point in the image, and overlapping surrounding pixel values according to corresponding weights to obtain blurred pixel values. And constructing images with different screen resolutions, wherein the weights of the adopted Gaussian kernels are different.
Step S23: respectively obtaining local maxima of each layer in a control element scale space to be detected and a screen background scale space as characteristic points of the layer;
The local maximum value of each layer in the scale space is obtained, a pixel point i is selected, 16 pixel points on the circumference are selected, the pixel points are selected as shown in fig. 5, if the pixel value of 9 continuous pixel points in the 16 pixel points is larger than the pixel value of the selected circle center pixel point plus a threshold value or smaller than the pixel value of the selected pixel point minus the threshold value, the selected pixel point is taken as the local maximum value point, otherwise, the pixel point is not taken as the candidate point of the feature point.
For example, an image with a size of 11×9 is shown in fig. 6, and a pixel value corresponding to each pixel point of the image is shown in fig. 7; the local maximum value of the current layer can be obtained through the method;
Step S24: and comparing the pixel values of the characteristic points of the ith layer with the pixel values of the characteristic points of the ith-1 layer and the pixel values of the characteristic points of the ith-1 layer, reserving the characteristic points corresponding to the maximum value, and screening out the rest characteristic points, so that a characteristic point set of each control element image to be detected and a characteristic point set of the screen background image are obtained.
For example, layer 2 compares feature point pixel values with layers 1 and 3 simultaneously, takes a reserve of maximum values, and so on, layer 3 compares layers 2 and 4 simultaneously, takes a maximum value. After the steps, the characteristic point set of one image can be obtained.
The pixel value after Gaussian blur of the weight x1 is shown in fig. 8;
The pixel values after the gaussian blur of the weight x2 are shown in fig. 9; the pixel values of the pixel points at the lower right side after Gaussian blur are basically unchanged, so that the points can be removed after comparison between layers, and meanwhile, no obvious characteristic points at the lower right side of the image can be seen according to the gray level image. And comparing the pixel values of the original image pixel points with the pixel values of the two layers of image pixel points to obtain a characteristic point set of the image.
In one embodiment, the step S3: binary coding is carried out on the gray level diagram of the control element to be detected and the characteristic points of the screen background, and the Euclidean distance between the gray level diagram and the characteristic points of the screen background is calculated, so that a matched characteristic point pair set is obtained, and the binary coding method specifically comprises the following steps:
Step S31: describing characteristic points in a characteristic point set of the element image of the control to be detected and a characteristic point set of the background image of the screen: comparing the gray values of two surrounding pixel points p1 and p2 around the characteristic point p according to a Gaussian distribution rule, wherein 1 is used for representing the gray value of p1 to be larger than the gray value of p2, 0 is used for representing the gray value of p1 to be smaller than the gray value of p2, and 128 pairs of pixels are compared around the characteristic point p to obtain 128 binary code descriptions consisting of 0 and 1 as the characteristic point p;
step S32: and calculating the Euclidean distance between the characteristic points of the element image of the control to be detected and the characteristic points of the background image of the screen according to the following formula:
Wherein X i represents the binary code of the ith feature point in the feature point set of the element image of the control to be detected, and Y i represents the binary code of the ith feature point in the feature point set of the background image of the screen; d is the number of feature points;
The smaller the D value obtained according to the formula is, the higher the similarity degree between the characteristic points in the element image of the control to be detected and the characteristic points in the screen background image is;
Step S33: taking the element image of the control to be detected as a reference image, and calculating Euclidean distances between the characteristic points in the characteristic point set of the element image of the control to be detected and the characteristic points in the characteristic point set of the background image of the screen according to the formula of the step S32; then, taking the screen background image as a reference image, and calculating Euclidean distances between the characteristic points in the characteristic point set of the screen background image and the characteristic points in the characteristic point set of the element image of the control to be detected according to the formula of the step S32; if the two calculation results of a pair of feature points are the same, reserving the matched feature point pair, and deleting the rest feature points; and finally obtaining a matched characteristic point pair set.
In one embodiment, step S4 above: optimizing the matched feature point set, removing incorrect matched feature point pairs to obtain optimized matched feature point pairs, and specifically comprising:
step S41: the width and the height of the element image of the control to be detected are w and h respectively, and the element image is uniformly divided into 2 x 2 blocks; the number of the feature points matched with the feature point set is K;
The embodiment of the invention is used for blocking the element image of the control to be detected, and aims to randomly select four characteristic points to fit an initial model in the subsequent circulation process so as to prevent the influence of the close proximity of the four characteristic points on the result, thereby blocking;
Step S42: randomly selecting a characteristic point from each block and matching characteristic points corresponding to the screen background image to form 4 point pairs, calculating a transformation matrix, and fitting to obtain a matching estimation model;
Step S43: sequentially selecting feature points from the 4 blocks, solving the re-projection coordinates of the selected feature points in the screen background image according to the transformation matrix, comparing the distance between the re-projection coordinates and the original matched feature point coordinates, and if the distance is smaller than a certain threshold value, considering that the feature points are matched into correct matched feature point pairs; otherwise, the matching is considered to be wrong, and the number of correctly matched characteristic point pairs is recorded;
Step S44: judging whether the number of the correctly matched characteristic point pairs in the current cycle is larger than the number of the correctly matched characteristic point pairs in the previous cycle, if so, updating a matching estimation model, and executing a step S45; otherwise, directly executing the step S45;
Step S45: judging whether the cycle times are greater than the set maximum cycle times, if so, outputting an optimal matching model, namely taking the result with the most number of correctly matched characteristic points as a final result, and eliminating the incorrectly matched result to obtain an optimized matched characteristic point pair; if not, step S41 is performed.
In one embodiment, the step S5 is as follows: judging whether a control element to be detected exists in the screen background image according to the optimized matching characteristic point pairs, and if so, calculating the relative coordinate position of the control element to be detected in the screen background image, wherein the method specifically comprises the following steps:
Step S51: dividing the number of the optimized matched characteristic point pairs by the number of the characteristic points in the element images of the control to be detected to obtain the similarity between the element images of the control to be detected and the screen background images;
Step S52: when the similarity is greater than or equal to 95%, judging that the element image of the control to be detected and the screen background image can be matched, and executing step S53; when the similarity is smaller than 95%, judging that the element image of the control to be detected is not matched with the screen background image, and returning an abnormal message which cannot be positioned;
Step S53: according to the width w and the height h of the element image of the control to be detected, selecting the same width and the same height from the background image of the screen as matching areas, and outputting X and Y coordinates of four vertexes of the element image of the control to be detected in the background image of the screen;
step S54: the method comprises the steps of respectively obtaining actual X-direction and Y-direction position proportion [ X ] and [ Y ] of four vertexes in a control element image to be detected, and calculating the relative coordinate position of the control element image to be detected in a screen background image:
x＝(upper_left_pos[x]+(upper_right_pos[x]-upper_left_pos[x])*proportion[x])
y＝(upper_left_pos[y]+(bottom_left_pos[y]-upper_left_pos[y])*proportion[y]
Wherein upper_left_pos [ x ] is the x-coordinate of the upper left vertex, upper_left_pos [ y ] is the y-coordinate of the upper left vertex, upper_right_pos [ x ] is the x-coordinate of the upper right vertex, and bottom_left_pos [ y ] is the y-coordinate of the lower left vertex.
The element identification method based on image feature matching, which is applied to the actual automatic test, comprises the following specific steps:
1. parameters required for automated testing are acquired: the method comprises the steps of obtaining a relative path source_img_path of an element control image to be tested, an absolute path of a test script parent directory, proportions of the element control to be tested in X and Y directions in manual screenshot [ X ] and the proportions of the element control to be tested in the Y directions ], waiting time sleep_time among operation steps, operation description of each operation step and other parameters.
2. And transmitting the source_img_ path, path, proportion [ x ] and the report [ y ] to an element positioning function, and calculating to obtain the similarity between the element image of the control to be detected and the screen background image and the relative coordinate position of the element of the control to be detected according to an element identification method based on image feature matching.
3. And transmitting the relative coordinate positions of the control elements to be tested to PyAutoGUI mouse-keyboard driving functions, and simulating mouse clicking and keyboard key operation.
4. And calling a screenshot function to intercept the latest screen background image in a full screen mode to be taken as an actual test result screenshot, and storing the latest screen background image in a test result folder to be taken as output of the module (a function library module).
The function library module comprises a mouse operation class, a keyboard input operation class and a shortcut key operation class. The mouse operation comprises mouse clicking, roller sliding, mouse moving, mouse dragging and other operations; the keyboard operation class mainly comprises character input of an analog input box, and characters to be input are transmitted to a keyboard input function in a character string mode; the shortcut key operation can simulate that a user presses a plurality of keys simultaneously, and different keys are separated by commas, for example, the shortcut key ctrl+c, and then the ("Ctrl", "C") is transmitted to the shortcut key operation function. When writing the script, the corresponding operation function is called, and the parameter value is input, so that the operation steps in the test case can be converted into script code sentences.
For example, one test case is: double click open WPS document
The operation steps are as follows: double-click WPS document icon with left mouse button
The code implementation:
mouse_double_click ("WPS document icon Path, proportion, sleep_time," mouse left click double click WPS document icon ")
Through the code, the mouse can automatically click the corresponding position, and after clicking, the application can be started to the WPS document starting interface, and the full-screen intercepting starting interface image is taken as output. If the to-be-tested application appears Bug, the to-be-tested application cannot be started to the starting interface, and then the actual test result image intercepted during screenshot is different from the expected result.
Further, the test results may be asserted by image matching.
And calling the function library module to execute the test step, obtaining an actual test result screenshot of the tested application, and performing image matching on the actual test result screenshot intercepted by the full screen and the expected result screenshot. If the similarity of the two images is larger than a specified threshold, the assertion is correct, and the test case result is marked as PASS; otherwise, the assertion FAILs, the test case result is marked as FAIL, and the exception that the test result FAILs is thrown out. The method comprises the following specific steps:
1. And before the automatic test, manually intercepting the expected result screenshot in a full screen mode, and storing the expected result screenshot in an expected result screenshot folder.
2. And acquiring a path actual_result_img_path of the actual test result screenshot and a path expected_result_img_path of the expected test result screenshot, performing image feature matching on the two pictures, respectively detecting feature points in the two pictures, and matching the feature points in the two pictures.
3. And calculating the characteristic point logarithm which can be successfully matched in the two images, marking as success_point_ pairs, dividing the characteristic point logarithm which can be successfully matched by the characteristic point number in the expected test result, and marking as the similarity of the two images.
4. Comparing the similarity of the two images with a specified threshold, if the similarity of the two images is greater than or equal to the specified threshold, declaring correct, wherein the test case is PASS; if the similarity of the two images is smaller than the specified threshold, the assertion FAILs, and the test case is FAIL. And returning the result judgment of the test case.
And step 4, judging the running result of the tested software after the execution of one test case is finished, and judging whether the problems of abnormal prompt box, application crash and blocking, inconsistent display of the actual result interface and the expected result UI interface and the like occur in the actual result.
After the steps are finished, a unittest unit test framework is used for collecting logs in the running process of the test script and the assertion result of each test case, and outputting the execution number of each test case, the passing and failure of the test cases and the like, and the abnormal information of the tested software in the running process.
Example two
As shown in fig. 10, an embodiment of the present invention provides an element recognition system based on image feature matching, which includes the following modules:
The control element image to be tested and screen background image module 61 is used for analyzing the control element to be tested in the operation step of the test case, obtaining the current screen background image and the control element image to be tested, and respectively recording the storage positions of the control element image to be tested and the screen background image;
The module 62 is configured to obtain a feature point of an element image and a background image of a screen of a control to be detected, gray-scale the element image and the background image of the screen to obtain a gray-scale image, construct a scale space based on the gray-scale image, and find feature points in each layer of the scale space to obtain a feature point set of the element image and the background image of the control to be detected;
The module 63 is used for obtaining a matching characteristic point of the element image of the control to be detected and the screen background image, performing binary coding on the gray level image of the element of the control to be detected and the characteristic point of the screen background, and calculating the Euclidean distance between the gray level image of the element of the control to be detected and the characteristic point of the screen background, so as to obtain a matching characteristic point pair set;
the optimized matching feature point module 64 is configured to optimize the matching feature point set, remove incorrect matching feature point pairs, and obtain optimized matching feature point pairs;
The module 65 is configured to calculate a relative coordinate position of the control element to be measured in the screen background, and determine whether the control element to be measured exists in the screen background image according to the optimized matching feature point pair, and if so, calculate a relative coordinate position of the control element to be measured in the screen background image.
Example III
The embodiment of the invention provides an element identification platform based on image feature matching, which can be used as an automatic test platform and run on a domestic operating system to finish the function automatic test of software.
The platform is realized by developing software with a graphical user interface, and comprises four modules of newly-built script files, simulated operation monitoring, parameter setting and script automatic generation:
newly-built script file module: the method comprises the steps of creating a script file input and display window, and opening up space in a machine memory for storing a test script file;
and a simulation operation monitoring module: the method comprises the steps of monitoring a mouse click event, and opening various setting windows after the mouse click event is monitored, wherein the various setting windows comprise a mouse click setting window, a keyboard input setting window, a roller sliding setting window, a mouse moving setting window, a mouse dragging setting window and the like;
Parameter setting module: the method comprises the steps of setting parameters such as clicking times, element image paths of control elements to be tested, coordinate proportion positions, waiting time for executing next operation, current step description and the like, of real parameters to be transmitted in operation steps of test cases in different setting windows, storing parameter data to a local place after the setting is completed, and uniformly managing the parameter data in each script;
The script automatic generation module: and transferring the parameters input in the parameter setting unit to the corresponding mouse and keyboard operation functions, processing the parameters, splicing the parameters into executable script sentences, and attaching the script sentences to a script display area created in the newly built file unit to complete automatic generation of test script sentences.
The function implementation of the element identification platform based on image feature matching is completed by the cooperation of a menu bar, a main window, a sub window and the connection of the main window and the sub window. Clicking a new script file of the menu bar, and displaying the new script file in a main window; inputting project information of a script file in a project management area in a main window, clicking a mouse or keyboard operation button to be simulated in a simulation operation part of the main window, and calling a parameter setting sub-window; and performing parameter setting in the parameter setting sub-window, generating script code sentences after data processing, and returning the generated code sentences to a code display area of the main window for display.
Menu bar: all functions of the application are included, including files, edits, operations, runs, help five menu items. Clicking on a different menu may enter a different secondary menu bar: the file menu is used for processing the test script file, including creating, opening, storing and exiting; the new function can create a blank file interface, the open function can open the saved code file, the save function can save the generated test script code as the py type, and the exit function can exit the software; editing menu including copying, cutting, pasting, deleting, cancelling, redoing, selecting, and editing code in code display area; the operation menu comprises python package import, an addition path, mouse click, keyboard input, mouse dragging, mouse movement, mouse wheel, shortcut key, combined operation, desktop display and other operations, and the corresponding operation setting flick frame can be opened for setting after different operations are clicked; the running menu is used for executing and debugging the script file; and the help menu is used for viewing the use method and version information of the application.
A main window: the main window UI interface layout is shown in fig. 11. The method comprises six parts of menu bar, project management, simulation operation, code editing, code display area, script running and result processing. The project management part manages the project name, the test case name and the test case number of the current test script, each project is provided with a plurality of test cases and test numbers, and the test case name and the test case numbers are unique; the simulation operation part is used for monitoring a mouse event, clicking a corresponding button and then calling up a corresponding parameter setting bullet frame; the code editing part is used for adding a third party library and an adding path used in the project; the code display area displays a test script which is automatically generated after the setting of the operation parameters is completed and can be used for display and input; the script operation and result processing is used for operating the generated test script and test result processing.
Sub-windows: the two modes of opening by clicking the secondary menu of the menu bar and opening by clicking a button in the main window can be realized. Buttons of the same name open up as the same sub-window. Parameter setting information of the function buttons is arranged in the sub-window. As shown in fig. 12, the clicking operation sets the UI interface of the sub-window, and the display of different sub-windows is different. The clicking operation setting window comprises four parts including clicking mode, element positioning, waiting time for executing the next step and current operation function description.
The clicking mode is displayed in a combined frame, and comprises four options of left click, right click and middle click, and can be switched by clicking a drop-down frame; the element positioning comprises two settings of element position and coordinate proportion, wherein the element position comprises an input frame and a browsing button, the window can be opened after clicking the browsing button, the absolute path of the image is selected, only the image name is displayed in the input frame, and all the absolute paths are not displayed; the coordinate proportion includes the proportion of the X direction and the Y direction; executing the waiting time of the next step, wherein a number can be input in an input box, and the waiting time is expressed in seconds(s); the current operation description may be entered within the input box. The right lower corner of the window is provided with a confirmation button and a cancel button, a test script is automatically generated after the confirmation button is clicked, and after the confirmation button is clicked, the setting in the window is not saved.
Connection of main window and sub window: taking a button in a menu bar or a main window as a transmitting signal control element, and calling a show_Win () function to open a sub-window or execute corresponding operation after a mouse event is monitored; after entering the parameters in the child window, the call generates script function GENERATESCRIPT () and will call display code function showScript () to display the code back into the main window. The role of the event processing function and the function name are shown in table 1:
Table 1: action of event handling function and function name
Taking the example of "double clicking Foxmail a desktop icon to open Foxmail an application" as a test case, a test script for implementing the test case is automatically generated:
Newly creating a script file, determining that a control element to be detected in the step is Foxmail desktop icons, and executing simulation operation as follows: double clicking the left mouse button, the expected result is a Foxmail application's login interface. Clicking a mouse click button in a main window, opening a mouse click setting window, selecting a click mode in the setting window to be a mouse left click, wherein the element position is Foxmail a path of a desktop icon, the coordinate ratio of X and Y directions is 1/2 and 1/2, the waiting time for executing the next step is 2 seconds, and the current operation is described as follows: double clicking Foxmail on the desktop icon opens Foxmail application, clicks on the ok button, tests the script code "mouse. Left_double_click (" foxmailicon. Png ", source_img_path, [1/2,1/2],2,
Decription = "double click Foxmail desktop icon open Foxmail application") "is displayed within the code display area of the main window.
The above examples are provided for the purpose of describing the present invention only and are not intended to limit the scope of the present invention. The scope of the invention is defined by the appended claims. Various equivalents and modifications that do not depart from the spirit and principles of the invention are intended to be included within the scope of the invention.
Claims (7)
1. An element identification method based on image feature matching is characterized by comprising the following steps:
Step S1: analyzing a control element to be tested in the operation step of the test case, acquiring a current screen background image and a control element image to be tested, and respectively recording the storage positions of the control element image to be tested and the screen background image;
Step S2: graying the control element image to be detected and the screen background image to obtain a gray image, constructing a scale space based on the gray image, and searching for characteristic points in each layer of the scale space to obtain a characteristic point set of the control element image to be detected and the screen background image;
Step S3: binary coding is carried out on the characteristic points of the control element gray level diagram to be detected and the screen background gray level diagram, and Euclidean distance between the characteristic points is calculated, so that a matched characteristic point pair set is obtained;
Step S4: optimizing the matched characteristic point pair set, removing incorrect matched characteristic point pairs, and obtaining optimized matched characteristic point pairs;
Step S5: judging whether the control element to be detected exists in the screen background image according to the optimized matching feature point pairs, and if so, calculating the relative coordinate position of the control element to be detected in the screen background image.
2. The method for identifying elements based on image feature matching according to claim 1, wherein said step S1: analyzing a control element to be tested in the operation step of the test case, obtaining a current screen background image and a control element image to be tested, and respectively recording the storage positions of the control element image to be tested and the screen background image, wherein the method specifically comprises the following steps:
step S11: capturing a screen background image of a control element to be tested in a full screen by calling a screenshot function, and storing the screen background image under the same directory as a test script;
step S12: intercepting a control element image to be detected according to the characteristics of the control element to be detected in the screen background image, and storing the control element image to be detected in a control element image list to be detected, wherein the three conditions are as follows:
a) The control element to be detected is unique and easy to identify in the screen background image, and a rectangular image containing the control element to be detected is intercepted to serve as a control element image to be detected;
b) The control element to be detected is unique but not easy to identify in the screen background image, and the elements which are easy to identify around the control element to be detected are intercepted together to be used as a control element image to be detected;
c) And the control element to be detected is not unique in the screen background image, and the elements which are easy to identify around the control element to be detected are intercepted together to be used as the control element image to be detected.
3. The method for identifying elements based on image feature matching according to claim 2, wherein said step S2: graying the control element image to be detected and the screen background image to obtain a gray image, constructing a scale space based on the gray image, searching feature points in each layer of the scale space, and comparing adjacent layers to obtain a local maximum value to obtain a feature point set of the control element image to be detected and the screen background image, wherein the method specifically comprises the following steps:
Step S21: extracting R, G and B three-color components of each control element image to be detected and each screen background image respectively, setting a weight for each component, multiplying the pixel values of the three components by the corresponding weights respectively, multiplying 255 and rounding after adding, and finally merging the three components together to obtain a control element gray level image to be detected and a screen background gray level image respectively;
Step S22: respectively constructing a scale space for the control element gray level diagram to be detected and the screen background gray level diagram by adopting a bilinear interpolation method and a Gaussian blur method to obtain the control element scale space to be detected and the screen background scale space to simulate the display effect of the control element to be detected under different screen resolutions, wherein each screen resolution is used as a layer of the scale space;
Step S23: respectively acquiring local maxima of each layer in the control element scale space to be detected and the screen background scale space as characteristic points of the layer;
Step S24: and comparing the pixel values of the characteristic points of the ith layer with the pixel values of the characteristic points of the ith-1 layer and the pixel values of the characteristic points of the ith-1 layer, reserving the characteristic points corresponding to the maximum value, and screening out the rest characteristic points, so that a characteristic point set of each control element image to be detected and a characteristic point set of the screen background image are obtained.
4. The method for identifying elements based on image feature matching according to claim 3, wherein said step S3: binary coding is carried out on the characteristic points of the control element gray level diagram to be detected and the screen background gray level diagram, and Euclidean distance between the characteristic points is calculated, so that a matched characteristic point pair set is obtained, and the binary coding method specifically comprises the following steps:
Step S31: describing characteristic points in the characteristic point set of the element image of the control to be detected and the characteristic point set of the screen background image: comparing the gray values of two surrounding pixel points p1 and p2 around the characteristic point p according to a Gaussian distribution rule, wherein 1 is used for representing the gray value of p1 to be larger than the gray value of p2, 0 is used for representing the gray value of p1 to be smaller than the gray value of p2, and 128 pairs of pixels are compared around the characteristic point p to obtain 128 binary code descriptions consisting of 0 and 1 as the characteristic point p;
step S32: and calculating the Euclidean distance between the characteristic points of the element image of the control to be detected and the characteristic points of the background image of the screen according to the following formula:
Wherein, X i represents the binary code of the ith feature point in the feature point set of the element image of the control to be detected, and Y i represents the binary code of the ith feature point in the feature point set of the background image of the screen; d is the number of feature points;
step S33: calculating Euclidean distances between the characteristic points in the characteristic point set of the control element image to be detected and the characteristic points in the characteristic point set of the screen background image according to the formula of the step S32 by taking the control element image to be detected as a reference image; then, taking the screen background image as a reference image, and calculating Euclidean distances between the characteristic points in the characteristic point set of the screen background image and the characteristic points in the characteristic point set of the element image of the control to be detected according to the formula of the step S32; if the two calculation results of a pair of feature points are the same, reserving the matched feature point pair, and deleting the rest feature points; and finally obtaining a matched characteristic point pair set.
5. The method for identifying elements based on image feature matching according to claim 4, wherein said step S4: optimizing the matched characteristic point pair set, removing incorrect matched characteristic point pairs, and obtaining optimized matched characteristic point pairs, wherein the method specifically comprises the following steps of:
Step S41: the width and the height of the element image of the control to be detected are w and h respectively, and the element image is uniformly divided into 2 x 2 blocks; the number of the characteristic points of the matched characteristic point set is K;
Step S42: randomly selecting a characteristic point from each block and matching characteristic points corresponding to the screen background image to form 4 point pairs, calculating a transformation matrix, and fitting a matching estimation model;
Step S43: sequentially selecting feature points from the 4 blocks, solving the re-projection coordinates of the selected feature points in the screen background image according to the transformation matrix, comparing the distance between the re-projection coordinates and the original matched feature point coordinates, and if the distance is smaller than a certain threshold value, considering that the feature points are matched to be correct matched feature point pairs; otherwise, the matching is considered to be wrong, and the number of correctly matched characteristic point pairs is recorded;
Step S44: judging whether the number of the correctly matched characteristic point pairs in the current cycle is larger than the number of the correctly matched characteristic point pairs in the previous cycle, if so, updating a matching estimation model, and executing a step S45; otherwise, directly executing the step S45;
Step S45: judging whether the cycle times are greater than the set maximum cycle times, if so, outputting an optimal matching model, namely taking the result with the most number of correctly matched characteristic points as a final result, and eliminating the incorrectly matched result to obtain an optimized matched characteristic point pair; if not, step S41 is performed.
6. The method for identifying elements based on image feature matching according to claim 5, wherein said step S5: judging whether the control element to be detected exists in the screen background image according to the optimized matching feature point pairs, and if so, calculating the relative coordinate position of the control element to be detected in the screen background image, wherein the method specifically comprises the following steps:
Step S51: dividing the number of the optimized matched characteristic point pairs by the number of characteristic points in the element image of the control to be detected to obtain the similarity between the element image of the control to be detected and the screen background image;
Step S52: when the similarity is greater than or equal to 95%, judging that the element image of the control to be detected and the screen background image can be matched, and executing step S53; when the similarity is smaller than 95%, judging that the element image of the control to be detected is not matched with the screen background image, and returning an abnormal message which cannot be positioned;
step S53: selecting the same width and height as matching areas in the screen background image according to the width w and the height h of the control element image to be detected, and outputting X and Y coordinates of four vertexes of the control element image to be detected in the screen background image;
step S54: the actual X-direction and Y-direction position proportion project [ X ] and project [ Y ] of the four vertexes in the control element image to be detected are respectively obtained, and the relative coordinate position of the control element image to be detected in the screen background image is calculated:
x＝(upper_left_pos[x]+(upper_right_pos[x]-upper_left_pos[x])*proportion[x])
y＝(upper_left_pos[y]+(bottom_left_pos[y]-upper_left_pos[y])*proportion[y]
Wherein upper_left_pos [ x ] is the x-coordinate of the upper left vertex, upper_left_pos [ y ] is the y-coordinate of the upper left vertex, upper_right_pos [ x ] is the x-coordinate of the upper right vertex, and bottom_left_pos [ y ] is the y-coordinate of the lower left vertex.
7. An element identification system based on image feature matching is characterized by comprising the following modules:
the control element image to be tested and the screen background image module are obtained and used for analyzing the control element to be tested in the operation step of the test case, obtaining the current screen background image and the control element image to be tested, and respectively recording the storage positions of the control element image to be tested and the screen background image;
The control element image to be detected and the screen background image feature point module is used for graying the control element image to be detected and the screen background image to obtain a gray image, constructing a scale space based on the gray image, searching feature points in each layer of the scale space, and obtaining feature point sets of the control element image to be detected and the screen background image;
The control element image to be detected and the screen background image matching feature point module is used for binary coding the feature points of the control element gray level image to be detected and the screen background gray level image and calculating the Euclidean distance between the two to obtain a matching feature point pair set;
The optimization matching characteristic point module is used for optimizing the matching characteristic point pair set, removing incorrect matching characteristic point pairs and obtaining optimized matching characteristic point pairs;
And the module is used for calculating the relative coordinate position of the control element to be detected in the screen background, judging whether the control element to be detected exists in the screen background image according to the optimized matching characteristic point pairs, and if so, calculating the relative coordinate position of the control element to be detected in the screen background image.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410158845.7A CN117934882A (en) | 2024-02-04 | 2024-02-04 | Element identification method and system based on image feature matching |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410158845.7A CN117934882A (en) | 2024-02-04 | 2024-02-04 | Element identification method and system based on image feature matching |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117934882A true CN117934882A (en) | 2024-04-26 |
Family
ID=90750764
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410158845.7A Pending CN117934882A (en) | 2024-02-04 | 2024-02-04 | Element identification method and system based on image feature matching |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117934882A (en) |
-
2024
- 2024-02-04 CN CN202410158845.7A patent/CN117934882A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP7398068B2 (en) | software testing | |
| US11886892B2 (en) | Machine learned retraining for detection of user interface controls via variance parameters | |
| CN113391871B (en) | RPA element intelligent fusion picking method and system | |
| US9424167B2 (en) | Automated testing of an application system | |
| US8429612B2 (en) | Graphical user interface (GUI) noise reduction in a cognitive control framework | |
| US7831542B2 (en) | Iterative search with data accumulation in a cognitive control framework | |
| CN109189519B (en) | Universal user desktop behavior simulation system and method | |
| CN104391797A (en) | GUI (graphical user interface) widget identification method and device | |
| CN107666987A (en) | Robotic process automates | |
| US20200059441A1 (en) | Robotic process automation | |
| US11221833B1 (en) | Automated object detection for user interface generation | |
| TWI842931B (en) | Automatic identification processing device, automatic identification processing method, inspection system, program, and recording medium | |
| US8392887B2 (en) | Systems and methods for identifying graphic user-interface components | |
| CN114049631A (en) | Data labeling method and device, computer equipment and storage medium | |
| US20090268968A1 (en) | Triangular method for hypotheses filtration in a cognitive control framework | |
| CN112817863B (en) | AI-aided automatic test method and system based on AI deep learning | |
| CN117934882A (en) | Element identification method and system based on image feature matching | |
| CN115905016A (en) | BIOS Setup search function test method and device, electronic equipment and storage medium | |
| Ali et al. | Mobile-UI-Repair: a deep learning based UI smell detection technique for mobile user interface | |
| CN113703637A (en) | Inspection task coding method and device, electronic equipment and computer storage medium | |
| Pan et al. | Research on functional test of mobile app based on robot | |
| Gamal et al. | Owl Eye: An AI-Driven Visual Testing Tool | |
| CN117112446B (en) | Editor debugging method and device, electronic equipment and medium | |
| US20240184692A1 (en) | Software testing | |
| Wu et al. | Intelligent Software Testing based on Visual Feedback |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |