缺陷检测方法、装置、电子设备及计算机存储介质Defect detection method, device, electronic device and computer storage medium
本申请要求于2020年8月17日提交中国专利局、申请号为202010835350.5,发明名称为“缺陷检测方法、装置、电子设备及计算机存储介质”的中国专利申请的优先权,其全部内容通过引用结合在本申请中。This application claims the priority of the Chinese patent application filed on August 17, 2020 with the application number 202010835350.5 and the title of the invention is "defect detection method, device, electronic device and computer storage medium", the entire content of which is by reference Incorporated in this application.
技术领域technical field
本申请涉及计算机视觉技术领域,尤其涉及一种缺陷检测方法、装置、电子设备及计算机存储介质。The present application relates to the field of computer vision technology, and in particular, to a defect detection method, apparatus, electronic device, and computer storage medium.
背景技术Background technique
随着高铁建设的不断发展,高铁供电***安全性和可靠性的要求也在逐步提高,高铁接触网的安全检测与维护就显得尤为重要。接触网悬挂监测装置的普及极大地提高了接触网运营维护中图像采集作业的效率,但是,面对海量的图像数据,传统采用人工排查的方式对接触网进行缺陷检测的方式不仅耗时长、效率低,而且还存在漏检率较高的问题,导致接触网缺陷检测的准确性较低。With the continuous development of high-speed railway construction, the requirements for the safety and reliability of the high-speed railway power supply system are gradually increasing, and the safety inspection and maintenance of the high-speed railway catenary network is particularly important. The popularization of catenary suspension monitoring devices has greatly improved the efficiency of image acquisition in the operation and maintenance of catenary. However, in the face of massive image data, the traditional method of detecting defects in catenary by manual inspection is not only time-consuming and efficient. It also has the problem of high missed detection rate, resulting in low accuracy of catenary defect detection.
发明内容SUMMARY OF THE INVENTION
针对上述问题,本申请提供了一种缺陷检测方法、装置、电子设备及存储介质,有利于降低高铁接触网缺陷检测的漏检率,提高接触网缺陷检测的准确性。In view of the above problems, the present application provides a defect detection method, device, electronic device and storage medium, which are beneficial to reduce the missed detection rate of high-speed rail catenary defect detection and improve the accuracy of catenary defect detection.
为实现上述目的,本申请实施例第一方面提供了一种缺陷检测方法,该方法包括:In order to achieve the above purpose, a first aspect of the embodiments of the present application provides a defect detection method, the method includes:
获取高铁接触网的待检测图像;Obtain the image to be inspected of the high-speed rail catenary;
从所述待检测图像中分割出高铁接触网的第一部件的图像;Segment the image of the first component of the high-speed rail catenary from the image to be detected;
从所述第一部件的图像中分割出高铁接触网的第二部件的图像;所述第二部件为所述第一部件的子部件;Segment the image of the second part of the high-speed rail catenary from the image of the first part; the second part is a sub-part of the first part;
基于所述第一部件的图像、所述第二部件的图像对所述第一部件、所述第二部件进行缺陷分类,得到缺陷分类结果。Defect classification is performed on the first part and the second part based on the image of the first part and the image of the second part, and a defect classification result is obtained.
结合第一方面,在一种可能的实施方式中,所述从所述待检测图像中分割出高铁接触网的第一部件的图像,包括:With reference to the first aspect, in a possible implementation manner, the segmentation of the image of the first component of the high-speed rail contact net from the to-be-detected image includes:
对所述待检测图像进行特征提取,得到第一特征图;Perform feature extraction on the to-be-detected image to obtain a first feature map;
基于所述第一特征图对所述第一部件进行定位和分类,得到所述第一部件的第一矩形检测框;Locating and classifying the first component based on the first feature map to obtain a first rectangular detection frame of the first component;
按照所述第一矩形检测框从所述待检测图像中分割出所述第一部件的图像。The image of the first component is segmented from the to-be-detected image according to the first rectangular detection frame.
结合第一方面,在一种可能的实施方式中,所述基于所述第一特征图对所述第一部件进行定位和分类,得到所述第一部件的第一矩形检测框,包括:With reference to the first aspect, in a possible implementation manner, the positioning and classification of the first component based on the first feature map to obtain a first rectangular detection frame of the first component includes:
在所述第一特征图上进行所述第一部件的候选区域坐标预测和前、背景分类,确定出所述第一部件的前景目标;Performing coordinate prediction and front and background classification of the candidate region of the first component on the first feature map, and determining the foreground target of the first component;
将所述第一部件的前景目标在所述第一特征图中对应的特征进行池化处理,得到第一池化特征;performing pooling processing on the corresponding features of the foreground target of the first component in the first feature map to obtain a first pooling feature;
基于所述第一池化特征对所述第一部件进行分类,得到所述第一部件的类别和所述第一矩形检测框。The first part is classified based on the first pooling feature to obtain the category of the first part and the first rectangular detection frame.
结合第一方面,在一种可能的实施方式中,所述从所述第一部件的图像中分割出高铁接触网的第二部件的图像,包括:With reference to the first aspect, in a possible implementation manner, the segmentation of the image of the second component of the high-speed rail contact net from the image of the first component includes:
对所述第一部件的图像进行伽马校验,得到所述第一部件的待分割图像;Perform gamma check on the image of the first part to obtain the image to be divided of the first part;
从所述第一部件的待分割图像中分割出所述第二部件的图像。The image of the second part is segmented from the image to be segmented of the first part.
结合第一方面,在一种可能的实施方式中,所述从所述第一部件的待分割图像中分割 出所述第二部件的图像,包括:In conjunction with the first aspect, in a possible implementation manner, segmenting the image of the second component from the to-be-segmented image of the first component includes:
对所述第一部件的待分割图像进行特征提取,得到第二特征图;performing feature extraction on the to-be-segmented image of the first component to obtain a second feature map;
基于所述第二特征图对所述第二部件进行定位和分类,得到所述第二部件的第二矩形检测框;Locating and classifying the second component based on the second feature map, to obtain a second rectangular detection frame of the second component;
按照所述第二矩形检测框从所述第一部件的待分割图像中分割出所述第二部件的图像。The image of the second part is segmented from the image to be segmented of the first part according to the second rectangular detection frame.
结合第一方面,在一种可能的实施方式中,所述基于所述第二特征图对所述第二部件进行定位和分类,得到所述第二部件的第二矩形检测框,包括:With reference to the first aspect, in a possible implementation manner, the positioning and classification of the second component based on the second feature map to obtain a second rectangular detection frame of the second component includes:
在所述第二特征图上进行所述第二部件的候选区域坐标预测和前、背景分类,确定出所述第二部件的前景目标;Performing coordinate prediction and front and background classification of the candidate region of the second component on the second feature map to determine the foreground target of the second component;
将所述第二部件的前景目标在所述第二特征图中对应的特征进行池化处理,得到第二池化特征;performing pooling processing on the corresponding features of the foreground target of the second component in the second feature map to obtain a second pooling feature;
基于所述第二池化特征对所述第二部件进行分类,得到所述第二部件的类别和所述第二矩形检测框。Classify the second part based on the second pooling feature to obtain the category of the second part and the second rectangular detection frame.
结合第一方面,在一种可能的实施方式中,所述基于所述第一部件的图像、所述第二部件的图像对所述第一部件、所述第二部件进行缺陷分类,得到缺陷分类结果,包括:With reference to the first aspect, in a possible implementation manner, the defect classification is performed on the first part and the second part based on the image of the first part and the image of the second part to obtain defects Classification results, including:
分别对所述第一部件的图像、所述第二部件的图像进行特征提取;Perform feature extraction on the image of the first component and the image of the second component respectively;
基于所述第一部件的图像提取出的特征对所述第一部件进行缺陷类型预测,得到所述第一部件的缺陷分类结果;所述第一部件的缺陷分类结果包括所述第一部件的缺陷类型和所述第一部件的类别;以及,Predict the defect type of the first part based on the features extracted from the image of the first part, and obtain the defect classification result of the first part; the defect classification result of the first part includes the defect classification result of the first part. the type of defect and the category of the first part; and,
基于所述第二部件的图像提取出的特征对所述第二部件进行缺陷类型预测,得到所述第二部件的缺陷分类结果;所述第二部件的缺陷分类结果包括所述第二部件的缺陷类型和所述第二部件的类别。Predict the defect type of the second part based on the features extracted from the image of the second part, and obtain the defect classification result of the second part; the defect classification result of the second part includes the defect classification result of the second part. Defect type and category of the second part.
结合第一方面,在一种可能的实施方式中,在基于所述第一部件的图像、所述第二部件的图像对所述第一部件、所述第二部件进行缺陷分类,得到缺陷分类结果之后,所述方法还包括:In combination with the first aspect, in a possible implementation manner, defect classification is performed on the first part and the second part based on the image of the first part and the image of the second part to obtain a defect classification After the result, the method further includes:
根据所述缺陷分类结果进行缺陷预警。Defect early warning is performed according to the defect classification result.
结合第一方面,在一种可能的实施方式中,所述根据所述缺陷分类结果进行缺陷预警,包括:With reference to the first aspect, in a possible implementation, the performing a defect early warning according to the defect classification result includes:
针对所述第一部件,输出所述第一部件的缺陷类型、所述第一部件在所述待检测图像中的位置、所述第一部件的类别和所述第一部件所在的高铁线路,以进行缺陷预警;For the first part, output the defect type of the first part, the position of the first part in the image to be inspected, the type of the first part, and the high-speed rail line where the first part is located, for defect warning;
针对所述第二部件,确定所述第二部件所属的目标第一部件,输出所述第二部件的缺陷类型、所述第二部件在所述待检测图像中的位置、所述第二部件的类别、所述目标第一部件在所述待检测图像中的位置、所述目标第一部件的类别和所述第二部件所在的高铁线路,以进行缺陷预警。For the second part, determine the target first part to which the second part belongs, and output the defect type of the second part, the position of the second part in the image to be inspected, the second part The category of the target first part, the position of the target first part in the to-be-detected image, the category of the target first part and the high-speed rail line where the second part is located, so as to carry out defect early warning.
结合第一方面,在一种可能的实施方式中,所述获取高铁接触网的待检测图像,包括:With reference to the first aspect, in a possible implementation manner, the obtaining of the image to be detected of the high-speed rail catenary includes:
获取成像设备采集的高铁接触网原始图像;Obtain the original image of the high-speed rail catenary collected by the imaging equipment;
对所述高铁接触网原始图像进行过滤,得到所述待检测图像。The original image of the high-speed rail catenary is filtered to obtain the to-be-detected image.
本申请实施例第二方面提供了一种缺陷检测装置,该装置包括:A second aspect of the embodiments of the present application provides a defect detection device, the device comprising:
图像获取模块,用于获取高铁接触网的待检测图像;The image acquisition module is used to acquire the image to be detected of the high-speed rail catenary;
第一检测模块,用于从所述待检测图像中分割出高铁接触网的第一部件的图像;a first detection module, used for segmenting the image of the first component of the high-speed rail catenary from the to-be-detected image;
第二检测模块,用于从所述第一部件的图像中分割出高铁接触网的第二部件的图像;所述第二部件为所述第一部件的子部件;a second detection module, configured to segment the image of the second part of the high-speed rail contact net from the image of the first part; the second part is a sub-component of the first part;
缺陷分类模块,用于基于所述第一部件的图像、所述第二部件的图像对所述第一部件、 所述第二部件进行缺陷分类,得到缺陷分类结果。The defect classification module is configured to perform defect classification on the first part and the second part based on the image of the first part and the image of the second part, and obtain a defect classification result.
本申请实施例第三方面提供了一种电子设备,该电子设备包括输入设备和输出设备,还包括处理器,适于实现一条或多条指令;以及,计算机存储介质,所述计算机存储介质存储有一条或多条指令,所述一条或多条指令适于由所述处理器加载并执行上述第一方面任一种实施方式中的步骤。A third aspect of the embodiments of the present application provides an electronic device, the electronic device includes an input device and an output device, and further includes a processor, adapted to implement one or more instructions; and, a computer storage medium, the computer storage medium storing There is one or more instructions adapted to be loaded by the processor and to perform the steps in any of the embodiments of the first aspect above.
本申请实施例第四方面提供了一种计算机存储介质,所述计算机存储介质存储有一条或多条指令,所述一条或多条指令适于由处理器加载并执行上述第一方面任一种实施方式中的步骤。A fourth aspect of the embodiments of the present application provides a computer storage medium, where the computer storage medium stores one or more instructions, and the one or more instructions are suitable for being loaded by a processor and executing any one of the foregoing first aspects steps in the implementation.
可以看出,本申请实施例通过获取高铁接触网的待检测图像;从所述待检测图像中分割出高铁接触网的第一部件的图像;从所述第一部件的图像中分割出高铁接触网的第二部件的图像;所述第二部件为所述第一部件的子部件;基于所述第一部件的图像、所述第二部件的图像对所述第一部件、所述第二部件进行缺陷分类,得到缺陷分类结果。这样对高铁接触网的待检测图像进行一级部件(即第一部件)的检测,从高铁接触网的待检测图像中分割出一级部件的图像,再对一级部件的图像进行二级部件(即第二部件)的检测,以识别出一级部件上的二级部件,分割出二级部件的图像,利用一级部件的图像和二级部件的图像进行缺陷分类,实现了级联式的高铁接触网缺陷检测,从而有利于降低高铁接触网缺陷检测的漏检率,进而提高接触网缺陷检测的准确性。同时,还有利于降低人工巡检成本、缩短检测时间、提高检测效率。It can be seen that the embodiment of the present application obtains the image to be detected of the high-speed rail catenary; the image of the first part of the high-speed rail catenary is segmented from the to-be-detected image; the high-speed rail contact is segmented from the image of the first part an image of a second part of the web; the second part is a sub-part of the first part; based on the image of the first part, the image of the second part Parts are classified into defects, and the result of defect classification is obtained. In this way, the first-level component (ie, the first component) is detected on the image to be detected of the high-speed rail contact net, the image of the first-level component is segmented from the to-be-detected image of the high-speed railway contact line, and then the image of the first-level component is processed for the second-level component. (that is, the second part), to identify the secondary part on the primary part, segment the image of the secondary part, use the image of the primary part and the image of the secondary part to classify the defects, and realize the cascade type The high-speed rail catenary defect detection is beneficial to reduce the missed detection rate of high-speed rail catenary defect detection, thereby improving the accuracy of catenary defect detection. At the same time, it is also beneficial to reduce the cost of manual inspection, shorten the detection time, and improve the detection efficiency.
附图说明Description of drawings
为了更清楚地说明本申请实施例或现有技术中的技术方案,下面将对实施例或现有技术描述中所需要使用的附图作简单地介绍,显而易见地,下面描述中的附图仅仅是本申请的一些实施例,对于本领域普通技术人员来讲,在不付出创造性劳动的前提下,还可以根据这些附图获得其他的附图。In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the accompanying drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only These are some embodiments of the present application. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings without any creative effort.
图1为本申请实施例提供的一种缺陷检测方法的流程示意图;1 is a schematic flowchart of a defect detection method provided by an embodiment of the present application;
图2为本申请实施例提供的一种高铁接触网缺陷检测的应用环境示意图;2 is a schematic diagram of an application environment for defect detection of a high-speed rail catenary provided by an embodiment of the present application;
图3为本申请实施例提供的一种对高铁接触网原始图像进行过滤的示意图;3 is a schematic diagram of filtering an original image of a high-speed rail catenary according to an embodiment of the present application;
图4为本申请实施例提供的一种高铁接触网缺陷检测模型的结构示意图;4 is a schematic structural diagram of a defect detection model for a high-speed rail catenary provided by an embodiment of the present application;
图5A为本申请实施例提供的一种分割第一部件的示意图;5A is a schematic diagram of dividing a first component according to an embodiment of the present application;
图5B为本申请实施例提供的一种分割第二部件的示意图;5B is a schematic diagram of dividing a second component according to an embodiment of the present application;
图6为本申请实施例提供的另一种缺陷检测方法的流程示意图;6 is a schematic flowchart of another defect detection method provided by an embodiment of the present application;
图7为申请实施例提供的一种基于特征图产生候选区域的示意图;7 is a schematic diagram of generating a candidate region based on a feature map according to an embodiment of the application;
图8为申请实施例提供的一种缺陷检测装置的结构示意图;FIG. 8 is a schematic structural diagram of a defect detection device provided by an embodiment of the application;
图9为申请实施例提供的另一种缺陷检测装置的结构示意图;FIG. 9 is a schematic structural diagram of another defect detection device provided by an embodiment of the application;
图10为本申请实施例提供的一种电子设备的结构示意图。FIG. 10 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
具体实施方式detailed description
为了使本技术领域的人员更好地理解本申请方案,下面将结合本申请实施例中的附图,对本申请实施例中的技术方案进行清楚、完整地描述,显然,所描述的实施例仅仅是本申请一部分的实施例,而不是全部的实施例。基于本申请中的实施例,本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例,都应当属于本申请保护的范围。In order to make those skilled in the art better understand the solutions of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only The embodiments are part of the present application, but not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the scope of protection of the present application.
本申请说明书、权利要求书和附图中出现的术语“包括”和“具有”以及它们任何变形,意图在于覆盖不排他的包含。例如包含了一系列步骤或单元的过程、方法、***、产品或设备没有限定于已列出的步骤或单元,而是可选地还包括没有列出的步骤或单元,或可选地还包括对于这些过程、方法、产品或设备固有的其它步骤或单元。此外,术语“第 一”、“第二”和“第三”等是用于区别不同的对象,而并非用于描述特定的顺序。The appearances of the terms "comprising" and "having" and any variations thereof in the specification, claims and drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes unlisted steps or units, or optionally also includes For other steps or units inherent to these processes, methods, products or devices. In addition, the terms "first", "second", "third", etc. are used to distinguish different objects, and not to describe a specific order.
本申请实施例提出一种高铁接触网的缺陷检测方案,以降低高铁接触网缺陷检测的漏检率,提高缺陷检测的准确性。在具体实施中,采用了基于深度学习的高铁接触网缺陷检测模型,先从高铁接触网待检测图像中定位出一级部件,再从一级部件的图像中定位出与一级部件具有级联关系的二级部件,有利于降低部件的漏检率,利用一级部件的图像、二级部件的图像进行一级部件和二级部件的缺陷类型预测,最后缺陷预警时能够输出缺陷的具***置、缺陷类型、缺陷所属的部件、缺陷所属部件的上级部件以及缺陷所在的具体线路段等,整个检测流程呈树结构,部件之间的结构化信息有利于运营维护人员快速确定缺陷位置、缺陷路线,从而展开接触网的维护工作,以保障高铁的安全运行,同时,有利于新部件、新缺陷的扩展。The embodiment of the present application proposes a defect detection scheme for a high-speed rail catenary, so as to reduce the missed detection rate of defect detection of the high-speed rail catenary and improve the accuracy of defect detection. In the specific implementation, a high-speed rail catenary defect detection model based on deep learning is adopted. First, the first-level components are located from the images of the high-speed rail catenary to be inspected, and then the first-level components are located from the images of the first-level components. The second-level components of the relationship are beneficial to reduce the missed detection rate of the components. The images of the first-level components and the images of the second-level components are used to predict the defect types of the first-level components and the second-level components, and the specific location of the defect can be output in the final defect warning. , defect type, the component to which the defect belongs, the superior component of the component to which the defect belongs, and the specific line segment where the defect is located. , so as to carry out the maintenance of the catenary to ensure the safe operation of the high-speed rail, and at the same time, it is conducive to the expansion of new components and new defects.
以下结合相关附图对本申请实施例提供的缺陷检测方法进行详细阐述。The defect detection method provided by the embodiments of the present application will be described in detail below with reference to the relevant drawings.
请参见图1,图1为本申请实施例提供的一种缺陷检测方法的流程示意图,该缺陷检测方法应用于服务器,例如:部署有基于深度学习的高铁接触网缺陷检测模型的服务器、电脑主机、云服务器等,如图1所示,包括步骤S11-S14:Please refer to FIG. 1. FIG. 1 is a schematic flowchart of a defect detection method provided by an embodiment of the application. The defect detection method is applied to a server, such as a server and a computer host where a deep learning-based high-speed rail catenary defect detection model is deployed. , cloud server, etc., as shown in Figure 1, including steps S11-S14:
S11,获取高铁接触网的待检测图像。S11, acquiring an image to be detected of the high-speed rail catenary.
本申请具体实施例中,如图2所示,高铁巡检车通常在夜晚作业,巡检车上安装有高清成像设备和车载传感器,巡检车在高铁线路上行进,每当车载传感器检测到路线两侧的支柱时,便触发成像设备对接触网进行图像采集,得到高铁接触网原始图像,例如:巡检车上通常包括车头和车尾两组成像设备,当车载传感器检测到预设范围内有支柱时,便触发两组成像设备对接触网的支持部件、悬挂部件等进行正反面及整体布局的成像,由此得到大批量来自不同角度的高铁接触网原始图像。高铁接触网原始图像的分辨率通常有一个较佳的值,例如:6576*4384像素,但由于夜间作业、雾气弥漫等环境因素,导致采集的高铁接触网原始图像中仍然会存在分辨率较低的图像,例如:分辨率长宽低于2000像素,因此,如图3所示,需要对采集的高铁接触网原始图像进行过滤,从中筛选出分辨率长宽达到预设像素值的高铁接触网原始图像作为后续缺陷检测的待检测图像,过滤掉分辨率长宽低于预设像素值的高铁接触网原始图像。In the specific embodiment of the present application, as shown in FIG. 2 , the high-speed rail inspection vehicle usually operates at night. The inspection vehicle is equipped with high-definition imaging equipment and on-board sensors. The inspection vehicle travels on the high-speed rail line. Whenever the on-board sensor detects When there are pillars on both sides of the route, the imaging device is triggered to collect images of the catenary, and the original image of the high-speed rail catenary is obtained. When there are pillars inside, two sets of imaging equipment are triggered to image the front and back and overall layout of the support parts and suspension parts of the catenary, thus obtaining a large number of original images of the high-speed rail catenary from different angles. The resolution of the original image of the high-speed rail catenary usually has a better value, for example: 6576*4384 pixels, but due to environmental factors such as night operations and foggy, the collected original images of the high-speed rail catenary still have low resolution. For example, the resolution length and width are less than 2000 pixels. Therefore, as shown in Figure 3, it is necessary to filter the collected original images of the high-speed rail catenary, and filter out the high-speed rail catenary whose resolution length and width reach the preset pixel value. The original image is used as the image to be detected for subsequent defect detection, and the original image of the high-speed rail catenary whose resolution length and width are lower than the preset pixel value is filtered out.
S12,从所述待检测图像中分割出高铁接触网的第一部件的图像。S12, segment the image of the first component of the high-speed rail catenary from the to-be-detected image.
本申请具体实施例中,采用预训练的基于深度学习的高铁接触网缺陷检测模型对步骤S11中获取的待检测图像中的各部件进行缺陷检测,该高铁接触网缺陷检测模型包括第一部件检测器、第二部件检测器、缺陷分类器和缺陷预警模块,如图4所示,第一部件检测器的输入为待检测图像,用于从待检测图像中检测出高铁接触网的第一部件,例如:柱顶盖板、绝缘子、环杆-直角挂板关节、臂腕底座、AF线肩架底座、接触线中心锚结线夹、坠陀限制架等等,第二部件检测器用于从第一部件检测器输出的第一部件的图像中检测出第一部件上的第二部件,例如:绝缘子两端的螺栓、螺母、开口销等,其输出为第二部件的图像,缺陷分类器用于根据第一部件的图像对第一部件进行缺陷分类、根据第二部件的图像对第二部件进行缺陷分类,缺陷预警模块用于根据缺陷分类器输出的缺陷分类结果进行缺陷预警,其输出包括缺陷所在的位置、缺陷类型(如臂腕底座上开口销角度不到位)、缺陷所属部件的上级部件、缺陷所处的高铁线路等。可选的,第一部件检测器可以是two-stage检测器,也可以是one-stage检测器,two-stage的检测器基于从待检测图像提取出的特征图产生候选区域,然后对候选区域进行分类预测,得到第一部件的类别和矩形检测框坐标,该矩形检测框坐标可以是左上角和右下角的坐标,也可以是中心点坐标和长宽等,具体不作限定,如图5A所示,按照矩形检测框从待检测图像中分割出第一部件的图像,如绝缘子、臂腕底座等。one-stage检测器,不需要产生候选区域,其针对输入的待检测图像直接进行分类预测,得到第一部件的类别和矩形检测框坐标,然后按照该矩形检测框分 割出第一部件的图像。可选的,第一部件检测器采用高铁接触网的样本图像进行训练,该样本图像中的第一部件带有类别标签,在训练过程中通过预设损失函数对该第一部件检测器进行调优。In the specific embodiment of the present application, a pre-trained deep learning-based high-speed rail catenary defect detection model is used to perform defect detection on each component in the to-be-detected image obtained in step S11, and the high-speed rail catenary defect detection model includes a first component detection model. As shown in Figure 4, the input of the first part detector is the image to be detected, which is used to detect the first part of the high-speed rail catenary from the image to be detected , such as: column top cover plate, insulator, ring rod-right angle hanging plate joint, arm wrist base, AF wire shoulder frame base, contact wire center anchor clamp, drop weight limit frame, etc., the second part detector is used to The second part on the first part is detected in the image of the first part output by the first part detector, such as bolts, nuts, cotter pins, etc. at both ends of the insulator, and the output is the image of the second part, and the defect classifier is used for The first part is classified according to the image of the first part, and the second part is classified according to the image of the second part; The location, the type of defect (such as the angle of the cotter pin on the arm base is not in place), the superior component of the component to which the defect belongs, the high-speed rail line where the defect is located, etc. Optionally, the first component detector may be a two-stage detector or a one-stage detector. The two-stage detector generates candidate regions based on the feature maps extracted from the images to be detected, and then analyzes the candidate regions. Perform classification prediction to obtain the category of the first component and the coordinates of the rectangular detection frame. The coordinates of the rectangular detection frame can be the coordinates of the upper left corner and the lower right corner, or the coordinates of the center point and length and width, etc., which are not limited in detail, as shown in Figure 5A As shown, the image of the first component, such as the insulator, the arm-wrist base, etc., is segmented from the image to be detected according to the rectangular detection frame. The one-stage detector does not need to generate a candidate area, it directly performs classification prediction for the input image to be detected, obtains the category of the first part and the coordinates of the rectangular detection frame, and then divides the image of the first part according to the rectangular detection frame. Optionally, the first component detector is trained using a sample image of the high-speed rail catenary, the first component in the sample image has a class label, and the first component detector is adjusted by a preset loss function during the training process. excellent.
S13,从所述第一部件的图像中分割出高铁接触网的第二部件的图像;所述第二部件为所述第一部件的子部件。S13, segment the image of the second part of the high-speed rail contact net from the image of the first part; the second part is a sub-part of the first part.
本申请具体实施例中,第二部件是指第一部件上的子部件,二者具有级联关系,由于二级部件具有像素占比小的特点,在光线不佳的环境下,如果直接对步骤S12中分割出的第一部件的图像进行检测,出现漏检的可能性较高,因此,需要对第一部件的图像进行伽马校验,以提高图像质量,得到第一部件的待分割图像(即伽马校验后得到的图像),然后通过第二部件检测器从待分割图像中分割出高铁接触网的第二部件的图像。可选的,该第二部件检测器可以与第一部件检测器相同,也可以不同,其可以与第一部件检测器一起训练,也可以单独训练,同理,得到第二部件的类别和矩形检测框后,如图5B所示,按照该矩形检测框从第一部件的图像中分割出第二部件的图像。In the specific embodiment of this application, the second component refers to the sub-component on the first component, and the two are in a cascade relationship. The image of the first part segmented in step S12 is detected, and the possibility of missed detection is high. Therefore, it is necessary to perform gamma check on the image of the first part to improve the image quality and obtain the to-be-segmented image of the first part. image (that is, the image obtained after gamma verification), and then segment the image of the second component of the high-speed rail catenary from the image to be segmented by the second component detector. Optionally, the second part detector can be the same as the first part detector, or it can be different, it can be trained together with the first part detector, or can be trained separately, in the same way, the category and rectangle of the second part can be obtained. After the frame is detected, as shown in FIG. 5B , the image of the second part is segmented from the image of the first part according to the rectangular detection frame.
S14,基于所述第一部件的图像、所述第二部件的图像对所述第一部件、所述第二部件进行缺陷分类,得到缺陷分类结果。S14: Perform defect classification on the first part and the second part based on the image of the first part and the image of the second part, to obtain a defect classification result.
本申请具体实施例中,可选的,上述基于所述第一部件的图像、所述第二部件的图像对所述第一部件、所述第二部件进行缺陷分类,得到缺陷分类结果,包括:In a specific embodiment of the present application, optionally, the above-mentioned performing defect classification on the first part and the second part based on the image of the first part and the image of the second part, to obtain a defect classification result, including :
分别对所述第一部件的图像、所述第二部件的图像进行特征提取;Perform feature extraction on the image of the first component and the image of the second component respectively;
基于所述第一部件的图像提取出的特征对所述第一部件进行缺陷类型预测,得到所述第一部件的缺陷分类结果;所述第一部件的缺陷分类结果包括所述第一部件的缺陷类型和所述第一部件的类别;以及,Predict the defect type of the first part based on the features extracted from the image of the first part, and obtain the defect classification result of the first part; the defect classification result of the first part includes the defect classification result of the first part. the type of defect and the category of the first part; and,
基于所述第二部件的图像提取出的特征对所述第二部件进行缺陷类型预测,得到所述第而部件的缺陷分类结果;所述第二部件的缺陷分类结果包括所述第二部件的缺陷类型和所述第二部件的类别。Predict the defect type of the second part based on the features extracted from the image of the second part, and obtain the defect classification result of the second part; the defect classification result of the second part includes the defect classification result of the second part. Defect type and category of the second part.
请继续参见图4,得到第一部件的图像和第二部件的图像后,将其输入缺陷分类器进行缺陷类型的概率预测。具体的,通过缺陷分类器的骨干网络提取出第一部件的图像的特征和第二部件的图像的特征,骨干网络主要进行卷积处理,然后基于提取出的特征,将该特征输入全连接层进行缺陷类型的概率预测,取概率最高的缺陷类型作为该部件的缺陷类型,例如:当前输入第一部件坠陀限制架的特征,全连接层通过分类处理,预测出该坠陀限制架存在裂纹的概率达到95%(最高),则该坠陀限制架的缺陷类型即为存在裂纹。当然,缺陷分类器最终的输出除了部件的缺陷类型外,还有部件的类别索引和矩形检测框坐标,例如:c05,存在裂纹,c05即表示部件的类别索引,其中,第一部件的类别索引可在第一部件检测器得到第一部件的类别时确定,第二部件的类别索引可在第二部件检测器得到第二部件的类别时确定。其中,缺陷分类器可与第一部件检测器一起训练,也可与第二部件检测器一起训练,或者单独训练。Please continue to refer to Figure 4. After obtaining the image of the first part and the image of the second part, they are input into the defect classifier for probability prediction of defect types. Specifically, the feature of the image of the first part and the feature of the image of the second part are extracted through the backbone network of the defect classifier. The backbone network mainly performs convolution processing, and then based on the extracted features, the features are input into the fully connected layer Predict the probability of defect types, and take the defect type with the highest probability as the defect type of the part. For example, the characteristics of the first part of the pendulum limit frame are currently input, and the fully connected layer is classified to predict the existence of cracks in the pendulum head restraint frame. If the probability reaches 95% (the highest), the defect type of the pendulum limit frame is the existence of cracks. Of course, in addition to the defect type of the part, the final output of the defect classifier also has the class index of the part and the coordinates of the rectangular detection frame, for example: c05, if there is a crack, c05 means the class index of the part, among which, the class index of the first part The class index of the second part can be determined when the class of the first part is obtained by the first part detector, and the class index of the second part can be determined when the class of the second part is obtained by the second part detector. Wherein, the defect classifier can be trained together with the first part detector, can also be trained together with the second part detector, or can be trained separately.
可选的,在基于所述第一部件的图像、所述第二部件的图像对所述第一部件、所述第二部件进行缺陷分类,得到缺陷分类结果之后,所述方法还包括:根据所述缺陷分类结果进行缺陷预警。Optionally, after the defect classification is performed on the first part and the second part based on the image of the first part and the image of the second part, and a defect classification result is obtained, the method further includes: according to The defect classification result is used for defect early warning.
本申请具体实施例中,所述根据所述缺陷分类结果进行缺陷预警,包括:In a specific embodiment of the present application, the performing defect early warning according to the defect classification result includes:
针对所述第一部件,输出所述第一部件的缺陷类型、所述第一部件在所述待检测图像中的位置、所述第一部件的类别和所述第一部件所在的高铁线路,以进行缺陷预警;For the first part, output the defect type of the first part, the position of the first part in the image to be inspected, the type of the first part, and the high-speed rail line where the first part is located, for defect warning;
针对所述第二部件,确定所述第二部件所属的目标第一部件,输出所述第二部件的缺陷类型、所述第二部件在所述待检测图像中的位置、所述第二部件的类别、所述目标第一部件在所述待检测图像中的位置、所述目标第一部件的类别和所述第二部件所在的高铁线 路,以进行缺陷预警。For the second part, determine the target first part to which the second part belongs, and output the defect type of the second part, the position of the second part in the image to be inspected, the second part The category of the target first part, the position of the target first part in the to-be-detected image, the category of the target first part and the high-speed rail line where the second part is located, so as to carry out defect early warning.
具体的,目标第一部件是指第二部件的上级部件,缺陷预警模块的输入包括第一部件和第二部件的缺陷分类结果、第一部件和第二部件的矩形检测框坐标,可输出部件的缺陷类型、部件在待检测图像中的位置、部件的类别、部件所属上级部件在待检测图像中的位置、部件的上级部件的类别以及部件所在的高铁线路。可选的,部件在待检测图像中的位置以及部件的上级部件在待检测图像中的位置均以部件的矩形检测框呈现,部件的类别以及部件的上级部件的类别均以类别索引呈现,部件所在的高铁线路可根据成像设备上传高铁接触网原始图像时携带的标识进行输出,例如:某张高铁接触网原始图像携带有AB1002(A市到B市的第1002段线路)的标识,该标识会存在于整个缺陷检测过程中,当然,此处仅仅为一种举例说明,并不对本申请实施例造成限定。Specifically, the target first part refers to the upper-level part of the second part. The input of the defect warning module includes the defect classification results of the first part and the second part, the coordinates of the rectangular detection frame of the first part and the second part, and the output part The type of defect, the location of the component in the image to be inspected, the category of the component, the location of the superior component to which the component belongs in the image to be inspected, the category of the superior component of the component, and the high-speed rail line where the component is located. Optionally, the position of the component in the image to be detected and the position of the upper-level component of the component in the image to be detected are presented in the rectangular detection frame of the component, the category of the component and the category of the parent component of the component are presented with the category index, the component The high-speed rail line you are in can output according to the logo carried when the imaging device uploads the original image of the high-speed railway catenary. It will exist in the entire defect detection process. Of course, this is only an example, and does not limit the embodiments of the present application.
另外,由于第一部件并没有所属的上级部件,因此第一部件的输出的预警信息包括第一部件的缺陷类型(如AF线肩架底座上螺母脱落)、第一部件在待检测图像中的位置(如AF线肩架底座的矩形检测框)、第一部件的类别和第一部件所在的高铁线路。而第二部件通常存上级部件,因此,需要确定第二部件所属的目标第一部件,在输出第二部件本身的缺陷类型、第二部件在待检测图像中的位置等信息,还需要输出该目标第一部件的类别、在待检测图像中的位置等。In addition, since the first part does not have a parent part to which it belongs, the early warning information output by the first part includes the defect type of the first part (for example, the nut on the AF wire shoulder frame has fallen off), the defect of the first part in the image to be detected. Position (such as the rectangular detection frame of the AF line shoulder frame base), the type of the first part, and the high-speed rail line where the first part is located. The second part usually stores the upper-level part. Therefore, it is necessary to determine the target first part to which the second part belongs. When outputting information such as the defect type of the second part itself, the position of the second part in the image to be inspected, etc., it is also necessary to output the The category of the first part of the target, the position in the image to be detected, etc.
进一步的,所述确定所述第二部件所属的目标第一部件,包括:Further, the determining of the target first component to which the second component belongs includes:
获取所述第二部件的第二矩形检测框与所有所述第一部件的第一矩形检测框的交集;obtaining the intersection of the second rectangular detection frame of the second component and the first rectangular detection frame of all the first components;
获取所述交集与所述第二矩形检测框之间的比值;obtaining the ratio between the intersection and the second rectangular detection frame;
根据所述第二矩形检测框之间的比值确定所述第二部件所属的目标第一部件。The target first part to which the second part belongs is determined according to the ratio between the second rectangular detection frames.
具体的,第一矩形检测框是指第一部件的边界框回归,第二矩形检测框是指第二部件的边界框回归,针对当前检测的定位钩(属于第二部件),假设其第二矩形检测框为A,所有第一部件的第一矩形检测框为B1,B2,B3…Bn,先获取该第二矩形检测框A与所有第一矩形检测框B1,B2,B3…Bn的交集C1,C2,C3…Cn,再分别计算交集C1,C2,C3…Cn与第二矩形检测框A的比值C1/A,C2/A,C3/A…Cn/A,若C1/A为最大值,则将第一矩形检测框B1所对应的第一部件作为该定位钩所属的目标第一部件。该实施方式中,由于缺陷预警模块输入的是第一部件和第二部件的缺陷分类结果、第一部件和第二部件的矩形检测框坐标,因此,可根据第二部件与第一部件的矩形检测框的交集同第二部件的矩形检测框的比值来确定第二部件的上级部件,对整体检测耗时并未造成影响,且准确度满足需求。Specifically, the first rectangular detection frame refers to the bounding box regression of the first component, and the second rectangular detection frame refers to the bounding box regression of the second component. For the currently detected positioning hook (belonging to the second component), it is assumed that its second The rectangular detection frame is A, and the first rectangular detection frames of all the first components are B1, B2, B3...Bn, and the intersection of the second rectangular detection frame A and all the first rectangular detection frames B1, B2, B3...Bn is obtained first. C1, C2, C3...Cn, and then calculate the ratios C1/A, C2/A, C3/A...Cn/A of the intersection C1, C2, C3...Cn and the second rectangular detection frame A respectively, if C1/A is the largest value, the first component corresponding to the first rectangular detection frame B1 is used as the target first component to which the positioning hook belongs. In this embodiment, since the defect early warning module inputs the defect classification results of the first part and the second part, and the coordinates of the rectangular detection frame of the first part and the second part, the rectangular detection frame coordinates of the first part and the second part can be The ratio of the intersection of the detection frame and the rectangular detection frame of the second component is used to determine the upper-level component of the second component, which does not affect the overall detection time, and the accuracy meets the requirements.
本申请实施例通过获取高铁接触网的待检测图像;从所述待检测图像中分割出高铁接触网的第一部件的图像;从所述第一部件的图像中分割出高铁接触网的第二部件的图像;所述第二部件为所述第一部件的子部件;基于所述第一部件的图像、所述第二部件的图像对所述第一部件、所述第二部件进行缺陷分类,得到缺陷分类结果。这样对高铁接触网的待检测图像进行一级部件的检测,从高铁接触网的待检测图像中分割出一级部件的图像,再对一级部件的图像进行二级部件的检测,以识别出一级部件上的二级部件,分割出二级部件的图像,利用一级部件的图像和二级部件的图像进行缺陷分类,实现了级联式的高铁接触网缺陷检测,从而有利于降低高铁接触网缺陷检测的漏检率,进而提高接触网缺陷检测的准确性。同时,还有利于降低人工巡检成本、缩短检测时间、提高检测效率。In the embodiment of the present application, the image to be detected of the high-speed rail catenary is obtained; the image of the first part of the high-speed rail catenary is segmented from the to-be-detected image; the second part of the high-speed rail catenary is segmented from the image of the first part an image of a part; the second part is a sub-part of the first part; the first part and the second part are classified as defective based on the image of the first part and the image of the second part , get the defect classification result. In this way, the first-level component is detected on the image to be detected of the high-speed rail contact line, the image of the first-level component is segmented from the to-be-detected image of the high-speed rail contact line, and then the second level component is detected on the image of the first-level component to identify The secondary part on the primary part, the image of the secondary part is segmented, and the image of the primary part and the image of the secondary part are used for defect classification, which realizes the cascading high-speed rail catenary defect detection, which is beneficial to reduce the high-speed rail. The missed detection rate of catenary defect detection, thereby improving the accuracy of catenary defect detection. At the same time, it is also beneficial to reduce the cost of manual inspection, shorten the detection time, and improve the detection efficiency.
请参见图6,图6为本申请实施例提供的另一种缺陷检测方法的流程示意图,如图6所示,包括步骤S61-S67:Please refer to FIG. 6, which is a schematic flowchart of another defect detection method provided by an embodiment of the present application, as shown in FIG. 6, including steps S61-S67:
S61,获取高铁接触网的待检测图像;S61, acquiring an image to be detected of the high-speed rail catenary;
S62,对所述待检测图像进行特征提取,得到第一特征图;S62, perform feature extraction on the to-be-detected image to obtain a first feature map;
S63,基于所述第一特征图对高铁接触网的第一部件进行定位和分类,得到所述第一部件的第一矩形检测框;S63, locating and classifying the first component of the high-speed rail catenary based on the first feature map, to obtain a first rectangular detection frame of the first component;
S64,按照所述第一矩形检测框从所述待检测图像中分割出所述第一部件的图像;S64, segment the image of the first component from the to-be-detected image according to the first rectangular detection frame;
S65,从所述第一部件的图像中分割出高铁接触网的第二部件的图像;所述第二部件为所述第一部件的子部件;S65, segment the image of the second component of the high-speed rail contact network from the image of the first component; the second component is a subcomponent of the first component;
S66,基于所述第一部件的图像、所述第二部件的图像对所述第一部件、所述第二部件进行缺陷分类,得到缺陷分类结果;S66, perform defect classification on the first part and the second part based on the image of the first part and the image of the second part, to obtain a defect classification result;
S67,根据所述缺陷分类结果进行缺陷预警。S67, perform a defect early warning according to the defect classification result.
本申请具体实施例中,针对输入第一部件检测器的待检测图像,通过第一部件检测器的骨干网络对其进行特征提取,骨干网络主要进行卷积处理,其输出的特征图即上述的第一特征图,在该第一特征图上进行第一部件的候选区域坐标预测,产生如图7所示的候选区域,在该候选区域内进行前、背景分类,得到第一部件的前景目标,然后将该前景目标在第一特征图中对应的特征进行池化处理,其输出的特征即上述的第一池化特征,将该第一池化特征输入全连接层进行最终的分类,输出待检测图像中第一部件的类别和矩形检测框(即第一矩形检测框),按照第一矩形检测框从待检测图像中分割出第一部件的图像,作为缺陷分类器的输入。该实施方式中,采用基于候选区域的第一部件检测器对待检测图像中的第一部件进行分类,精度更高。In the specific embodiment of the present application, for the image to be detected that is input to the first component detector, feature extraction is performed on it through the backbone network of the first component detector. The backbone network mainly performs convolution processing, and the output feature map is the above-mentioned feature map. The first feature map, the coordinates of the candidate region of the first component are predicted on the first feature map, and the candidate region shown in Figure 7 is generated, and the front and background classification is performed in the candidate region to obtain the foreground target of the first component. , and then the corresponding features of the foreground target in the first feature map are pooled, and the output features are the above-mentioned first pooling features, the first pooling features are input into the fully connected layer for final classification, and the output The category of the first part in the image to be inspected and the rectangular inspection frame (ie, the first rectangular inspection frame), and the image of the first part is segmented from the image to be inspected according to the first rectangular inspection frame, as the input of the defect classifier. In this embodiment, the first part detector based on the candidate region is used to classify the first part in the image to be detected, and the accuracy is higher.
可选的,所述从所述第一部件的图像中分割出高铁接触网的第二部件的图像,包括:Optionally, the segmentation of the image of the second component of the high-speed rail catenary from the image of the first component includes:
对所述第一部件的图像进行伽马校验,得到所述第一部件的待分割图像;Perform gamma check on the image of the first part to obtain the image to be divided of the first part;
从所述第一部件的待分割图像中分割出所述第二部件的图像。The image of the second part is segmented from the image to be segmented of the first part.
该实施方式中,对第一部件的图像进行伽马校验能够得到质量更好的待分割图像,有利于克服光线不佳给检测带来的影响,以准确分割出第二部件的图像,降低第二部件的漏检率。In this embodiment, performing gamma verification on the image of the first component can obtain an image to be segmented with better quality, which is beneficial to overcome the influence of poor light on detection, so as to accurately segment the image of the second component, reduce The miss rate of the second part.
可选的,所述从所述第一部件的待分割图像中分割出所述第二部件的图像,包括:Optionally, segmenting the image of the second part from the image to be segmented of the first part includes:
对所述第一部件的待分割图像进行特征提取,得到第二特征图;performing feature extraction on the to-be-segmented image of the first component to obtain a second feature map;
基于所述第二特征图对所述第二部件进行定位和分类,得到所述第二部件的第二矩形检测框;Locating and classifying the second component based on the second feature map, to obtain a second rectangular detection frame of the second component;
按照所述第二矩形检测框从所述第一部件的待分割图像中分割出所述第二部件的图像。The image of the second part is segmented from the image to be segmented of the first part according to the second rectangular detection frame.
可选的,所述基于所述第二特征图对所述第二部件进行定位和分类,得到所述第二部件的第二矩形检测框,包括:Optionally, the positioning and classification of the second component based on the second feature map to obtain a second rectangular detection frame of the second component includes:
在所述第二特征图上进行所述第二部件的候选区域坐标预测和前、背景分类,确定出所述第二部件的前景目标;Performing coordinate prediction and front and background classification of the candidate region of the second component on the second feature map to determine the foreground target of the second component;
将所述第二部件的前景目标在所述第二特征图中对应的特征进行池化处理,得到第二池化特征;performing pooling processing on the corresponding features of the foreground target of the second component in the second feature map to obtain a second pooling feature;
基于所述第二池化特征对所述第二部件进行分类,得到所述第二部件的类别和所述第二矩形检测框。Classify the second part based on the second pooling feature to obtain the category of the second part and the second rectangular detection frame.
本申请具体实施例中,第二特征图是指第二部件检测器通过骨干网络从第一部件的子图提取出的特征图,第二池化特征是指第二部件检测器对第二部件的前景目标在第二特征图中对应的特征进行池化处理后得到的特征。需要说明的是,第二部件检测器与第一部件检测器的处理过程相同,同样是基于产生的候选区域进行第二部件的类别预测,输出第二部件的类别和第二矩形检测框。In the specific embodiment of the present application, the second feature map refers to the feature map extracted by the second component detector from the sub-graph of the first component through the backbone network, and the second pooling feature refers to the second component detector's effect on the second component. The features obtained by pooling the corresponding features of the foreground target in the second feature map. It should be noted that the processing procedure of the second component detector is the same as that of the first component detector. It also predicts the category of the second component based on the generated candidate area, and outputs the category of the second component and the second rectangular detection frame.
其中,上述步骤S61-S67的具体实施方式,在图1所示的实施例中已有相关说明,且能达到相同或相似的有益效果,此处不再赘述。The specific implementations of the above steps S61 to S67 have been described in the embodiment shown in FIG. 1 , and can achieve the same or similar beneficial effects, and will not be repeated here.
基于图1或图6所示方法实施例的描述,本申请实施例还提供一种缺陷检测装置,请 参见图8,图8为本申请实施例提供的一种缺陷检测装置的结构示意图,如图8所示,该装置包括:Based on the description of the method embodiment shown in FIG. 1 or FIG. 6 , an embodiment of the present application further provides a defect detection apparatus. Please refer to FIG. 8 . FIG. 8 is a schematic structural diagram of a defect detection apparatus provided by an embodiment of the present application. As shown in Figure 8, the device includes:
图像获取模块81,用于获取高铁接触网的待检测图像;The image acquisition module 81 is used to acquire the to-be-detected image of the high-speed rail catenary;
第一检测模块82,用于从所述待检测图像中分割出高铁接触网的第一部件的图像;The first detection module 82 is used for segmenting the image of the first component of the high-speed rail catenary from the to-be-detected image;
第二检测模块83,用于从所述第一部件的图像中分割出高铁接触网的第二部件的图像;所述第二部件为所述第一部件的子部件;The second detection module 83 is configured to segment the image of the second part of the high-speed rail contact net from the image of the first part; the second part is a sub-component of the first part;
缺陷分类模块84,用于基于所述第一部件的图像、所述第二部件的图像对所述第一部件、所述第二部件进行缺陷分类,得到缺陷分类结果。The defect classification module 84 is configured to perform defect classification on the first part and the second part based on the image of the first part and the image of the second part to obtain a defect classification result.
在一种可能的实施方式中,在从所述待检测图像中分割出高铁接触网的第一部件的图像方面,第一检测模块82具体用于:In a possible implementation manner, in terms of segmenting the image of the first component of the high-speed rail catenary from the to-be-detected image, the first detection module 82 is specifically configured to:
对所述待检测图像进行特征提取,得到第一特征图;performing feature extraction on the to-be-detected image to obtain a first feature map;
基于所述第一特征图对所述第一部件进行定位和分类,得到所述第一部件的第一矩形检测框;Locating and classifying the first component based on the first feature map to obtain a first rectangular detection frame of the first component;
按照所述第一矩形检测框从所述待检测图像中分割出所述第一部件的图像。The image of the first component is segmented from the to-be-detected image according to the first rectangular detection frame.
在一种可能的实施方式中,在基于所述第一特征图对所述第一部件进行定位和分类,得到所述第一部件的第一矩形检测框方面,第一检测模块82具体用于:In a possible implementation manner, in terms of locating and classifying the first component based on the first feature map to obtain a first rectangular detection frame of the first component, the first detection module 82 is specifically configured to :
在所述第一特征图上进行所述第一部件的候选区域坐标预测和前、背景分类,确定出所述第一部件的前景目标;Performing coordinate prediction and front and background classification of the candidate region of the first component on the first feature map, and determining the foreground target of the first component;
将所述第一部件的前景目标在所述第一特征图中对应的特征进行池化处理,得到第一池化特征;performing pooling processing on the corresponding features of the foreground target of the first component in the first feature map to obtain a first pooling feature;
基于所述第一池化特征对所述第一部件进行分类,得到所述第一部件的类别和所述第一矩形检测框。The first part is classified based on the first pooling feature to obtain the category of the first part and the first rectangular detection frame.
在一种可能的实施方式中,在从所述第一部件的图像中分割出高铁接触网的第二部件的图像方面,第二检测模块83具体用于:In a possible implementation manner, in terms of segmenting the image of the second component of the high-speed rail catenary from the image of the first component, the second detection module 83 is specifically configured to:
对所述第一部件的图像进行伽马校验,得到所述第一部件的待分割图像;Perform gamma check on the image of the first part to obtain the image to be divided of the first part;
从所述第一部件的待分割图像中分割出所述第二部件的图像。The image of the second part is segmented from the image to be segmented of the first part.
在一种可能的实施方式中,在从所述第一部件的待分割图像中分割出所述第二部件的图像方面,第二检测模块83具体用于:In a possible implementation manner, in terms of segmenting the image of the second part from the image to be segmented of the first part, the second detection module 83 is specifically configured to:
对所述第一部件的待分割图像进行特征提取,得到第二特征图;performing feature extraction on the to-be-segmented image of the first component to obtain a second feature map;
基于所述第二特征图对所述第二部件进行定位和分类,得到所述第二部件的第二矩形检测框;Locating and classifying the second component based on the second feature map, to obtain a second rectangular detection frame of the second component;
按照所述第二矩形检测框从所述第一部件的待分割图像中分割出所述第二部件的图像。The image of the second part is segmented from the image to be segmented of the first part according to the second rectangular detection frame.
在一种可能的实施方式中,基于所述第二特征图对所述第二部件进行定位和分类,得到所述第二部件的第二矩形检测框方面,第二检测模块83具体用于:In a possible implementation manner, the second detection module 83 is specifically configured to:
在所述第二特征图上进行所述第二部件的候选区域坐标预测和前、背景分类,确定出所述第二部件的前景目标;Performing coordinate prediction and front and background classification of the candidate region of the second component on the second feature map to determine the foreground target of the second component;
将所述第二部件的前景目标在所述第二特征图中对应的特征进行池化处理,得到第二池化特征;performing pooling processing on the corresponding features of the foreground target of the second component in the second feature map to obtain a second pooling feature;
基于所述第二池化特征对所述第二部件进行分类,得到所述第二部件的类别和所述第二矩形检测框。Classify the second part based on the second pooling feature to obtain the category of the second part and the second rectangular detection frame.
在一种可能的实施方式中,在基于所述第一部件的图像、所述第二部件的图像对所述第一部件、所述第二部件进行缺陷分类,得到缺陷分类结果方面,缺陷分类模块84具体用于:In a possible implementation manner, in terms of performing defect classification on the first part and the second part based on the image of the first part and the image of the second part, and obtaining a defect classification result, the defect classification Module 84 is specifically used to:
分别对所述第一部件的图像、所述第二部件的图像进行特征提取;Perform feature extraction on the image of the first component and the image of the second component respectively;
基于所述第一部件的图像提取出的特征对所述第一部件进行缺陷类型预测,得到所述第一部件的缺陷分类结果;所述第一部件的缺陷分类结果包括所述第一部件的缺陷类型和所述第一部件的类别;以及,Predict the defect type of the first part based on the features extracted from the image of the first part, and obtain the defect classification result of the first part; the defect classification result of the first part includes the defect classification result of the first part. the type of defect and the category of the first part; and,
基于所述第二部件的图像提取出的特征对所述第二部件进行缺陷类型预测,得到所述第二部件的缺陷分类结果;所述第二部件的缺陷分类结果包括所述第二部件的缺陷类型和所述第二部件的类别。Predict the defect type of the second part based on the features extracted from the image of the second part, and obtain the defect classification result of the second part; the defect classification result of the second part includes the defect classification result of the second part. Defect type and category of the second part.
在一种可能的实施方式中,如图9所示,该装置还包括缺陷预警模块85;所述缺陷预警模块85具体用于:In a possible implementation manner, as shown in FIG. 9 , the device further includes a defect early warning module 85; the defect early warning module 85 is specifically used for:
根据所述缺陷分类结果进行缺陷预警。Defect early warning is performed according to the defect classification result.
在一种可能的实施方式中,在根据所述缺陷分类结果进行缺陷预警方面,缺陷预警模块85具体用于:In a possible implementation manner, in terms of performing a defect early warning according to the defect classification result, the defect early warning module 85 is specifically used for:
针对所述第一部件,输出所述第一部件的缺陷类型、所述第一部件在所述待检测图像中的位置、所述第一部件的类别和所述第一部件所在的高铁线路,以进行缺陷预警;For the first part, output the defect type of the first part, the position of the first part in the image to be inspected, the type of the first part, and the high-speed rail line where the first part is located, for defect warning;
针对所述第二部件,确定所述第二部件所属的目标第一部件,输出所述第二部件的缺陷类型、所述第二部件在所述待检测图像中的位置、所述第二部件的类别、所述目标第一部件在所述待检测图像中的位置、所述目标第一部件的类别和所述第二部件所在的高铁线路,以进行缺陷预警。For the second part, determine the target first part to which the second part belongs, and output the defect type of the second part, the position of the second part in the image to be inspected, the second part The category of the target first part, the position of the target first part in the to-be-detected image, the category of the target first part and the high-speed rail line where the second part is located, so as to carry out defect early warning.
在一种可能的实施方式中,在获取高铁接触网的待检测图像方面,图像获取模块81具体用于:In a possible implementation manner, in terms of acquiring the to-be-detected image of the high-speed rail catenary, the image acquisition module 81 is specifically used for:
获取成像设备采集的高铁接触网原始图像;Obtain the original image of the high-speed rail catenary collected by the imaging equipment;
对所述高铁接触网原始图像进行过滤,得到所述待检测图像。The original image of the high-speed rail catenary is filtered to obtain the to-be-detected image.
根据本申请的一个实施例,图8或图9所示的缺陷检测装置中的各个单元可以分别或全部合并为一个或若干个另外的单元来构成,或者其中的某个(些)单元还可以再拆分为功能上更小的多个单元来构成,这可以实现同样的操作,而不影响本申请的实施例的技术效果的实现。上述单元是基于逻辑功能划分的,在实际应用中,一个单元的功能也可以由多个单元来实现,或者多个单元的功能由一个单元实现。在本申请的其它实施例中,基于缺陷检测装置也可以包括其它单元,在实际应用中,这些功能也可以由其它单元协助实现,并且可以由多个单元协作实现。According to an embodiment of the present application, each unit in the defect detection apparatus shown in FIG. 8 or FIG. 9 may be respectively or all merged into one or several other units to form, or some unit(s) may also be It is further divided into multiple units with smaller functions, which can realize the same operation without affecting the realization of the technical effects of the embodiments of the present application. The above-mentioned units are divided based on logical functions. In practical applications, the function of one unit may also be implemented by multiple units, or the functions of multiple units may be implemented by one unit. In other embodiments of the present application, the defect-based detection device may also include other units, and in practical applications, these functions may also be implemented with the assistance of other units, and may be implemented by cooperation of multiple units.
根据本申请的另一个实施例,可以通过在包括中央处理单元(CPU)、随机存取存储介质(RAM)、只读存储介质(ROM)等处理元件和存储元件的例如计算机的通用计算设备上运行能够执行如图1或图6中所示的相应方法所涉及的各步骤的计算机程序(包括程序代码),来构造如图8或图9中所示的缺陷检测装置设备,以及来实现本申请实施例的缺陷检测方法。所述计算机程序可以记载于例如计算机可读记录介质上,并通过计算机可读记录介质装载于上述计算设备中,并在其中运行。According to another embodiment of the present application, a general-purpose computing device, such as a computer, may be implemented on a general-purpose computing device including a central processing unit (CPU), a random access storage medium (RAM), a read-only storage medium (ROM), and other processing elements and storage elements. Running a computer program (including program code) capable of executing the steps involved in the corresponding method as shown in FIG. 1 or FIG. 6, to construct the defect detection apparatus as shown in FIG. 8 or FIG. 9, and to realize the present invention. The defect detection method of the application embodiment is provided. The computer program can be recorded on, for example, a computer-readable recording medium, and loaded in the above-mentioned computing device through the computer-readable recording medium, and executed therein.
基于上述方法实施例和装置实施例的描述,本申请实施例还提供一种电子设备。请参见图10,该电子设备至少包括处理器1001、输入设备1002、输出设备1003以及计算机存储介质1004。其中,电子设备内的处理器1001、输入设备1002、输出设备1003以及计算机存储介质1004可通过总线或其他方式连接。Based on the descriptions of the foregoing method embodiments and apparatus embodiments, the embodiments of the present application further provide an electronic device. Referring to FIG. 10 , the electronic device includes at least a processor 1001 , an input device 1002 , an output device 1003 and a computer storage medium 1004 . The processor 1001 , the input device 1002 , the output device 1003 and the computer storage medium 1004 in the electronic device may be connected through a bus or other means.
计算机存储介质1004可以存储在电子设备的存储器中,所述计算机存储介质1004用于存储计算机程序,所述计算机程序包括程序指令,所述处理器1001用于执行所述计算机存储介质1004存储的程序指令。处理器1001(或称CPU(Central Processing Unit,中央处理器))是电子设备的计算核心以及控制核心,其适于实现一条或多条指令,具体适于加载并执行一条或多条指令从而实现相应方法流程或相应功能。The computer storage medium 1004 can be stored in the memory of the electronic device, the computer storage medium 1004 is used for storing a computer program, the computer program includes program instructions, and the processor 1001 is used for executing the program stored in the computer storage medium 1004 instruction. The processor 1001 (or called CPU (Central Processing Unit, central processing unit)) is the computing core and the control core of the electronic device, which is suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to achieve Corresponding method flow or corresponding function.
在一个实施例中,本申请实施例提供的电子设备的处理器1001可以用于进行一系列缺陷检测处理:In one embodiment, the processor 1001 of the electronic device provided in this embodiment of the present application may be configured to perform a series of defect detection processes:
获取高铁接触网的待检测图像;Obtain the image to be inspected of the high-speed rail catenary;
从所述待检测图像中分割出高铁接触网的第一部件的图像;Segment the image of the first component of the high-speed rail catenary from the image to be detected;
从所述第一部件的图像中分割出高铁接触网的第二部件的图像;所述第二部件为所述第一部件的子部件;Segment the image of the second part of the high-speed rail catenary from the image of the first part; the second part is a sub-part of the first part;
基于所述第一部件的图像、所述第二部件的图像对所述第一部件、所述第二部件进行缺陷分类,得到缺陷分类结果。Defect classification is performed on the first part and the second part based on the image of the first part and the image of the second part, and a defect classification result is obtained.
再一个实施例中,处理器1001执行所述从所述待检测图像中分割出高铁接触网的第一部件的图像,包括:In yet another embodiment, the processor 1001 executes the segmentation of the image of the first component of the high-speed rail catenary from the to-be-detected image, including:
对所述待检测图像进行特征提取,得到第一特征图;Perform feature extraction on the to-be-detected image to obtain a first feature map;
基于所述第一特征图对所述第一部件进行定位和分类,得到所述第一部件的第一矩形检测框;Locating and classifying the first component based on the first feature map to obtain a first rectangular detection frame of the first component;
按照所述第一矩形检测框从所述待检测图像中分割出所述第一部件的图像。The image of the first component is segmented from the to-be-detected image according to the first rectangular detection frame.
再一个实施例中,处理器1001执行所述基于所述第一特征图对所述第一部件进行定位和分类,得到所述第一部件的第一矩形检测框,包括:In yet another embodiment, the processor 1001 executes the positioning and classification of the first component based on the first feature map to obtain a first rectangular detection frame of the first component, including:
在所述第一特征图上进行所述第一部件的候选区域坐标预测和前、背景分类,确定出所述第一部件的前景目标;Performing coordinate prediction and front and background classification of the candidate region of the first component on the first feature map, and determining the foreground target of the first component;
将所述第一部件的前景目标在所述第一特征图中对应的特征进行池化处理,得到第一池化特征;performing pooling processing on the corresponding features of the foreground target of the first component in the first feature map to obtain a first pooling feature;
基于所述第一池化特征对所述第一部件进行分类,得到所述第一部件的类别和所述第一矩形检测框。The first part is classified based on the first pooling feature to obtain the category of the first part and the first rectangular detection frame.
再一个实施例中,处理器1001执行所述从所述第一部件的图像中分割出高铁接触网的第二部件的图像,包括:In yet another embodiment, the processor 1001 executes the segmentation of the image of the second component of the high-speed rail catenary from the image of the first component, including:
对所述第一部件的图像进行伽马校验,得到所述第一部件的待分割图像;Perform gamma check on the image of the first part to obtain the image to be divided of the first part;
从所述第一部件的待分割图像中分割出所述第二部件的图像。The image of the second part is segmented from the image to be segmented of the first part.
再一个实施例中,处理器1001执行所述从所述第一部件的待分割图像中分割出所述第二部件的图像,包括:In yet another embodiment, the processor 1001 performing the segmenting of the image of the second part from the image to be segmented of the first part includes:
对所述第一部件的待分割图像进行特征提取,得到第二特征图;performing feature extraction on the to-be-segmented image of the first component to obtain a second feature map;
基于所述第二特征图对所述第二部件进行定位和分类,得到所述第二部件的第二矩形检测框;Locating and classifying the second component based on the second feature map, to obtain a second rectangular detection frame of the second component;
按照所述第二矩形检测框从所述第一部件的待分割图像中分割出所述第二部件的图像。The image of the second part is segmented from the image to be segmented of the first part according to the second rectangular detection frame.
再一个实施例中,处理器1001执行所述基于所述第二特征图对所述第二部件进行定位和分类,得到所述第二部件的第二矩形检测框,包括:In yet another embodiment, the processor 1001 executes the positioning and classification of the second component based on the second feature map to obtain a second rectangular detection frame of the second component, including:
在所述第二特征图上进行所述第二部件的候选区域坐标预测和前、背景分类,确定出所述第二部件的前景目标;Performing coordinate prediction and front and background classification of the candidate region of the second component on the second feature map to determine the foreground target of the second component;
将所述第二部件的前景目标在所述第二特征图中对应的特征进行池化处理,得到第二池化特征;performing pooling processing on the corresponding features of the foreground target of the second component in the second feature map to obtain a second pooling feature;
基于所述第二池化特征对所述第二部件进行分类,得到所述第二部件的类别和所述第二矩形检测框。Classify the second part based on the second pooling feature to obtain the category of the second part and the second rectangular detection frame.
再一个实施例中,处理器1001执行所述基于所述第一部件的图像、所述第二部件的图像对所述第一部件、所述第二部件进行缺陷分类,得到缺陷分类结果,包括:In yet another embodiment, the processor 1001 performs the defect classification on the first part and the second part based on the image of the first part and the image of the second part, to obtain a defect classification result, including: :
分别对所述第一部件的图像、所述第二部件的图像进行特征提取;Perform feature extraction on the image of the first component and the image of the second component respectively;
基于所述第一部件的图像提取出的特征对所述第一部件进行缺陷类型预测,得到所述第一部件的缺陷分类结果;所述第一部件的缺陷分类结果包括所述第一部件的缺陷类型和所述第一部件的类别;以及,Predict the defect type of the first part based on the features extracted from the image of the first part, and obtain the defect classification result of the first part; the defect classification result of the first part includes the defect classification result of the first part. the type of defect and the category of the first part; and,
基于所述第二部件的图像提取出的特征对所述第二部件进行缺陷类型预测,得到所述第二部件的缺陷分类结果;所述第二部件的缺陷分类结果包括所述第二部件的缺陷类型和所述第二部件的类别。Predict the defect type of the second part based on the features extracted from the image of the second part, and obtain the defect classification result of the second part; the defect classification result of the second part includes the defect classification result of the second part. Defect type and category of the second part.
再一个实施例中,在基于所述第一部件的图像、所述第二部件的图像对所述第一部件、所述第二部件进行缺陷分类,得到缺陷分类结果之后,处理器1001还用于执行:In yet another embodiment, after the defect classification is performed on the first part and the second part based on the image of the first part and the image of the second part, and a defect classification result is obtained, the processor 1001 further uses To execute:
根据所述缺陷分类结果进行缺陷预警。Defect early warning is performed according to the defect classification result.
再一个实施例中,处理器1001执行所述根据所述缺陷分类结果进行缺陷预警,包括:In yet another embodiment, the processor 1001 executes the defect early warning according to the defect classification result, including:
针对所述第一部件,输出所述第一部件的缺陷类型、所述第一部件在所述待检测图像中的位置、所述第一部件的类别和所述第一部件所在的高铁线路,以进行缺陷预警;For the first part, output the defect type of the first part, the position of the first part in the image to be inspected, the type of the first part, and the high-speed rail line where the first part is located, for defect warning;
针对所述第二部件,确定所述第二部件所属的目标第一部件,输出所述第二部件的缺陷类型、所述第二部件在所述待检测图像中的位置、所述第二部件的类别、所述目标第一部件在所述待检测图像中的位置、所述目标第一部件的类别和所述第二部件所在的高铁线路,以进行缺陷预警。For the second part, determine the target first part to which the second part belongs, and output the defect type of the second part, the position of the second part in the image to be inspected, the second part The category of the target first part, the position of the target first part in the to-be-detected image, the category of the target first part and the high-speed rail line where the second part is located, so as to carry out defect early warning.
再一个实施例中,处理器1001执行所述获取高铁接触网的待检测图像,包括:In yet another embodiment, the processor 1001 executes the obtaining of the image to be detected of the high-speed rail catenary, including:
获取成像设备采集的高铁接触网原始图像;Obtain the original image of the high-speed rail catenary collected by the imaging equipment;
对所述高铁接触网原始图像进行过滤,得到所述待检测图像。The original image of the high-speed rail catenary is filtered to obtain the to-be-detected image.
示例性的,上述电子设备可以是电脑、电脑主机、服务器、云服务器、服务器集群等,电子设备可包括但不仅限于处理器1001、输入设备1002、输出设备1003以及计算机存储介质1004,输入设备1002可以是键盘、触摸屏等,输出设备1003可以是扬声器、显示器、射频发送器等。本领域技术人员可以理解,所述示意图仅仅是电子设备的示例,并不构成对电子设备的限定,可以包括比图示更多或更少的部件,或者组合某些部件,或者不同的部件。Exemplarily, the above-mentioned electronic device may be a computer, a computer host, a server, a cloud server, a server cluster, etc. The electronic device may include but is not limited to a processor 1001, an input device 1002, an output device 1003, and a computer storage medium 1004. The input device 1002 It can be a keyboard, a touch screen, etc., and the output device 1003 can be a speaker, a display, a radio frequency transmitter, and the like. Those skilled in the art can understand that the schematic diagram is only an example of an electronic device, and does not constitute a limitation to the electronic device, and may include more or less components than the one shown, or combine some components, or different components.
需要说明的是,由于电子设备的处理器1001执行计算机程序时实现上述的缺陷检测方法中的步骤,因此上述缺陷检测方法的实施例均适用于该电子设备,且均能达到相同或相似的有益效果。It should be noted that, since the processor 1001 of the electronic device implements the steps in the above-mentioned defect detection method when executing the computer program, the embodiments of the above-mentioned defect detection method are all applicable to the electronic device, and can achieve the same or similar benefits. Effect.
本申请实施例还提供了一种计算机存储介质(Memory),所述计算机存储介质是电子设备中的记忆设备,用于存放程序和数据。可以理解的是,此处的计算机存储介质既可以包括终端中的内置存储介质,当然也可以包括终端所支持的扩展存储介质。计算机存储介质提供存储空间,该存储空间存储了终端的操作***。并且,在该存储空间中还存放了适于被处理器1001加载并执行的一条或多条的指令,这些指令可以是一个或一个以上的计算机程序(包括程序代码)。需要说明的是,此处的计算机存储介质可以是高速RAM存储器,也可以是非不稳定的存储器(non-volatile memory),例如至少一个磁盘存储器;可选的,还可以是至少一个位于远离前述处理器1001的计算机存储介质。在一个实施例中,可由处理器1001加载并执行计算机存储介质中存放的一条或多条指令,以实现上述有关缺陷检测方法的相应步骤。Embodiments of the present application further provide a computer storage medium (Memory), where the computer storage medium is a memory device in an electronic device and is used to store programs and data. It can be understood that, the computer storage medium here may include both a built-in storage medium in the terminal, and certainly also an extended storage medium supported by the terminal. The computer storage medium provides storage space, and the storage space stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor 1001 are also stored in the storage space, and these instructions may be one or more computer programs (including program codes). It should be noted that the computer storage medium here can be a high-speed RAM memory, or a non-volatile memory (non-volatile memory), such as at least one disk memory; optionally, it can also be at least one memory located far away from the aforementioned processing The computer storage medium of the device 1001 . In one embodiment, one or more instructions stored in a computer storage medium can be loaded and executed by the processor 1001 to implement the corresponding steps of the above-mentioned defect detection method.
示例性的,计算机存储介质的计算机程序包括计算机程序代码,所述计算机程序代码可以为源代码形式、对象代码形式、可执行文件或某些中间形式等。所述计算机可读介质可以包括:能够携带所述计算机程序代码的任何实体或装置、记录介质、U盘、移动硬盘、磁碟、光盘、计算机存储器、只读存储器(ROM,Read-Only Memory)、随机存取存储器(RAM,Random Access Memory)、电载波信号、电信信号以及软件分发介质等。Exemplarily, the computer program of the computer storage medium includes computer program code, which may be in source code form, object code form, executable file or some intermediate form, and the like. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, removable hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory) , Random Access Memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc.
需要说明的是,由于计算机存储介质的计算机程序被处理器执行时实现上述的缺陷检 测方法中的步骤,因此上述缺陷检测方法的所有实施例均适用于该计算机存储介质,且均能达到相同或相似的有益效果。It should be noted that, since the computer program of the computer storage medium is executed by the processor to realize the steps in the above-mentioned defect detection method, all the embodiments of the above-mentioned defect detection method are applicable to the computer storage medium, and can achieve the same or similar beneficial effects.
以上对本申请实施例进行了详细介绍,本文中应用了具体个例对本申请的原理及实施方式进行了阐述,以上实施例的说明只是用于帮助理解本申请的方法及其核心思想;同时,对于本领域的一般技术人员,依据本申请的思想,在具体实施方式及应用范围上均会有改变之处,综上所述,本说明书内容不应理解为对本申请的限制。The embodiments of the present application are described in detail above, and specific examples are used in this paper to illustrate the principles and implementations of the present application. The descriptions of the above embodiments are only used to help understand the methods and core ideas of the present application; at the same time, for Persons of ordinary skill in the art, based on the idea of the present application, will have changes in the specific implementation manner and application scope. In summary, the contents of this specification should not be construed as limitations on the present application.