WO2020215241A1 - 一种面向计算资源局限平台部署的实时目标检测的方法 - Google Patents

一种面向计算资源局限平台部署的实时目标检测的方法 Download PDF

Info

Publication number
WO2020215241A1
WO2020215241A1 PCT/CN2019/084060 CN2019084060W WO2020215241A1 WO 2020215241 A1 WO2020215241 A1 WO 2020215241A1 CN 2019084060 W CN2019084060 W CN 2019084060W WO 2020215241 A1 WO2020215241 A1 WO 2020215241A1
Authority
WO
WIPO (PCT)
Prior art keywords
layer
output
yolo
fire
pass
Prior art date
Application number
PCT/CN2019/084060
Other languages
English (en)
French (fr)
Inventor
方伟
任培铭
王林
孙俊
吴小俊
Original Assignee
江南大学
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 江南大学 filed Critical 江南大学
Priority to PCT/CN2019/084060 priority Critical patent/WO2020215241A1/zh
Priority to US16/885,450 priority patent/US11410035B2/en
Publication of WO2020215241A1 publication Critical patent/WO2020215241A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/06—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
    • G06N3/063—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons using electronic means
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/94—Hardware or software architectures specially adapted for image or video understanding
    • G06V10/955—Hardware or software architectures specially adapted for image or video understanding using specific electronic processors

Definitions

  • the invention relates to a method for real-time target detection oriented to the deployment of a computing resource limited platform, belonging to the field of deep learning and image processing.
  • Object detection is an important task in many emerging fields, such as robot navigation and autonomous driving.
  • object detection methods based on deep learning methods have greater advantages than traditional methods, and object detection algorithms based on deep learning continue to rise, such as R-CNN, SPPNet, fast-R-CNN, faster-R- CNN, R-FCN and FPN.
  • object detection algorithms have achieved unprecedented accuracy, the detection speed is not fast, far from meeting the real-time requirements on devices with low computing power.
  • the size of the deep learning model usually takes up a lot of storage space and requires powerful GPU computing capabilities. However, in most practical application scenarios, it is impossible to place a powerful GPU workstation on the device.
  • YOLO is one of the fastest object detection methods with real-time performance and high precision. You Only Look Once (YOLO) has been continuously optimized since its introduction. YOLO-V1 has two fully connected layers and 24 convolutional layers. The scale reaches 1GB, it takes up a lot of storage space, and the performance requirements of the operating platform are very high. On this basis, YOLO-V2 deletes the fully connected layers and uses anchor boxes to predict the bounding box. YOLO-V3 uses the residual structure to further deepen the network layer and achieve a breakthrough in accuracy. At the same time, the tiny version of YOLO takes up With less storage space, Tiny-YOLO-V3 is only 34MB, which is unprecedentedly lightweight, but the storage capacity of mobile terminals is still not small.
  • Both Highway and ResNet structures propose a data bypass (skip-layer) technology to enable signals to circulate at high speed between the input layer and the output layer.
  • the core idea is to create a cross-layer connection to connect the network.
  • the "Densely Connected Convolutional Networks" co-authored by Cornell University, Tsinghua University, and Facebook FAIR Lab elaborated on it in detail.
  • the author will All layers in the network are connected in pairs, so that each layer in the network accepts the features of all the layers before it as input. Because there are a large number of dense connections in the network, the author calls this network structure Dense connection. Dense connection has the advantages of reducing the problem of gradient dispersion, making the model not easy to overfit; it can enhance the flow of features between layers, greatly reduce the number of parameters, and improve training efficiency.
  • SqueezeNet is a small model network structure that is based on the existing Convolutional Neural Networks (CNN) model and compressed in a lossy manner. Use a small number of parameters to train the network model to achieve model compression. It uses the Fire Modle model structure, which is divided into a compression part and an expansion part. The compression part and the expansion part are connected to form a convolution filter in the Fire module.
  • the usual SqueezeNet starts with a separate convolutional layer (conv1), then 8 Fire modules, and finally a final conversion layer (conv10).
  • the present invention provides a method that can be used for real-time detection of multiple targets.
  • the method of the present invention can run on embedded devices such as Jetson TX1, Jetson TX2 and mobile devices for target detection, with high accuracy and real-time performance.
  • Jetson TX1, Jetson TX2 and mobile devices for target detection with high accuracy and real-time performance.
  • YOLO tiny version of YOLO.
  • the technical solution of the present invention is: a method for target detection based on image processing, the method including the following steps:
  • Tinier-YOLO reads the image collected by the camera
  • the Tinier-YOLO is an improved YOLO-v3-tiny network structure: the alternate operation of the first five convolutional layers and the pooling layer of the YOLO-v3-tiny network structure is retained, and then the five SqueezeNets are connected in turn Fire module, output to the first pass-through layer, and then the pass-through layer connects the Fire module in the sixth SqueezeNet, and uses the Dense connection to connect the output characteristic maps of the five Fire modules to the input of the sixth Fire module.
  • the data of the six Fire modules are output to the second pass-through layer and a 1*1 bottleneck layer.
  • the seventh and eighth Fire modules in SqueezeNet perform data compression, and then the data is output to the 1*1 bottleneck layer, connected to the output end, and the output feature map size is 26*26.
  • the first pass-through layer and the third Each pass-through layer is connected to the fifth convolutional layer to obtain the output features of the fifth convolutional layer; the Fire module in the sixth SqueezeNet is also connected to the output terminal, and the output feature map size is 13*13; re Train the network to get Tinier-YOLO.
  • the pass-through layer is introduced from the YOLO-v2 algorithm.
  • the use of Dense connection to connect the output characteristic diagrams of the five Fire modules with the input of the sixth Fire module means that the outputs of the first to fifth Fire modules are simultaneously used as the first Inputs for six Fire modules.
  • the third through layer performs feature fusion on the enlarged image and the output of the feature map of the fifth convolutional layer.
  • the picture is a picture or a picture in a video.
  • the target is a person, a moving object or a stationary object.
  • the target is preferably a person or a moving object.
  • the moving object is a vehicle or an animal.
  • the transportation means are airplanes, ships, trains, buses, automobiles, motorcycles or bicycles.
  • the animal is a cat, dog, sheep, horse, cow, bird, etc.
  • the GPU-containing platform is any device with a storage space of not less than 10MB, such as Jetson TX1, Jetson TX2, iPone, Huawei and other smart phones; in particular, the method of the present invention can be applied to Platforms with limited computing resources, that is, embedded platforms or mobile devices with GPU processing performance.
  • the embedded platform with GPU processing performance is an embedded device with equivalent performance such as Jetson TX1 or Jetson TX2.
  • the present invention also provides a device for target detection based on image processing.
  • the device includes an acquisition device, a calculation module, and an output module.
  • the calculation module includes a calculation network and hardware equipment, and the image acquisition module is used to collect data.
  • the computing network runs on the hardware equipment to read the collected images and detect the line targets, and then output the detection information through the hardware equipment or image acquisition device;
  • the Tinier-YOLO is an improved YOLO-v3-tiny network structure: the alternate operation of the first five convolutional layers and the pooling layer of the YOLO-v3-tiny network structure is retained, and then the five SqueezeNets are connected in turn Fire module, output to the first pass-through layer, and then the pass-through layer connects the Fire module in the sixth SqueezeNet, and uses the Dense connection to connect the output characteristic maps of the five Fire modules to the input of the sixth Fire module.
  • the data of the six Fire modules are output to the second pass-through layer and a 1*1 bottleneck layer.
  • the seventh and eighth Fire modules in SqueezeNet perform data compression, and then the data is output to the 1*1 bottleneck layer, connected to the output end, and the output feature map size is 26*26.
  • the first pass-through layer and the third Each pass-through layer is connected to the fifth convolutional layer to obtain the output features of the fifth convolutional layer; the Fire module in the sixth SqueezeNet is also connected to the output terminal, and the output feature map size is 13*13; re Train the network to get Tinier-YOLO.
  • the pass-through layer is introduced from the YOLO-v2 algorithm.
  • the use of Dense connection to connect the output characteristic diagrams of the five Fire modules with the input of the sixth Fire module means that the outputs of the first to fifth Fire modules are simultaneously used as the first Inputs for six Fire modules.
  • the third through layer performs feature fusion of the enlarged image with the output of the feature map of the fifth convolutional layer.
  • the hardware device includes a GPU-containing platform
  • the GPU-containing platform is any device with a storage space of not less than 10MB, such as Jetson TX1, Jetson TX2, iPone, Huawei, etc.
  • Mobile phone in particular, the method of the present invention can be applied to platforms with limited computing resources, that is, embedded platforms or mobile devices with GPU processing performance.
  • the embedded platform or mobile device with GPU processing performance is preferably Jetson TX1 or Jetson TX2.
  • Tinier-YOLO of the present invention is only 7.9MB, which is only 1/4 of 34.9MB of YOLO-v3-tiny and 1/8 of YOLO-v2-tiny; its model size is reduced and The real-time performance and accuracy are not affected.
  • Tinier-YOLO of the present invention is improved by 21.8% compared with YOLO-v3-tiny, and by 70.8% compared with YOLO-v2-tiny; Compared with YOLO-v3-tiny, Tinier-YOLO's average accuracy of mAP is increased by 10.1%, and compared with YOLO-v2-tiny, it is increased by nearly 18.2%; it can be seen that Tinier-YOLO of the present invention can be realized in limited computing resources. The purpose of real-time detection can still be carried out on the platform, and the effect is better.
  • the present invention introduces a fire layer into the computing network, reduces the number of model parameters, increases the depth and width of the entire network, and ensures the accuracy of model detection; in addition, the present invention uses Dense connections to improve accuracy. Through a proper Dense connection, the Tinier-YOLO of the present invention can ensure that the real-time performance is not lost on the basis of improving the accuracy.
  • the Tinier-YOLO of the present invention realizes the installation and real-time monitoring on the embedded platform, and has a small demand for server communication. After training, it can accurately detect more than 80 objects, which overcomes the calculation in the prior art. The network cannot perform real-time calculations on embedded platforms.
  • Figure 1 Schematic diagram of the network structure of YOLO-v3-tiny.
  • Figure 2 is a schematic diagram of the network structure of Tinier-YOLO of the present invention, where (1) is the first Fire module, (2) is the second Fire module, (3) is the third Fire module, and (4) is the first Fire module.
  • Four Fire modules (5) is the fifth Fire module, (6) is the sixth Fire module, (7) is the seventh Fire module, (8) is the eighth Fire module, and (9) is the A through layer, (10) is the second through layer, and (11) is the third through layer.
  • P(i) refers to the accuracy when the threshold i is given
  • ⁇ r(i) refers to the recall change value between k and k-1.
  • the data set for training and testing comes from PASCALVOC (The pattern analysis, statistical modelling and computational learning Visual Object Classes Project), which is divided into two parts: VOC 2007 and VOC 2012.
  • PASCALVOC The pattern analysis, statistical modelling and computational learning Visual Object Classes Project
  • the present invention can train different object categories according to needs, using VOC 2007 data
  • the 5011 pictures in the set and 11540 pictures in the VOC 2012 data set are used as training data.
  • the training data is 16,551 in total, and the test set is the VOC 2007 test set, with a total of 4952 test pictures.
  • the technical solution of the present invention is: an object detection method based on image processing, the method includes the following steps:
  • Tinier-YOLO reads the image collected by the camera, among which,
  • the Tinier-YOLO is an improved YOLO-v3-tiny network structure: the alternate operation of the first five convolutional layers and pooling layers of the YOLO-v3-tiny network structure is retained, and then the five Fire modules in SqueezeNet are connected in turn , Output to the first pass-through layer, and then the pass-through layer connects to the Fire module in the sixth SqueezeNet, and uses the Dense connection to connect the output feature maps of the five Fire modules to the input of the sixth Fire module, the sixth The data of the Fire module is output to the second pass-through layer and a 1*1 bottleneck layer.
  • the subsequent data is then enlarged through the up-sampling layer to obtain the third pass-through layer with a feature map size of 26*26, and then connect to the seventh layer in turn.
  • the Fire module in the first and eighth SqueezeNet performs data compression, and then the data is output to the 1*1 bottleneck layer, connected to the output terminal, and the output feature map size is 26*26.
  • the first pass-through layer and the third pass-through The layers are respectively connected to the fifth convolutional layer to obtain the output features of the fifth convolutional layer; the Fire module in the sixth SqueezeNet is also connected to the output terminal, and the output feature map size is 13*13; retrain the network , You can get Tinier-YOLO; the pass-through layer is introduced from the YOLO-v2 algorithm; the use of Dense connection to connect the output feature maps of the five Fire modules to the input of the sixth Fire module means: The output from the first to the fifth Fire module is also used as the input of the sixth Fire module; the third pass-through layer combines the enlarged image with the output of the fifth convolutional layer's feature map.
  • the result information of the object detection is output to the computer or the display screen in real time through Jetson TX1.
  • the YOLO-v2-tiny and YOLO-v3-tiny neural networks are used to perform the same object detection, and the three different neural networks and the detection result data are compared.
  • FLOPS floating point operations per second
  • the Tinier-YOLO of the present invention can detect objects at 26.3fps on the Jetson TX1 platform. As shown in Table 1, compared with YOLO-v3-tiny, the real-time performance is improved 21.8%, an increase of 70.8% compared with YOLO-v2-tiny. It can be seen that the Tinier-YOLO of the present invention can not only be used on an embedded platform, but also can achieve real-time monitoring performance.
  • the Tinier-YOLO of the present invention not only greatly reduces the size of the model, but also improves the real-time performance. At the same time, it can improve the accuracy of detection. It can be seen that the present invention provides a real-time model that can be used on embedded platforms. The monitoring method solves the problem that the existing technology cannot perform real-time monitoring on the embedded platform.
  • Table 2 The average accuracy of object detection by different neural network models
  • the input of the second convolutional layer is the output of the first convolutional layer
  • the input of the third convolutional layer is the first
  • the input of the fourth convolutional layer is the output of the first to third convolutional layer
  • the input of the fifth convolutional layer is the output of the first to fourth convolutional layer
  • the rest The invented Tinier-YOLO network structure is consistent, and the first five Fire modules are not densely connected with the input of the sixth Fire module; the network is trained, and the object is monitored according to the method of embodiment 1.
  • the real-time performance is tested, and the results are shown in Table 1. It can be seen that this change not only greatly increases the amount of calculation, but also greatly affects the real-time performance.
  • the detection speed (real-time performance) is only 13.3FPS, and the model is relatively large and accurate. Not enough promotion.
  • the poor speed performance of Comparative Example 1 is caused by the large size of the feature map of the convolutional layer in the front part of the network, which results in the doubled calculation amount.
  • the input of the second Fire module is the output of the first Fire module
  • the input of the third Fire module is the first and The output of the second Fire module, and so on.
  • the output of the fifth maximum pooling layer is the input of the first to fifth Fire modules, and the rest is consistent with the Tinier-YOLO of the present invention and the first five Each Fire module is not densely connected with the input of the sixth Fire module, the network is trained, and the object monitoring is performed according to the method of embodiment 1.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Health & Medical Sciences (AREA)
  • Software Systems (AREA)
  • General Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • Computing Systems (AREA)
  • Artificial Intelligence (AREA)
  • General Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Molecular Biology (AREA)
  • General Engineering & Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Mathematical Physics (AREA)
  • Computational Linguistics (AREA)
  • Multimedia (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Databases & Information Systems (AREA)
  • Medical Informatics (AREA)
  • Neurology (AREA)
  • Image Analysis (AREA)

Abstract

一种基于图像处理的目标检测的方法，方法中Tinier-YOLO结构是通过对YOLO-v3-tiny神经网络进行改进形成的，保留了YOLO-v3-tiny的前五个卷积层和池化层，引入了SqueezeNet中的Fire模块、1*1瓶颈层和Dense连接。Tinier-YOLO结构能够在嵌入式设备和移动设备上运行进行目标检测。还披露了一种基于图像处理的目标检测的装置，以及一种基于图像处理的目标检测的方法或装置在自动驾驶或安防、建筑领域的应用。

Description

一种面向计算资源局限平台部署的实时目标检测的方法 技术领域
本发明涉及一种面向计算资源局限平台部署的实时目标检测的方法，属于深度学习和图像处理领域。
背景技术
物体检测是许多新兴领域的重要任务，如机器人导航，自动驾驶等。在这些复杂场景中，基于深度学习方法的对象检测方法比传统方法具有更大的优势，基于深度学习的目标检测算法不断兴起，如R-CNN、SPPNet、fast-R-CNN、faster-R-CNN、R-FCN和FPN。虽然这些物体检测算法达到了前所未有的准确度，但检测速度并不快，远远不能满足在低计算能力设备上的实时性的要求。同时，深度学习模型大小通常占用大量存储空间并且需要强大的GPU计算能力，然而在大多数的实际应用场景中，无法在设备上放置功能强大的GPU工作站。
因此，需要寻找一种同时具有出色的实时性能和更小的模型尺寸的物体检测算法。YOLO是具有实时性能和高精度的最快的物体检测方法之一，You Only Look Once(YOLO)自引入以来一直在不断优化，YOLO-V1有两个全连接层和24个卷积层，其规模达到1GB，占用存储空间非常大，对运行平台的性能要求非常高。在此基础上，YOLO-V2删除完全连接的图层并使用锚框来预测边界框，YOLO-V3利用残差结构进一步加深网络层，实现精度的突破，与此同时，YOLO的tiny版本占用的存储空间更少，Tiny-YOLO-V3只有34MB，达到了前所未有的轻量级，但移动终端的存储容量仍然不小。
但是，目前所有版本的YOLO都无法在嵌入式和移动设备上实现实时性能。YOLO仍然需要在轻量级和实时性方面进行大量改进。根据YOLO的发展趋势和当前的实际应用场景，减少模型参数、减少存储空间和提高精度是当前的发展趋势。如何实现YOLO在嵌入式和移动设备上实现实时和精确的性能，实现多个物体的实时监测，仍是一个重大的挑战。
Highway和ResNet结构中均提出了一种数据旁路(skip-layer)的技术来使得信号可以在输入层和输出层之间高速流通，核心思想都是创建了一个跨层连接来连通网路中前后层，之后由康奈尔大学、清华大学、Facebook FAIR实验室合著的《Densely Connected Convolutional Networks》对其进行了详细的阐述，为了最大化网络中所有层之间的信息流，作者将网络中的所有层两两都进行了连接，使得网络中每一层都接受它前面所有层的特征作为输入。由于网络中存在着大量密集的连接，作者将这种网络结构称为Dense连接。Dense连接具有能够减轻梯度弥散的问题，使模型不容易过拟合；能够增强特征在各个层之间的流动，大大减少 参数个数，提高了训练效率等优点。
SqueezeNet是在利用现有的基于卷积神经网络(Convolutional Neural Networks，CNN)模型并以有损的方式压缩的一种小型模型的网络结构。利用少量的参数训练网络模型，实现模型的压缩。它采用Fire Modle模型结构，分为压缩部分和扩展部分，利用压缩部分和扩展部分相连接形成一种Fire模块中组织卷积过滤器。通常的SqueezeNet开始于一个独立的卷积层(conv1)，然后是8个Fire模块，最后是一个最终的转换层(conv10)。
发明内容
为了解决上述问题，本发明提供了一种可用于多目标实时检测的方法，本发明方法能够在嵌入式设备如Jetson TX1、Jetson TX2和移动设备上运行进行目标检测，准确率和实时性均高于YOLO的tiny版本。
具体的，本发明的技术方案为：一种基于图像处理的目标检测的方法，所述方法包括以下步骤：
(1)在含有GPU的平台部署Tinier-YOLO，通过摄像头采集画面；
(2)Tinier-YOLO读取摄像头采集到的图像；
(3)Tinier-YOLO检测识别目标信息；
(4)将目标检测的结果信息实时输出到设备屏幕或摄像头自带的屏幕上；
其中，所述Tinier-YOLO为改进的YOLO-v3-tiny网络结构：保留YOLO-v3-tiny网络结构的前五个卷积层和池化层的交替运算，其后依次连接五个SqueezeNet中的Fire模块，输出至第一个直通层，之后所述直通层连接第六个SqueezeNet中的Fire模块，并使用Dense连接将五个Fire模块的输出特征图和第六个Fire模块的输入相连，第六个Fire模块的数据输出至第二个直通层和一个1*1瓶颈层，之后的数据再经过上采样层放大图像得到特征图大小为26*26的第三个直通层，之后再依次连接第七个和第八个SqueezeNet中的Fire模块进行数据压缩，之后数据输出至1*1瓶颈层，连接输出端，输出的特征图大小为26*26，此外，第一个直通层和第三个直通层分别与第五个卷积层连接，获取第五个卷积层的输出特征；第六个SqueezeNet中的Fire模块处也连接有输出端，输出的特征图大小为13*13；重新训练网络，即可得到Tinier-YOLO。
在本发明的一种实施方式中，所述直通层是自YOLO-v2算法中引入的。
在本发明的一种实施方式中，所述使用Dense连接将五个Fire模块的输出特征图和第六个Fire模块的输入相连是指：第一个到第五个Fire模块的输出同时作为第六个Fire模块的输入。
在本发明的一种实施方式中，所述第三个直通层将放大后的图像与第五个卷积层的特征图的输出进行了特征融合。
在本发明的一种实施方式中，所述画面为图片或者视频中的画面。
在本发明的一种实施方式中，所述目标为人、移动物体或者静止物体。
在本发明的一种实施方式中，所述目标优选为人或移动物体。
在本发明的一种实施方式中，所述移动物体为交通工具或动物。
在本发明的一种实施方式中，所述交通工具为飞机、船舶、火车、公共汽车、汽车、摩托车或自行车等。
在本发明的一种实施方式中，所述动物为猫、狗、羊、马、牛、鸟等。
在本发明的一种实施方式中，所述含有GPU的平台为任一存储空间不小于10MB的设备，例如Jetson TX1、Jetson TX2、iPone、华为等智能手机；尤其的，本发明方法能够适用于计算资源局限的平台，即具有GPU处理性能的嵌入式平台或移动设备。
在本发明的一种实施方式中，所述具有GPU处理性能的嵌入式平台为Jetson TX1或Jetson TX2等性能相当的嵌入式设备。
本发明还提供了一种基于图像处理的目标检测的装置，所述装置包括采集装置、计算模块和输出模块，其中，计算模块包括计算网络和硬件设备，所述图像采集模块用于采集数据，计算网络在硬件设备上运行以读取采集得到的图像并对行目标进行检测，之后将检测信息通过硬件设备或图像采集装置输出；
其中，所述Tinier-YOLO为改进的YOLO-v3-tiny网络结构：保留YOLO-v3-tiny网络结构的前五个卷积层和池化层的交替运算，其后依次连接五个SqueezeNet中的Fire模块，输出至第一个直通层，之后所述直通层连接第六个SqueezeNet中的Fire模块，并使用Dense连接将五个Fire模块的输出特征图和第六个Fire模块的输入相连，第六个Fire模块的数据输出至第二个直通层和一个1*1瓶颈层，之后的数据再经过上采样层放大图像得到特征图大小为26*26的第三个直通层，之后再依次连接第七个和第八个SqueezeNet中的Fire模块进行数据压缩，之后数据输出至1*1瓶颈层，连接输出端，输出的特征图大小为26*26，此外，第一个直通层和第三个直通层分别与第五个卷积层连接，获取第五个卷积层的输出特征；第六个SqueezeNet中的Fire模块处也连接有输出端，输出的特征图大小为13*13；重新训练网络，即可得到Tinier-YOLO。
在本发明的一种实施方式中，所述直通层是自YOLO-v2算法中引入的。
在本发明的一种实施方式中，所述使用Dense连接将五个Fire模块的输出特征图和第六 个Fire模块的输入相连是指：第一个到第五个Fire模块的输出同时作为第六个Fire模块的输入。
在本发明的一种实施方式中，第三个直通层将放大后的图像与第五个卷积层的特征图的输出进行了特征融合。
在本发明的一种实施方式中，所述硬件设备所述含有GPU的平台，所述含有GPU的平台为任一存储空间不小于10MB的设备，例如Jetson TX1、Jetson TX2、iPone、华为等智能手机；尤其的，本发明方法能够适用于计算资源局限的平台，即具有GPU处理性能的嵌入式平台或移动设备。
在本发明的一种实施方式中，所述有GPU处理性能的嵌入式平台或移动设备优选为Jetson TX1或Jetson TX2。
本发明取得的有益技术效果：
(1)本发明的Tinier-YOLO的模型尺寸仅为7.9MB，仅为YOLO-v3-tiny的34.9MB的1/4，为YOLO-v2-tiny的1/8；其模型尺寸的减小并未影响其实时性和准确度，与之相反，本发明的Tinier-YOLO的实时性能与YOLO-v3-tiny相比提高了21.8％，与YOLO-v2-tiny相比提高了70.8％；准确度与YOLO-v3-tiny相比，Tinier-YOLO的mAP平均精度均值提高了10.1％，与YOLO-v2-tiny相比提高了近18.2％；可见，本发明的Tinier-YOLO能够实现在计算资源有限的平台上仍然可以进行实时检测的目的，且效果更好。
(2)本发明在计算网络中引入火层，减少了模型参数的数量，增加了整个网络的深度和宽度，保证了模型检测的准确性；此外，本发明利用Dense连接提高准确率，本发明通过合适的Dense连接使得本发明的Tinier-YOLO在提高准确率的基础上，保证实时性不受损失。
(3)本发明的Tinier-YOLO实现了在嵌入式平台上的安装和实时监测，对服务器通信需求较小，经过训练，能够准确的检测出80余种物体，克服了现有技术中的计算网络无法在嵌入式平台上进行时实运算的问题。
附图说明
图1 YOLO-v3-tiny的网络结构示意图。
图2本发明的Tinier-YOLO的网络结构示意图，其中，(1)为第一个Fire模块，(2)为第二个Fire模块，(3)为第三个Fire模块，(4)为第四个Fire模块，(5)为第五个Fire模块，(6)为第六个Fire模块，(7)为第七个Fire模块，(8)为第八个Fire模块，(9)为第一个直通层，(10)为第二个直通层，(11)为第三个直通层。
具体实施方式
AP：即Average Precision，平均精度，其计算公式如下：
Figure PCTCN2019084060-appb-000001
其中，P(i)是指给定阈值i的时的精度，Δr(i)是指k和k-1之间的recall变化值。
训练和测试的数据集来源于PASCALVOC(The pattern analysis，statistical modelling and computational learning Visual Object Classes Project)，分为VOC 2007和VOC 2012两部分，本发明可以根据需要训练不同物体的类别，使用VOC 2007数据集的5011张图片和VOC 2012数据集的11540张图片做训练数据，训练数据共16551张，测试集为VOC 2007测试集，共4952张测试图片。
实施例1
本发明的技术方案为：一种基于图像处理的物体检测的方法，所述方法包括以下步骤：
(1)在Jetson TX1部署Tinier-YOLO，通过摄像头采集画面；
(2)Tinier-YOLO读取摄像头采集到的图像，其中，
所述Tinier-YOLO为改进的YOLO-v3-tiny网络结构：保留YOLO-v3-tiny网络结构的前五个卷积层和池化层的交替运算，其后依次连接五个SqueezeNet中的Fire模块，输出至第一个直通层，之后所述直通层连接第六个SqueezeNet中的Fire模块，并使用Dense连接将五个Fire模块的输出特征图和第六个Fire模块的输入相连，第六个Fire模块的数据输出至第二个直通层和一个1*1瓶颈层，之后的数据再经过上采样层放大图像得到特征图大小为26*26的第三个直通层，之后再依次连接第七个和第八个SqueezeNet中的Fire模块进行数据压缩，之后数据输出至1*1瓶颈层，连接输出端，输出的特征图大小为26*26，此外，第一个直通层和第三个直通层分别与第五个卷积层连接，获取第五个卷积层的输出特征；第六个SqueezeNet中的Fire模块处也连接有输出端，输出的特征图大小为13*13；重新训练网络，即可得到Tinier-YOLO；所述直通层是自YOLO-v2算法中引入的；所述使用Dense连接将五个Fire模块的输出特征图和第六个Fire模块的输入相连是指：第一个到第五个Fire模块的输出同时作为第六个Fire模块的输入；第三个直通层将放大后的图像与第五个卷积层的特征图的输出进行了特征融合。
(3)Tinier-YOLO检测识别物体信息；
(4)通过Jetson TX1将物体检测的结果信息实时输出到电脑上或者显示屏上。
按照同样的方法利用YOLO-v2-tiny和YOLO-v3-tiny神经网络进行同样的物体检测，并对三种不同神经网络以及检测的结果数据进行比较。
(1)神经网络的模型尺寸
对YOLO-v2、YOLO-v2-tiny、YOLO-v3-tiny和Tinier-YOLO这四种神经网络模型的模型尺寸和FLOPS(每秒浮点运算次数)进行对比，结果如表1所示，可见Tinier-YOLO神经网络的模型尺寸仅为7.9MB，为YOLO-v3-tiny的34.9MB的1/4，为YOLO-v2-tiny的1/8。
FLOPS(每秒浮点运算次数)通常用于衡量模型所需的计算能力，FLOPS越大，对器件的要求越高，从表1可见，Tinier-YOLO的FLOPS非常低，较其余算法更适合嵌入式AI环境中。
表1 不同神经网络模型的存储大小和FLOPS值
Figure PCTCN2019084060-appb-000002
—表示暂无数据。
(2)实时性能
对不同的神经网络进行实时性能测试，可见，本发明的Tinier-YOLO在Jetson TX1的平台上，可以检测26.3fps的物体，如表1所示，与YOLO-v3-tiny相比，实时性能提高了21.8％，与YOLO-v2-tiny相比提高了70.8％。可见，本发明的Tinier-YOLO不仅能够在嵌入式平台上使用，且能够实现实时监测的性能。
(3)平均精度mAP
针对三种不同的神经网络YOLO-v2-tiny、YOLO-v3-tiny和Tinier-YOLO，按照实施例1的步骤进行不同的物体进行检测，平均精度见表1和表2，可见，与YOLO-v3-tiny相比，Tinier-YOLO的准确度提高了6.2％，对鸟类单项提升12.3％，对瓶子和盆栽植物等物体的检测平均精度也提升达到10％以上，见表2。与YOLO-v2-tiny相比平均精度提高了近10.4％，对瓶子类别单项的检测平均精度提升25.8％。Tinier-YOLO通过直通层和多尺度预测获取了更多的细粒度特征，提升了对小目标的检测能力，表中加粗数值为Tinier-YOLO表现较优数值。
综上可知，本发明的Tinier-YOLO不仅仅模型尺寸大幅度减小，且实时性能有所提高，同时能够提高检测的准确性，可见，本发明提供了一种能够用于嵌入式平台的实时监测的方法，解决了现有技术无法在在嵌入式平台上进行实时监测的问题。
表2 不同神经网络模型进行物体检测的平均精度
Figure PCTCN2019084060-appb-000003
对比例1
当Dense连接使用在保留的YOLO-v3-tiny网络结构的前五个卷积层时，即第二卷积层的输入为第一卷积层的输出，第三卷积层的输入为第一和第二卷积层的输出，第四卷积层的输入为第一到第三卷积层的输出，第五卷积层的输入为第一到第四卷积层的输出，其余和本发明的Tinier-YOLO网络结构一致且前五个Fire模块不与第六个Fire模块的输入进行Dense连接；训练网络，并按照实施例1的方法进行物体监测。
对其实时性能进行测试，结果见表1，可见，此改动不仅大大增加了计算量，且这极大地影响了实时性能，检测速度(实时性能)仅为13.3FPS，模型也相对较大，精度提升不够。对比例1的速度性能较差是由于网络前部分的卷积层特征图尺寸较大，其所带来的计算量倍增所导致。
对比例2
当使用Dense连接将5个Fire模块和第五层最大池化层进行连接时，即第二个Fire模块的输入是第一个Fire模块的输出，第三个Fire模块的输入是第一个和第二个Fire模块的输出， 以此类推，同时，第五层最大池化层的输出分别为第一个到第五个Fire模块的输入，其余部分和本发明的Tinier-YOLO一致且前五个Fire模块不与第六个Fire模块的输入进行Dense连接，训练网络，并按照实施例1的方法进行物体监测。
对其实时性能进行测试，结果见表1，可以发现，本对比例对较小的"13×13"特征图的火层模块之间进行了频繁的密集连接，检测速度已经达到实时，模型大小和mAP也得到了提升。但是对比本发明的Tinier-YOLO，对比例2中即使使用如此频繁的连接，精度并不会有所提高，反而会导致参数和计算量的增加，方式2存在冗余连接。本发明的Tinier-YOLO使用前五层火层的输出特征图密集连接到第六层火层的输入，较方式2具有更好的实时性能，mAP提高将近2％，且模型减少了1MB。
虽然本发明已以较佳实施例公开如上，但其并非用以限定本发明，任何熟悉此技术的人，在不脱离本发明的精神和范围内，都可做各种的改动与修饰，因此本发明的保护范围应该以权利要求书所界定的为准。

Claims (10)

  1. 一种基于图像处理的目标检测的方法，其特征在于，所述方法包括以下步骤：
    (1)在含有GPU的平台部署Tinier-YOLO，通过摄像头采集画面；
    (2)Tinier-YOLO读取摄像头采集到的图像；
    (3)Tinier-YOLO检测识别目标信息；
    (4)将目标检测的结果信息实时输出到设备屏幕或摄像头自带的屏幕上；
    其中，：保留YOLO-v3-tiny网络结构的前五个卷积层和池化层的交替运算，其后依次连接五个SqueezeNet中的Fire模块，输出至第一个直通层，之后所述直通层连接第六个SqueezeNet中的Fire模块，并使用Dense连接将五个Fire模块的输出特征图和第六个Fire模块的输入相连，第六个Fire模块的数据输出至第二个直通层和一个1*1瓶颈层，之后的数据再经过上采样层放大图像得到特征图大小为26*26的第三个直通层，之后再依次连接第七个和第八个SqueezeNet中的Fire模块进行数据压缩，之后数据输出至1*1瓶颈层，连接输出端，输出的特征图大小为26*26，此外，第一个直通层和第三个直通层分别与第五个卷积层连接，获取第五个卷积层的输出特征；第六个SqueezeNet中的Fire模块处也连接有输出端，输出的特征图大小为13*13；重新训练网络，即可得到Tinier-YOLO。
  2. 根据权利要求1所述的一种基于图像处理的目标检测的方法，其特征在于，使用Dense连接将五个Fire模块的输出特征图和第六个Fire模块的输入相连是指：第一个到第五个Fire模块的输出同时作为第六个Fire模块的输入。
  3. 根据权利要求1或2所述的一种基于图像处理的目标检测的方法，其特征在于，所述第三个直通层将放大后的图像与第五个卷积层的特征图的输出进行了特征融合。
  4. 根据权利要求1-3任一所述的一种基于图像处理的目标检测的方法，其特征在于，所述目标为人、移动物体或者静止物体。
  5. 根据权利要求1～4任一所述的一种基于图像处理的目标检测的方法，其特征在于，所述含有GPU的平台为任一存储空间不小于10MB的设备。
  6. 根据权利要求5所述的一种基于图像处理的目标检测的方法，其特征在于，所述含有GPU的平台包括具有GPU处理性能的嵌入式平台或移动设备。
  7. 一种基于图像处理的目标检测的装置，其特征在于，所述装置包括采集装置、计算模块和输出模块，其中，所述图像采集模块用于采集数据，所述计算模块包括计算网络和硬件设备，计算网络在硬件设备上运行以读取采集得到的图像并对行目标进行检测，之后将检测信息通过硬件设备或图像采集装置输出；
    其中，所述Tinier-YOLO为改进的YOLO-v3-tiny网络结构：保留YOLO-v3-tiny网络结构的前五个卷积层和池化层的交替运算，其后依次连接五个SqueezeNet中的Fire模块，输出 至第一个直通层，之后所述直通层连接第六个SqueezeNet中的Fire模块，并使用Dense连接将五个Fire模块的输出特征图和第六个Fire模块的输入相连，第六个Fire模块的数据输出至第二个直通层和一个1*1瓶颈层，之后的数据再经过上采样层放大图像得到特征图大小为26*26的第三个直通层，之后再依次连接第七个和第八个SqueezeNet中的Fire模块进行数据压缩，之后数据输出至1*1瓶颈层，连接输出端，输出的特征图大小为26*26，此外，第一个直通层和第三个直通层分别与第五个卷积层连接，获取第五个卷积层的输出特征；第六个SqueezeNet中的Fire模块处也连接有输出端，输出的特征图大小为13*13；重新训练网络，即可得到Tinier-YOLO。
  8. 根据权利要求7所述的一种基于图像处理的目标检测的装置，其特征在于，所述硬件设备为含有GPU的平台，所述含有GPU的平台为任一个存储空间不小于10MB的设备。
  9. 权利要求1～6任一所述的一种基于图像处理的物体检测的方法在自动驾驶或安防、建筑领域的应用。
  10. 权利要求7或8所述的一种基于图像处理的目标检测的装置在自动驾驶或安防、建筑领域的应用。
PCT/CN2019/084060 2019-04-24 2019-04-24 一种面向计算资源局限平台部署的实时目标检测的方法 WO2020215241A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
PCT/CN2019/084060 WO2020215241A1 (zh) 2019-04-24 2019-04-24 一种面向计算资源局限平台部署的实时目标检测的方法
US16/885,450 US11410035B2 (en) 2019-04-24 2020-05-28 Real-time target detection method deployed on platform with limited computing resources

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2019/084060 WO2020215241A1 (zh) 2019-04-24 2019-04-24 一种面向计算资源局限平台部署的实时目标检测的方法

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US16/885,450 Continuation US11410035B2 (en) 2019-04-24 2020-05-28 Real-time target detection method deployed on platform with limited computing resources

Publications (1)

Publication Number Publication Date
WO2020215241A1 true WO2020215241A1 (zh) 2020-10-29

Family

ID=72423756

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/084060 WO2020215241A1 (zh) 2019-04-24 2019-04-24 一种面向计算资源局限平台部署的实时目标检测的方法

Country Status (2)

Country Link
US (1) US11410035B2 (zh)
WO (1) WO2020215241A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113177937A (zh) * 2021-05-24 2021-07-27 河南大学 基于改进YOLOv4-tiny的布匹缺陷检测模型及方法

Families Citing this family (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112101322B (zh) * 2020-11-17 2021-03-02 深圳市优必选科技股份有限公司 目标物体检测模型、训练方法、物体检测方法和相关设备
CN112613504A (zh) * 2020-12-17 2021-04-06 上海大学 一种声呐水下目标检测方法
CN113033284B (zh) * 2020-12-22 2022-10-25 迪比（重庆）智能科技研究院有限公司 一种基于卷积神经网络的车辆实时超载检测方法
CN112990317B (zh) * 2021-03-18 2022-08-30 中国科学院长春光学精密机械与物理研究所 一种弱小目标检测方法
CN113011365A (zh) * 2021-03-31 2021-06-22 中国科学院光电技术研究所 一种结合轻量化网络的目标检测方法
CN113378890B (zh) * 2021-05-17 2024-03-22 浙江工业大学 一种基于改进YOLO v4的轻量级行人车辆检测方法
CN113807472B (zh) * 2021-11-19 2022-02-22 智道网联科技(北京)有限公司 分级目标检测方法及装置
CN115171006B (zh) * 2022-06-15 2023-04-14 武汉纺织大学 基于深度学习的自动识别人员进入电力危险区的检测方法
CN115346169B (zh) * 2022-08-08 2023-04-07 航天神舟智慧***技术有限公司 一种睡岗行为检测方法及***
CN115223130B (zh) * 2022-09-20 2023-02-03 南京理工大学 基于改进YOLOv5的多任务全景驾驶感知方法与***
CN115457540B (zh) * 2022-11-11 2023-03-24 整数智能信息技术(杭州)有限责任公司 点云目标检测模型的构建方法、目标检测标注方法及装置
CN115661614B (zh) * 2022-12-09 2024-05-24 江苏稻源科技集团有限公司 一种基于轻量化YOLO v1的目标检测方法
CN116205895B (zh) * 2023-03-16 2024-04-02 四川轻化工大学 一种基于改进YOLOv5的变压器漏油检测方法
CN117496475B (zh) * 2023-12-29 2024-04-02 武汉科技大学 一种应用于自动驾驶的目标检测方法及***

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108960067A (zh) * 2018-06-05 2018-12-07 北京华纵科技有限公司 基于深度学习的实时的列车驾驶员动作识别***和方法
US20190012551A1 (en) * 2017-03-06 2019-01-10 Honda Motor Co., Ltd. System and method for vehicle control based on object and color detection
CN109583355A (zh) * 2018-11-23 2019-04-05 江南大学 一种基于边界选择的人流统计的装置和方法

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20190012551A1 (en) * 2017-03-06 2019-01-10 Honda Motor Co., Ltd. System and method for vehicle control based on object and color detection
CN108960067A (zh) * 2018-06-05 2018-12-07 北京华纵科技有限公司 基于深度学习的实时的列车驾驶员动作识别***和方法
CN109583355A (zh) * 2018-11-23 2019-04-05 江南大学 一种基于边界选择的人流统计的装置和方法

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113177937A (zh) * 2021-05-24 2021-07-27 河南大学 基于改进YOLOv4-tiny的布匹缺陷检测模型及方法

Also Published As

Publication number Publication date
US11410035B2 (en) 2022-08-09
US20200293891A1 (en) 2020-09-17

Similar Documents

Publication Publication Date Title
WO2020215241A1 (zh) 一种面向计算资源局限平台部署的实时目标检测的方法
CN109815868B (zh) 一种图像目标检测方法、装置及存储介质
CN110110627B (zh) 一种面向计算资源局限平台部署的实时目标检测的方法
CN114202672A (zh) 一种基于注意力机制的小目标检测方法
Liu et al. A high-precision positioning approach for catenary support components with multiscale difference
CN111784685A (zh) 一种基于云边协同检测的输电线路缺陷图像识别方法
CN108647665A (zh) 基于深度学习的航拍车辆实时检测方法
CN109801265B (zh) 一种基于卷积神经网络的实时输电设备异物检测***
CN111507150A (zh) 利用基于深度神经网络的多重图像块组合识别人脸的方法
CN112329881B (zh) 车牌识别模型训练方法、车牌识别方法及装置
CN112258512A (zh) 点云分割方法、装置、设备和存储介质
Ma et al. Mdcn: Multi-scale, deep inception convolutional neural networks for efficient object detection
CN115294476B (zh) 面向无人机电力巡检的边缘计算智能检测方法及设备
CN116503318A (zh) 一种融合CAT-BiFPN与注意力机制的航拍绝缘子多缺陷检测方法、***及设备
Wang et al. Improved YOLOv5 with BiFPN on PCB defect detection
CN113989939B (zh) 一种基于改进yolo算法的小目标行人检测***
Mubarak et al. Smart tourism: A proof of concept for cyprus museum of modern arts in the iot era
Cao et al. Edge-Cloud collaborated object detection via difficult-case discriminator
Li et al. MobileNetV3-CenterNet: A target recognition method for avoiding missed detection effectively based on a lightweight network
CN116229406B (zh) 车道线检测方法、***、电子设备及存储介质
CN112101366A (zh) 基于混合扩张网络的实时分割***与方法
CN116580324A (zh) 一种基于YOLOv5的无人机对地目标检测方法
CN110046632A (zh) 模型训练方法和装置
CN115170803A (zh) 一种基于e-solo的城市街景实例分割方法
Hou et al. Real‐time defect detection method based on YOLO‐GSS at the edge end of a transmission line

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19925658

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19925658

Country of ref document: EP

Kind code of ref document: A1