WO2018120446A1 - 一种面向实时目标识别的异构处理机并行协调处理方法 - Google Patents

一种面向实时目标识别的异构处理机并行协调处理方法 Download PDF

Info

Publication number
WO2018120446A1
WO2018120446A1 PCT/CN2017/077106 CN2017077106W WO2018120446A1 WO 2018120446 A1 WO2018120446 A1 WO 2018120446A1 CN 2017077106 W CN2017077106 W CN 2017077106W WO 2018120446 A1 WO2018120446 A1 WO 2018120446A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
fpga
dsp
processing
frame
Prior art date
Application number
PCT/CN2017/077106
Other languages
English (en)
French (fr)
Inventor
张天序
李欢
郑畅
张培阳
向叮
Original Assignee
华中科技大学
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华中科技大学 filed Critical 华中科技大学
Publication of WO2018120446A1 publication Critical patent/WO2018120446A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T1/00—General purpose image data processing
    • G06T1/20—Processor architectures; Processor configuration, e.g. pipelining
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T1/00—General purpose image data processing
    • G06T1/60—Memory management
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N1/00—Scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission; Details thereof
    • H04N1/387—Composing, repositioning or otherwise geometrically modifying originals
    • H04N1/3877—Image rotation

Definitions

  • the invention belongs to the field of cross-scientific technology of navigation guidance, image recognition and image processing, and more specifically relates to a parallel coordination processing method of a heterogeneous processor for real-time target recognition.
  • Moving target detection, tracking and recognition are important research issues in computer vision, target detection and tracking, and navigation guidance.
  • the moving target detection and recognition technology in the static platform is relatively mature, and the moving target detection and recognition technology under the dynamic platform is still developing.
  • static platform moving target detection tasks the image background remains unchanged, the background model is generally known, and moving target detection is relatively easy.
  • Moving platform moving target detection is much more complicated than static platform moving target detection.
  • the platform motion is divided into rolling, rotating and moving. In the case of a platform scrolling, the sequence image will have a significant rotation effect. Therefore, the sequence image must be rotated and preprocessed to eliminate the problems caused by platform scrolling.
  • the rotation and movement of the platform will cause an apparent change in the background of the image, while the target to be detected is also moving.
  • the foreground motion of the entire image is mixed with the background changes produced by the platform motion. Only the two motions are well separated. In order to accurately detect moving targets and achieve tracking tasks, this poses a great challenge to the background compensation problem.
  • DSP digital signal processor
  • multi-DSP also called DSPs
  • FIG. 1 A series of algorithms, such as image rotation, downsampling and edge detection, multi-stage filtering, visual nonlinear segmentation, connected region labeling, feature extraction and matching, are sequentially executed on one DSP. This not only increases the complexity of the program, but the real-time difference caused by sequential execution is unavoidable.
  • DSP processor due to the limited versatility of the DSP processor, it has a bottleneck in the optimization of image processing and target detection and recognition algorithms. Therefore, there is a need in the art for a parallel coordinated processing method for heterogeneous processors for real-time target recognition.
  • the present invention aims to provide a processing method for solving the problems of complicated calculation, long time consumption, large power consumption, and the like in the homogeneous structure processing system.
  • the present invention provides a parallel coordination processing method for a heterogeneous processor for real-time target recognition, comprising the following steps:
  • the sequence image is input into the buffer in the FPGA;
  • the image acquisition device collects the inertial navigation information of the target image and inputs it to the DSP;
  • the DSP receives the inertial navigation information and obtains the rotation angle parameter therefrom; at the same time, the DSP determines whether the error feedback signal of the subsequent processing step is received; if the error feedback signal is received, the DSP resets the FPGA and jumps to step (3). If the error feedback signal is not received, the DSP determines whether it needs to control the FPGA to call the rotating ASIC chip to rotate the image according to the rotation angle parameter of the image; if it needs to rotate the image, skip to step (3), if it does not need to rotate, jump Go to step (4);
  • the DSP calls the FPGA to call the rotating ASIC chip, rotates the image, and writes the image data after the rotation processing to the external DPRAM, and the FPGA reads the rotated image in the DPRAM to perform verification.
  • the rotation result is correct, skip to step (4); when the rotating chip is not working properly or the rotation result is incorrect, skip to step (2)
  • the FPGA performs preprocessing on the rotated image; if the previous image does not need to be rotated, the FPGA preprocesses the cached original image;
  • the DSP calls the FPGA to call the multi-stage filter ASIC to perform multi-stage filtering on the image after the pre-processing in step (4); the image processed by the multi-stage filter ASIC is written to the external DPRAM, and the FPGA reads the DPRAM inside. After multi-stage filtering, the FPGA then buffers the image data processed by the multi-stage filtering and transmits it to the DSP;
  • the DSP performs nonlinear segmentation processing on the image processed by the multi-stage filtering in step (5); the image after the nonlinear segmentation process is called by the DSP to perform morphological filtering processing on the FPGA;
  • the DSP calls the FPGA, and the image after the nonlinear segmentation and the morphological filtering in step (6) is transferred to the contour tracking and marking ASIC chip, and the contour tracking and labeling processing is performed, and the marked image is buffered to the FPGA. Transfer to the DSP;
  • the DSP obtains the processing result and geometric features of the marked image, and calls the pattern classifier to perform subsequent target classification; finally, the processed image is transmitted to the PAL video output for display, that is, the real-time recognition and tracking result of the target.
  • the image data of the first frame is rotated and the image preprocessing of the FPGA is performed, the image of the second frame enters the rotating chip for rotation; because the processing speed of the FPGA is fast, when the image processing of the first frame is completed, the image of the second frame is completed.
  • the image data of the first frame is buffered, and the image of the second frame is preprocessed by the FPGA after the rotation of the image of the second frame is completed; when the FPGA performs the preprocessing of the second frame image, the FPGA simultaneously The cached data of the first frame image is read for further processing.
  • each individual module of the FPGA is executed in parallel.
  • the FPGA, the DSP, and the ASICs perform data transmission through a switched network constructed by the FIFO, and each of the input FIFOs selects one of the three-state gates with an enable signal to implement an output interface.
  • the data is simultaneously transferred to multiple input FIFOs.
  • the DSP is used as the main control chip, and the specific integrated chip is called by the FPGA to realize certain specific algorithm functions.
  • the communication interconnection between the DSP and the FPGA can timely feedback each chip. Whether it works properly, the DSP issues instructions to terminate or continue the system. In this way, it is possible to avoid the spread of errors caused by the previous chip not working properly.
  • the present invention decomposes the complex algorithm required for image object recognition into a basic calculation module, implements a rotation algorithm using a rotating ASIC chip, implements a multi-stage filtering algorithm with a multi-stage filtering ASIC chip, and preprocesses with an FPGA.
  • the module implements morphological filtering, marks the PIC image with marker ASIC, etc.
  • the final feature matching algorithm is implemented by DSP.
  • Figure 1 is a flow chart of a conventional algorithm processing
  • 3 is a structural diagram of synchronization processing of data
  • Figure 4 is a high bandwidth dynamic full interconnection structure diagram
  • FIG. 5 is a parallel processing flow diagram
  • Figure 6 (a) is an infrared image of a building in an embodiment
  • Figure 6 (b) is an infrared image of the aircraft in the embodiment
  • Figure 6 (c) is an infrared image of the ship in the embodiment
  • Figure 6 (d) is an infrared image of the armored vehicle in the embodiment
  • Figure 7 (a) is a diagram of a building identification result in the embodiment
  • Figure 7 (b) is a diagram of aircraft recognition results in the embodiment.
  • Figure 7 (c) is a diagram of the ship identification result in the embodiment.
  • Figure 7 (d) is a diagram showing the recognition result of the armored vehicle in the embodiment.
  • Figure 8 (a) is an infrared image of the original human hand after imaging by the image capture device
  • Figure 8 (b) is an image of Figure 8 (a) after the corrected SOC
  • Figure 9 (a) is an infrared image of the original building after imaging by the image capture device
  • Figure 9 (b) is an image of Figure 9 (a) after rotating the ASIC
  • Figure 10 (a) is an infrared image of the original ship after imaging by the image capture device
  • Fig. 10(b) is an image of Fig. 10(a) subjected to multi-stage filtering by a multi-stage filter ASIC.
  • the heterogeneous processor is a heterogeneous parallel architecture based on "FPGA+DSPs+ASICs".
  • the whole information processing machine is mainly composed of processing board, embedded operating system and database. It has the characteristics of modularization, standardization, small size and low power consumption, which can meet the requirements of real-time processing.
  • the system uses three ASICs, a rotating ASIC, a multi-stage filter ASIC, and a contour tracking and tag ASIC. They replace the DSP to realize the algorithm of non-uniformity correction, image rotation, image multi-level filtering and contour tracking and marking.
  • the system processing flow chart is shown in Figure 2, and the architecture diagram is shown in Figure 3.
  • the sequence image is input to the FPGA of the processing board, and the FPGA converts the input image data into a format and caches it.
  • the DSP receives the inertial navigation information of the image acquisition device and obtains the rotation angle parameter. At the same time, the DSP will determine whether an error feedback signal for subsequent processing steps is received. If an error feedback signal is detected, the DSP resets the entire FPGA processing board. Control whether the FPGA calls the rotating ASIC chip to rotate the image according to the rotation angle parameter of the image. If you need to rotate, skip to step (3). If you don't need to rotate, skip to step (4).
  • the DSP calls the FPGA to call the rotating ASIC chip, rotates the image, and writes the image data after the rotation processing to the external DPRAM, and the FPGA reads the rotated image in the DPRAM and performs verification. If the rotation result is correct, skip to step (4); when the rotation chip is not working properly, the rotation result is incorrect, for example, the rotated image has a wrong frame, a field or an image with obvious error information, etc. Will send an error feedback signal to the DSP, skip to step (2)
  • the FPGA preprocesses the rotated image, for example, downsampling and edge detection; if the previous image does not need to be rotated, the original image of the cache to the FPGA Pretreatment is performed.
  • the DSP calls the FPGA to call the multi-stage filtering ASIC to perform multi-stage filtering on the image after the preprocessing in step (4).
  • the processed image of the multi-stage filter ASIC is written to the external DPRAM, and the FPGA reads the image processed by the multi-stage filtering in the DPRAM, and then the FPGA buffers the image data processed by the multi-stage filtering and transmits it to the DSP.
  • the image data is too large, the data can be stored in the memory chip DDR2 of the DSP.
  • the DSP performs different algorithms to perform nonlinear segmentation processing on the image.
  • the image after nonlinear segmentation processing is processed by the DSP through the FPGA for morphological filtering.
  • the DSP calls the FPGA, and the nonlinearly segmented and morphologically filtered processed image in step (6) is transferred to the contour tracking and marking ASIC chip for contour tracking and label processing, and the marked image is buffered to the FPGA and transmitted.
  • the DSP obtains the processing result and geometric features of the marked image, and calls the pattern classifier to perform subsequent target recognition classification processing. Finally, the processed image is transmitted to the PAL video output through the image display unit for display.
  • the processing system uses the FIFO resources and DPRAM resources inside the FPGA, and also uses two external DPRAM memory chips to solve the data synchronization problem by processing the data and the like.
  • step (3) and step (4) of different frames can be parallel, and processing time synchronization can be guaranteed by means of caching. Therefore, the use of FIFO and DPRAM plays a crucial role in the operation of the pipeline.
  • each independent module of the FPGA is executed in parallel, for example, when the first frame image is multi-stage filtered, the second frame image is rotating.
  • the system adopts a FIFO-built switched network configuration as shown in FIG. 4, wherein at each of the input FIFOs, an enable signal can be selected to select one of the three-state gates, that is, for each input FIFO. You can choose which output interface the data comes from to avoid conflicts. At the same time, the data of one output interface can be simultaneously transmitted to multiple input FIFOs to realize the broadcast transmission of data.
  • Such a switched network reduces the FIFO memory that requires a large number of memory cells, but also achieves full interconnection between processing components. It can be realized that during the task process, the data flow is changed according to the needs of the algorithm, and the data flow of the algorithm is reconstructed.
  • (b) is the image after the rotation of the ASIC; as shown in Figure 10, (a) is the infrared image of the original ship after imaging by the image acquisition device, (b) is multi-stage filtered The image after the ASIC.
  • the invention implements certain specific algorithms (for example: rotation algorithm, connected region labeling algorithm, small target detection, etc.) with a dedicated integrated chip, which not only reduces the algorithm complexity of the DSP, but also these different heterogeneous devices.
  • rotation algorithm for example: rotation algorithm, connected region labeling algorithm, small target detection, etc.
  • small target detection etc.
  • the ability to coordinate parallel work greatly improves the real-time performance of target detection and recognition.
  • ASICs Dedicated integrated chips
  • ASICs feature high processing efficiency and low power consumption. Designed specifically for image processing algorithm applications, ASICs can quickly and efficiently perform the corresponding image processing algorithms at very low power.
  • the use of a dedicated image processing ASIC in the target imaging automatic target recognition system can improve the real-time performance of the system and reduce the power consumption of the system.
  • the DSP is used as the main control chip, and the specific integrated chip is called by the FPGA to realize certain specific algorithm functions.
  • the communication interconnection between the DSP and the FPGA can timely feedback each chip. Whether it works properly, the DSP issues instructions to terminate or continue the system. In this way, it is possible to avoid the spread of errors caused by the previous chip not working properly.
  • the invention decomposes the complex algorithm required for image object recognition into basic calculation modules, for example, using a rotating ASIC chip to implement a rotation algorithm, a multi-stage filtering ASIC chip to implement a multi-stage filtering algorithm, and an FPGA preprocessing module.
  • the final feature matching algorithm is implemented by the DSP, as shown in Figure 2.
  • DSP digital signal processor
  • the DSP is used as the real-time target recognition heterogeneous processing system of the main control chip, because FPGA, DSP, ASICs and the like have different working clocks, and the ASIC is used as a dedicated processor, input and output interface and general processing.
  • DSP digital signal processor
  • FPGA field-programmable gate array
  • ASIC application-specific integrated circuit
  • the rotary ASIC inputs the image data
  • the output is the image data after the image processing
  • the image data has a bit width of 8bit-14bit
  • the mark ASIC inputs the binary (multi-value) divided image data
  • the output is a feature
  • the contour information and coordinates of the target, etc., the bit width of the image data is 3 bits.
  • the ASIC's input and output interface caches are also different.
  • the tag ASIC does not require an external memory chip to store data
  • the rotating ASIC and multi-stage filter ASIC require an external memory chip to store data.
  • the separation between the logical layer and the physical layer and the dynamic interconnection structure of the logical layer are an effective method for solving heterogeneous interconnections.
  • the physical layer and the logical layer select different interfaces according to different requirements.
  • the inter-board connection uses the LVDS interface, and the serial-to-parallel conversion is connected to the logic layer;
  • the intra-board connection uses a synchronous memory interface, such as FIFO, DPRAM, and the like.
  • the cache structure guarantees asynchronous data caching problems caused by differences in processing speed between different processors.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Image Processing (AREA)

Abstract

一种面向实时目标识别的异构处理机并行协调处理方法，该方法基于高效的互联结构以及高低速、多总线数据同步传输设计，对复杂算法进行分解，合理分配到相应的软硬件资源上，通过DSP作为主控器件，根据实际需求适时的调用各个异构器件，使得各个异构器件能够协调同步有序工作，解决了同构结构处理***存在的计算复杂、耗时长、功耗大等问题。

Description

一种面向实时目标识别的异构处理机并行协调处理方法 【技术领域】
本发明属于导航制导、图像识别和图像处理的交叉科学技术领域，更具体地，涉及一种实时目标识别的异构处理机并行协调处理方法。
【背景技术】
运动目标检测、跟踪与识别是计算机视觉、目标检测与跟踪、导航制导等应用领域的重要研究问题。运动目标检测主要有静平台和动平台两种模式。静平台情况下的运动目标检测与识别技术已经比较成熟，动平台情况下的运动目标检测与识别技术还在发展之中。对于静平台运动目标检测任务，图像背景保持不变，背景模型通常已知，运动目标检测相对容易。动平台运动目标检测要比静平台运动目标检测复杂得多，平台的运动分为滚动、转动和平动。在平台滚动的情况下，序列图像会存在明显的旋转效果。因此必须对序列图像进行旋转预处理，消除平台滚动带来的问题。平台的转动和平动会导致图像背景的表观变化，同时待检测的目标也在运动，整幅图像的前景运动和平台运动产生的背景变化混合在一起，只有很好地把这两种运动分开，才能够准确的检测出运动目标并实现跟踪任务，这给背景的补偿问题提出了很大的挑战。
传统的实时目标检测与识别多采用DSP或者多DSP(也叫DSPs)来进行算法的处理，流程如图1所示。将图像的旋转、降采样及边缘检测、多级滤波、视觉非线性分割、连通区域标记、特征提取与匹配等一系列的算法均在一个DSP上顺序执行。这样不仅加大了程序的复杂度，由顺序执行带来的实时性差是无法避免的，同时由于DSP处理器的通用性有限，使得它在图像处理与目标检测识别算法优化方面存在瓶颈。因此，本领域亟需一种面向实时目标识别的异构处理机的并行协调处理方法。
【发明内容】
针对现有技术的缺陷，本发明旨在提供一种解决同构结构处理***存在的计算复杂、耗时长、功耗大等问题的处理方法。
为了实现上述目的，本发明提供了一种面向实时目标识别的异构处理机并行协调处理方法，包括如下步骤：
(1)序列图像输入到FPGA中缓存；图像采集装置采集目标图像的惯导信息并输入到DSP；
(2)DSP接收惯导信息，从中获取旋转角度参数；同时，DSP判断是否接收到后续处理步骤的出错反馈信号；若接收到出错反馈信号，DSP对FPGA进行复位，并跳到步骤(3)；若没有接收到出错反馈信号，DSP根据图像的旋转角度参数来判断是否需要控制FPGA调用旋转ASIC芯片进行图像的旋转；若需要进行图像旋转则跳到步骤(3)，若不需要旋转则跳到步骤(4)；
(3)DSP调用FPGA调用旋转ASIC芯片，对图像进行旋转操作，并将旋转处理完成后的图像数据写往外部的DPRAM，同时FPGA读取DPRAM里面经过旋转处理后的图像，进行校验，若旋转结果正确，跳到步骤(4)；当旋转芯片无法正常工作或旋转结果不正确时，跳到步骤(2)
(4)若旋转ASIC对图像进行旋转操作成功，FPGA对旋转处理后的图像进行预处理；若之前图像不需要旋转操作，FPGA对缓存的原始图像进行预处理；
(5)DSP调用FPGA调用多级滤波ASIC对步骤(4)中经过预处理之后的图像进行多级滤波处理；多级滤波ASIC处理完后的图像写往外部的DPRAM，同时FPGA读取DPRAM里面经过多级滤波处理后的图像，然后FPGA将经过多级滤波处理后的图像数据进行缓存并传送到DSP中；
(6)DSP对步骤(5)中经过多级滤波处理后的图像进行非线性分割处理；非线性分割处理后的图像由DSP调用FPGA进行形态学滤波处理；
(7)DSP调用FPGA，将步骤(6)中经非线性分割、形态学滤波处理后的图像传送到轮廓跟踪与标记ASIC芯片，进行轮廓跟踪与标记处理，标记完后的图像缓存到FPGA并传送给DSP；
(8)DSP获取标记完后的图像的处理结果及几何特征，并调用模式分类器进行后续目标分类；最后将处理完的图像传送到PAL视频输出进行显示，即目标的实时识别与跟踪结果。
进一步地，使用FPGA内部的FIFO资源以及DPRAM资源，还使用两个外部的DPRAM存储芯片，通过对数据的缓存使得在处理某一帧图片的某一步骤的同时，还能够并行处理其他帧的其他步骤。
进一步地，所述并行处理的方法如下：
当第一帧图像数据旋转完成，进行FPGA的图像预处理时，第二帧图像便进入到旋转芯片进行旋转；由于FPGA的处理速度较快，第一帧图像预处理完成时，第二帧图像还未旋转完成，此时对第一帧图像数据进行缓存，待第二帧图像旋转完成后再由FPGA进行第二帧图像的预处理；当FPGA进行第二帧图像的预处理时，FPGA同时读取第一帧图像的缓存数据进行下一步处理。
进一步地，FPGA的各个独立模块都是并行执行的。
进一步地，FPGA、DSP、各ASIC之间均通过FIFO构建的交换式网络进行数据传输，每个输入FIFO的选择端均用使能信号选择其中的一个三态门接通，以实现一个输出接口的数据同时传输到多个输入FIFO中。
与现有技术相比，本发明的效果如下：
(1)将复杂算法进行分解，由DSP作为主控芯片，合理的通过FPGA来调用专用集成芯片来实现某些特定的算法功能；同时DSP与FPGA之间的通信互联，可以及时的反馈各个芯片是否正常工作，DSP由此发出指令来终止或者继续***的工作。这样一来，就可以避免由前一个芯片无法正常工作而导致的错误蔓延。
(2)本发明将完成图像目标识别所需要的复杂算法分解成一个个的基本计算模块，用旋转ASIC芯片来实现旋转算法、用多级滤波ASIC芯片来实现多级滤波算法、用FPGA预处理模块来实现形态学滤波、用标记ASIC来完成对分割后图像的标记等，最终的特征匹配算法由DSP来实现。这样合理的分工以及协调调度既保证了***的实时性，又降低了***的整体功耗，同时降低了对DSP和FPGA等处理芯片对运算能力的要求。
(3)流水线操作与并行操作相结合。由DSP作为主控芯片，合理的通过FPGA来调用专用集成芯片来实现某些特定的算法功能的关键就是适时调用各个模块，对图像的处理采用流水线操作，图像数据依次有序地经过各个处理单元处理，各处理单元协同工作，以保证每一个时刻都有多帧图像在流水线上被有序地处理。这种异构器件的流水线结合并行操作的特点，大大地提高了***的实时性。
【附图说明】
图1是传统算法处理流程图；
图2是异构处理机处理流程图；
图3是数据的同步处理结构图；
图4是高带宽动态全互联结构图；
图5是并行处理流水图；
图6(a)是实施例中建筑物红外图像；
图6(b)是实施例中飞机红外图像；
图6(c)是实施例中舰船红外图像；
图6(d)是实施例中装甲车红外图像；
图7(a)是实施例中建筑物识别结果图；
图7(b)是实施例中飞机识别结果图；
图7(c)是实施例中舰船识别结果图；
图7(d)是实施例中装甲车识别结果图；
图8(a)是图像采集装置成像后的原始人手的红外图像；
图8(b)是图8(a)经过校正SOC后的图像；
图9(a)是图像采集装置成像后的原始建筑物的红外图像；
图9(b)是图9(a)经过旋转ASIC后的图像；
图10(a)是图像采集装置成像后的原始舰船的红外图像；
图10(b)是图10(a)经过多级滤波ASIC进行多级滤波后的图像。
【具体实施方式】
为了使本发明的目的、技术方案及优点更加清楚明白，以下结合附图及实施例，对本发明进行进一步详细说明。应当理解，此处所描述的具体实施例仅仅用以解释本发明，并不用于限定本发明。此外，下面所描述的本发明各个实施方式中所涉及到的技术特征只要彼此之间未构成冲突就可以相互组合。
图2、3是用于本发明的方法的一款实时目标识别的异构处理机，主要是用于对红外图像进行实时地处理与识别。该异构处理机是基于“FPGA+DSPs+ASICs”的异构并行架构。整个信息处理机主要由处理板、嵌入式操作***和数据库组成，具有模块化、标准化、体积小、功耗低的特点，能够满足实时处理的要求。同时，***采用了三种ASIC，分别是旋转ASIC、多级滤波ASIC和轮廓跟踪与标记ASIC。它们分别代替DSP来实现非均匀性校正、图像旋转、图像多级滤波及轮廓跟踪与标记的算法，该***处理流程图如图2所示，架构图如图3所示。
请参照图2，本发明的具体步骤如下：
(1)***工作时，序列图像输入到处理板的FPGA中，FPGA将输入的图像数据转换格式并缓存。
(2)DSP作为主控器件，接收到图像采集装置的惯导信息，获取旋转角度参数。同时，DSP会判断是否接收到后续处理步骤的出错反馈信号。 若检测到出错反馈信号，DSP会对整个FPGA处理板进行复位操作。根据图像的旋转角度参数来控制FPGA是否调用旋转ASIC芯片进行图像的旋转。若需要旋转则跳到步骤(3)，若不需要旋转则跳到步骤(4)
(3)DSP调用FPGA调用旋转ASIC芯片，对图像进行旋转操作，并将经旋转处理完成后的图像数据写往外部的DPRAM，同时FPGA读取DPRAM里面经过旋转处理后的图像，并进行校验，若旋转结果正确，跳到步骤(4)；当旋转芯片无法正常工作旋转结果不正确时，例如：旋转后的图像出现错帧、半帧或者图像有明显的错误信息等，此时FPGA就会向DSP发送一个出错反馈信号，跳到步骤(2)
(4)若旋转ASIC对图像进行旋转操作成功，此时FPGA对旋转处理后的图像进行预处理，例如进行降采样并边缘检测；若之前图像不需要旋转操作，此时FPGA对缓存的原始图像进行预处理。
(5)DSP调用FPGA调用多级滤波ASIC对步骤(4)中经过预处理之后的图像进行多级滤波处理。多级滤波ASIC处理完后的图像写往外部的DPRAM，同时FPGA读取DPRAM里面经过多级滤波处理后的图像，然后FPGA将经过多级滤波处理后的图像数据进行缓存并传送到DSP中，当图像数据过大时，可以将数据存储到DSP的存储芯片DDR2中。
(6)对于步骤(5)中经过多级滤波处理后的图像，根据不同的应用场景的需求，DSP执行不同的算法对图像进行非线性分割处理。非线性分割处理后的图像由DSP经过FPGA做形态学滤波处理。
(7)DSP调用FPGA，将步骤(6)中非线性分割、形态学滤波后的处理图像传送到轮廓跟踪与标记ASIC芯片，进行轮廓跟踪与标记处理，标记完后的图像缓存到FPGA并传送给DSP；
(8)DSP获取标记完后的图像的处理结果及几何特征，并调用模式分类器进行后续目标识别分类的处理。最后将处理完的图像经过图像显示单元传送到PAL视频输出进行显示。
为了保证数据的有效处理，保证流水线操作的有序执行。该处理***使用了FPGA内部的FIFO资源以及DPRAM资源，还使用了两个外部的DPRAM存储芯片，通过对数据的缓存等处理主要解决了数据的同步问题。
例如：如图5所示，当第一帧图像数据旋转完成进行FPGA的图像预处理时，此时第二帧图像便进入到旋转芯片进行旋转，由于FPGA的处理速度较快，第一帧图像预处理完成时，第二帧图像还未旋转完成，此时就应当对图像数据进行缓存，待旋转完成后再进行FPGA的图像预处理。即不同帧的步骤(3)和步骤(4)可以并行，并且可以通过缓存的方式保证处理时间同步。因此，FIFO和DPRAM的使用在流水线的操作中起着至关重要的作用。同时，FPGA的各个独立模块都是并行执行的，例如：当第一帧图像进行多级滤波的时第二帧图像正在进行旋转。
此外，该***采用FIFO构建的交换式网络构成形式如图4所示，其中在每个输入FIFO的选择端可用使能信号选择其中的一个三态门接通，即对每个输入FIFO来说可以选择数据来自哪个输出接口，避免冲突。同时，可实现一个输出接口的数据同时传输到多个输入FIFO中，实现数据的广播发送。这样的交换式网络减少了需要占用大量存储单元的FIFO存储器，但是同样实现了各处理部件间的全互联。可以实现在任务过程中，数据流根据算法的需要进行改变，重构算法的数据流。
最后，对处理结果图像序列进行人工标定，统计被正确识别目标的图像帧数n，设测试图像帧数为N，则正确识别概率η＝n/N，结果如表1所示。
表1
Figure PCTCN2017077106-appb-000001
基于以上测试方法，对建筑物、机场跑道上的飞机、舰船和装甲车辆等四类目标的序列图进行测试。它们的原始红外图像如图6(a)～(d)，目标识别后的结果图如图7(a)～(d)所示。如图8所示，(a)是图像采集装置成像后的原始人手的红外图像，手掌轮廓模糊(b)是经过校正SOC后的图像；如图9所示，(a)是图像采集装置成像后的原始建筑物的红外图像，(b)是经过旋转ASIC后的图像；图10所示，(a)是图像采集装置成像后的原始舰船的红外图像，(b)是经过多级滤波ASIC后的图像。
本发明将某些特定的算法(例如：旋转算法、连通区域标记算法，小目标检测等)用专用集成芯片来实现，这样一来不仅减轻了DSP的算法复杂度，而且这些不同的异构器件能够协调并行工作，大大地提高了目标检测识别的实时性。
专用集成芯片(ASIC)具有处理效率高、功耗低的特点。专门针对图像处理算法应用而设计的ASIC可以在很低的功耗下，快速高效地完成相应的图像处理算法。在目标成像自动目标识别***中采用专门的图像处理ASIC可以提高***的实时性，并降低***的功耗。
本发明的创新点体现在以下几个方面：
(1)将复杂算法进行分解，由DSP作为主控芯片，合理的通过FPGA来调用专用集成芯片来实现某些特定的算法功能；同时DSP与FPGA之间的通信互联，可以及时的反馈各个芯片是否正常工作，DSP由此发出指令来终止或者继续***的工作。这样一来，就可以避免由前一个芯片无法正常工作而导致的错误蔓延。本发明将完成图像目标识别所需要的复杂算法分解成一个个的基本计算模块，例如：用旋转ASIC芯片来实现旋转算法、用多级滤波ASIC芯片来实现多级滤波算法、用FPGA预处理模块来实现形态学滤波、用标记ASIC来完成对分割后图像的标记等，最终的特征匹配算法由DSP来实现，如图2所示。这样合理的分工以及协调调度既保证了***的实时性，又降低了***的整体功耗，同时降低了对DSP和FPGA等处 理芯片对运算能力的要求。
(2)高低速、多总线数据同步传输设计。本发明中，由DSP作为主控芯片的实时目标识别的异构处理***中，由于FPGA、DSP、ASICs等器件具有不同的工作时钟，同时ASIC作为专用的处理器，输入输出接口和通用的处理器DSP和FPGA相比有很大的差异，不同ASIC之间输入输出、位宽等差异也很大。例如：旋转ASIC输入的是图像数据，输出是图处理后的像数据，图像数据的位宽8bit-14bit，而标记ASIC输入的是二值(多值)分割后的图像数据，输出是特征(如目标的轮廓信息和坐标等)，图像数据的位宽3bit。此外，ASIC的输入输出接口缓存也存在着差异，例如：标记ASIC不需要外接存储芯片对数据进行存储，而旋转ASIC和多级滤波ASIC则需要外接存储芯片对数据进行存储。这些差异就导致不同ASIC之间通过FPGA进行互联以及不同ASIC与DSP以及FPGA之间的互联设计是一个设计的难点。
不同的处理器之间高效的互联，对提高整个***的性能有着重要的意义。逻辑层与物理层之间进行分隔、逻辑层的动态互联结构是解决异构互联的一个有效方法。物理层和逻辑层根据需求不同选择不同的接口，例如：板间连接使用LVDS接口，采用串并转换后和逻辑层相连；板内连接使用同步存储器接口，如：FIFO、DPRAM等。缓存结构保证不同处理器之间处理速度差异造成的异步数据缓存问题。
(3)流水线操作与并行操作相结合。由DSP作为主控芯片，合理的通过FPGA来调用专用集成芯片来实现某些特定的算法功能的关键就是适时调用各个模块，对图像的处理采用流水线操作，图像数据依次有序地经过各个处理单元处理，各处理单元协同工作，以保证每一个时刻都有多帧图像在流水线上被有序地处理。这种异构器件的流水线结合并行操作的特点，大大地提高了***的实时性。
本领域的技术人员容易理解，以上所述仅为本发明的较佳实施例而已，并不用以限制本发明，凡在本发明的精神和原则之内所作的任何修改、等同替换和改进等，均应包含在本发明的保护范围之内。

Claims (5)

  1. 一种面向实时目标识别的异构处理机并行协调处理方法，其特征在于，包括如下步骤：
    (1)序列图像输入到FPGA中缓存；图像采集装置采集目标图像的惯导信息并输入到DSP；
    (2)DSP接收惯导信息，从中获取旋转角度参数；同时，DSP判断是否接收到后续处理步骤的出错反馈信号；若接收到出错反馈信号，DSP对FPGA进行复位，并跳到步骤(3)；若没有接收到出错反馈信号，DSP根据图像的旋转角度参数来判断是否需要控制FPGA调用旋转ASIC芯片进行图像的旋转；若需要进行图像旋转则跳到步骤(3)，若不需要旋转则跳到步骤(4)；
    (3)DSP调用FPGA调用旋转ASIC芯片，对图像进行旋转操作，并将旋转处理完成后的图像数据写往外部的DPRAM，同时FPGA读取DPRAM里面经过旋转处理后的图像，进行校验，若旋转结果正确，跳到步骤(4)；当旋转芯片无法正常工作或旋转结果不正确时，跳到步骤(2)
    (4)若旋转ASIC对图像进行旋转操作成功，FPGA对旋转处理后的图像进行预处理；若之前图像不需要旋转操作，FPGA对缓存的原始图像进行预处理；
    (5)DSP调用FPGA调用多级滤波ASIC对步骤(4)中经过预处理之后的图像进行多级滤波处理；多级滤波ASIC处理完后的图像写往外部的DPRAM，同时FPGA读取DPRAM里面经过多级滤波处理后的图像，然后FPGA将经过多级滤波处理后的图像数据进行缓存并传送到DSP中；
    (6)DSP对步骤(5)中经过多级滤波处理后的图像进行非线性分割处理；非线性分割处理后的图像由DSP调用FPGA进行形态学滤波处理；
    (7)DSP调用FPGA，将步骤(6)中经非线性分割、形态学滤波处理 后的图像传送到轮廓跟踪与标记ASIC芯片，进行轮廓跟踪与标记处理，标记完后的图像缓存到FPGA并传送给DSP；
    (8)DSP获取标记完后的图像的处理结果及几何特征，并调用模式分类器进行后续目标分类；最后将处理完的图像传送到PAL视频输出进行显示，即目标的实时识别与跟踪结果。
  2. 如权利要求1所述的一种面向实时目标识别的异构处理机并行协调处理方法，其特征在于，使用FPGA内部的FIFO资源以及DPRAM资源，还使用两个外部的DPRAM存储芯片，通过对数据的缓存使得在处理某一帧图片的某一步骤的同时，还能够并行处理其他帧的其他步骤。
  3. 如权利要求2所述的一种面向实时目标识别的异构处理机并行协调处理方法，其特征在于，所述并行处理的方法如下：
    当第一帧图像数据旋转完成，进行FPGA的图像预处理时，第二帧图像便进入到旋转芯片进行旋转；由于FPGA的处理速度较快，第一帧图像预处理完成时，第二帧图像还未旋转完成，此时对第一帧图像数据进行缓存，待第二帧图像旋转完成后再由FPGA进行第二帧图像的预处理；当FPGA进行第二帧图像的预处理时，FPGA同时读取第一帧图像的缓存数据进行下一步处理。
  4. 如权利要求2所述的一种面向实时目标识别的异构处理机并行协调处理方法，其特征在于，FPGA的各个独立模块都是并行执行的。
  5. 如权利要求1～4任意一项所述的一种面向实时目标识别的异构处理机并行协调处理方法，其特征在于，FPGA、DSP、各ASIC之间均通过FIFO构建的交换式网络进行数据传输，每个输入FIFO的选择端均用使能信号选择其中的一个三态门接通，以实现一个输出接口的数据同时传输到多个输入FIFO中。
PCT/CN2017/077106 2016-12-31 2017-03-17 一种面向实时目标识别的异构处理机并行协调处理方法 WO2018120446A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201611268937.2 2016-12-31
CN201611268937.2A CN106686278A (zh) 2016-12-31 2016-12-31 一种面向实时目标识别的异构处理机并行协调处理方法

Publications (1)

Publication Number Publication Date
WO2018120446A1 true WO2018120446A1 (zh) 2018-07-05

Family

ID=58848883

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2017/077106 WO2018120446A1 (zh) 2016-12-31 2017-03-17 一种面向实时目标识别的异构处理机并行协调处理方法

Country Status (2)

Country Link
CN (1) CN106686278A (zh)
WO (1) WO2018120446A1 (zh)

Cited By (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109785220A (zh) * 2018-12-30 2019-05-21 武汉极目智能技术有限公司 基于fpga的单目摄像头图像多通道采集装置和方法
CN110275842A (zh) * 2018-07-09 2019-09-24 西北工业大学 基于fpga的高光谱目标跟踪***及方法
CN110825687A (zh) * 2019-10-25 2020-02-21 天津津航技术物理研究所 一种基于dsp多核架构双模跟踪方法
CN110992239A (zh) * 2019-11-14 2020-04-10 中国航空工业集团公司洛阳电光设备研究所 一种基于单片ddr3芯片的图像时域滤波与显示的方法
CN111260553A (zh) * 2020-01-13 2020-06-09 哈尔滨工程大学 一种基于远距离无损视频传输的国产视觉计算***
CN112330524A (zh) * 2020-10-26 2021-02-05 沈阳上博智像科技有限公司 一种在图像跟踪***中快速实现卷积的装置和方法
CN112667717A (zh) * 2020-12-23 2021-04-16 贵州电网有限责任公司电力科学研究院 变电站巡检信息处理方法、装置、计算机设备和存储介质
CN112750066A (zh) * 2019-10-31 2021-05-04 华中科技大学 一种用于图像目标检测的可扩展协处理器架构
CN114691579A (zh) * 2022-04-15 2022-07-01 南京邮电大学 一种室内无人车的异构处理器及其通信方法
CN114820630A (zh) * 2022-07-04 2022-07-29 国网浙江省电力有限公司电力科学研究院 一种基于fpga的目标跟踪算法模型管道加速方法以及电路

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114170553A (zh) * 2021-12-09 2022-03-11 北京字节跳动网络技术有限公司 图像处理方法、装置和电子设备

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8582805B2 (en) * 2007-11-05 2013-11-12 California Institute Of Technology Synthetic foveal imaging technology
CN103544470A (zh) * 2013-08-05 2014-01-29 华中科技大学 一种动平台对空目标双色红外异构并行自动目标识别器
CN103544471A (zh) * 2013-08-05 2014-01-29 华中科技大学 一种动平台对地固定目标异构并行自动目标识别器

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7954114B2 (en) * 2006-01-26 2011-05-31 Exegy Incorporated Firmware socket module for FPGA-based pipeline processing
CN103646232B (zh) * 2013-09-30 2016-08-17 华中科技大学 飞行器地面运动目标红外图像识别装置

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8582805B2 (en) * 2007-11-05 2013-11-12 California Institute Of Technology Synthetic foveal imaging technology
CN103544470A (zh) * 2013-08-05 2014-01-29 华中科技大学 一种动平台对空目标双色红外异构并行自动目标识别器
CN103544471A (zh) * 2013-08-05 2014-01-29 华中科技大学 一种动平台对地固定目标异构并行自动目标识别器

Cited By (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110275842B (zh) * 2018-07-09 2022-10-21 西北工业大学 基于fpga的高光谱目标跟踪***及方法
CN110275842A (zh) * 2018-07-09 2019-09-24 西北工业大学 基于fpga的高光谱目标跟踪***及方法
CN109785220A (zh) * 2018-12-30 2019-05-21 武汉极目智能技术有限公司 基于fpga的单目摄像头图像多通道采集装置和方法
CN109785220B (zh) * 2018-12-30 2023-04-07 武汉极目智能技术有限公司 基于fpga的单目摄像头图像多通道采集装置和方法
CN110825687A (zh) * 2019-10-25 2020-02-21 天津津航技术物理研究所 一种基于dsp多核架构双模跟踪方法
CN110825687B (zh) * 2019-10-25 2024-01-05 天津津航技术物理研究所 一种基于dsp多核架构双模跟踪方法
CN112750066A (zh) * 2019-10-31 2021-05-04 华中科技大学 一种用于图像目标检测的可扩展协处理器架构
CN110992239A (zh) * 2019-11-14 2020-04-10 中国航空工业集团公司洛阳电光设备研究所 一种基于单片ddr3芯片的图像时域滤波与显示的方法
CN110992239B (zh) * 2019-11-14 2023-03-24 中国航空工业集团公司洛阳电光设备研究所 一种基于单片ddr3芯片的图像时域滤波与显示的方法
CN111260553A (zh) * 2020-01-13 2020-06-09 哈尔滨工程大学 一种基于远距离无损视频传输的国产视觉计算***
CN112330524A (zh) * 2020-10-26 2021-02-05 沈阳上博智像科技有限公司 一种在图像跟踪***中快速实现卷积的装置和方法
CN112667717A (zh) * 2020-12-23 2021-04-16 贵州电网有限责任公司电力科学研究院 变电站巡检信息处理方法、装置、计算机设备和存储介质
CN112667717B (zh) * 2020-12-23 2023-04-07 贵州电网有限责任公司电力科学研究院 变电站巡检信息处理方法、装置、计算机设备和存储介质
CN114691579A (zh) * 2022-04-15 2022-07-01 南京邮电大学 一种室内无人车的异构处理器及其通信方法
CN114691579B (zh) * 2022-04-15 2023-11-10 南京邮电大学 一种室内无人车的异构处理器及其通信方法
CN114820630B (zh) * 2022-07-04 2022-09-06 国网浙江省电力有限公司电力科学研究院 一种基于fpga的目标跟踪算法模型管道加速方法以及电路
CN114820630A (zh) * 2022-07-04 2022-07-29 国网浙江省电力有限公司电力科学研究院 一种基于fpga的目标跟踪算法模型管道加速方法以及电路

Also Published As

Publication number Publication date
CN106686278A (zh) 2017-05-17

Similar Documents

Publication Publication Date Title
WO2018120446A1 (zh) 一种面向实时目标识别的异构处理机并行协调处理方法
CN110176054B (zh) 用于训练神经网络模型的合成图像的生成
CN110046537B (zh) 使用递归神经网络进行动态面部分析的***和方法
US10521928B2 (en) Real-time gesture recognition method and apparatus
US11954904B2 (en) Real-time gesture recognition method and apparatus
US20190114546A1 (en) Refining labeling of time-associated data
CN108734194B (zh) 一种面向虚拟现实的基于单深度图的人体关节点识别方法
WO2015043363A1 (zh) 飞行器地面运动目标红外图像识别装置
CN106980477B (zh) 一种卫星序列图像中点目标实时检测与跟踪***及方法
WO2017113540A1 (zh) 一种气动光学效应校正识别一体化实时处理的***和方法
CN114092665A (zh) 从视频重建三维对象
CN104751421B (zh) 一种在fpga上实现的图像去雾方法
CN105631870A (zh) 基于NoC架构的星图配准与目标轨迹提取方法及装置
US20220198622A1 (en) High dynamic range support for legacy applications
CN109919825B (zh) 一种orb-slam硬件加速器
US10861243B1 (en) Context-sensitive augmented reality
CN108171734A (zh) 一种orb特征提取与匹配的方法及装置
CN105915804A (zh) 视频拼接方法及***
CN112836597A (zh) 基于级联并行卷积神经网络的多手姿态关键点估计方法
CN113392968A (zh) 针对神经网络的迭代式小样本精细化的微训练
Wan et al. An energy-efficient quad-camera visual system for autonomous machines on fpga platform
CN114647527A (zh) 在基于云的流式应用程序中测量和检测空闲处理时段并确定其根本原因
CN104567865B (zh) 一种星敏感器空间粒子干扰条件下的姿态捕获方法
CN116433580A (zh) 一种基于智能无人机的大坝裂缝检测方法
CN103544471B (zh) 一种动平台对地固定目标异构并行自动目标识别器

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 17886936

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 17886936

Country of ref document: EP

Kind code of ref document: A1