CN113890977B - Airborne video processing device and unmanned aerial vehicle with same - Google Patents
Airborne video processing device and unmanned aerial vehicle with same Download PDFInfo
- Publication number
- CN113890977B CN113890977B CN202111193580.7A CN202111193580A CN113890977B CN 113890977 B CN113890977 B CN 113890977B CN 202111193580 A CN202111193580 A CN 202111193580A CN 113890977 B CN113890977 B CN 113890977B
- Authority
- CN
- China
- Prior art keywords
- video
- video processing
- data
- fpga
- processing device
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000012545 processing Methods 0.000 title claims abstract description 122
- 238000001514 detection method Methods 0.000 claims abstract description 27
- 230000006835 compression Effects 0.000 claims abstract description 20
- 238000007906 compression Methods 0.000 claims abstract description 20
- 230000015654 memory Effects 0.000 claims abstract description 17
- 238000006243 chemical reaction Methods 0.000 claims abstract description 15
- 230000002452 interceptive effect Effects 0.000 claims abstract description 4
- 238000004891 communication Methods 0.000 claims description 18
- RZVHIXYEVGDQDX-UHFFFAOYSA-N 9,10-anthraquinone Chemical compound C1=CC=C2C(=O)C3=CC=CC=C3C(=O)C2=C1 RZVHIXYEVGDQDX-UHFFFAOYSA-N 0.000 claims description 8
- 230000001360 synchronised effect Effects 0.000 claims description 8
- 238000001914 filtration Methods 0.000 claims description 6
- 230000001419 dependent effect Effects 0.000 claims description 4
- 230000005693 optoelectronics Effects 0.000 claims description 3
- 230000001133 acceleration Effects 0.000 claims description 2
- 238000007781 pre-processing Methods 0.000 claims description 2
- 230000006870 function Effects 0.000 abstract description 49
- 238000013461 design Methods 0.000 abstract description 7
- 230000005540 biological transmission Effects 0.000 description 15
- 238000000034 method Methods 0.000 description 14
- 230000010354 integration Effects 0.000 description 11
- 238000010586 diagram Methods 0.000 description 7
- 230000003287 optical effect Effects 0.000 description 5
- 238000012634 optical imaging Methods 0.000 description 5
- 230000008569 process Effects 0.000 description 4
- 238000012549 training Methods 0.000 description 4
- 238000012360 testing method Methods 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 2
- 238000003384 imaging method Methods 0.000 description 2
- 230000004075 alteration Effects 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000001934 delay Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 210000001503 joint Anatomy 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000004806 packaging method and process Methods 0.000 description 1
- 230000000737 periodic effect Effects 0.000 description 1
- 230000006340 racemization Effects 0.000 description 1
- 238000010008 shearing Methods 0.000 description 1
- 230000000087 stabilizing effect Effects 0.000 description 1
Abstract
The invention provides an airborne video processing device and an unmanned aerial vehicle with the same, wherein the airborne video processing device comprises an FPGA and a video processing chip which are integrally arranged, the FPGA receives visible light video signals and thermal infrared imager video signals to obtain video data after format conversion, the video data are transmitted to the video processing chip through an interactive interface, the video processing chip encodes the video data according to a preset protocol and then stores the encoded video data in a local memory, and the encoded video data are transmitted to ground receiving equipment through a wireless data link, or the video processing chip transmits the encoded video data to the FPGA, and the FPGA transmits the encoded video data to the ground receiving equipment through a wireless data link. The on-board video processing device adopts an FPGA+SOC hardware architecture, can realize the functions of video compression storage, multi-target detection, target tracking, target speed prediction and the like, reserves a plurality of interfaces, realizes integrated, multifunctional and miniaturized design, and meets the actual requirements of unmanned aerial vehicles on video processing functions.
Description
Technical Field
The invention relates to the technical field of video processing, in particular to an onboard video processing device and an unmanned aerial vehicle with the same.
Background
At present, unmanned aerial vehicles are increasingly widely applied in the fields of military, police and civil use, and when an unmanned aerial vehicle-mounted photoelectric pod performs target reconnaissance, detection, identification and tracking, acquired target video images are required to be transmitted to a ground station in real time through wireless communication link equipment so as to enable operators to grasp the target state and the surrounding environment in a flight area in real time to make decisions; meanwhile, the acquired target video image is compressed and stored on the onboard photoelectric equipment for further analysis after the fact.
The limitation of wireless link bandwidth, for the target image collected by optical loads such as a high-definition camera, a thermal infrared imager and the like, the data volume of hundreds megabits per second or even the data volume of the upper gigabit (G) bit cannot be directly transmitted through the wireless link, and the video data must be encoded and compressed so as to ensure the real-time performance of a transmission picture and the normal operation of a system; meanwhile, as the external interfaces of each wireless link device are different, the compression coding system is required to be configured with different external interfaces so as to improve the adaptability of the photoelectric pod device; in addition, the weight and the power consumption of the photoelectric pod are very critical technical indexes for the unmanned aerial vehicle, and the endurance time of the unmanned aerial vehicle is directly influenced.
The method is limited by the calculation running speed of a core processing chip and the resources of a processor, the general photoelectric pod compression coding function and the equipment target detection and tracking function are discrete modules, namely, the image compression coding function, the video storage function, the target detection and identification tracking function are respectively independent processing modules, the original video and the tracking video overlapped with the target state information are required to be transmitted through corresponding video interfaces, the integration degree is low, the size and the weight of the equipment are increased, the delay of data transmission among the circuit modules is increased, and the power consumption and the cost of the equipment are increased.
Disclosure of Invention
The invention provides an airborne video processing device and an unmanned aerial vehicle with the same, and aims to solve the technical problem of realizing the light weight and integrated design of the airborne video processing device.
The airborne video processing device comprises an FPGA and a video processing chip which are integrally arranged, wherein the FPGA receives a visible light video signal and a thermal infrared imager video signal to obtain video data after format conversion, the video data is transmitted to the video processing chip through an interactive interface, the video processing chip encodes the video data according to a preset protocol and stores the encoded video data in a local memory, and the encoded video data is transmitted to ground receiving equipment through a wireless data link, or the video processing chip transmits the encoded video data to the FPGA, and the FPGA transmits the encoded video data to the ground receiving equipment through the wireless data link.
According to some embodiments of the invention, the on-board video processing device is further provided integrally with: the system comprises an SDI decoding chip and an A/D decoding chip, wherein the SDI decoding chip is used for decoding a received visible photoelectric video signal and then transmitting the video decoded signal to the FPGA, and the A/D decoding chip is used for decoding a received thermal infrared imager video signal and then transmitting the decoded signal to the FPGA.
In some embodiments of the invention, the video processing chip has: a video input module, a video processing module and a video coding module,
The FPGA converts the received decoded visible light television video signal into a BT.1120 time sequence, and converts the received decoded infrared thermal imager video signal into a BT.656 time sequence, the BT.656 time sequence is input into the video input module to obtain image data, the image data is transmitted to the video processing module to be preprocessed to obtain video processing data, and the video encoding module encodes the video processing data according to an H.264/H.265 protocol.
According to some embodiments of the present invention, the video processing chip adopts an online mode to directly transmit the video data stream of the video input module to the video processing module, and the video processing module uses a mode of transmitting while collecting to transmit the video processing data to the video encoding module for encoding in a row unit.
In some embodiments of the present invention, the video processing chip transmits the encoded video data to the wireless data link through an ethernet port or to the FPGA through a PCIe expansion bus.
According to some embodiments of the present invention, when the video processing module encodes, the video data, the aircraft data and the payload status data are formed into a video data stream according to a preset protocol, and stored in a local memory, and the FPGA reads the video data stream in the local memory through a PCIe interface, performs parallel-serial conversion, and then sends a clock and drives to output according to a set code stream.
In some embodiments of the present invention, the FPGA has a multi-way asynchronous communication 422 interface for enabling communication with the carrier flight control, inertial navigation, servo systems and host computers.
According to some embodiments of the invention, the video processing chip has a target tracking module that performs target tracking using a kernel-dependent filtering algorithm based on directional gradient histogram features.
In some embodiments of the invention, the video processing chip employs Hi3519A or Hi35559A.
According to the unmanned aerial vehicle provided by the embodiment of the invention, the unmanned aerial vehicle is provided with the airborne photoelectric pod, and the airborne photoelectric pod adopts the airborne video processing device to carry out video compression storage, multi-target detection, target tracking and target speed prediction.
The airborne video processing device provided by the invention has the following advantages:
The on-board video processing device adopts an FPGA+SOC (Hua Hai Si Hi 3519A) hardware architecture, combines the respective performance and functional advantages of the FPGA and the Hi3519A, realizes a video multifunctional processing module, reserves a plurality of hardware input and output interfaces, and meets the actual requirements of the current unmanned aerial vehicle on video processing functions; the functions of video compression storage, multi-target detection, target tracking, target speed prediction and the like can be realized on software, the functions of modules are enriched, and integration, multifunction and miniaturization are realized; the KCF target tracking algorithm based on HOG characteristics is adopted, so that the long-time target tracking capacity and tracking precision of the photoelectric pod are improved; the design method is easy to expand functions, improves the integration and miniaturization levels of equipment, has more abundant module functions and has stronger adaptability; the method is suitable for realizing video transmission and control communication of photoelectric pod equipment of the current unmanned aerial vehicle or tethered unmanned aerial vehicle, and meets the practical application of the system.
Drawings
Fig. 1 is a schematic diagram of an on-board video processing device according to an embodiment of the present invention;
fig. 2 is a schematic diagram of hardware components of an on-board video processing device according to an embodiment of the present invention;
Fig. 3 is a schematic diagram of a video encoding flow of an on-board video processing device according to an embodiment of the invention;
fig. 4 is a schematic diagram of a high-speed synchronization 422 data transmission flow of an on-board video processing device according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of a functional architecture of a Hai Si processor of an on-board video processing device according to an embodiment of the present invention;
FIG. 6 is a schematic front view of an on-board video processing device according to an embodiment of the present invention;
fig. 7 is a schematic diagram of a real back of an on-board video processing device according to an embodiment of the invention.
Detailed Description
In order to further describe the technical means and effects adopted by the present invention for achieving the intended purpose, the following detailed description of the present invention is given with reference to the accompanying drawings and preferred embodiments.
The unmanned aerial vehicle-mounted photoelectric pod is provided with a visible light television and a thermal infrared imager imaging sensor, and the visible light television and the thermal infrared imager imaging sensor are respectively an SDI digital video interface and a PAL analog video interface. When target detection, identification and tracking are carried out, the acquired target video image is required to be transmitted to a ground station in real time through wireless communication link equipment, so that operators can grasp the target state in real time and make decisions; meanwhile, the acquired target video image is required to be compressed and stored on the photoelectric pod equipment body for further analysis after the fact. The method is limited by the bandwidth of a wireless communication link, and for target images acquired by optical loads such as a high-definition camera, a thermal infrared imager and the like, data volume of hundreds of megabits per second or even kilomega bit order cannot be directly transmitted through the wireless link, and video data must be compressed and encoded; meanwhile, as the video interface forms of each wireless link device are different, the compression coding system is required to be configured with different external interfaces so as to improve the adaptability of the photoelectric pod device; in addition, the weight and the power consumption of the photoelectric pod are very critical technical indexes for the unmanned aerial vehicle, and the endurance time of the unmanned aerial vehicle is directly influenced.
In the prior art, the core processing chip calculates the running speed and limits the processor resources, the general photoelectric pod compression coding function and the equipment target detection and tracking function are discrete modules, namely, the image compression coding function, the video storage function and the target detection and identification tracking are respectively independent processing modules, the original video and the tracking video overlapped with the target state information are required to be transmitted through corresponding video interfaces, the integration degree is low, the size and the weight of the equipment are increased, the data transmission delay among the circuit modules is increased, and the power consumption and the cost of the equipment are increased.
The disadvantages of the prior art are as follows:
The image processing module has single function, only has compression coding and storage functions, and has no target detection, identification and tracking functions; the number of external interfaces is small, and the adaptability of the carrier is not strong; the integration degree is not high, and the hardware cost, the power consumption, the volume, the weight and the like are increased.
Aiming at the defects, the invention aims to provide the multifunctional, lightweight, miniaturized and highly integrated image processing module which is applied to the photoelectric pod equipment, has rich external hardware interface functions and software functions, has video compression and encoding functions on the premise of realizing target reconnaissance, detection and identification tracking functions, reduces the volume, power consumption and cost of the equipment, and solves the actual requirements of unmanned aerial vehicles on the photoelectric pod.
The airborne video processing device comprises an FPGA and a video processing chip which are integrally arranged, wherein the FPGA receives a visible light video signal and a thermal infrared imager video signal to obtain video data after format conversion, the video data is transmitted to the video processing chip through an interactive interface, the video processing chip encodes the video data according to a preset protocol and stores the encoded video data in a local memory, and the encoded video data is transmitted to ground receiving equipment through a wireless data link, or the video processing chip transmits the encoded video data to the FPGA, and the FPGA transmits the encoded video data to the ground receiving equipment through the wireless data link.
According to some embodiments of the invention, the on-board video processing device is further integrally provided with: the system comprises an SDI decoding chip and an A/D decoding chip, wherein the SDI decoding chip is used for decoding a received visible photoelectric video signal and then transmitting the video decoded signal to the FPGA, and the A/D decoding chip is used for decoding a received thermal infrared imager video signal and then transmitting the decoded signal to the FPGA.
In some embodiments of the invention, a video processing chip has: a video input module, a video processing module and a video coding module,
The FPGA converts the received decoded visible light video signal into a BT.1120 time sequence, converts the received decoded video signal of the thermal infrared imager into a BT.656 time sequence, inputs the image data into a video input module, and transmits the image data to a video processing module for preprocessing to obtain video processing data, and a video encoding module encodes the video processing data according to an H.264/H.265 protocol.
According to some embodiments of the present invention, the video processing chip adopts an on-line mode to directly transmit the video data stream of the video input module to the video processing module, and the video processing module transmits the video processing data to the video encoding module for encoding in a line unit in a manner of collecting and transmitting.
In some embodiments of the present invention, the video processing chip transmits the encoded video data to the wireless data link via the ethernet port or to the FPGA via the PCIe expansion bus.
According to some embodiments of the present invention, when the video processing module encodes, the video data, the aircraft data and the payload status data are formed into a video data stream according to a preset protocol, and stored in the local memory, and the FPGA reads the video data stream in the local memory through the PCIe interface, performs parallel-serial conversion, and then sends a clock and drives to output according to a set code stream.
In some embodiments of the present invention, the FPGA has a multi-way asynchronous communication 422 interface for enabling communication with the carrier flight control, inertial navigation, servo systems and host computers.
According to some embodiments of the invention, a video processing chip has a target tracking module that performs target tracking using a kernel-dependent filtering algorithm based on directional gradient histogram features.
In some embodiments of the invention, the video processing chip employs Hi3519A or Hi35559A.
According to the unmanned aerial vehicle provided by the embodiment of the invention, the unmanned aerial vehicle is provided with the airborne photoelectric pod, and the airborne photoelectric pod adopts the airborne video processing device to carry out video compression storage, multi-target detection, target tracking and target speed prediction.
The airborne video processing device provided by the invention has the following advantages:
The on-board video processing device adopts an FPGA+SOC (Hua Hai Si Hi 3519A) hardware architecture, combines the respective performance and functional advantages of the FPGA and the Hi3519A, realizes a video multifunctional processing module, reserves a plurality of hardware input and output interfaces, and meets the actual requirements of the current unmanned aerial vehicle on video processing functions; the functions of video compression storage, multi-target detection, target tracking, target speed prediction and the like can be realized on software, the functions of modules are enriched, and integration, multifunction and miniaturization are realized; the KCF target tracking algorithm based on HOG characteristics is adopted, so that the long-time target tracking capacity and tracking precision of the photoelectric pod are improved; the design method is easy to expand functions, improves the integration and miniaturization levels of equipment, has more abundant module functions and has stronger adaptability; the method is suitable for realizing video transmission and control communication of photoelectric pod equipment of the current unmanned aerial vehicle or tethered unmanned aerial vehicle, and meets the practical application of the system.
The on-board video processing device according to the present invention is described in detail below with reference to the accompanying drawings. It is to be understood that the following description is exemplary only and is not to be taken as limiting the invention in any way.
Fig. 1 shows a block diagram of a two-way video compression and storage system. An FPGA+SOC (Hua Hai Si Hi 3519A) architecture is adopted. The two-way video signals of the visible light television and the thermal infrared imager are transmitted to the FPGA through a special decoding chip, the FPGA carries out format conversion On 2 paths of videos, the 2 paths of videos are transmitted to the Hi3519A SOC through a BT.1120 interface mode, the Hi3519A can carry out OSD (On-SCREEN DISPLAY) information superposition according to actual needs, the OSD (On-SCREEN DISPLAY) information superposition is carried out according to H.264/H.265 protocol compression coding and packaging, one path of the OSD information is stored in a local memory, the other path of the OSD information is transmitted to a wireless data link through an Ethernet port On the SOC or transmitted to the FPGA through a PCIe expansion bus, the FPGA transmits received data to wireless data link equipment through a high-speed synchronous 422 interface (the selection of the synchronous 422 interface or the Ethernet port is dependent On the video interface form reserved by the wireless link equipment), the video is transmitted to a ground display control station in real time, and the ground station decodes according to the protocol after receiving the compressed video frequency, and video image display is restored. The module also has a plurality of asynchronous 422 interfaces, which can realize communication with an optoelectronic pod servo system, an optical sensor, a carrier inertial navigation, a flight control, an upper computer and the like.
1. And (3) inputting a video analysis module:
The video output by the visible light television is in the HD-SDI format of 1080p@30Hz, the onboard video processing device adopts a special SDI decoding chip (LMH 0387) to finish video decoding, and the decoded video effective data is output to the FPGA in a CMOS level mode along with F, V, H synchronous signals by 20 bits.
The thermal infrared imager outputs PAL system video data, the onboard video processing device adopts a special A/D chip (TW 9912) to complete video analog-digital conversion, and the video data is output to the FPGA along with HS, VS, PIXCLK in the form of 8bit data.
The FPGA analyzes and converts the format of the received video data and transmits the video data to the Haishu Hi3519A SOC processor in a BT.1120 mode.
2. Video compression coding module:
The FPGA converts 1 path of SDI video and 1 path of infrared video into BT.1120 and BT.656 time sequences respectively, inputs the BT.1120 and BT.656 time sequences into a Video Input (VI) module of Hi3519A, outputs processed image data to a Video Processing (VPSS) module through interface conversion, shearing, scaling and the like, uniformly pre-processes (denoising, de-interlacing and the like) the input image by a VPSS module, performs scaling, sharpening and the like, and finally outputs an image with set resolution to a video coding (VENC) module. The VENC module carries out H.264/H.265 coding and code stream control on the image data, and the coded data is sent to the FPGA to realize transmission. All modules in the whole process are realized by hardware modules in Hi3519A, and the whole coding flow can be realized only by calling and parameter setting through software programming.
In the functional module, the Hi3519A uses an online mode to directly transmit the video input VI data stream to the VPSS module, the VPSS module transmits the video data to the VENC module for encoding in a line unit by adopting a mode of collecting and transmitting at the same time, and thus, the data delay in the process of processing the complete frame image by the VPSS module and transmitting the complete frame image to the VENC module can be reduced.
3. Video data transmitting module:
The coded video data, the aircraft data and the load state data are packaged and then transmitted to a wireless data link through a high-speed synchronous 422 interface or an Ethernet interface to realize wireless transmission to a ground control station. When compression encoding is performed in Hi3519A, the video data, the aircraft data and the load state data are added with frame header and frame tail according to the protocol to form a complete video data stream, the complete video data stream is stored in the DDR memory, and the FPGA reads the data therein through the PCIe interface to perform parallel-serial conversion and then sends clock and drive output according to the set code stream, as shown in fig. 4. Likewise, the video data stream may also be transmitted to the ground station via the ethernet interface of the Hi3519A SOC.
The high-speed synchronous 422 sending module processes data according to 3 independent levels of frames, bytes and bits, and mainly comprises the following functions:
the processor communicates: the sync 422 controls the mapping of the slave device mode to the processor memory control through PCIe, supporting the processor to directly perform transmission and reception control of the data frame;
Frame processing: the transmitting end reads out the transmission data, adds the frame head and the frame tail to form a complete frame;
byte processing: according to the protocol, the transmitting end adds corresponding stuffing bytes between frames;
BIT processing: the transmitting end completes parallel-serial conversion from bytes to BIT, and clock and data output drive;
HDMI video output: the uncompressed mixed coded visible light and infrared two-way video stream can be output to the equipment of the corresponding interface.
4. The target tracking module realizes:
the target tracking module runs in a floating point processing unit (FPU) built in the Hi3519A SOC in an algorithm plug-in calling mode, and the CPU controls the calling time and data exchange of the module. The input data of the module is the memory address of the image data of the current frame of the original video, and the pixel error quantity and the tracking state of the target are output.
The target tracking module employs a kernel correlation filtering (Kernel Correlation Filter, KCF) algorithm based on directional gradient histogram (Histogram of Oriented Gradient, HOG) features. Compared with the gray scale feature, the HOG feature is adopted to endow the algorithm with robustness to geometric deformation, illumination brightness and color change, and the stability to a moving target can be improved by utilizing the texture structure. The KCF algorithm has relatively low operand and good instantaneity, and can avoid the problem of target tracking failure possibly caused by the conditions of rapid movement, deformation, shielding and the like of the target through the filter training in the initialization stage and the real-time correction and learning of the tracking result.
5. The target speed prediction module is used for realizing:
In order to improve the target tracking precision, link delays such as circuit D/A conversion, data frame access, off-target calculation and the like are reduced, and the on-board video processing device utilizes a target speed prediction module to introduce feedforward control for a servo system so as to compensate the time delay. The speed prediction module is operated in a DSP (digital signal processor) built in the Hi3519A SOC, and realizes speed prediction by reading a series of data such as azimuth pitch angle, azimuth pitch angle speed, target tracking error amount and the like of the photoelectric pod and adopting a Kalman filtering algorithm. And introducing the speed and acceleration signals of the signals at the input end of the speed loop channel to form feedforward control, and improving the servo performance so as to improve the target tracking precision.
6. The target detection module realizes:
the target detection module operates on a Neural Network Interface Engine (NNIE) built in the Hi3519A SOC, so that multi-target automatic detection and identification under a complex background can be realized. The adopted target detection model is YOLO V3, the model is realized in a single step, namely, the target detection problem is converted into a single regression problem of extracting the grouping boxes and the class probabilities from the image, and a single convolutional neural network is adopted to predict a plurality of grouping boxes and the class probabilities. Therefore, the model can achieve a fast detection speed and high accuracy.
The target detection module is realized by adopting an off-line training and on-line detection mode. And training a YOLO V3 model built by using the Caffe frame by using a required image library offline, converting model parameters into a special binary file by adopting Ruyi Studio development tools provided by Hai Si after training, and importing the binary file into an on-board storage space. When the device works, the CPU transmits the image to be detected into NNIE, the latter loads the imported model parameters, the target detection is completed in a hardware realization mode, and the information of the category, the confidence level, the position, the size and the like of a plurality of targets is output.
7. The communication module realizes:
the on-board video processing device can communicate with multiple devices or systems in real time through multiple asynchronous 422 serial ports extended by the FPGA.
1) And is communicated with the flight control system of the carrier. The module can send information such as the servo states of the azimuth, the pitching, the working mode and the like of the photoelectric pod equipment, the focal length of the optical sensor, the residual recording time of the storage module, the number of stored video clips and the like to the carrier flight control system; the carrier flight control transmits information such as the course angle, pitch angle, roll angle, longitude and latitude, altitude and the like of the airplane to the module.
2) And is communicated with the inertial navigation system of the carrier. The module can receive information such as course angle, pitch angle, roll angle, longitude and latitude, altitude and the like of the airplane sent by the inertial navigation system of the carrier, and the information can be used for racemization of electronic images and geographic tracking and positioning.
3) And is communicated with an electro-optic pod servo system. The module can receive servo state information, optical imaging sensor field angle and current working optical imaging sensor information and laser ranging working state information sent by a servo system; the module sends control instructions of photoelectric pod equipment, working mode, optical imaging sensor and laser range finder to a servo system.
4) And is communicated with an upper computer display control system. The module receives a servo control instruction of photoelectric pod equipment, an optical imaging sensor control instruction, a laser range finder control instruction and a video storage module control instruction which are sent by a display control system; the state information sent to the display control system by the module comprises a servo state of the optoelectronic pod device, a state of the optical imaging sensor, a state of the laser range finder and a working state of the video storage module.
8. Software functions:
Hi3519ASOC is a main control processor for completing video coding, storage, communication management and other functions. The FPGA realizes the processing of the video and the communication interface, completes the format conversion of the video and the data, and is in butt joint with the main control processor. Fig. 5 is a Hi3519A software functional architecture, based on a hardware platform, mainly including a system boot, a Linux kernel, a driving layer, and an application layer, and the basic functions implemented are as follows:
1) Video coding: finishing the coding of video data (1-path SDI+1-path PAL video), wherein the coding mode H.264/H.265 is configurable, and the code rate is configurable;
2) Video playback function: supporting 1-path video playback, namely reading recorded video files for playback;
3) OSD module: superimposing the attitude information of the carrier, the information of the carrier nacelle and the like into the video for coding, and realizing local storage and transmission;
4) Video file storage management function: the coded video can be stored in an electronic magnetic disk according to a corresponding format, and can be automatically covered (under the condition of full disk);
5) RS422 interface control instruction resolution: receiving and analyzing the control instruction, and realizing the functions of video recording, video playback, OSD superposition, synchronous transmission of packed data and the like according to the requirements;
6) BIT acquisition function: and reporting the state information of the storage system in a timing period, wherein the state information comprises the information of the remaining recording time, the number of the recorded files, the state of the storage system and the like.
7) Self-checking function: the power-on self-test can realize the initial state detection and the periodic self-test of the equipment, and the module timing period self-test can be realized, so that the reliability and the stability of the equipment are improved.
The software is based on Linux 2.6.38SMP to develop the package, has strong reliability and is convenient for upgrading, expanding and maintaining.
In summary, the on-board video processing device provided by the invention adopts an fpga+soc (huashi Hi 3519A) hardware architecture, designs a multifunctional video image processing module, reserves a corresponding synchronous high-speed 422 video output interface, an ethernet video output port and an HDMI video output port, reserves a multi-channel asynchronous communication 422 interface, can realize communication with an on-board flight control, inertial navigation, a servo system and an upper computer, improves the installation adaptability of the module, and meets the application requirements of wireless or tethered unmanned aerial vehicle wired video transmission and equipment control.
The on-board video processing device can realize the functions of video compression storage, multi-target detection, target tracking, target speed prediction and the like on software, enriches the functions of modules, and realizes integration, multifunction and miniaturization.
The onboard video processing device adopts a KCF target tracking algorithm based on HOG characteristics, and improves the long-time target tracking capability and tracking precision of the photoelectric pod.
The method for multi-target detection and target speed prediction is provided, and meanwhile, the module design method is easy to expand functions in hardware and software, for example, a plurality of RS422 interfaces can be expanded through an FPGA in hardware, so that direct control of more optical sensors is realized; the software can be added with functions of stabilizing aiming and the like, so that the integration and miniaturization level of equipment is improved, the functions of the modules are more abundant, and the adaptability is stronger.
The airborne video processing device provided by the invention has the following advantages:
1) The FPGA+SOC (Hua Hai Si Hi 3519A) hardware architecture is adopted, so that the respective performance and functional advantages of the FPGA and the Hi3519A are considered, a video multifunctional processing module is realized, a plurality of hardware input and output interfaces are reserved, and the actual requirement of the unmanned aerial vehicle on the video processing function is met.
2) The module software can realize functions of video compression storage, multi-target detection, target tracking, target speed prediction and the like, enriches the functions of the module, and realizes integration, multifunction and miniaturization.
3) The module adopts a KCF target tracking algorithm based on HOG characteristics, and improves the long-time target tracking capacity and tracking precision of the photoelectric pod.
4) The module design method is easy to expand functions, improves the integration and miniaturization levels of equipment, and is richer in module functions and stronger in adaptability.
5) The method is suitable for realizing video transmission and control communication of photoelectric pod equipment of the current unmanned aerial vehicle or tethered unmanned aerial vehicle, and meets the practical application of the system.
The following terms are used in the description of the application:
1、GTP transceiver：
GTP TRANSCEIVER is a transceiver with a linear speed of 500Mb/s to 6.6Gb/s in the FPGA, and can be flexibly configured by utilizing programmable resources in the FPGA, so that the transceiver is suitable for different requirements such as Ethernet, SATA1.0 interface and the like, and has the function of a physical layer of various high-speed serial interfaces.
2. HDMI interface:
the high definition multimedia interface is a full digital video and sound transmitting interface. It supports various television and computer video formats including SDTV, HDTV video pictures, plus multi-channel digital audio, and can transmit uncompressed audio and video signals.
3. PAL system video, SDI video:
PAL is also called PAL, an abbreviation for english Phase Alteration line. Sometimes referred to as 625-line, 25-grid per second, interlaced, PAL color coded television system.
SDI is a digital interface, an abbreviation for english SERIAL DIGITAL INTERFACE, is a standard for transmitting digital video on a coaxial line. The three formats SD-SDI, HD-SDI and 3G-SDI have corresponding transmission rates of 270Mb/s, 1.485Gb/s and 2.97Gb/s, are the same as HDMI and are used for transmitting uncompressed audio and video signals, and have the characteristics of low loss and strong anti-interference capability, and are widely applied to the fields of broadcast television and monitoring.
4. SOC: the System on Chip is called as a System on Chip, and the System on Chip is called as a System on Chip;
5. Maritime 3519A SOC related terminology:
VI: video input;
VPSS: video processing;
VENC: video coding;
PHY: fully called Port PHYSICAL LAYER, port physical layer;
U-Boot: full scale Universal Boot Loader, an open source item following the terms of the universal public license agreement GPL (General Public License), which functions as a system boot;
MMZ: one of two physical memories of the Hai Si chip is a multimedia memory area, which is called Media Memory Zone in English and is called MMZ for short.
While the invention has been described in connection with specific embodiments thereof, it is to be understood that these drawings are included in the spirit and scope of the invention, it is not to be limited thereto.
Claims (9)
1. The airborne video processing device is characterized by comprising an FPGA and a video processing chip which are integrally arranged, wherein the FPGA receives a visible light video signal and a thermal infrared imager video signal to obtain video data after format conversion, the video data is transmitted to the video processing chip through an interactive interface, the video processing chip encodes the video data according to a preset protocol and then stores the encoded video data in a local memory, and the encoded video data is transmitted to ground receiving equipment through a wireless data link, or the video processing chip transmits the encoded video data to the FPGA, and the FPGA transmits the encoded video data to the ground receiving equipment through the wireless data link;
The on-board video processing device utilizes a target speed prediction module to introduce feedforward control into a servo system, compensates time delay, operates in a DSP (digital signal processor) arranged in the video processing chip, realizes speed prediction by reading a series of data of azimuth pitch angle, azimuth pitch angle speed and target tracking error amount of a photoelectric pod and adopting a Kalman filtering algorithm, and introduces speed and acceleration signals of signals at the input end of a speed loop channel to form feedforward control, so that servo performance is improved to improve target tracking precision;
the on-board video processing device is also integrally provided with: the system comprises an SDI decoding chip and an A/D decoding chip, wherein the SDI decoding chip is used for carrying out video decoding on a received visible photoelectric video signal, transmitting decoded video effective data to the FPGA in a CMOS level mode by 20bit along with F, V, H synchronous signals, and the A/D decoding chip is used for carrying out video analog-to-digital conversion on a received thermal infrared imager video signal, and transmitting video data to the FPGA in an 8bit data mode along with HS, VS, PIXCLK.
2. The on-board video processing device of claim 1, wherein the video processing chip has: the FPGA converts the received decoded visible light video signal into a BT.1120 time sequence, converts the received decoded thermal infrared imager video signal into a BT.656 time sequence, inputs the BT.1120 time sequence and the BT.656 time sequence, and transmits the BT.656 time sequence to the video processing module for preprocessing to obtain video processing data, and the video encoding module encodes the video processing data according to an H.264/H.265 protocol.
3. The on-board video processing device according to claim 2, wherein the video processing chip adopts an on-line mode to directly transmit the video data stream of the video input module to the video processing module, and the video processing module transmits the video processing data to the video encoding module for encoding in a line unit in a manner of transmitting while collecting.
4. The on-board video processing device of claim 1, wherein the video processing chip transmits the encoded video data to the wireless data link via an ethernet port or to the FPGA via a PCIe expansion bus.
5. The device according to claim 4, wherein when the video processing chip encodes, the video data, the aircraft data and the payload status data are formed into a video data stream according to a preset protocol, the video data stream is stored in a local memory, and the FPGA reads the video data stream in the local memory through a PCIe interface, performs parallel-to-serial conversion, and then transmits a clock and a driving output according to a set code stream.
6. The on-board video processing device of claim 1, wherein the FPGA has a multi-way asynchronous communication 422 interface for enabling communication with the on-board flight control, inertial navigation, servo system and host computer.
7. The on-board video processing device of claim 1, wherein the video processing chip has a target tracking module that performs target tracking using a kernel-dependent filtering algorithm based on directional gradient histogram features.
8. The on-board video processing device of any one of claims 1-7, wherein the video processing chip employs Hi3519A or Hi35559A.
9. A drone provided with an on-board optoelectronic pod for video compression storage, multi-target detection, target tracking and target speed prediction using an on-board video processing device according to any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111193580.7A CN113890977B (en) | 2021-10-13 | Airborne video processing device and unmanned aerial vehicle with same |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111193580.7A CN113890977B (en) | 2021-10-13 | Airborne video processing device and unmanned aerial vehicle with same |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113890977A CN113890977A (en) | 2022-01-04 |
| CN113890977B true CN113890977B (en) | 2024-06-21 |
Family
ID=
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107993257A (en) * | 2017-12-28 | 2018-05-04 | 中国科学院西安光学精密机械研究所 | A kind of intelligence IMM Kalman filtering feedforward compensation target tracking methods and system |
| CN111932588A (en) * | 2020-08-07 | 2020-11-13 | 浙江大学 | Tracking method of airborne unmanned aerial vehicle multi-target tracking system based on deep learning |
| CN112751585A (en) * | 2021-01-15 | 2021-05-04 | 天津航天中为数据***科技有限公司 | Unmanned aerial vehicle wireless communication terminal |
| CN113449566A (en) * | 2020-03-27 | 2021-09-28 | 北京机械设备研究所 | Intelligent image tracking method and system for low-speed small target in human-in-loop |
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107993257A (en) * | 2017-12-28 | 2018-05-04 | 中国科学院西安光学精密机械研究所 | A kind of intelligence IMM Kalman filtering feedforward compensation target tracking methods and system |
| CN113449566A (en) * | 2020-03-27 | 2021-09-28 | 北京机械设备研究所 | Intelligent image tracking method and system for low-speed small target in human-in-loop |
| CN111932588A (en) * | 2020-08-07 | 2020-11-13 | 浙江大学 | Tracking method of airborne unmanned aerial vehicle multi-target tracking system based on deep learning |
| CN112751585A (en) * | 2021-01-15 | 2021-05-04 | 天津航天中为数据***科技有限公司 | Unmanned aerial vehicle wireless communication terminal |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11869234B2 (en) | Subject tracking systems for a movable imaging system | |
| US10636150B2 (en) | Subject tracking systems for a movable imaging system | |
| CN2917134Y (en) | DSP-based embedded real-time panoramic image acquisition and processing device | |
| US11258949B1 (en) | Electronic image stabilization to improve video analytics accuracy | |
| CN111307291B (en) | Surface temperature anomaly detection and positioning method, device and system based on unmanned aerial vehicle | |
| CN112714281A (en) | Unmanned aerial vehicle carries VR video acquisition transmission device based on 5G network | |
| US11514371B2 (en) | Low latency image processing using byproduct decompressed images | |
| CN210526874U (en) | Airborne three-light photoelectric pod system | |
| CN112584092B (en) | Data acquisition device and data acquisition system | |
| KR102192405B1 (en) | Real-time frame alignment of video data | |
| CN109788214B (en) | Multi-channel video seamless switching system and method based on FPGA | |
| CN110740257A (en) | high-speed high-definition CMOS imaging system and imaging method thereof | |
| US10681303B2 (en) | Multi-camera serial video data conversion for graphics processing unit (GPU) interface | |
| CN113890977B (en) | Airborne video processing device and unmanned aerial vehicle with same | |
| CN101765006B (en) | Remote detection video format real-time conversion equipment | |
| CN113890977A (en) | Airborne video processing device and unmanned aerial vehicle with same | |
| CN104159015A (en) | Image pickup system based on unmanned aerial vehicle | |
| CN116797716A (en) | Real-time acquisition method, device and system for mapping model | |
| CN114326764A (en) | Rtmp transmission-based smart forestry unmanned aerial vehicle fixed-point live broadcast method and unmanned aerial vehicle system | |
| CN107948476A (en) | Camera system based on unmanned plane | |
| CN204180158U (en) | Based on the camera system of unmanned plane | |
| WO2023029567A1 (en) | Visualization method and system for various data collected by sensor | |
| KR102462709B1 (en) | Common platform for communication of open electric vehicles | |
| Liu et al. | Research on the Application of UAV Wide-angle Shooting Technology in Multimedia Video Playing | |
| CN219039781U (en) | Vehicle-mounted panoramic double-spectrum vision enhancement system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |