CN108154523B - A kind of real-time modeling method system and method in airborne photoelectric platform - Google Patents
A kind of real-time modeling method system and method in airborne photoelectric platform Download PDFInfo
- Publication number
- CN108154523B CN108154523B CN201711422365.3A CN201711422365A CN108154523B CN 108154523 B CN108154523 B CN 108154523B CN 201711422365 A CN201711422365 A CN 201711422365A CN 108154523 B CN108154523 B CN 108154523B
- Authority
- CN
- China
- Prior art keywords
- target
- data
- image
- visible images
- infrared
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/23—Clustering techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10048—Infrared image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10052—Images from lightfield camera
Landscapes
- Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- General Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Image Analysis (AREA)
Abstract
The present invention provides the real-time modeling method system and methods in a kind of airborne photoelectric platform, belong to intelligent video processing technology field, and system includes image capture module, image decoder module, data communication module and target tracking module.Image capture module is responsible for the acquisition of visible images and infrared picture data, after completing acquisition, infrared picture data and visible images data input picture cache module, data communication module is responsible for command information and image information transmitting, the communication with host computer and rear end servo-control system between dsp chip, target tracking module receives host computer instruction, reads the infrared picture data and visible images data cached, and output phase answers processing result, that is target bearing information, finally, target position information is passed to rear end servo-control system.The present invention is easily achieved, and can carry out real-time modeling method processing in airborne photoelectric platform immediately after collecting image, precision is high, stability is strong, output delay is low, can replace the artificial intelligent control realized to servo-system.
Description
Technical field
The present invention relates to intelligent video process fields, and in particular to be a kind of real-time target in airborne photoelectric platform with
Track system and method.
Background technique
Target following refers to the position for providing target in a certain frame image of video, and then algorithm Continuous plus goes out in subsequent frame
The task of target position.Target following technology is widely used scene.In modern Airborne photoelectric platform, Target Tracking System
It plays an important role.Conventional on-board photoelectric platform does not have Target Tracking System, by the fortune for manually controlling photoelectric nacelle
It is dynamic, to track target, continuous observation is carried out to target.In modern Airborne photoelectric platform, intelligent Target tracking system is responsible for continuous
The location information of selected target is provided, to adjust orientation, the pitch angle of photoelectric nacelle immediately, target is made to be in video always
Picture center is observed convenient for user.Existing Target Tracking System generally first passes video data back local high-performance computer
Or server, target following processing is then carried out again, finally returns result.This processing mode video data transmission difficulty
Greatly, height is required to hardware computing capability, is not able to satisfy the requirement of real-time modeling method.Existing some airborne target tracking systems
It is generally difficult to handle images above data all the way simultaneously, is unsatisfactory for requirement of real-time, or in the case where meeting requirement of real-time,
It cannot be guaranteed good target following quality.
Summary of the invention
Technology of the invention solves the problems, such as: overcoming the deficiencies of the prior art and provide real-time in a kind of airborne photoelectric platform
Target Tracking System and method realize there is the spies such as small in size, low in energy consumption based on the hardware platform of fpga chip plus dsp chip
Point is directly installed on airborne photoelectric platform rear end, directly carries out real-time modeling method processing at the scene, can be with extremely low delay
Target position continuously is provided, instructs the ray machine servo-control system follow-up motion of rear end.
The adopted technical solution is that: the real-time modeling method system in a kind of airborne photoelectric platform, packet
Include image capture module, image decoder module, data communication module and target tracking module.Have for multi-core DSP chip more
The design feature of a independent kernel carries out parallel optimization, realizes stable and accurate real-time modeling method.Main wound of the invention
Newly it is that the scheduling of combination decision target tracking algorism and algorithm used in target tracking module in par-ticular processor is real
It is existing.
Described image acquisition module includes: visible light image sensor and infrared image sensor, it is seen that light image sensing
Device and infrared image sensor are installed with optical axis；Optical axis calibrator error is within 2 pixels；Infrared picture data and visible images
Data are transmitted using SDI agreement；Infrared image sensor and visible light image sensor acquire infrared band and visible light respectively
The destination image data of wave band is used for target following；
Described image decoder module includes: that two SDI receive chips, SRAM array, fpga chip and in fpga chip
The infrared picture data and visible images data stream algorithm of operation, SDI receive chip and receive from image capture module
SDI infrared picture data and visible images data be converted to parallel data stream and be transferred to fpga chip, in fpga chip
Infrared picture data is decoded from infrared picture data and visible images data flow with visible images data stream algorithm
Effective infrared picture data and visible images data out, include in the data flow valid data, Elided data, frame synchronization,
Row synchrodata, and data buffer storage is carried out using SRAM array；
The data communication module includes: serial communication chip, the data communication program realized in fpga chip, data
There are two functions for signal procedure, first is that the serial communication with exterior, defeated including parsing the instruction received and coding transmission
Information out, second is that the data interaction between fpga chip and dsp chip, including being transmitted to the infrared of dsp chip from fpga chip
Image data and visible images data, PC control instruct, and the target following knot of fpga chip is transmitted to from dsp chip
Fruit；
The target tracking module includes: multi-core DSP chip and target tracking algorism, the target run on dsp chip with
Track algorithm according to the infrared picture data that is received from data communication module and visible images data, PC control instruct into
Row target following operation, the position that Automatic solution image middle finger sets the goal obtain target following as a result, and being transferred to data communication
Module output；It works to complete instruction response, data communication and target following in real time, the work on multi-core DSP chip is divided
Two tasks are controlled for target following and system, wherein 0~n-1 core completes target following task, the last one n core completes system
Control task；The target tracking algorism is used based on infrared and visible images combination decision visual target tracking algorithms,
Decision model is constructed respectively for visible images and infrared image, is determined that sample collected is target or background, is resolved
Target position；The probability that single model misjudgment causes tracking to fail is larger, is combined decision using two decision models
The probability of tracking failure can be greatly reduced, realize stable and accurate target following, there are multiple independences for multi-core DSP chip
The design feature of kernel carries out parallel optimization to the different task for needing while running, realizes real-time modeling method.The present invention
Main innovation be that combination decision target tracking algorism and algorithm are in par-ticular processor used in target tracking module
Scheduling realize.
It is real based on infrared and visible images combination decision visual target tracking algorithms in the target tracking module
It is existing that steps are as follows:
(1) according to target initial position, and initial infrared picture data and visible images data, acquire initial infrared figure
As the target image block in data and visible images data is as positive negative training sample, sample characteristics are extracted, building is determined respectively
Plan model Dv、Dir；
(2) receive a new frame infrared picture data and visible images data after, from target in previous frame position
Surrounding acquires candidate samples, judges whether candidate samples are target using decision model, determines target position in a new frame；
(3) determine that the result which decision model provides is optimizing decision as a result, two decisions of combination according to loss function
The differentiation of model is as a result, obtain final output, and provide the decision model of sub-optimal result, elimination using optimal result amendment
The error message in the decision model of sub-optimal result is generated, the decision model for generating sub-optimal result is enable to track in succeeding target
More accurate target following is provided in the process as a result, boosting algorithm robustness.
Step (1)-(3) further realize as follows:
(1) when extracting sample characteristics, sample image block is divided into nonoverlapping zonule, respectively according to gradient direction
Gradient magnitude in statistical regions at pixel, the original feature vector C of composition one 27 calculate standard using following formula later
Change operator, then C be standardized, obtained Standardization Operator N (i, j):
N (i, j)=(| | C (i, j) | |2+||C(i+1,j)||2+||C(i-1,j)||2+||C(i,j+1)||22+||C(i,
j-1)|||2)2
Wherein C (i, j) is the image area characteristics vector of the i-th row j column, is standardized, is obtained using following formula
To final feature vector F (i, j), the feature vector of each image block collectively constitutes clarification of objective representing matrix X；
F (i, j)=max (α, C (i, j)/N (i, j))
Wherein α is an intercept term, for eliminating the excessive noise item of eigenmatrix intermediate value, so that the image extracted is special
Sign being capable of more robust expression target；
(2) initial target image block is acquired, using scaling, rotation, translation, overturning, affine transformation mode, generates a collection of mesh
Logo image positive sample, is denoted as Tp, while the background area of image extract at random it is some be overlapped with target image it is less or not
The image block of coincidence, as negative sample Tn；It is trained using a variety of obtained positive samples that convert, greatly enhances decision model
Type is to the robustness accordingly converted；
(3) in new frame image, from stochastical sampling around target previous frame position, a collection of candidate samples are obtained, uniformly
Sampling obtains a collection of candidate samples, collectively constitutes candidate target sample, stochastical sampling can increase track algorithm for target with
The robustness that machine quickly moves still is able to accurately be captured after uniform sampling can guarantee that target is moved to any direction；
(4) it is directed to visible images and infrared image, constructs decision model D respectivelyv、Dir,
Wherein θv, θirFor model parameter, the feature of x sample.In each frame, two are calculated as a result, being denoted as R respectivelyv、
Rir:
It determines that the result which decision model provides is optimizing decision as a result, as output according to loss function, uses simultaneously
Optimizing decision result goes amendment to generate the decision model of sub-optimal result, and the decision model for generating sub-optimal result is made in subsequent frames
There is better performance；A loss function can be calculated in n-th frame in each result of decisionSentence to be promoted
Disconnected accuracy judges optimizing decision result using aggregated loss function:
WhereinFor the loss function of model D, D ∈ D herev,Dir, D*For optimal decision model, Δ n is aggregated loss letter
Several time window lengths；In object tracking process, the result of decision of two decision models is combined, wherein optimal result is made for selection
To export, and optimal result is utilized, amendment updates suboptimum decision-making model, corrects the mistake that suboptimum decision-making model running introduces in the process
False information, two decision models are cooperated, are corrected mutually, realize stable and accurate target following.
The dsp chip is the TMS320C6678 multi-core processor of TI, has 8 independent kernels that can be run parallel, point
It is not denoted as core 0-7 and is transmitted to the dsp chip of target tracking module after data communication module receives the instruction of host computer sending,
Dsp chip needs the instruction of real-time response host computer, after starting target following, dsp chip needs be completed at the same time instruction response, from
Video decoding module carries infrared picture data and visible images data and target following works；These work are can be with
It independently carries out, the system features that the present invention realizes are: devising two subtasks operated on different kernels, wherein mesh
Mark tracing task is responsible for target position resolving, and system control tasks communication control task is responsible for instruction response, result exports and red
The carrying of outer image data and visible images data, two tasks are separately operable on different kernels, system control tasks
Infrared picture data and visible images data and control instruction are provided for target following task, target following task utilizes infrared
Image data and visible images data are according to control instruction solving target position, and output is to system control tasks, system control
Task completes the passback of target following result.Running on different kernels for task will not seize mutually process resource, will not generate
Unnecessary obstruction causes time delay, to ensure that the real-time of target following and instruction response；Core 0-6 operational objective with
Track task is completed target position and is resolved using based on infrared and visible images combination decision visual target tracking algorithms,
Center 0 is used as main core, completes the initialization of dsp chip and the overall operation of target following task, and core 7 runs communication control
Task, response system instruction, carries infrared picture data and visible images data, output tracking result.
The step of target following task run on dsp chip in the target tracking module are as follows:
(1) after system electrification, the initialization of dsp chip is completed, starts target following task, is ready for target position solution
It calculates；
(2) when target following task is in non-tracking state, task is in idle condition, and receives the mesh of host computer sending
After marking trace command, (3) are gone to；
(3) target initial coordinate information R is extracted from from the Command Information Flow of host computer0, trigger inside dsp chip
Enhancing memory directly accesses (EDMA) data transmission mechanism, from infrared picture data and visible light figure from data communication module
As extracting initial target region P in data flowv、Pir；
(4) using the infrared picture data and visible images data in initial target region, initialization based on it is infrared with
The combination decision visual target tracking algorithm of visible images constructs decision model D for infrared and visible images respectivelyv、
Dir；
(5) when n-th frame image arrives, in the tracking result R of previous framen-1Surrounding acquires candidate samples, extracts each sample
Feature, judge whether it is target using decision model, obtain the output result R of two decision-making devicesvWith Rir；
(6) aggregated loss function is used to determine optimizing decision as a result, as final output Rn, while in final result neighborhood
Interior extraction sample, is updated decision model, boosting algorithm robustness；
(7) electric under system, target following task terminates.
The step of system control communication control task run on dsp chip in the target tracking module are as follows:
(1) system electrification starts core 7, bring into operation communication control task after core 0 completes dsp chip initialization；
(2) when not having instruction to arrive, control task is in idle condition, right after receiving the instruction from host computer
Instruction is parsed, and effective information is passed to core 0 and is specifically executed；After starting target following, core 7 is from data communication module
It carries in real time and comes from front-end image sensor, by the infrared picture data and visible images data of image decoder module, supply
Target following task uses；
(3) electric under system, communication control task terminates.
Real-time modeling method method in a kind of airborne photoelectric platform of the present invention realizes that steps are as follows:
(1) infrared sensor and visible light sensor in image capture module acquires the infrared picture data with optical axis
With visible images data, image decoder module is transferred to by SDI agreement；
(2) the serial infrared image data from image capture module and visible images data flow, first pass around SDI and connect
Chip is received, parallel infrared picture data and visible images data flow are become, is then passed to fpga chip；Figure in fpga chip
As decoding program use state machine, effective infrared image is decoded from parallel infrared picture data and visible images data flow
Data and visible images data are stored in SRAM array；Image decoding state machine include stIDLE, stFRAME, stLINE,
StWAIT state, state count CntV, horizontal direction meter according to vertical synchronizing signal VS, data valid signal DE, vertical direction
Number CntH, picturedeep Rows, picturewide Cols signal shift；
(3) it after the data communication program in fpga chip receives target following instruction, is read from SRAM array infrared
Buffer area in image data and visible images data to piece, notice dsp chip take infrared picture data away from buffer area
With visible images data, while recently received host computer instruction is passed into dsp chip；
(4) dsp chip receive infrared picture data and visible images data, target initial position message and starting with
After track instruction, it is based respectively on infrared image and visible images, extracts positive negative sample, training decision model constructs decision model,
When a new frame image arrives, candidate samples are acquired near target position in previous frame image, extract each candidate samples
Feature, determine that it is target or background using decision model, obtain target position in a new frame.Finally according to target following knot
Fruit and infrared picture data are updated with visible images data match plan model；
(5) target following result is transmitted to data communication module from dsp chip, after being transferred to later by data communication module
End system.
Target Tracking System of the invention compared with prior art the advantages of have:
(1) present invention is based on infrared using based on infrared and visible images combination decision visual target tracking algorithms
With visible images, two decision models are constructed, to carry out target following.Final goal tracking knot is determined by combination decision
Fruit.Infrared image and visible images have respective Pros and Cons in different scenes, and by combination decision, algorithm can be
Target is accurately tracked in scene abundant, while during each frame arithmetic, can use optimal result amendment and generate suboptimum
As a result decision model, has good robustness algorithm in the process of running, being capable of stable and accurate carry out target following.
(2) dominant frequency and computing capability of dsp chip are limited, the present invention realize dsp chip on target following program when,
According to realistic objective, specific aim optimization has been carried out based on multi-core processor, has been divided into the work of dsp chip using reasonable manner
Target following task and communication control task cause unnecessary runing time expense so that two tasks will not generate conflict.
Simultaneously in operational objective tracing task, parallel processing will be capable of in algorithm and is partially distributed to execute parallel in multiple kernels,
The real-time that ensure that system each task, realizes real-time modeling method.
Detailed description of the invention
Fig. 1 is system structure diagram of the invention；
Fig. 2 is the state transition diagram of video data decoding state machine；
Fig. 3 is target following task run flow chart on dsp chip；
Fig. 4 is experiment effect figure of the present invention.
Specific embodiment
The following describes the present invention in detail with reference to the accompanying drawings and embodiments.
As shown in Figure 1, present system includes image decoder module, data communication module and target tracking module；Image
Acquisition module includes visible light image sensor, infrared image sensor, and wherein visible light image sensor acquires visible light wave
The information of section, imaging have color abundant, texture and gradient information, and infrared image sensor is imaged according to heat radiation,
When illumination condition is poor, it still is able to clear display environment and object, plays the role of supplying to visible images；When installation, it is seen that
Light image and infrared image sensor require same optical axis, and optical axis calibrator error is within 2 pixels.Wherein visible light image sensor
The image data of 30 frame 1920*1080 resolution ratio of output per second, infrared image sensor 30 frame 640*512 resolution ratio of output per second
Image data.The image data of visible images and infrared image sensor is transmitted using SDI agreement.
Image decoder module include SDI receive chip, the infrared picture data realized in SRAM array and fpga chip with
Visible images data stream program, SDI receive the infrared figure of high speed serialization SDI that chip will be received from image capture module
It is transferred to fpga chip as data and visible images data are converted to parallel data stream, the decoding program in fpga chip is from red
Valid data are decoded in outer image data and visible images data, and carry out data buffer storage using SRAM array.The SDI
It receives chip and uses GS2971A 3G-SDI video decoding chip.Fpga chip uses XC7K325T, uses shape in fpga chip
State machine realizes the decoding of infrared picture data Yu visible images data, state machine include stIDLE, stFRAME, stLINE,
StWAIT state, state is according to VS (vertical synchronizing signal), DE (data valid signal), CntV (vertical direction counting), CntH
(horizontal direction counting), Rows (picturedeep), Cols (picturewide) signal shift, and the original state of state machine is
StIDLE, into stFRAME state, prepares the image data for decoding a new frame when VS and DE is 0, when VS and DE is 1,
Into stLINE state, prepare the image data for decoding new a line, when it is 0 that VS, which is 1, DE, into stWAIT state, under waiting
The arrival of data line is again introduced into stFRAME state when VS and DE is 0, prepares the image data for decoding a new frame, tool
Body state transfer case such as Fig. 2.
Data communication module includes serial communication chip, in the data communication program and dsp chip realized in fpga chip
The data communication program of realization, serial communication chip complete the conversion before single-ended signal and differential signal, fpga chip are sent out
Single-ended signal out switchs to differential signal transmission, or the differential signal received is switched to single-ended signal transmission to fpga chip,
There are two functions for the data communication program realized in fpga chip, first is that the serial communication with exterior, including parsing receive
The instruction arrived and coding send output information, second is that the data interaction between fpga chip and dsp chip, including from fpga chip
Be transmitted to the infrared picture data of dsp chip and the transmission of visible images data, PC control instructs, and from dsp chip
It is transmitted to the target following result of fpga chip.The serial communication chip uses MAX3077E chip.
Target tracking module includes multi-core DSP chip and the target following program wherein realized.Multi-core DSP chip uses TI
The TMS320C6678 multi-core DSP chip of company, chip include 8 independent kernels, and each core operating frequency is 1GHz.Target
Tracking module is instructed according to the infrared picture data and visible images data, the PC control that receive from data communication module
Carry out target following operation, the position that Automatic solution image middle finger sets the goal obtains target following as a result, and to be transferred to data logical
Believe module output；To complete the work such as instruction response, data communication and target following in real time, run simultaneously on multi-core DSP chip
Target following and system control two subtasks, wherein target following task is responsible for target position resolving, communication control task
It is responsible for instruction response, result output and the carrying of infrared picture data and visible images data, two tasks to be separately operable
On different kernels.Target following program is used based on infrared and visible images combination decision visual target tracking algorithms,
There is the design feature of multiple independent kernels for multi-core DSP chip, carry out parallel optimization, realize stable and accurate real-time mesh
Mark tracking.
As shown in Figure 1, the combination decision visual target tracking method step that the present invention is based on infrared with visible images is such as
Under:
(1) according to the infrared image and visible images received, the basic tracker of each of combination decision is constructed respectively.
Extract the Gradient Features of target image according to the following formula first,
Gx=I (x+1, y)-I (x-1, y)
Gy=I (x, y+1)-I (x, y-1)
Wherein Gx, Gy are the gradient magnitude that position (x, y) is on the direction x and the direction y respectively, and I indicates image.
Next gradient magnitude and direction at each pixel are calculated according to Gx, Gy, divides the image into and does not overlap
4x4 pocket, in each region respectively by the gradient magnitude at each pixel by the cumulative statistics in direction to difference
Section.Using 9 sections in direction and the section in 18 directions in the present invention.When such as using the section in 9 directions, by gradient magnitude
Enter sections such as (0 °~40 °, 40 °~80 °, 320 °~0 °) by directional statistics.9 interval statistics results of each image block and 18th area
Between statistical result collectively constitute one 27 original feature vector C, following formula normalized operator is used later, then to C
It is standardized,
N (i, j)=(| | C (i, j) | |2+||C(i+1,j)||2+||C(i-1,j)||2+||C(i,j+1)||2+||C(i,
j-1)||2)2
Wherein C (i, j) is the image area characteristics vector of the i-th row j column, and N (i, j) is the Standardization Operator being calculated.
It is standardized using following formula
F (i, j)=max (α, C (i, j)/N (i, j))
Wherein α is an intercept term, for eliminating the excessive noise item of eigenmatrix intermediate value, so that the image extracted is special
Sign being capable of more robust expression target.F (i, j) is the feature vector finally acquired, common group of the feature vector of each image block
At clarification of objective representing matrix X.
(2) a collection of target figure is generated using modes such as scaling, rotation, translation, overturning, affine transformations according to target image
The positive sample of picture, is denoted as Tp, each sample according to transformation degree, be based respectively on value for 0~1 label, with original object figure
As closer, the resulting label value of sample is bigger.It extracts and some is overlapped with target image at random in the background area of image simultaneously
Region that is less or not being overlapped, as negative sample Tn, by being standardized computing cross-correlation with target image, obtain its with
The similarity of target, the as label value of negative sample.When being standardized mutual, sample image is zoomed into 16*16 picture first
Plain size.Then it calculates as follows:
Wherein T represents initial target image sample, TnAny one negative sample is represented, two samples of ⊙ are multiplied simultaneously pixel-by-pixel
Adduction.Conf is the similarity that this negative sample and target is calculated.
The feature and its label of each sample are extracted using the method described in step 1, training decision model obtains one
Group weight vector.Training process uses stochastic gradient descent method.
The discriminant equation of decision model are as follows:
hθ(x)=g (θTx)
The wherein feature vector that x transforms into for the eigenmatrix of a certain sample, θ are the weight vector that training obtains, and g is using such as
Minor function can make the value being calculated in the section of [0,1], when h is greater than a certain threshold value TconfWhen, sample is judged as mesh
Mark, otherwise, sample is considered as background.
(3) a collection of sample, uniform sampling are obtained from stochastical sampling around target previous frame position in a new frame image
A collection of sample is obtained, candidate target sample is collectively constituted.Stochastical sampling can increase track algorithm and target is quickly transported at random
Dynamic robustness still is able to accurately be captured after uniform sampling can guarantee that target is moved to any direction.Stochastical sampling is adopted
With normal distyribution function, respectively with the x on the boundary up and down of target, centered on y-coordinate value, the seat of candidate samples is randomly generated
Scale value.It uniformly uses centered on target initial position, step-length is 2 pixels, and sliding window extracts candidate samples.Obtain candidate samples
Afterwards, sample characteristics are extracted, judge that it is background or target according to decision model.If having multiple targets simultaneously, according to result
Position is clustered, and the biggish candidate samples of deviation are left out, and the similar candidate samples in position are weighted and averaged according to confidence level,
Obtain final goal position.
(4) finally, after obtaining new target position, the method described in (1) obtains a collection of positive sample and negative sample, extracts
Feature, and the training method described in (2) is updated decision model, with the robust for keeping algorithm to change target appearance
Property.
(5) it is directed to visible images and infrared image, constructs decision model D respectively according to (1) to (4)v、Dir,
Wherein subscript v represents visible light, and ir represents infrared.
For visible images and infrared image, decision model D is constructed respectivelyv、Dir,
Wherein θv, θirFor model parameter, the feature of x sample.In each frame, two are calculated as a result, being denoted as R respectivelyv、
Rir:
Under normal conditions, the goodness of fit of two kinds of results is higher, deviation (school axis error containing 2 pixels) within 3 elements.But
In certain scenes, if illumination condition is poor, background and when more uniform target temperature in image, a certain result is it is possible that larger
Drift error, at this moment, the present invention according to loss function determines optimizing decision as a result, exporting as system.Simultaneously with optimal knot
Fruit goes to correct another decision model, makes it that can have better performance in subsequent frames.Each result of decision, can in n-th frame
A loss function is enough calculatedFor the accuracy for promoting judgement, the present invention uses aggregated loss function, and judgement is most
The excellent result of decision:
WhereinFor the loss function of model D, D ∈ D herev,Dir, wherein D*For optimal decision model, Δ n is accumulation damage
Lose the time window length of function.After obtaining optimal result, using optimal result position, using method restoration updating described in (4)
Suboptimum decision-making model corrects the error message introduced in its operational process.In object tracking process, two decision models are mutual
Cooperation, mutually amendment, realize stable and accurate target following.
Method for tracking target operation of the invention includes following operating procedure, target following task run flow chart such as Fig. 3
It is shown:
(1) after system electrification, the initialization of dsp chip is completed, starts target following task, is ready for target position solution
It calculates.
(2) when target following task is in non-tracking state, task is in idle condition.Receive the mesh of host computer sending
After marking trace command, (3) are gone to.
(3) target initial coordinate information R is extracted from from the Command Information Flow of host computer0, trigger inside dsp chip
Enhancing memory directly accesses (EDMA) data transmission mechanism.From infrared picture data and visible light figure from data communication module
As extracting initial target region P in data flowv、Pir。
(4) using the infrared picture data and visible images data in initial target region, initialization based on it is infrared with
The combination decision visual target tracking algorithm of visible images.According to 2, for infrared and visible images, difference structure
Build decision model Dv、Dir.Its center 0 carries out the decision model D based on visible imagesvRelated operation, core 1 carries out based on red
The decision model D of outer imageirRelated operation, core 2-6 provides parallel support for the operation of algorithm on core 0-1.Including initially just
Negative sample Tp、TnAcquisition, each sample feature extraction and subsequent step in sample classification.
(5) when n-th frame image arrives, in the tracking result R of previous framen-1Surrounding acquires candidate samples, extracts each sample
Feature, judge whether it is target using decision model.Obtain the output result R of two decision-making devicesvWith Rir。
(6) aggregated loss function is used to determine optimizing decision as a result, as final output Rn.Simultaneously in final result neighborhood
Interior extraction sample, is updated decision model, boosting algorithm robustness.
(7) electric under system, real-time modeling method task terminates.
The step of communication control task run on dsp chip in target tracking module are as follows:
(1) system electrification starts core 7, bring into operation communication control task after core 0 completes dsp chip initialization；
(2) when not having instruction to arrive, system control tasks are in idle condition.Receive the instruction from host computer
Afterwards, instruction is parsed, effective information is passed into core 0 and is specifically executed.After starting target following, core 7 is logical from data
Letter module is carried in real time from front-end image sensor, by the infrared picture data and visible images number of image decoder module
According to for the use of target following task.
(3) electric under system, communication control task terminates.
The part of test results of Target Tracking System of the invention is as shown in figure 4, in Fig. 4 in the first behavior visible images
Experimental result, it can be seen that when target is by partial occlusion, be still able to maintain stable objects tracking.Second and the third line in Fig. 4
For the experimental result in infrared image, it can be seen that when deformation occurs for target, be still able to maintain stable objects tracking, target is complete
It is blocked when occurring again entirely, target can be given for change again, continue target following.
Target Tracking System in the present invention can handle infrared picture data and visible images data simultaneously, can
Reply target by partial occlusion, blocked completely by target well, and the scene seriously changed occurs for target shape, keeps to target
Tenacious tracking.
Above embodiments are provided just for the sake of the description purpose of the present invention, and are not intended to limit the scope of the invention.This
The range of invention is defined by the following claims.It does not depart from spirit and principles of the present invention and the various equivalent replacements made and repairs
Change, should all cover within the scope of the present invention.
Claims (6)
1. the real-time modeling method system in a kind of airborne photoelectric platform, it is characterised in that: including image capture module, image solution
Code module, data communication module and target tracking module:
Described image acquisition module includes: visible light image sensor and infrared image sensor, it is seen that optical image sensor with
Infrared image sensor is installed with optical axis；Infrared picture data and visible images data are transmitted using SDI agreement；Infrared image
Sensor and visible light image sensor acquire the destination image data of infrared band and visible light wave range respectively, for target following
It uses；
Described image decoder module includes: that two SDI receive chip, SRAM array, fpga chip and run in fpga chip
Infrared picture data and visible images data stream algorithm, SDI receive the SDI that receives from image capture module of chip
Infrared picture data and visible images data are converted to parallel data stream and are transferred to fpga chip, the infrared figure in fpga chip
As data and visible images data stream algorithm decode effectively from infrared picture data and visible images data flow
Infrared picture data and visible images data include that valid data, Elided data, frame synchronization, row synchronize in the data flow
Data, and data buffer storage is carried out using SRAM array；
The data communication module includes: serial communication chip, the data communication program realized in fpga chip, data communication
There are two functions for program, first is that the serial communication with exterior, sends output letter including parsing the instruction received and coding
Breath, second is that the data interaction between fpga chip and dsp chip, the infrared image including being transmitted to dsp chip from fpga chip
Data and visible images data, PC control instruct, and the target following result of fpga chip is transmitted to from dsp chip；
The target tracking module includes: multi-core DSP chip and target tracking algorism, and the target following run on dsp chip is calculated
Method instructs according to the infrared picture data and visible images data, the PC control that receive from data communication module and carries out mesh
Mark tracking operation, the position that Automatic solution image middle finger sets the goal obtains target following as a result, and being transferred to data communication module
Output；It works to complete instruction response, data communication and target following in real time, the work on multi-core DSP chip is divided into mesh
Mark tracking and system control two tasks, wherein 0~n-1 core completes target following task, the last one n core completes system control
Task；The target tracking algorism is used based on infrared with visible images combination decision visual target tracking algorithms, for
Visible images and infrared image construct decision model respectively, determine that sample collected is target or background, solving target
Position；The probability that single model misjudgment causes tracking to fail is larger, and being combined decision using two decision models can
The probability of tracking failure is greatly reduced, realizes stable and accurate target following, there are multiple independent kernels for multi-core DSP chip
Design feature, to need simultaneously operation different task progress parallel optimization, realize real-time modeling method；
In the target tracking module, based on infrared and visible images combination decision visual target tracking algorithms, step is realized
It is rapid as follows:
(1) according to target initial position, and initial infrared picture data and visible images data, acquire initial infrared image number
According to the target image block in visible images data, as positive negative training sample, sample characteristics is extracted, construct decision model respectively
Type Dv、Dir；
(2) after the infrared picture data and visible images data that receive a new frame, from target in previous frame around position
Candidate samples are acquired, judges whether candidate samples are target using decision model, determines target position in a new frame；
(3) determine that the result which decision model provides is optimizing decision as a result, two decision models of combination according to loss function
Differentiation as a result, obtain final output, and provide using optimal result amendment the decision model of sub-optimal result, eliminate and generate
Error message in the decision model of sub-optimal result enables the decision model for generating sub-optimal result to track process in succeeding target
In provide more accurate target following as a result, boosting algorithm robustness.
2. the real-time modeling method system in airborne photoelectric platform according to claim 1, it is characterised in that: the step
(1)-(3) further realize as follows:
(1) when extracting sample characteristics, sample image block is divided into nonoverlapping zonule, is counted respectively according to gradient direction
Gradient magnitude in region at pixel, the original feature vector C of composition one 27 are calculated using following formula normalized later
Then son is standardized C, obtained Standardization Operator N (i, j):
N (i, j)=(| | C (i, j) | |2+||C(i+1,j)||2+||C(i-1,j)||2+||C(i,j+1)||2+||C(i,j-1)|
|2)2
Wherein C (i, j) is the image area characteristics vector of the i-th row j column, is standardized, is obtained most using following formula
Whole feature vector F (i, j), the feature vector of each image block collectively constitute clarification of objective representing matrix X；
F (i, j)=max (α, C (i, j)/N (i, j))
Wherein α is an intercept term, for eliminating the excessive noise item of eigenmatrix intermediate value, enables the characteristics of image extracted
Enough more robust expression targets；
(2) initial target image block is acquired, using scaling, rotation, translation, overturning, affine transformation mode, generates a collection of target figure
As positive sample, it is denoted as Tp, while being extracted at random in the background area of image and some being overlapped less with target image or be not overlapped
Image block, as negative sample Tn；It is trained using a variety of obtained positive samples that convert, greatly enhances decision model pair
The robustness accordingly converted；
(3) in new frame image, from stochastical sampling around target previous frame position, a collection of candidate samples, uniform sampling are obtained
A collection of candidate samples are obtained, collectively constitute candidate target sample, it is fast at random for target that stochastical sampling can increase track algorithm
The robustness of speed movement still is able to accurately be captured after uniform sampling can guarantee that target is moved to any direction；
(4) it is directed to visible images and infrared image, constructs decision model D respectivelyv、Dir,
Wherein θv, θirFor model parameter, the feature of x sample in each frame, is calculated two as a result, being denoted as R respectivelyv、Rir:
Determine that result which decision model provides is optimizing decision as a result, as output according to loss function, while with optimal
The result of decision goes amendment to generate the decision model of sub-optimal result, there is the decision model for generating sub-optimal result can in subsequent frames more
Good performance；A loss function can be calculated in n-th frame in each result of decisionTo promote judgement
Accuracy judges optimizing decision result using aggregated loss function:
WhereinFor the loss function of model D, D ∈ Dv,Dir, D*For optimal decision model, Δ n is the time of aggregated loss function
Window length；In object tracking process, combine the result of decision of two decision models, select wherein optimal result as output,
And optimal result is utilized, and amendment updates suboptimum decision-making model, the error message introduced during suboptimum decision-making model running is corrected,
Two decision models are cooperated, are corrected mutually, realize stable and accurate target following.
3. the real-time modeling method system in a kind of airborne photoelectric platform according to claim 1, it is characterised in that: described
Dsp chip is the TMS320C6678 multi-core processor of TI, has 8 independent kernels that can be run parallel, is denoted as core 0-7 respectively,
After data communication module receives the instruction of host computer sending, it is transmitted to the dsp chip of target tracking module, dsp chip needs
The instruction of real-time response host computer, after starting target following, dsp chip needs are completed at the same time instruction response, decode mould from video
Block carries infrared picture data and visible images data and target following works；It devises two and operates in different kernels
On subtask, wherein target following task is responsible for target position resolving, and system control tasks communication control task is responsible for instruction
Response, result output and the carrying of infrared picture data and visible images data, two tasks are separately operable in different
On core, system control tasks provide infrared picture data and visible images data and control instruction, mesh for target following task
Tracing task is marked using infrared picture data and visible images data according to control instruction solving target position, is exported to system
Control task, system control tasks complete the passback of target following result；Core 0-6 operational objective tracing task, using based on infrared
It with the combination decision visual target tracking algorithm of visible images, completes target position and resolves, center 0 is used as main core, completes
The initialization of dsp chip and the overall operation of target following task, core 7 run communication control task, and response system instructs,
Carry infrared picture data and visible images data, output tracking result.
4. the real-time modeling method system in a kind of airborne photoelectric platform according to claim 1, it is characterised in that: described
The step of target following task run on dsp chip in target tracking module are as follows:
(1) after system electrification, the initialization of dsp chip is completed, starts target following task, is ready for target position resolving；
(2) when target following task is in non-tracking state, task is in idle condition, receive host computer sending target with
After track instruction, (3) are gone to；
(3) target initial coordinate information R is extracted from from the Command Information Flow of host computer0, trigger the enhancing inside dsp chip
Memory directly accesses data transmission mechanism, from infrared picture data and visible images data flow from data communication module
Extract initial target region Pv、Pir；
(4) using the infrared picture data and visible images data in initial target region, initialization based on it is infrared with it is visible
The combination decision visual target tracking algorithm of light image constructs decision model D for infrared and visible images respectivelyv、Dir；
(5) when n-th frame image arrives, in the tracking result R of previous framen-1Surrounding acquires candidate samples, extracts the spy of each sample
Sign, judges whether it is target using decision model, obtains the output result R of two decision-making devicesvWith Rir；
(6) aggregated loss function is used to determine optimizing decision as a result, as final output Rn, while being mentioned in final result neighborhood
Sampling originally, is updated decision model, boosting algorithm robustness.
5. the real-time modeling method system in a kind of airborne photoelectric platform according to claim 1, it is characterised in that: described
The step of system control communication control task run on dsp chip in target tracking module are as follows:
(1) system electrification starts core 7, bring into operation communication control task after core 0 completes dsp chip initialization；
(2) when not having instruction to arrive, control task is in idle condition, after receiving the instruction from host computer, to instruction
It is parsed, effective information is passed into core 0 and is specifically executed；After starting target following, core 7 is real-time from data communication module
It carries and comes from front-end image sensor, by the infrared picture data and visible images data of image decoder module, for target
Tracing task uses.
6. a kind of real-time modeling method method in airborne photoelectric platform, which is characterized in that realize that steps are as follows:
(1) infrared sensor and visible light sensor in image capture module, acquire with optical axis infrared picture data with can
Light-exposed image data is transferred to image decoder module by SDI agreement；
(2) the serial infrared image data from image capture module and visible images data flow first pass around SDI and receive core
Piece becomes parallel infrared picture data and visible images data flow, is then passed to fpga chip；Image solution in fpga chip
Coded program use state machine decodes effective infrared picture data from parallel infrared picture data and visible images data flow
With visible images data, it is stored in SRAM array；Image decoding state machine includes stIDLE, stFRAME, stLINE, stWAIT
State, state counts CntV according to vertical synchronizing signal VS, data valid signal DE, vertical direction, horizontal direction counts CntH,
Picturedeep Rows, picturewide Cols signal shift；
(3) after the data communication program in fpga chip receives target following instruction, infrared image is read from SRAM array
Buffer area in data and visible images data to piece, notice dsp chip take infrared picture data away from buffer area and can
Light-exposed image data, while recently received host computer instruction is passed into dsp chip；
(4) dsp chip receives infrared picture data and refers to visible images data, target initial position message and start-up trace
After order, it is based respectively on infrared image and visible images, extracts positive negative sample, training decision model constructs decision model, newly
When one frame image arrives, candidate samples are acquired near target position in previous frame image, extract the spy of each candidate samples
Sign, determines that it is target or background using decision model, obtains target position in a new frame, finally according to target following result and
Infrared picture data is updated with visible images data match plan model；
(5) target following result is transmitted to data communication module from dsp chip, is transferred to rear end system by data communication module later
System.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711422365.3A CN108154523B (en) | 2017-12-25 | 2017-12-25 | A kind of real-time modeling method system and method in airborne photoelectric platform |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711422365.3A CN108154523B (en) | 2017-12-25 | 2017-12-25 | A kind of real-time modeling method system and method in airborne photoelectric platform |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108154523A CN108154523A (en) | 2018-06-12 |
| CN108154523B true CN108154523B (en) | 2018-12-18 |
Family
ID=62465881
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711422365.3A Active CN108154523B (en) | 2017-12-25 | 2017-12-25 | A kind of real-time modeling method system and method in airborne photoelectric platform |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108154523B (en) |
Families Citing this family (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111383231B (en) * | 2018-12-28 | 2023-10-27 | 成都皓图智能科技有限责任公司 | Image segmentation method, device and system based on 3D image |
| CN109978910A (en) * | 2019-02-21 | 2019-07-05 | 华东计算技术研究所（中国电子科技集团公司第三十二研究所） | Target tracking system and method based on DSP |
| CN110084822A (en) * | 2019-05-05 | 2019-08-02 | 中国人民解放军战略支援部队航天工程大学 | A kind of target acquisition real time processing system and method towards the in-orbit application of satellite |
| CN110570441B (en) * | 2019-09-16 | 2020-07-24 | 广州波视信息科技股份有限公司 | Ultra-high definition low-delay video control method and system |
| CN110794713A (en) * | 2019-12-04 | 2020-02-14 | 中国航天空气动力技术研究院 | Reconnaissance type unmanned aerial vehicle photoelectric load simulation training system |
| CN112380079A (en) * | 2020-11-27 | 2021-02-19 | 武汉高德红外股份有限公司 | Injection type image algorithm test system and method based on SDI |
| CN112492373A (en) * | 2020-11-30 | 2021-03-12 | 凯迈（洛阳）测控有限公司 | General type high definition display system based on FPGA for photoelectric platform |
| CN112991200B (en) * | 2021-02-08 | 2024-02-02 | 中国科学院西安光学精密机械研究所 | Method and device for adaptively enhancing infrared image |
| CN113255538B (en) * | 2021-06-01 | 2024-06-14 | 大连理工大学 | FPGA-based infrared weak and small target detection tracking device and method |
| CN113395448B (en) * | 2021-06-15 | 2023-02-21 | 西安视成航空科技有限公司 | Airborne pod image searching, tracking and processing system |
| CN113709399B (en) * | 2021-08-31 | 2024-03-08 | 中国电子科技集团公司第五十八研究所 | Visual target tracking system based on DSP+FPGA |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102314693A (en) * | 2011-07-27 | 2012-01-11 | 中国科学院长春光学精密机械与物理研究所 | Dual mode video target identification tracking system |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080177507A1 (en) * | 2006-10-10 | 2008-07-24 | Mian Zahid F | Sensor data processing using dsp and fpga |
-
2017
- 2017-12-25 CN CN201711422365.3A patent/CN108154523B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102314693A (en) * | 2011-07-27 | 2012-01-11 | 中国科学院长春光学精密机械与物理研究所 | Dual mode video target identification tracking system |
Non-Patent Citations (2)
| Title |
|---|
| A TWO-CHANNEL DETECTION AND TRACK SYSTEM DESIGN BASED ON FPGA AND DSP;HONG ZHANG 等;《4TH INTERNATIONAL CONFERENCE ON SOFTWARE TECHNOLOGY AND ENGINEERING (ICSTE 2012)》;20121231;摘要，第1-5节、图1-2 * |
| 基于TMS320C6678平台的粒子滤波目标跟踪设计与仿真;王诗琴 等;《传感器世界》;20150531;第21卷(第5期);第36-40页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108154523A (en) | 2018-06-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108154523B (en) | A kind of real-time modeling method system and method in airborne photoelectric platform | |
| US11055521B2 (en) | Real-time gesture recognition method and apparatus | |
| US20220036050A1 (en) | Real-time gesture recognition method and apparatus | |
| He et al. | Bounding box regression with uncertainty for accurate object detection | |
| CN111104960B (en) | Sign language identification method based on millimeter wave radar and machine vision | |
| CN105631870A (en) | Star image registration and target track extraction method based on NoC framework and device thereof | |
| CN104821010A (en) | Binocular-vision-based real-time extraction method and system for three-dimensional hand information | |
| CN109871813B (en) | Real-time image tracking method and system | |
| CN110555408A (en) | Single-camera real-time three-dimensional human body posture detection method based on self-adaptive mapping relation | |
| CN112668492A (en) | Behavior identification method for self-supervised learning and skeletal information | |
| CN116524593A (en) | Dynamic gesture recognition method, system, equipment and medium | |
| CN112950673A (en) | Target object detection and tracking system and method | |
| Zhang et al. | Gesture recognition system based on improved YOLO v3 | |
| Li et al. | IIE-SegNet: Deep semantic segmentation network with enhanced boundary based on image information entropy | |
| Tian et al. | High-speed tiny tennis ball detection based on deep convolutional neural networks | |
| Lv et al. | A feedback-driven DNN inference acceleration system for edge-assisted video analytics | |
| CN102077268A (en) | Digital video filter and image processing | |
| Wang et al. | Summary of object detection based on convolutional neural network | |
| CN112884803A (en) | Real-time intelligent monitoring target detection method and device based on DSP | |
| Li et al. | Lightweight particle filter for robust visual tracking | |
| Han et al. | Cigarette Detection Algorithm Based on Improved Faster R-CNN | |
| Gao et al. | Study of improved Yolov5 algorithms for gesture recognition | |
| Shi et al. | Explicit Visual Prompts for Visual Object Tracking | |
| Cereda et al. | Improving the generalization capability of dnns for ultra-low power autonomous nano-uavs | |
| CN112215174A (en) | Sanitation vehicle state analysis method based on computer vision |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |