CN112241763A - Multi-source multi-mode dynamic information fusion and cognition method and system - Google Patents

Multi-source multi-mode dynamic information fusion and cognition method and system Download PDF

Info

Publication number
CN112241763A
CN112241763A CN202011120817.4A CN202011120817A CN112241763A CN 112241763 A CN112241763 A CN 112241763A CN 202011120817 A CN202011120817 A CN 202011120817A CN 112241763 A CN112241763 A CN 112241763A
Authority
CN
China
Prior art keywords
information
fusion
data
target
radar
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202011120817.4A
Other languages
Chinese (zh)
Inventor
高洪波
黄鹏博
李智军
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
University of Science and Technology of China USTC
Original Assignee
University of Science and Technology of China USTC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by University of Science and Technology of China USTC filed Critical University of Science and Technology of China USTC
Priority to CN202011120817.4A priority Critical patent/CN112241763A/en
Publication of CN112241763A publication Critical patent/CN112241763A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/25—Fusion techniques
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
    • G01S7/02—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S13/00
    • G01S7/41—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S13/00 using analysis of echo signal for target characterisation; Target signature; Target cross-section
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/25—Determination of region of interest [ROI] or a volume of interest [VOI]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20212—Image combination
    • G06T2207/20221—Image fusion; Image merging

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Evolutionary Biology (AREA)
  • Evolutionary Computation (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • General Engineering & Computer Science (AREA)
  • Artificial Intelligence (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Multimedia (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Remote Sensing (AREA)
  • Measurement Of Velocity Or Position Using Acoustic Or Ultrasonic Waves (AREA)

Abstract

The invention provides a multi-source multi-mode dynamic information fusion and cognition method and a system, which comprises the following steps: a camera image fusion step: using a plurality of cameras to acquire camera images, and preprocessing and fusing the acquired images to obtain an image information fusion result; radar information fusion: collecting target information by using a plurality of radars, and fusing and detecting the collected target information to obtain a target detection result; acoustic vector sensor tracking: acquiring target sound signals by using a plurality of sound vector sensors, and fusing and tracking target sound signal information to obtain a target tracking result; radar information and visual information fusion step: and fusing the image information fusion result, the target detection result and the target tracking result to generate a target region of interest. The invention is respectively used for collecting image information, radar information and sound information, the three types of information are mutually combined and have complementary advantages, and the limitation of a single sensor is made up.

Description

Multi-source multi-mode dynamic information fusion and cognition method and system
Technical Field
The invention relates to the technical field of epidemic prevention robots, in particular to a multi-source multi-mode dynamic information fusion and cognition method and system. In particular to a multi-source multi-mode dynamic information fusion and cognition method for an epidemic prevention robot in a complex environment.
Background
At present, under the condition of large waves and tides of information technology, robots are very commonly applied, and epidemic prevention robots are one type of robots. The epidemic prevention robot plays an important role in hospitals and areas after earthquake, can serve as a medicine delivery worker, a disinfector and the like, and can also be better competent for transporting and carrying some medical instruments. Because the working environment of the epidemic prevention robot is generally complex, a plurality of sensors are generally arranged on the epidemic prevention robot, the advantages of the sensors are complementary, information fusion is carried out, and the epidemic prevention task is completed together.
A large number of sensors are arranged on the epidemic prevention robot body, the sensors have the functions of eyes, ears and other organs of a human body and are mainly used for capturing external information. Information fusion is a method for processing information aiming at multiple sensors. Sensor information fusion is the key point of research in all countries in recent years, and a great amount of human resources and material resources are put into all countries in the world to carry out theoretical and practical research on the sensor information fusion technology. The information fusion of the sensors can be divided into two types according to different fusion types, wherein one type is the information fusion among the sensors of the same type, and the other type is the information fusion among the sensors of different types. For the information fusion of the similar sensors, the method mainly aims to obtain more accurate targets and more extensive information; for heterogeneous sensor information fusion, the main function is to combine the advantages of each sensor and complement the advantages of the functions of a single sensor, so that the effect of the sensor can be fully exerted.
The schematic diagram of the multi-sensor information fusion is shown in fig. 2, information which cannot be acquired by a single sensor can be realized by using the multi-sensor information fusion, the capability of the system for processing information can be effectively expanded, and meanwhile, by using redundant information provided by the multi-sensor, when one sensor fails, the information provided by the other sensor can be used, so that the robustness of the system is improved. In addition, a plurality of same-type sensors or a plurality of different-type sensors carry out comprehensive processing on information through information fusion, the detection area of the sensors can be enlarged, more comprehensive, more accurate and information data at different moments can be obtained, the detection capability of the system can be improved, and the limitation of information measurement caused by the adoption of a single sensor can be avoided.
When the sensor information fusion is implemented specifically, the sensor information fusion mainly comprises two aspects of processing, firstly, when the data of the sensor is processed, data integration needs to be carried out on the data of the same type, data of different types are subjected to data type conversion and converted into the data of the same type, and then when the data processed in the last step is fused, some methods need to be adopted to optimize data results, so that effective information can be accurately obtained.
The target detection is used for detecting a specific target, and the data measured by multiple sensors are fused by a fusion algorithm by utilizing a multi-sensor information fusion technology, so that more accurate information than that of a single sensor can be obtained. The information fusion structure comprises data level fusion, characteristic level fusion and the like, wherein the sensors of the data level information fusion are required to be of the same type, original observation information is correlated and then subjected to data layer fusion, and the characteristic level fusion is to extract the characteristics of targets observed by the sensors and then perform fusion processing on the extracted characteristic information.
Compared with the conventional sound pressure sensor which cannot utilize the sound wave vibration velocity information, the sound vector sensor can simultaneously acquire the sound pressure and vibration velocity information of target sound, can measure abundant information and has stronger direction sensitivity.
Patent document CN108392795A (application number: 201810109937.0) discloses a multi-modal control method for a rehabilitation robot based on multi-information fusion, which uses biological information and motion information of a patient as the basis of the adjustment of training parameters of the rehabilitation robot and parameters of a control system, i.e. motion commands or given signals of the control system. The method obtains information of surface electromyographic signals, electroencephalogram signals, electrocardiosignals, joint angles, joint angular velocities, joint moments, contact forces and the like of a patient, dynamically adjusts training parameters and control system parameters by using an information fusion algorithm, automatically switches training modes, and can realize upper and lower limb cooperative intelligent control and flexible control of the rehabilitation robot.
Disclosure of Invention
Aiming at the defects in the prior art, the invention aims to provide a multi-source multi-mode dynamic information fusion and cognition method and a multi-source multi-mode dynamic information fusion and cognition system.
The multi-source multi-mode dynamic information fusion and cognition method provided by the invention comprises the following steps:
a camera image fusion step: using a plurality of cameras to acquire camera images, and preprocessing and fusing the acquired images to obtain an image information fusion result;
radar information fusion: collecting target information by using a plurality of radars, and fusing and detecting the collected target information to obtain a target detection result;
acoustic vector sensor tracking: acquiring target sound signals by using a plurality of sound vector sensors, and fusing and tracking target sound signal information to obtain a target tracking result;
radar information and visual information fusion step: and fusing the image information fusion result, the target detection result and the target tracking result to generate a target region of interest.
Preferably, the plurality of cameras form a parallel structure, image acquisition is respectively carried out, acquired images are preprocessed to obtain data-level image information, the data-level image information is fused, image information with the precision meeting the preset requirement is obtained, and the time coverage range is expanded.
Preferably, the plurality of radars form a parallel structure, respectively acquire information of the target, preprocess the acquired target information to obtain data-level target information, fuse the data-level target information, acquire target detection information with accuracy meeting preset requirements, and expand a target detection range.
Preferably, the plurality of acoustic vector sensors form a parallel structure, respectively collect target sound signals, preprocess the collected target sound signals to obtain data-level target sound signals, fuse the data-level target sound signals, and obtain target sound information with precision meeting preset requirements.
Preferably, the fusing comprises:
adopting a pixel-level fusion algorithm based on a weighted average method, wherein the formula is as follows:
F(i,j)＝ωA×A(i,j)+ωB×B(i,j)
wherein A, B is the two images and F is the image after fusion; i represents the ith row of the image; j represents the jth column of the image; omegaARepresenting the weight applied to image a.
Preferably, two radars R are providedhAnd RlWherein the high-precision radar RhWith a sampling period of 2s, low-precision radar RlThe sampling period of (3) s, and the data are time-sequenced by using an interpolation extrapolation method;
the method comprises the steps of performing incremental sequencing on each data of the radar, interpolating and extrapolating the data on the observation time of the high-precision radar to the data on the time point of the low-precision radar, taking the minimum time interval after interpolation and extrapolation as a basic time unit, and interpolating and extrapolating the data on other time points according to the basic time unit to obtain the data with equal intervals.
Preferably, the data after time sequencing is subjected to data level fusion, a plurality of groups of target information data collected by the radar are fused into a group of data, and a least square method is adopted for fusion during radar information fusion, wherein the fusion method comprises the following steps:
is provided with Zk＝(z1,z2,…,zn)TIs tk-1To tkN data sets collected by the radar array at a moment
Figure BDA0002731962660000031
Z in (1) and
Figure BDA0002731962660000032
denotes z1,z2,…,znFused measured values and derivatives thereof, radar measured values ziExpressed as:
Figure BDA0002731962660000033
vtrepresenting the measurement noise, the above equation is written in matrix form:
Zn＝WnU+Vn(ii) a Wherein, Vn＝(v1,v2,…,vn)TThe mean is 0 and the covariance matrix is:
Figure BDA0002731962660000034
wherein σrMeasuring the noise variance before fusion;
Figure BDA0002731962660000035
according to the least squares criterion:
Figure BDA0002731962660000041
j is to
Figure BDA0002731962660000042
A minimum defined variable;
Figure BDA0002731962660000043
and U have the same meaning;
to minimize J, the derivation is done on both sides of the above equation:
Figure BDA0002731962660000044
then there are:
Figure BDA0002731962660000045
Figure BDA0002731962660000046
and z have the same meaning;
Figure BDA0002731962660000047
and
Figure BDA0002731962660000048
have the same meaning;
the variance matrix estimates are:
Figure BDA0002731962660000049
fusing the n measured values to obtain the measured value at the k moment and the noise variance as follows:
Figure BDA00027319626600000410
Figure BDA00027319626600000411
Figure BDA00027319626600000412
z (k) represents the fused measurement value at time k.
The multi-source multi-mode dynamic information fusion and cognition system provided by the invention comprises the following components:
a camera image fusion module: using a plurality of cameras to acquire camera images, and preprocessing and fusing the acquired images to obtain an image information fusion result;
the radar information fusion module: collecting target information by using a plurality of radars, and fusing and detecting the collected target information to obtain a target detection result;
acoustic vector sensor tracking module: acquiring target sound signals by using a plurality of sound vector sensors, and fusing and tracking target sound signal information to obtain a target tracking result;
radar information and visual information fusion module: and fusing the image information fusion result, the target detection result and the target tracking result to generate a target region of interest.
Preferably, the plurality of cameras form a parallel structure, image acquisition is respectively carried out, acquired images are preprocessed to obtain data-level image information, the data-level image information is fused, and image information with the precision meeting the preset requirement and the time coverage range are expanded;
the multiple radars form a parallel structure, respectively acquire information of a target, preprocess the acquired target information to obtain data-level target information, fuse the data-level target information to obtain target detection information with the precision meeting the preset requirement and expand the target detection range;
the plurality of acoustic vector sensors form a parallel structure, respectively collect target sound signals, preprocess the collected target sound signals to obtain data-level target sound signals, fuse the data-level target sound signals, and obtain target sound information with the precision meeting the preset requirement.
Preferably, the fusing comprises:
adopting a pixel-level fusion algorithm based on a weighted average method, wherein the formula is as follows:
F(i,j)＝ωA×A(i,j)+ωB×B(i,j)
wherein A, B is the two images and F is the image after fusion; i represents the ith row of the image; j represents the jth column of the image; omegaARepresents the weight applied to image a;
is provided with two radars RhAnd RlWherein the high-precision radar RhWith a sampling period of 2s, low-precision radar RlThe sampling period of (3) s, and the data are time-sequenced by using an interpolation extrapolation method;
the method comprises the steps of firstly, carrying out increment sequencing on each data of the radar, then carrying out interpolation extrapolation on the data at the observation time of the high-precision radar to the data at the time point of the low-precision radar, finally taking the minimum time interval after interpolation extrapolation as a basic time unit, and carrying out interpolation extrapolation according to the basic time unit to obtain the data at other time points, thereby obtaining the data at equal intervals;
performing data level fusion on the data after time sequencing, fusing a plurality of groups of target information data acquired by the radar into a group of data, and fusing by adopting a least square method during radar information fusion, wherein the fusion method comprises the following steps:
is provided with Zk＝(z1,z2,…,zn)TIs tk-1To tkN data sets collected by the radar array at a moment
Figure BDA0002731962660000051
Z in (1) and
Figure BDA0002731962660000052
denotes z1,z2,…,znFused measured values and derivatives thereof, radar measured values ziExpressed as:
Figure BDA0002731962660000053
vtrepresenting the measurement noise, the above equation is written in matrix form:
Zn＝WnU+Vn(ii) a Wherein, Vn＝(v1,v2,…,vn)TThe mean is 0 and the covariance matrix is:
Figure BDA0002731962660000054
wherein σrMeasuring the noise variance before fusion;
Figure BDA0002731962660000055
according to the least squares criterion:
Figure BDA0002731962660000056
j is to
Figure BDA0002731962660000057
A minimum defined variable;
Figure BDA0002731962660000058
and U have the same meaning;
to minimize J, the derivation is done on both sides of the above equation:
Figure BDA0002731962660000059
then there are:
Figure BDA00027319626600000510
Figure BDA0002731962660000061
and z have the same meaning;
Figure BDA0002731962660000062
and
Figure BDA0002731962660000063
have the same meaning;
the variance matrix estimates are:
Figure BDA0002731962660000064
fusing the n measured values to obtain the measured value at the k moment and the noise variance as follows:
Figure BDA0002731962660000065
Figure BDA0002731962660000066
Figure BDA0002731962660000067
z (k) represents the fused measurement value at time k.
Compared with the prior art, the invention has the following beneficial effects:
1. the three sensors, namely the camera, the acoustic vector sensor and the radar, are used for tracking the target and generating the region of interest, have advantages and are respectively used for acquiring image information, radar information and sound information, and the three types of information are combined with one another and have complementary advantages, so that the limitation of a single sensor is overcome;
2. according to the invention, by using a data level fusion method, image information, target information acquired by a radar and sound information acquired by a sound vector sensor are respectively subjected to data level fusion, so that the space and time scope of expanded information can be obtained, and more accurate target information can be obtained;
3. the method fuses radar information and visual information, utilizes the radar to detect the target, then projects the information onto the collected image to generate the region of interest, and the image only processes the target in the region of interest, thereby greatly reducing the operation amount;
4. the invention can be used for detecting the sound information of the target and tracking the target by designing the sound vector sensor part, and simultaneously can track the target and generate the interested region on the image by combining with the generated interested region.
Drawings
Other features, objects and advantages of the invention will become more apparent upon reading of the detailed description of non-limiting embodiments with reference to the following drawings:
FIG. 1 is an overall schematic view of the present invention;
FIG. 2 is a schematic diagram of multi-sensor information fusion in accordance with the present invention;
FIG. 3 is a schematic diagram of data level fusion according to the present invention;
FIG. 4 is a schematic view of a portion of a camera image of the present invention;
FIG. 5 is a schematic diagram of a portion of radar information according to the present invention;
FIG. 6 is a schematic diagram of an interpolation-extrapolation temporal registration of the present invention;
FIG. 7 is a schematic view of the tracking portion of the acoustic vector sensor of the present invention;
FIG. 8 is a schematic diagram of the fusion of radar and visual information according to the present invention.
Detailed Description
The present invention will be described in detail with reference to specific examples. The following examples will assist those skilled in the art in further understanding the invention, but are not intended to limit the invention in any way. It should be noted that it would be obvious to those skilled in the art that various changes and modifications can be made without departing from the spirit of the invention. All falling within the scope of the present invention.
Example (b):
referring to fig. 1, the multi-source multi-mode dynamic information fusion and cognition method for complex environment of epidemic prevention robot provided by the invention can be divided into a camera image part, a radar information part, an acoustic vector sensor tracking part and a radar information and visual information fusion part. The four parts specifically comprise a plurality of cameras, camera image acquisition, acquired image preprocessing, image information data level fusion, a plurality of radars, target information acquisition by using the radars, data level fusion of the acquired information of the targets, target detection, radar and visual information fusion, target interesting region generation, a plurality of acoustic vector sensors, target sound signal acquisition by using the acoustic vector sensors, data level fusion of the sound signal information, target tracking and combination of the generated interesting regions. For the camera image part, the camera image part is composed of a plurality of cameras, camera image acquisition, acquired image preprocessing and image information data level fusion, wherein the plurality of cameras form a parallel structure, respectively acquire images and then perform image data level fusion; the radar information part is composed of a plurality of radars, target information acquisition is carried out by utilizing the radars, data-level fusion and target detection are carried out on the information acquired by the targets, and as with the camera image part, the radars in the radar information part also form a parallel structure, respectively carry out information acquisition on the targets and then carry out data-level fusion; for the acoustic vector sensor tracking part, the acoustic vector sensor tracking part is composed of a plurality of acoustic vector sensors, utilizes the acoustic vector sensors to collect target sound signals, performs data level fusion on sound signal information and tracks a target, is the same as the camera image part, and the acoustic vector sensors form a parallel structure, respectively collects the target sound signals and then performs data level fusion; and for the radar information and visual information fusion part, connecting the radar information part and the camera image part, combining the target detection result obtained by the radar information part and the image information fusion result obtained by the camera image part, and fusing the radar and visual information to generate a target region of interest. The four parts are combined into one block as an abstract figure to form the complex environment multi-source multi-mode dynamic information fusion and cognition method of the epidemic prevention robot.
Referring to fig. 1, 2, 3 and 4, a multisource multimode dynamic information fusion and cognition method for complex environment of epidemic prevention robot is characterized in that: by constructing an image part, a parallel structure is formed by a plurality of cameras, image information is acquired by each camera, and then preprocessing and image information data-level fusion are performed to obtain more accurate image information and expand the image information space and time coverage.
In the parallel structure, the original homogeneous data obtained by the sensor is directly fused, so that the data loss is low, more comprehensive information can be provided for follow-up, and the parallel structure has the characteristic of high precision.
For data-level fusion, more accurate and wider information can be obtained by fusing data of a plurality of sensors of the same type. Because the cameras form a parallel structure, the images are acquired by the cameras independently, and the sampling periods of the cameras are not necessarily the same, so that the time difference problem exists when the data acquired by the cameras are fused, the image data fusion is performed in two steps, wherein the first step is firstly time registration and then fusion. For the time registration, in the image data level fusion, the current image information is extracted from the image information sequences acquired by the cameras respectively at the same time T, so that the time registration can be carried out. For the fusion algorithm, a pixel-level fusion algorithm based on a weighted average method is adopted, the algorithm can effectively suppress the noise of the input image, and the principle of the weighted average method is as follows:
F(i,j)＝ωA×A(i,j)+ωB×B(i,j)
where A, B is the two images and F is the image after fusion.
Referring to fig. 1, 2, 3 and 5, a multisource multimode dynamic information fusion and cognition method for complex environment of epidemic prevention robot is characterized in that: by constructing a radar information part, forming a parallel structure by a plurality of radars, collecting target information in parallel, and then carrying out data-level information fusion on the information, more accurate target detection information can be obtained and the range of target detection can be enlarged.
The method is characterized in that the data on the measurement time of the high-precision radar are calculated to the low-precision observation time by adopting an interpolation extrapolation method for radar information time alignment, so that the time synchronization of the two radars is achieved. Is provided with two radars RhAnd RlWherein the high-precision radar RhWith a sampling period of 2s, low-precision radar RlThe sampling period of (2) is 3s, and the principle of using interpolation extrapolation is as follows: the method comprises the steps of performing incremental sequencing on each data of the radar, interpolating and extrapolating the data on the observation time of the high-precision radar to the data on the time point of the low-precision radar, taking the minimum time interval after interpolation and extrapolation as a basic time unit, and interpolating and extrapolating the data on other time points according to the time unit to obtain a series of data with equal intervals. This principle is illustrated in fig. 6.
At a time of usingAfter inter-registration, data-level fusion can be performed, a plurality of groups of target information data collected by the radar are fused into a group of data, and a least square method is adopted for fusion during radar information fusion, wherein the specific fusion method comprises the following steps: is provided with Zk＝(z1,z2,…,zn)TIs tk-1To tkN data sets collected by the radar array at a moment
Figure BDA0002731962660000081
Z in (1) and
Figure BDA0002731962660000091
denotes z1,z2,…,znFused measured values and derivatives thereof, radar measured values ziCan be expressed in the following form:
Figure BDA0002731962660000092
wherein v istExpressed as measurement noise, the above equation is written in matrix form:
Zn＝WnU+Vn
wherein Vn＝(v1,v2,…,vn)TThe mean is 0 and the covariance matrix is:
Figure BDA0002731962660000093
wherein sigmarThe noise variance is measured before fusion.
Figure BDA0002731962660000094
According to the least squares criterion:
Figure BDA0002731962660000095
to minimize J, the derivation is done on both sides of the above equation:
Figure BDA0002731962660000096
then there are:
Figure BDA0002731962660000097
the variance matrix estimate is:
Figure BDA0002731962660000098
fusing the n measured values to obtain a measured value at the moment k and the noise variance as follows:
Figure BDA0002731962660000099
Figure BDA00027319626600000910
wherein
Figure BDA00027319626600000911
After data fusion, more accurate target detection information after fusion can be obtained.
Referring to fig. 1, 2, 3 and 7, a multisource multimode dynamic information fusion and cognition method for complex environment of epidemic prevention robot is characterized in that: by constructing an acoustic vector sensor tracking part, forming a parallel structure by a plurality of acoustic vector sensors, acquiring target sound signals in parallel, and then performing data-level fusion on the acquired information, more accurate target sound information can be obtained.
The method is characterized in that a plurality of acoustic vector sensors form a parallel structure as a radar information part and a camera image part, and after target sound signals are collected by each acoustic vector sensor, data collected by each acoustic vector sensor needs to be subjected to time registration before data level fusion is carried out, and then fusion is carried out. In the invention, the time registration and the data fusion of the tracking part of the acoustic vector sensor are the same as the radar information part, the time registration is carried out by adopting an interpolation extrapolation method, and the data fusion is carried out by adopting a least square method.
Referring to fig. 1, 2, 3 and 8, a multisource multimode dynamic information fusion and cognition method for complex environment of epidemic prevention robot is characterized in that: by constructing a radar information and visual information fusion part, target detection information output by the radar information part and fusion image information output by the camera image part are combined for information fusion, so that the region of interest is generated.
The image data level fusion information and the target detection information come from different sensors, direct interaction does not exist between the information of the two sensors, and sampling frequencies of the different sensors are different, so that the target position information acquired by the respective sensors needs to be aligned in time and integrated when the information is fused. In the time alignment strategy, the interpolation extrapolation method is still adopted to align time, radar information is fused into image information, and actually the radar information is converted into an image information coordinate system, namely the radar information and the image information are converted into a uniform space dimension. By mapping the radar coordinate system into the image coordinate system, the fusion of the target detected by the radar into the image information is completed, thereby generating the target region of interest.
The multi-source multi-mode dynamic information fusion and cognition system provided by the invention comprises the following components:
a camera image fusion module: using a plurality of cameras to acquire camera images, and preprocessing and fusing the acquired images to obtain an image information fusion result;
the radar information fusion module: collecting target information by using a plurality of radars, and fusing and detecting the collected target information to obtain a target detection result;
acoustic vector sensor tracking module: acquiring target sound signals by using a plurality of sound vector sensors, and fusing and tracking target sound signal information to obtain a target tracking result;
radar information and visual information fusion module: and fusing the image information fusion result, the target detection result and the target tracking result to generate a target region of interest.
Those skilled in the art will appreciate that, in addition to implementing the systems, apparatus, and various modules thereof provided by the present invention in purely computer readable program code, the same procedures can be implemented entirely by logically programming method steps such that the systems, apparatus, and various modules thereof are provided in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers and the like. Therefore, the system, the device and the modules thereof provided by the present invention can be considered as a hardware component, and the modules included in the system, the device and the modules thereof for implementing various programs can also be considered as structures in the hardware component; modules for performing various functions may also be considered to be both software programs for performing the methods and structures within hardware components.
The foregoing description of specific embodiments of the present invention has been presented. It is to be understood that the present invention is not limited to the specific embodiments described above, and that various changes or modifications may be made by one skilled in the art within the scope of the appended claims without departing from the spirit of the invention. The embodiments and features of the embodiments of the present application may be combined with each other arbitrarily without conflict.

Claims (10)

1. A multi-source multi-mode dynamic information fusion and cognition method is characterized by comprising the following steps:
a camera image fusion step: using a plurality of cameras to acquire camera images, and preprocessing and fusing the acquired images to obtain an image information fusion result;
radar information fusion: collecting target information by using a plurality of radars, and fusing and detecting the collected target information to obtain a target detection result;
acoustic vector sensor tracking: acquiring target sound signals by using a plurality of sound vector sensors, and fusing and tracking target sound signal information to obtain a target tracking result;
radar information and visual information fusion step: and fusing the image information fusion result, the target detection result and the target tracking result to generate a target region of interest.
2. The multi-source multi-mode dynamic information fusion and cognition method according to claim 1, wherein the plurality of cameras form a parallel structure, image acquisition is respectively performed, the acquired images are preprocessed to obtain data-level image information, the data-level image information is fused to obtain image information with precision meeting preset requirements, and the time coverage range is expanded.
3. The multi-source multi-mode dynamic information fusion and cognition method according to claim 1, wherein the plurality of radars form a parallel structure, respectively collect information of targets, preprocess the collected target information to obtain data-level target information and perform fusion, obtain target detection information with accuracy meeting preset requirements, and expand target detection range.
4. The multi-source multi-mode dynamic information fusion and cognition method according to claim 1, wherein the plurality of acoustic vector sensors form a parallel structure, respectively collect target sound signals, preprocess the collected target sound signals to obtain data-level target sound signals, perform fusion, and obtain target sound information with precision meeting preset requirements.
5. The multi-source multimodal dynamic information fusion and cognition method according to claim 1, wherein the fusing comprises:
adopting a pixel-level fusion algorithm based on a weighted average method, wherein the formula is as follows:
F(i，j)＝ωA×A(i，j)+ωB×B(i，j)
wherein A, B is the two images and F is the image after fusion; i represents the ith row of the image; j represents the jth column of the image; omegaARepresenting the weight applied to image a.
6. The multi-source multi-mode dynamic information fusion and cognition method according to claim 5, wherein two radars R are providedhAnd RlWherein the high-precision radar RhWith a sampling period of 2s, low-precision radar RlThe sampling period of (3) s, and the data are time-sequenced by using an interpolation extrapolation method;
the method comprises the steps of performing incremental sequencing on each data of the radar, interpolating and extrapolating the data on the observation time of the high-precision radar to the data on the time point of the low-precision radar, taking the minimum time interval after interpolation and extrapolation as a basic time unit, and interpolating and extrapolating the data on other time points according to the basic time unit to obtain the data with equal intervals.
7. The multi-source multi-mode dynamic information fusion and cognition method according to claim 6, characterized in that data level fusion is performed on time-sequenced data, a plurality of groups of target information data collected by radar are fused into a group of data, and a least square method is adopted for fusion during radar information fusion, and the fusion method is as follows:
is provided with Zk＝(z1，z2，…，zn)TIs tk-1To tkN data sets collected by the radar array at a moment,
Figure FDA0002731962650000021
z in (1) and
Figure FDA0002731962650000022
denotes z1，z2，…，znFused measured values and derivatives thereof, radar measured values ziExpressed as:
Figure FDA0002731962650000023
vtrepresenting the measurement noise, the above equation is written in matrix form:
Zn＝WnU+Vn(ii) a Wherein, Vn＝(v1，v2…，vn)TThe mean is 0 and the covariance matrix is:
Figure FDA0002731962650000024
wherein σrMeasuring the noise variance before fusion;
Figure FDA0002731962650000025
according to the least squares criterion:
Figure FDA0002731962650000026
j is to
Figure FDA0002731962650000027
A minimum defined variable;
Figure FDA0002731962650000028
and U have the same meaning;
to minimize J, the derivation is done on both sides of the above equation:
Figure FDA0002731962650000029
then there are:
Figure FDA00027319626500000210
Figure FDA00027319626500000211
and z have the same meaning;
Figure FDA00027319626500000212
and
Figure FDA00027319626500000217
have the same meaning;
the variance matrix estimates are:
Figure FDA00027319626500000213
fusing the n measured values to obtain the measured value at the k moment and the noise variance as follows:
Figure FDA00027319626500000214
Figure FDA00027319626500000215
Figure FDA00027319626500000216
z (k) represents the fused measurement value at time k.
8. A multi-source multi-mode dynamic information fusion and cognition system is characterized by comprising:
a camera image fusion module: using a plurality of cameras to acquire camera images, and preprocessing and fusing the acquired images to obtain an image information fusion result;
the radar information fusion module: collecting target information by using a plurality of radars, and fusing and detecting the collected target information to obtain a target detection result;
acoustic vector sensor tracking module: acquiring target sound signals by using a plurality of sound vector sensors, and fusing and tracking target sound signal information to obtain a target tracking result;
radar information and visual information fusion module: and fusing the image information fusion result, the target detection result and the target tracking result to generate a target region of interest.
9. The multi-source multi-mode dynamic information fusion and cognition system according to claim 8, wherein the plurality of cameras form a parallel structure, image acquisition is performed respectively, the acquired images are preprocessed to obtain data-level image information, the data-level image information is fused to obtain image information with precision meeting preset requirements, and the time coverage is expanded;
the multiple radars form a parallel structure, respectively acquire information of a target, preprocess the acquired target information to obtain data-level target information, fuse the data-level target information to obtain target detection information with the precision meeting the preset requirement and expand the target detection range;
the plurality of acoustic vector sensors form a parallel structure, respectively collect target sound signals, preprocess the collected target sound signals to obtain data-level target sound signals, fuse the data-level target sound signals, and obtain target sound information with the precision meeting the preset requirement.
10. The multi-source multimodal dynamic information fusion and cognition system according to claim 9 wherein the fusing comprises:
adopting a pixel-level fusion algorithm based on a weighted average method, wherein the formula is as follows:
F(i，j)＝ωA×A(i，j)+ωB×B(i，j)
wherein A, B is the two images and F is the image after fusion; i represents the ith row of the image; j represents the jth column of the image; omegaARepresents the weight applied to image a;
is provided with two radars RhAnd RlWherein the high-precision radar RhWith a sampling period of 2s, low-precision radar RlThe sampling period of (3) s, and the data are time-sequenced by using an interpolation extrapolation method;
the method comprises the steps of firstly, carrying out increment sequencing on each data of the radar, then carrying out interpolation extrapolation on the data at the observation time of the high-precision radar to the data at the time point of the low-precision radar, finally taking the minimum time interval after interpolation extrapolation as a basic time unit, and carrying out interpolation extrapolation according to the basic time unit to obtain the data at other time points, thereby obtaining the data at equal intervals;
performing data level fusion on the data after time sequencing, fusing a plurality of groups of target information data acquired by the radar into a group of data, and fusing by adopting a least square method during radar information fusion, wherein the fusion method comprises the following steps:
is provided with Zk＝(z1，z2，…，zn)TIs tk-1To tkN data sets collected by the radar array at a moment
Figure FDA0002731962650000041
Z in (1) and
Figure FDA0002731962650000042
denotes z1，z2，…，znFused measured values and derivatives thereof, radar measured values ziExpressed as:
Figure FDA0002731962650000043
vtrepresenting the measurement noise, the above equation is written in matrix form:
Zn＝WnU+Vn(ii) a Wherein, Vn＝(v1，v2…，vn)TThe mean is 0 and the covariance matrix is:
Figure FDA0002731962650000044
wherein σrMeasuring the noise variance before fusion;
Figure FDA0002731962650000045
according to the least squares criterion:
Figure FDA0002731962650000046
j is to
Figure FDA0002731962650000047
A minimum defined variable;
Figure FDA0002731962650000048
and U have the same meaning;
to minimize J, the derivation is done on both sides of the above equation:
Figure FDA0002731962650000049
then there are:
Figure FDA00027319626500000410
Figure FDA00027319626500000411
and z have the same meaning;
Figure FDA00027319626500000412
and
Figure FDA00027319626500000413
have the same meaning;
the variance matrix estimates are:
Figure FDA00027319626500000414
fusing the n measured values to obtain the measured value at the k moment and the noise variance as follows:
Figure FDA00027319626500000415
Figure FDA00027319626500000416
Figure FDA00027319626500000417
z (k) represents the fused measurement value at time k.
CN202011120817.4A 2020-10-19 2020-10-19 Multi-source multi-mode dynamic information fusion and cognition method and system Pending CN112241763A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202011120817.4A CN112241763A (en) 2020-10-19 2020-10-19 Multi-source multi-mode dynamic information fusion and cognition method and system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011120817.4A CN112241763A (en) 2020-10-19 2020-10-19 Multi-source multi-mode dynamic information fusion and cognition method and system

Publications (1)

Publication Number Publication Date
CN112241763A true CN112241763A (en) 2021-01-19

Family

ID=74169027

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202011120817.4A Pending CN112241763A (en) 2020-10-19 2020-10-19 Multi-source multi-mode dynamic information fusion and cognition method and system

Country Status (1)

Country Link
CN (1) CN112241763A (en)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN1389710A (en) * 2002-07-18 2003-01-08 上海交通大学 Multiple-sensor and multiple-object information fusing method
CN105491122A (en) * 2015-12-02 2016-04-13 中国民用航空总局第二研究所 System for fusing and exchanging data among multiple control centers
CN110428008A (en) * 2019-08-02 2019-11-08 深圳市唯特视科技有限公司 A kind of target detection and identification device and method based on more merge sensors
KR20200095367A (en) * 2019-01-31 2020-08-10 주식회사 스트라드비젼 Learning method and learning device for sensor fusion to integrate information acquired by radar capable of distance estimation and information acquired by camera to thereby improve neural network for supporting autonomous driving, and testing method and testing device using the same
CN111516605A (en) * 2020-04-28 2020-08-11 上汽大众汽车有限公司 Multi-sensor monitoring equipment and monitoring method

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN1389710A (en) * 2002-07-18 2003-01-08 上海交通大学 Multiple-sensor and multiple-object information fusing method
CN105491122A (en) * 2015-12-02 2016-04-13 中国民用航空总局第二研究所 System for fusing and exchanging data among multiple control centers
KR20200095367A (en) * 2019-01-31 2020-08-10 주식회사 스트라드비젼 Learning method and learning device for sensor fusion to integrate information acquired by radar capable of distance estimation and information acquired by camera to thereby improve neural network for supporting autonomous driving, and testing method and testing device using the same
CN110428008A (en) * 2019-08-02 2019-11-08 深圳市唯特视科技有限公司 A kind of target detection and identification device and method based on more merge sensors
CN111516605A (en) * 2020-04-28 2020-08-11 上汽大众汽车有限公司 Multi-sensor monitoring equipment and monitoring method

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
丁建江等: "《雷达组网技术》", 31 December 2017, 国防工业出版社 *
章新友: "《全国中医药行业高等教育"十三五"规划教材 医学图形图像处理 新世纪第3版》", 30 June 2018, 中国中医药出版社 *
马利庄等: "《数字动画创作与后期视频处理技术》", 31 August 2014, 上海交通大学出版社 *

Similar Documents

Publication Publication Date Title
CN108245164B (en) Human body gait information acquisition and calculation method for wearable inertial device
Macellari CoSTEL: a computer peripheral remote sensing device for 3-dimensional monitoring of human motion
CN107582098B (en) three-dimensional ultrasonic imaging method for two-dimensional ultrasonic image set reconstruction
US9759539B2 (en) Method of motion tracking
CN109345542A (en) A kind of wearable visual fixations target locating set and method
Surer et al. Methods and technologies for gait analysis
CN111194122A (en) Somatosensory interactive light control system
CN106125908A (en) A kind of motion capture calibration system
CN106767750B (en) Navigation method and system for intelligent robot
Mallat et al. Upper limbs kinematics estimation using affordable visual-inertial sensors
El-Gohary et al. Joint angle tracking with inertial sensors
CN108992072B (en) Human body lower limb movement intention identification method based on gait event
CN114964259A (en) Attitude calculation method and system based on infrared focal plane array time domain signal and infrared image
Choppin et al. Marker-less tracking of human movement using Microsoft Kinect
CN112241763A (en) Multi-source multi-mode dynamic information fusion and cognition method and system
Li et al. Marker-based shape estimation of a continuum manipulator using binocular vision and its error compensation
CN109343713B (en) Human body action mapping method based on inertial measurement unit
KR20200069218A (en) Motion capture apparatus using movement of human centre of gravity and method thereof
CN203630717U (en) Interaction system based on a plurality of light inertial navigation sensing input devices
Tao et al. Biomechanical model-based multi-sensor motion estimation
Reinschmidt et al. Realtime Hand-Gesture Recognition Based on Novel Charge Variation Sensor and IMU
Kim et al. Validation of feasibility of two depth sensor-based Microsoft Kinect cameras for human abduction-adduction motion analysis
Hachaj et al. Heuristic Method for Calculating the Translation of Human Body Recordings Using Data from an Inertial Motion Capture Costume
CN115248446A (en) Three-dimensional ultrasonic imaging method and system based on laser radar
Nguyen et al. Development of a smart shoe for building a real-time 3d map

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication
RJ01 Rejection of invention patent application after publication

Application publication date: 20210119