CN111121946B - Method for accurately determining abnormal value at multiple points in large dynamic range and large discrete single area - Google Patents

Method for accurately determining abnormal value at multiple points in large dynamic range and large discrete single area Download PDF

Info

Publication number
CN111121946B
CN111121946B CN201911307015.1A CN201911307015A CN111121946B CN 111121946 B CN111121946 B CN 111121946B CN 201911307015 A CN201911307015 A CN 201911307015A CN 111121946 B CN111121946 B CN 111121946B
Authority
CN
China
Prior art keywords
standard deviation
points
point
ith
optical fiber
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201911307015.1A
Other languages
Chinese (zh)
Other versions
CN111121946A (en
Inventor
孙小菡
徐宛丽
吴宝锋
刘华伟
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Southeast University
Original Assignee
Southeast University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Southeast University filed Critical Southeast University
Priority to CN201911307015.1A priority Critical patent/CN111121946B/en
Publication of CN111121946A publication Critical patent/CN111121946A/en
Application granted granted Critical
Publication of CN111121946B publication Critical patent/CN111121946B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01H—MEASUREMENT OF MECHANICAL VIBRATIONS OR ULTRASONIC, SONIC OR INFRASONIC WAVES
    • G01H9/00—Measuring mechanical vibrations or ultrasonic, sonic or infrasonic waves by using radiation-sensitive means, e.g. optical means
    • G01H9/004—Measuring mechanical vibrations or ultrasonic, sonic or infrasonic waves by using radiation-sensitive means, e.g. optical means using fibre optic sensors
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04B—TRANSMISSION
    • H04B10/00—Transmission systems employing electromagnetic waves other than radio-waves, e.g. infrared, visible or ultraviolet light, or employing corpuscular radiation, e.g. quantum communication
    • H04B10/07—Arrangements for monitoring or testing transmission systems; Arrangements for fault measurement of transmission systems
    • H04B10/075—Arrangements for monitoring or testing transmission systems; Arrangements for fault measurement of transmission systems using an in-service signal
    • H04B10/079—Arrangements for monitoring or testing transmission systems; Arrangements for fault measurement of transmission systems using an in-service signal using measurements of the data signal
    • H04B10/0791—Fault location on the transmission path

Landscapes

  • Physics & Mathematics (AREA)
  • Electromagnetism (AREA)
  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • General Physics & Mathematics (AREA)
  • Measurement Of Mechanical Vibrations Or Ultrasonic Waves (AREA)

Abstract

The invention discloses a method for accurately determining abnormal values in a large dynamic range and a large discrete single area at multiple points. Collecting a single-area sample with a large dynamic range and a large dispersion, extracting the standard deviation and the dispersion difference of each point in the single-area sample, and carrying out threshold processing on the standard deviation and the dispersion difference; processing the standard deviation of each point in the single-area sample through standard deviation mean forward translation operation; calculating the confidence rates of all points in the single-region sample; and comparing the confidence rates of all the points with the threshold values of the confidence rates respectively, judging abnormal points in the sample, and accurately determining abnormal values by using the large dynamic range and the large discrete single-region multipoint. The invention overcomes the problems of large data volume and unobvious abnormality and can quickly and accurately determine the abnormal numerical value.

Description

Method for accurately determining abnormal value at multiple points in large dynamic range and large discrete single area
Technical Field
The invention belongs to the technical field of computer data processing, and particularly relates to a method for determining abnormal values of large dynamic range and large discrete data.
Background
In computer modeling, it is important to clean up data samples to ensure that observations adequately represent problems. Sometimes, a data set may contain extreme values outside of the expected range, which are often referred to as outliers. By understanding and even removing these outliers, modeling and model skills can be improved.
Currently proposed methods of determining data outliers include distance-based methods, bias-based methods, and density-based methods, among others. Although the methods can find out abnormal values, the methods are complicated and require a large amount of machine learning, and particularly when the dynamic range is large and abnormal data are not obvious, the abnormal values are difficult to find out by the existing methods.
For example, in the distributed optical fiber vibration sensing positioning technology, the false alarm rate is a prominent problem, and when the system makes a false alarm, the vibration point measured by the system is different from the actual vibration point by hundreds of meters. Especially, when the transmission distance is long, the signal and the background noise cannot be distinguished because the back scattering signal is weak, and the false alarm of vibration positioning is caused, so a method for accurately positioning the vibration point is needed.
Disclosure of Invention
In order to solve the technical problems in the background art, the invention provides a method for accurately determining abnormal values at multiple points in a large dynamic range and a large discrete single area, which overcomes the problems of large data volume and unobvious abnormality and can quickly and accurately determine abnormal values.
In order to achieve the technical purpose, the technical scheme of the invention is as follows:
the method for accurately determining the abnormal value of the large dynamic range and the large discrete single area multipoint comprises the following steps:
(1) collecting a single-area sample with a large dynamic range and a large dispersion, extracting the standard deviation and the dispersion difference of each point in the single-area sample, and carrying out threshold processing on the standard deviation and the dispersion difference;
(2) processing the standard deviation of each point in the single-area sample through standard deviation mean forward translation operation;
(3) calculating the confidence rates of all points in the single-region sample;
(4) and comparing the confidence rates of all the points with the threshold values of the confidence rates respectively, judging abnormal points in the sample, and accurately determining abnormal values by using the large dynamic range and the large discrete single-region multipoint.
Further, in step (1), N single-region samples are collected at a time, and the average value of each point is calculated
Figure BDA0002323443580000021
Standard deviation sigmaiAnd a difference of dispersion dij：
Figure BDA0002323443580000022
Figure BDA0002323443580000023
Figure BDA0002323443580000024
Wherein x isijThe ith sample is represented as ith point value, i is 1,2, …, M is the number of sampling points, and j is 1,2, …, N.
Further, in step (1), the method of thresholding the standard deviation and the discrete deviation is as follows:
setting a threshold t1And t2；
Standard deviation sigma of the ith sample pointiSatisfy sigmai≤t1When, will σiIs set to t1；
Discrete difference d of ith sampling point of jth sampleijSatisfy dij≤t2When d is greater than dijIs set to 0.
Further, the process of step (2) is as follows:
setting T1And T2The ith-T1～i-T2The standard deviation of all the internal sampling points is weighted and averaged to be used as the standard deviation sigma of the ith sampling pointi。
Further, in step (3), the confidence rates D of the respective points are calculated as followsij：
Dij＝dij-g*σi
Wherein g is a preset multiple; if D isijIf greater than 0, let DijIf not, let Dij＝0；
Will DijStored in an array D, DijI.e. the element in the ith row and the jth column of the array D.
Further, in step (4), a confidence threshold D is set, and the sum D of each row element of the array D is calculatediIf d isiIf > d, the ith sampling point is considered as an abnormal point.
Drawings
FIG. 1 is a flow chart of an embodiment distributed optical fiber vibration sensing positioning method;
FIG. 2 is a graph of signal versus disturbance location at 10km of disturbance provided by an embodiment;
FIG. 3 is a graph of standard deviation at 10km of perturbation versus perturbation position provided by an embodiment;
FIG. 4 is a graph of the dispersion difference at 10km of perturbation versus the location of the perturbation provided by the embodiment;
FIG. 5 is a ring diagram of the criterion and disturbance position at 10km of disturbance provided by the embodiment;
FIG. 6 is a graph of confidence rate at 10km versus disturbance location for an embodiment;
FIG. 7 is a graph of location dispersion at a perturbation of 10km provided by an embodiment;
FIG. 8 is a diagram illustrating the relationship between the maximum localization dispersion and the perturbation position according to the embodiment.
Detailed Description
The technical scheme of the invention is explained in detail in the following with the accompanying drawings.
In reality, many scenes need to locate abnormal values of large data, such as the distributed optical fiber vibration sensing location technology mentioned in the aforementioned background art. In this embodiment, a method for accurately determining an abnormal value at multiple points in a large dynamic range and a large discrete single area is applied to a distributed optical fiber vibration sensing positioning technology, so as to obtain a distributed optical fiber vibration sensing positioning method, as shown in fig. 1, the steps of which are as follows:
step 1: taking N curves collected by the distributed optical fiber vibration sensing system as a group, calculating the average value, the standard deviation and the dispersion difference of M points on the N curves, and carrying out threshold processing on the standard deviation and the dispersion difference.
The following preferred scheme is adopted to realize the step 1:
average value of each point
Figure BDA0002323443580000041
Standard deviation sigmaiAnd a difference of dispersion dijThe calculation formula of (a) is as follows:
Figure BDA0002323443580000042
Figure BDA0002323443580000043
Figure BDA0002323443580000044
xijthe ith point value in the jth curve is represented, i is 1,2, …, M is the number of sampling points, and j is 1,2, …, N. N-20, M-65536.
The method for thresholding the standard deviation and the dispersion deviation is as follows:
setting a threshold t1And t2Standard deviation σ of the ith sample pointiSatisfy sigmai≤t1When, will σiIs set to t1(ii) a Discrete difference d of ith sampling point of jth sampleijSatisfy dij≤t2When d is greater than dijIs set to 0. Further, set t1＝30，t2＝72。
Step 2: and processing the standard deviation of each sampling point through a standard deviation mean forward translation operation.
The following preferred scheme is adopted to realize the step 2:
setting T1And T2The ith-T1～i-T2The standard deviation of all the internal sampling points is weighted and averaged to be used as the standard deviation sigma of the ith sampling pointi. Further, T is set1＝1000，T2When 500, then:
Figure BDA0002323443580000045
and step 3: the confidence rates of all the sample points are calculated.
The following preferred scheme is adopted to realize the step 3:
the confidence rate D of each point is calculated according to the following formulaij：
Dij＝dij-g*σi
Wherein g is a preset multiple; if D isijIf greater than 0, let DijIf not, let Dij＝0；
Will DijStored in an array D, DijI.e. the element in the ith row and the jth column of the array D.
In distributed fiber optic vibration sensing, the preferred value of the multiple g is 2.4.
And 4, step 4: and comparing the confidence rates of all the points with threshold values of the confidence rates respectively, thereby judging the vibration points.
The following preferred scheme is adopted to realize the step 4:
setting a confidence rate threshold value D, and calculating the sum D of each row element of the array DiIf d isiIf the sampling point is greater than d, the ith sampling point is considered to be possibly disturbed, the number of all the possible disturbed points is counted to be t, and the average of all the possible disturbed points is taken as the real vibrated point. Further, d is set to 6.
To verify the present invention, the following experiment was performed.
Continuously applying disturbance at 10km, continuously acquiring 2000 curves, and performing method processing on the acquired 2000 curves, wherein fig. 2 is an acquired signal original curve diagram, and because a disturbance signal is small and a dynamic range is large, it is difficult to directly obtain a disturbance position from fig. 2. By adopting the invention, the original data of FIG. 2 is subjected to feature extraction, FIG. 3 is a standard deviation graph thereof, and the standard deviation of the disturbance position and the subsequent point can be obviously changed; FIG. 4 is a graph of the dispersion difference of 20 samples taken, from above, it can be seen that the dispersion difference of the disturbance position and the point after the disturbance position has a sudden change; FIG. 5 is a graph of 20 samples taken after each point criterion, and it can be seen from the graph that only the criterion of the disturbance point position is distinguished from other position points, and the characteristics are obvious; FIG. 6 shows the confidence rates of 20 samples taken at each point, and it can be seen that the confidence rate at the perturbed point is greater than 6, as distinguished from the un-perturbed point. Fig. 7 shows the positioning dispersion of 100 sets of data at 10km of disturbance points by using the method of the present invention, and it can be seen that the maximum dispersion is 4 and the minimum dispersion is 0.
In addition, in this embodiment, only the disturbance position is changed under the condition that the laboratory conditions are kept unchanged, and the data of 2000 curves at different positions are collected and processed, so as to obtain the maximum dispersion at 2.2km, 5km, 10km, 15km and 20 km. Fig. 8 is a relationship diagram between the positioning dispersion and the disturbance position, and it can be seen that, along with the increase of the disturbance position, the positioning dispersion is always kept within 4.8 meters, which indicates that the positioning method can achieve accurate positioning, i.e. the abnormal point can be accurately found.
The above experiment shows that the method for accurately determining the abnormal value at the large dynamic range and the large discrete single area and the multiple points is applied to the distributed optical fiber vibration sensing system for locating the disturbance point, when the optical fiber is subjected to perturbation, the signal at the disturbance position is not obviously changed, the threshold value comparison method cannot determine the vibration point, but the standard deviation, the discrete difference and the confidence rate at the position are obviously changed due to the small change of the signal, so that the abnormal value can be accurately determined through the method, and the possibility of misjudgment is reduced.
The embodiments are only for illustrating the technical idea of the present invention, and the technical idea of the present invention is not limited thereto, and any modifications made on the basis of the technical scheme according to the technical idea of the present invention fall within the scope of the present invention.

Claims (6)

1. A distributed optical fiber vibration sensing positioning method is characterized by comprising the following steps:
(1) taking N curves collected by the distributed optical fiber vibration sensing system as a group, calculating the average value, the standard deviation and the dispersion difference of M points on the N curves, and carrying out threshold processing on the standard deviation and the dispersion difference;
average value of each point
Figure FDA0003177003840000011
Standard deviation sigmaiAnd a difference of dispersion dijIs calculated as follows:
Figure FDA0003177003840000012
Figure FDA0003177003840000013
Figure FDA0003177003840000014
wherein x isijThe ith point value in the jth curve is represented, i is 1,2, …, M is the number of sampling points, j is 1,2, …, N;
the method for thresholding the standard deviation and the dispersion deviation is as follows:
setting a threshold t1And t2Standard deviation σ of the ith sample pointiSatisfy sigmai≤t1When, will σiIs set to t1(ii) a Discrete difference d of ith sampling point of jth sampleijSatisfy dij≤t2When d is greater than dijSet to 0;
(2) and processing the standard deviation of each sampling point through standard deviation mean forward translation operation:
setting T1And T2The ith-T1～i-T2The standard deviation of all the internal sampling points is weighted and averaged to be used as the standard deviation sigma of the ith sampling pointi；
(3) Calculating the confidence rates of all sampling points:
the confidence rate D of each point is calculated according to the following formulaij：
Dij＝dij-g*σi
Wherein g is a preset multiple; if D isijIf greater than 0, let DijIf not, let Dij＝0；
Will DijStored in an array D, DijThe element is the element of the ith row and the jth column in the array D;
(4) comparing the confidence rates of all the points with threshold values of the confidence rates respectively, thereby judging vibration points:
setting a confidence rate threshold value D, and calculating the sum D of each row element of the array DiIf d isiIf the sampling point is greater than d, the probability of disturbance of the ith sampling point is considered, the number of all the points with the probability of disturbance is counted to be t, and all the points are averaged to be used as real vibration points.
2. The distributed optical fiber vibration sensing and positioning method according to claim 1, wherein in step (1), N-20 and M-65536.
3. The distributed optical fiber vibration sensing and positioning method according to claim 1, wherein in step (1), t is set1＝30，t2＝72。
4. The distributed optical fiber vibration sensing and positioning method according to claim 1, wherein in step (2), T is set1＝1000，T2＝500。
5. The distributed optical fiber vibration sensing and positioning method according to claim 1, wherein in step (3), the multiple g is 2.4.
6. The distributed optical fiber vibration sensing and positioning method according to claim 1, wherein in step (4), a confidence rate threshold value d-6 is set.
CN201911307015.1A 2019-12-18 2019-12-18 Method for accurately determining abnormal value at multiple points in large dynamic range and large discrete single area Active CN111121946B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201911307015.1A CN111121946B (en) 2019-12-18 2019-12-18 Method for accurately determining abnormal value at multiple points in large dynamic range and large discrete single area

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201911307015.1A CN111121946B (en) 2019-12-18 2019-12-18 Method for accurately determining abnormal value at multiple points in large dynamic range and large discrete single area

Publications (2)

Publication Number Publication Date
CN111121946A CN111121946A (en) 2020-05-08
CN111121946B true CN111121946B (en) 2021-09-28

Family

ID=70499428

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201911307015.1A Active CN111121946B (en) 2019-12-18 2019-12-18 Method for accurately determining abnormal value at multiple points in large dynamic range and large discrete single area

Country Status (1)

Country Link
CN (1) CN111121946B (en)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111735528B (en) * 2020-06-05 2022-05-20 南京曦光信息科技有限公司 Method for accurately positioning disturbance point based on KNN voting
CN115112866B (en) * 2022-08-29 2022-11-22 山东省鲁南地质工程勘察院(山东省地质矿产勘查开发局第二地质大队) Hydrothermal deposit investigation system based on electromagnetic radiation

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5689067A (en) * 1995-02-03 1997-11-18 Daimler-Benz Ag Diagnostic method and apparatus for monitoring the wear of at least an engine timing chain
CN102288286A (en) * 2011-06-16 2011-12-21 中国科学院沈阳自动化研究所 Method for analyzing and evaluating measure point precision of gearbox in vibration acceleration sensor
KR101696789B1 (en) * 2015-10-28 2017-01-16 한국생산기술연구원 Method for judging abnormal condition of waterjet nozzle and apparatus for monitering abnormal condition of waterjet nozzle
CN107092582A (en) * 2017-03-31 2017-08-25 江苏方天电力技术有限公司 One kind is based on the posterior exceptional value on-line checking of residual error and method for evaluating confidence

Family Cites Families (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6131013A (en) * 1998-01-30 2000-10-10 Motorola, Inc. Method and apparatus for performing targeted interference suppression
CN104240455B (en) * 2014-08-07 2016-08-17 北京航天控制仪器研究所 A kind of disturbance event recognition methods in distribution type fiber-optic pipeline safety early warning system
CN104217513B (en) * 2014-09-02 2016-08-17 浙江中欣动力测控技术有限公司 The method improving phase sensitive optical time domain reflectometer identification intrusion event accuracy rate
CN106803137A (en) * 2017-01-25 2017-06-06 东南大学 Urban track traffic AFC system enters the station volume of the flow of passengers method for detecting abnormality in real time
JP6869065B2 (en) * 2017-03-22 2021-05-12 大阪瓦斯株式会社 Abnormality judgment device and abnormality judgment method
CN110010202B (en) * 2017-09-22 2023-04-21 北京蓝标一成科技有限公司 Establishment and judgment standard and judgment method for judging similarity of pure varieties of dendrobium fimbriatum
CN108509850B (en) * 2018-02-24 2022-03-29 华南理工大学 Intrusion signal identification method based on distributed optical fiber system
CN109711551A (en) * 2018-12-28 2019-05-03 北京旷视科技有限公司 Data processing method and device

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5689067A (en) * 1995-02-03 1997-11-18 Daimler-Benz Ag Diagnostic method and apparatus for monitoring the wear of at least an engine timing chain
CN102288286A (en) * 2011-06-16 2011-12-21 中国科学院沈阳自动化研究所 Method for analyzing and evaluating measure point precision of gearbox in vibration acceleration sensor
KR101696789B1 (en) * 2015-10-28 2017-01-16 한국생산기술연구원 Method for judging abnormal condition of waterjet nozzle and apparatus for monitering abnormal condition of waterjet nozzle
CN107092582A (en) * 2017-03-31 2017-08-25 江苏方天电力技术有限公司 One kind is based on the posterior exceptional value on-line checking of residual error and method for evaluating confidence

Also Published As

Publication number Publication date
CN111121946A (en) 2020-05-08

Similar Documents

Publication Publication Date Title
CN112508105B (en) Fault detection and retrieval method for oil extraction machine
CN109359690B (en) Vehicle travel track identification method based on checkpoint data
CN111121946B (en) Method for accurately determining abnormal value at multiple points in large dynamic range and large discrete single area
CN103473540B (en) The modeling of intelligent transportation system track of vehicle increment type and online method for detecting abnormality
CN113344134B (en) Low-voltage distribution monitoring terminal data acquisition abnormality detection method and system
CN112911627B (en) Wireless network performance detection method, device and storage medium
CN103761748A (en) Method and device for detecting abnormal behaviors
EP1958034B1 (en) Use of sequential clustering for instance selection in machine condition monitoring
CN116308305B (en) Bridge health monitoring data management system
CN113657737B (en) Water quality evaluation method and system based on automatic monitoring data, equipment and storage medium
CN112414694A (en) Equipment multistage abnormal state identification method and device based on multivariate state estimation technology
CN103217172A (en) Fault detection method for Kalman filtering sensor information fusion
CN106935038B (en) Parking detection system and detection method
CN108257365A (en) A kind of industrial alarm designs method based on global nonspecific evidence dynamic fusion
CN114093171B (en) Traffic running state monitoring method and device based on multi-source data fusion
CN113221282A (en) Aero-engine service life prediction method based on Bayesian residual convolutional network
CN116701130A (en) Dynamic baseline optimization method and device based on index portrait and electronic equipment
CN106033643B (en) A kind of data processing method and device
CN112100037B (en) Alarm level identification method, device, electronic equipment and storage medium
CN112329810B (en) Image recognition model training method and device based on significance detection
CN111949496A (en) Data detection method and device
CN115932144B (en) Chromatograph performance detection method, chromatograph performance detection device, chromatograph performance detection equipment and computer medium
CN114579519A (en) File system abnormality detection method and device, storage medium and electronic device
CN112149546B (en) Information processing method, device, electronic equipment and storage medium
CN113420813B (en) Diagnostic method for particulate matter filter cotton state of vehicle tail gas detection equipment

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant