CN107644069A - High density Monitoring Data vacuates method - Google Patents
High density Monitoring Data vacuates method Download PDFInfo
- Publication number
- CN107644069A CN107644069A CN201710815056.6A CN201710815056A CN107644069A CN 107644069 A CN107644069 A CN 107644069A CN 201710815056 A CN201710815056 A CN 201710815056A CN 107644069 A CN107644069 A CN 107644069A
- Authority
- CN
- China
- Prior art keywords
- data
- sectional
- max
- high density
- density monitoring
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Landscapes
- Mobile Radio Communication Systems (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
What the present invention disclosed high density Monitoring Data vacuates method.This method tries to achieve required set of data points using the general gram of algorithm of Douglas for adding predictive pruning condition from high density Monitoring Data.This method can preferably describe the trend of data fluctuations with point as few as possible, using the Douglas algorithm of predictive pruning, can also effectively reduce the computation complexity for the method for vacuating, efficiently extract data point interested.
Description
Technical field
The present invention relates to the method that vacuates of data, more particularly to high density Monitoring Data vacuates method, further relates to be used for
The high density Monitoring Data of satellite navigation foundation enhancing difference base station vacuates method.
Background technology
With satellite navigation, Internet technology and economic fast development is shared, high accuracy positioning demand is more strong.For
Meet high accuracy positioning demand, ground enhancing technology can be used to carry out difference correction to satellite data.Stable base station clothes
Business is to ensure the important step of final service quality., it is necessary to observe the (central of base station in the O＆M support process of base station
Processing unit, CPU), the situation of the utilization rate of other equipment such as internal memory, and the sample frequency of data be typically the second very
To being millisecond, so high frequency causes data volume very huge, and therefore, such usual data have highdensity characteristic.Together
When, the superelevation sample frequency of monitoring process, which determines, will produce substantial amounts of temporal redundancy data.If complete to extract it is a certain compared with
The data visualization operation of the data of long period change at observed number strong point on browser page, it will be difficult in the page plus
Substantial amounts of data are carried, and can not meet the needs of low latency inquiry.If for example, CPU usage change that observe one day
Situation, sample frequency are the second, then the number of data point will be 86400.If all numbers are drawn in browser page
Strong point, the problems such as causing browser page loading slowly or even can not load.And the substantial amounts of period in its real data be present
Data fluctuations it is very small, can only be represented with a small amount of data point.So in order to meet this demand, it is necessary to data
Vacuated, the trend of data fluctuations is preferably described with point as few as possible.
The content of the invention
The present invention solves the problems, such as it is that high density Monitoring Data needs to vacuate data, and use point as few as possible is preferable
Description data fluctuations trend the problem of.
The present invention solve another problem is how reduce vacuate the computation complexity of method the problem of.
To solve the above problems, a kind of high density Monitoring Data of present invention offer vacuates method.This method utilizes increase
Douglas-Pu Ke the algorithms of predictive pruning condition try to achieve from high density Monitoring Data required set of data points.
Compared with prior art, the present invention at least has advantages below：
1st, the present invention obtains required data point by using the Douglas algorithm of increase predictive pruning from Monitoring Data
Set, so, meets to vacuate data, the trend of data fluctuations is preferably described with point as few as possible, using preshearing
The Douglas algorithm of branch, can also effectively reduce the computation complexity for the method for vacuating, efficiently extract number interested
Strong point.
2nd, this method can pre-set required maximum and can use number of data points, and the set of data points that this method is drawn is not
Set maximum number strong point number can be exceeded, avoid cumbersome last handling process.
3rd, the fluctuation that the fluctuation in view of base station high density data will not be very violent for a long time, so smaller for fluctuating
Data sectional, less data point can be taken, or even only take two end points of head and the tail, so that more data points be distributed to
Fluctuate larger data sectional.
4th, using the strategy of segment processing, according to difference standard deviation, effectively by more data points distribute to fluctuation compared with
Big and frequently data sectional, and difference standard deviation can effectively estimate the violent journey of each data sectional fluctuation
Degree,.Meanwhile partition strategy also makes it possible parallel processing or distributed treatment.
Embodiment
To describe technology contents, construction feature, institute's reached purpose and effect of the present invention in detail, below in conjunction with embodiment
It is described in detail.
The purpose of the present invention is to extract data point interested, such data point from the high density Monitoring Data of base station
Line by the fluctuating change of good response data.This method such as passes through at the segmentation method cutting data of data volume, and calculates
The difference standard deviation of each data sectional carrys out the severe degree of reactions change.According to standard deviation size and be previously set
The desirable number of data points of maximum can use data point number to distribute the maximum of each data segment.Used in each data sectional
Improved Douglas-Pu Ke algorithms take a little.The detailed process of method is as follows：
1. maximum desirable data point number n, data segment complexity threshold ε are set0, finally take point set S and be initialized as
Empty set.
2. calculate data sectional numberAnd data are segmented, the data point number being each segmented is
Wherein N represents the number of all data points.
3. calculating data sectional i difference result, difference is with+1 data point P of kthk+1Subtract k-th of data point
Pk。
4. calculate the standard deviation sigma of data sectional i difference resultiIf σiLess than data segment complexity threshold ε0, then will
The difference standard deviation sigma of the data segmentiIt is arranged to 0, wherein σiCalculation formula it is as follows：
5. calculating the desirable data point number of each segmentation, the maximum of segmentation i distribution can use data point number to be designated as
ni,
6. using improved Douglas-Pu Ke algorithms in each data sectional, detailed process is described below：
● the maximal distance threshold ε of initialization Douglas-Pu Ke algorithms1, current recursion depth d is 1, each segmentation
Contained minimum data point number nmin。
● calculate segmentation in each data point to be segmented both ends data point PSAnd PELine Euclidean distance and by PSAnd PE
It is added in set S.It is noted herein that the abscissa of data point is the time, ordinate is the size of data point.If
The data point for obtaining ultimate range is Pmax, and it is d that it, which arrives the distance of line,max.If dmax＜ ε1, recurrence termination.Otherwise
Calculate the at most desirable data point number n of current depthmax, i.e., 2d-1+1.If nmax≤ni, recurrence termination.Otherwise assume to work as
The data point number of preceding segmentation is ni′If ni′≤nmin, recurrence termination.Otherwise by PmaxIt is added in S, and to new segmentation
(PS, Pmax) and (Pmax, PE) above-mentioned recursive procedure is repeated, and depth adds 1.
Embodiment of the present invention compared with prior art, at least with following difference and effect：
First, this method can pre-set required maximum and can use number of data points, the data point set that this method is drawn
Close not over set maximum number strong point number, avoid cumbersome last handling process.Secondly, it is contemplated that base station is highly dense
The fluctuation of degrees of data will not very violent fluctuation for a long time, so for fluctuating less segmentation, less data can be taken
Point, or even two end points of head and the tail are only taken, fluctuate larger segmentation so as to which more data points be distributed to.Again, using segmentation
The strategy of processing, the severe degree of each data sectional fluctuation can be effectively estimated according to difference standard deviation.Finally, for
Traditional Douglas algorithm, add the function of predictive pruning：1) application scenarios of base station are considered, adjacent data point is not
Fluctuation larger suddenly is had, so when the data point number of segmentation is less, recurrence can be stopped.2) consider to set in advance
The maximum put can use data point number, according to all downward recursive situation of each segmentation is assumed, obtain recurrence to current layer when institute
The data point number taken, i.e., 2 power add 1.If it is assumed that the number of the data point taken is more than or equal to maximum desirable data
The quantity of point, then stop recurrence, so, can use relatively low computation complexity cost, efficiently extract data interested
Point, that is, the extreme point changed greatly.
Claims (8)
1. strengthen the high density Monitoring Data of difference base station for aeronautical satellite ground vacuates method, it is characterised in that：The party
Method tries to achieve required data point using the Douglas-Pu Ke algorithms for adding predictive pruning condition from high density Monitoring Data
Set.
2. the high density Monitoring Data of aeronautical satellite ground enhancing difference base station as claimed in claim 1 vacuates method, its
It is characterised by：Methods described also comprises the following steps：
(1) parameter initialization, including maximum desirable number of data points n, data sectional complexity threshold ε0With finally take point set S
And it is initialized as empty set.
(2) data sectional number is calculated, and carries out data sectional.
(3) the difference standard deviation of the difference result of each data sectional and the difference result of the data sectional is calculated, and is calculated every
The desirable data point number of individual data sectional, the maximum of segmentation i distribution can use data point number to be designated as ni.
(4) using the Douglas-Pu Ke algorithms for adding predictive pruning condition the data of each data sectional take a little with
Form the set of data points needed.
3. strengthen the side of vacuating of the high density Monitoring Data of difference base station for aeronautical satellite ground as claimed in claim 2
Method, it is characterised in that：The data volume of each data sectional is equal.
4. strengthen the side of vacuating of the high density Monitoring Data of difference base station for satellite navigation foundation as claimed in claim 2
Method, it is characterised in that：The step (4) comprises the following steps：
Initialize the maximal distance threshold ε of Douglas-Pu Ke algorithms1, current recursion depth d is 1, contained by each data sectional
Minimum data point number nmin。
Each data point is calculated in data sectional to data sectional both ends data point PSAnd PELine Euclidean distance and by PSAnd PE
It is added in set S, if the data point for obtaining ultimate range is Pmax, and it is d that it, which arrives the distance of line,maxIf dmax＜
ε1, recurrence termination, otherwise calculate current depth at most can use data point number nmaxIf nmax≤ni, recurrence termination, otherwise
Assuming that the data point number of current data segmentation is ni′If ni′≤nmin, recurrence terminates, otherwise by PmaxIt is added in S, and
To new data sectional (PS, Pmax) and (Pmax, PE) above-mentioned recursive procedure is repeated, and depth adds 1.
5. high density Monitoring Data vacuates method, it is characterised in that：This method is drawn using the Doug for adding predictive pruning condition
Si-Pu Ke algorithms try to achieve required set of data points from high density Monitoring Data.
6. high density Monitoring Data as claimed in claim 5 vacuates method, it is characterised in that：Methods described also includes as follows
Step：
(1) parameter initialization, including maximum desirable number of data points n, data sectional complexity threshold ε0With finally take point set S
And it is initialized as empty set.
(2) data sectional number is calculated, and carries out data sectional.
(3) the difference standard deviation of the difference result of each data sectional and the difference result of the data sectional is calculated, and is calculated every
The desirable data point number of individual data sectional, the maximum of segmentation i distribution can use data point number to be designated as n_i.
(4) using the Douglas-Pu Ke algorithms for adding beta pruning condition the data of each data sectional take a little with
Form the set of data points needed.
7. strengthen the side of vacuating of the high density Monitoring Data of difference base station for aeronautical satellite ground as claimed in claim 6
Method, it is characterised in that：The data volume of each data sectional is equal.
8. strengthen the side of vacuating of the high density Monitoring Data of difference base station for satellite navigation foundation as claimed in claim 6
Method, it is characterised in that：The step (4) comprises the following steps：
Initialize the maximal distance threshold ε of Douglas-Pu Ke algorithms1, current recursion depth d is 1, minimum contained by each segmentation
Data point number nmin。
Each data point is calculated in data sectional to data sectional both ends data point PSAnd PELine Euclidean distance and by PSAnd PE
It is added in set S, if the data point for obtaining ultimate range is Pmax, and it is d that it, which arrives the distance of line,maxIf dmax＜
ε1, recurrence termination, otherwise calculate current depth at most can use data point number nmaxIf nmax≤ni, recurrence termination, otherwise
Assuming that the data point number of current data segmentation is ni′If ni′≤nmin, recurrence terminates, otherwise by PmaxIt is added in S, and
To new data sectional (PS, Pmax) and (Pmax, PE) above-mentioned recursive procedure is repeated, and depth adds 1.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710815056.6A CN107644069B (en) | 2017-09-11 | 2017-09-11 | Method for thinning high-density monitoring data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710815056.6A CN107644069B (en) | 2017-09-11 | 2017-09-11 | Method for thinning high-density monitoring data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107644069A true CN107644069A (en) | 2018-01-30 |
| CN107644069B CN107644069B (en) | 2021-03-19 |
Family
ID=61111391
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710815056.6A Active CN107644069B (en) | 2017-09-11 | 2017-09-11 | Method for thinning high-density monitoring data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107644069B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111159498A (en) * | 2019-12-31 | 2020-05-15 | 北京蛙鸣华清环保科技有限公司 | Data point thinning method and device and electronic equipment |
| CN117150217A (en) * | 2023-11-01 | 2023-12-01 | 苏州爱雄斯通信技术有限公司 | Data compression processing method based on big data |
| CN117560015A (en) * | 2024-01-12 | 2024-02-13 | 山东易方云冷热工程技术有限公司 | Efficient processing system for air conditioner installation engineering data |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6606621B2 (en) * | 2001-05-30 | 2003-08-12 | Oracle International Corp. | Methods and apparatus for aggregating sparse data |
| CN102568009A (en) * | 2010-12-17 | 2012-07-11 | 上海博泰悦臻电子设备制造有限公司 | Line segment reducing device and method for electronic map |
| CN104766081A (en) * | 2015-03-19 | 2015-07-08 | 南京大学 | Method for extracting territorial sea baseline based on natural coastline data |
| CN105205288A (en) * | 2015-10-28 | 2015-12-30 | 中国人民解放军国防科学技术大学 | Mode evolution-based forecasting method for satellite long-term on-orbit running state |
-
2017
- 2017-09-11 CN CN201710815056.6A patent/CN107644069B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6606621B2 (en) * | 2001-05-30 | 2003-08-12 | Oracle International Corp. | Methods and apparatus for aggregating sparse data |
| CN102568009A (en) * | 2010-12-17 | 2012-07-11 | 上海博泰悦臻电子设备制造有限公司 | Line segment reducing device and method for electronic map |
| CN104766081A (en) * | 2015-03-19 | 2015-07-08 | 南京大学 | Method for extracting territorial sea baseline based on natural coastline data |
| CN105205288A (en) * | 2015-10-28 | 2015-12-30 | 中国人民解放军国防科学技术大学 | Mode evolution-based forecasting method for satellite long-term on-orbit running state |
Non-Patent Citations (1)
| Title |
|---|
| 胡新玲等: ""管线设计中断面数据的压缩算法"", 《地理空间信息》 * |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111159498A (en) * | 2019-12-31 | 2020-05-15 | 北京蛙鸣华清环保科技有限公司 | Data point thinning method and device and electronic equipment |
| CN117150217A (en) * | 2023-11-01 | 2023-12-01 | 苏州爱雄斯通信技术有限公司 | Data compression processing method based on big data |
| CN117150217B (en) * | 2023-11-01 | 2024-01-26 | 苏州爱雄斯通信技术有限公司 | Data compression processing method based on big data |
| CN117560015A (en) * | 2024-01-12 | 2024-02-13 | 山东易方云冷热工程技术有限公司 | Efficient processing system for air conditioner installation engineering data |
| CN117560015B (en) * | 2024-01-12 | 2024-03-22 | 山东易方云冷热工程技术有限公司 | Efficient processing system for air conditioner installation engineering data |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107644069B (en) | 2021-03-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111242282B (en) | Deep learning model training acceleration method based on end edge cloud cooperation | |
| CN107644069A (en) | High density Monitoring Data vacuates method | |
| CN110929865B (en) | Network quantification method, service processing method and related product | |
| WO2017185412A1 (en) | Neural network operation device and method supporting few-bit fixed-point number | |
| CN108805266A (en) | A kind of restructural CNN high concurrents convolution accelerator | |
| CN107273917A (en) | A kind of Method of Data with Adding Windows based on parallelization Principal Component Analysis Algorithm | |
| CN110233755B (en) | Computing resource and frequency spectrum resource allocation method for fog computing in Internet of things | |
| CN104199942B (en) | A kind of Hadoop platform time series data incremental calculation method and system | |
| CN109144719A (en) | Cooperation discharging method based on markov decision process in mobile cloud computing system | |
| CN111158912A (en) | Task unloading decision method based on deep learning in cloud and mist collaborative computing environment | |
| CN115577790A (en) | Hamiltonian simulation method, hamiltonian simulation device, hamiltonian simulation equipment and storage medium | |
| Wen et al. | Probabilistic wind power forecasting via Bayesian deep learning based prediction intervals | |
| CN107240100A (en) | A kind of image partition method and system based on genetic algorithm | |
| CN106990913A (en) | A kind of distributed approach of extensive streaming collective data | |
| CN112712164B (en) | Non-uniform quantization method of neural network | |
| CN113676917B (en) | Game theory-based energy consumption optimization method for unmanned aerial vehicle hierarchical mobile edge computing network | |
| CN113434209B (en) | End edge two-layer collaborative computing unloading method, device, terminal and storage medium | |
| Tie et al. | To accurately and lightly downscale the temperature field by deep learning | |
| CN108123893A (en) | A kind of multiple target bandwidth allocation methods of real time virtual machine migration | |
| CN111538560B (en) | Virtual machine deployment method and device, electronic equipment and storage medium thereof | |
| CN104270777B (en) | The performance statistics appraisal procedure of base station resource pond physical layer algorithm encapsulation scheme | |
| CN113420831A (en) | Tropical cyclone analysis method, device, equipment and storage medium | |
| CN111159902B (en) | Method and device for determining power of photovoltaic station and computer equipment | |
| CN104683817B (en) | Parallel transformation and inverse transform method based on AVS | |
| Li et al. | Adaptive Binary Whale Optimization Algorithm for Computation Offloading Optimization in Mobile Edge Computing |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |