CN103116599A - Urban mass data flow fast redundancy elimination method based on improved Bloom filter structure - Google Patents

Abstract

An urban mass data flow fast redundancy elimination method based on an improved Bloom filter structure comprises the steps of (1) dataset storage based on the Bloom filter structure, (2) a counting and storage method of massive dynamic datasets, (3) an acceleration processing mechanism of aging elements, (4) a dual-pointer dynamic control method, (5) new data insertion, (6) redundancy judgment of a new element, (7) data flow counting update, (8) periodical deletion of historical data, and (9) output of result data flow after redundancy optimization if no new arrival data. The urban mass data flow fast redundancy elimination method based on the improved Bloom filter structure facilitates urban administrative staff to process continuously rapid growing urban massive data flow quickly and efficiently.

Description

A kind of city mass data flow fast-speed redundancy removing method based on improving Bloom Filter structure

Technical field

The present invention relates to Intelligent Information Processing, space store compressed technical field knowledge, be specifically related to a kind of city high amount of traffic fast-speed redundancy removing method based on improving Bloom Filter structure.Be specially adapted to the city high amount of traffic problem that the processing of city management personnel quickness and high efficiency grows continuously and fast.

Background technology

Along with the development of computer technology and the widespread use of advanced computing technique, the Data Source form of required processing presents diversified characteristics, the large data environment that mass data arrives with swiftly flowing form has appearred in many professional domains, such as be widely used in urban traffic information, earthquake prediction, biosome reference breath detects and the sensor network of military information detection etc. produces detects continuously in a large number data.This new data mode has the features such as real-time, continuity, single pass, unlimitedness, and seeking the various emerging efficient data stream treatment technologies that are applicable to this field practice demand has become the focus that more and more researchers pay close attention to.

If exist the data object that had occurred just to be called the data stream redundancy in the data stream infinite sequence that arrives in real time.In the epoch that can continue automatically to produce a large amount of detail data, multiple magnanimity data stream type data at a high speed need to be carried out complex analyses in nearly real-time mode.And the data stream of redundancy not only can cause the waste of ample resources also directly to reduce the handling property of machine, thereby has affected largely the real-time analysis of the accuracy of this data stream.New data stream redundancy processing method need to adapt to magnanimity large data environment at a high speed, and coordinate Various types of data stream method for digging with the characteristic of low complex degree, high-accuracy, thereby produce the enlarge-effect of " 1+1〉2 ", make data stream obtain the analyzing and processing of real real-time high-efficiency.

Often need to solve following two large problems in research to the elimination of data stream redundancy.At first, traditional comparative approach line by line can not adapt to a large amount of groupings and concurrent data stream environment, and we must obtain all redundant recordings of data stream in single ergodic.Secondly, internal memory can not be processed all data streams, needs to adopt the method for window based in limited memory headroom, gets nearest one piece of data flowmeter calculation and obtains its approximation.Yet because the various data stream redundancy eliminating methods of the restriction of computational resource often reduce the complexity of method by the accuracy of sacrificing the redundancy elimination.The several data method for digging such as the classification under the high amount of traffic environment, cluster, Frequent Pattern Mining more pay attention to seeking optimum equilibrium point between counting yield and accuracy, raise the efficiency as much as possible under the prerequisite that rationally reduces redundancy elimination accuracy, make technology for eliminating can adapt to swiftly flowing high amount of traffic environment.

Summary of the invention

The present invention will overcome the shortcoming that existing Urban Data stream treatment capacity is large, redundance is high, the present invention proposes a kind of city high amount of traffic fast-speed redundancy removing method based on improving Bloom Filter structure, the city high amount of traffic that the processing of support city management personnel quickness and high efficiency grows continuously and fast.

The technical solution adopted for the present invention to solve the technical problems is:

A kind of city mass data flow fast-speed redundancy removing method based on improving Bloom Filter structure comprises the following steps:

1) store based on the data set of Bloom Filter structure: setting data stream is expressed as S _N=x ₁, x ₂..., x _n, N is the number of element in data stream, during original state, Bloom Filter is a bit array that comprises the m position, each all is set to 0, and is as follows: b[i]=0, i=0...m-1; Bloom Filter uses k separate hash function, each element map during they will gathers respectively to (1 ..., in scope m), to any one element x, the position that i hash function shines upon will be set to 1, i.e. h _i(x)=1,1≤i≤k;

2) magnanimity dynamic data set counting storage means: each of standard Bloom Filter bit array is expanded to a little counter, add respectively 1 for when inserting element the value of k corresponding Counter, subtract respectively 1 for the value of k corresponding Counter when the deletion element:

$h_i\left(x\right)=\left\{\begin{array}{c}\mathrm{counter}+1\\ \mathrm{counter}-1\end{array}\right.$

3) aging element accelerates treatment mechanism: adopt the jump window scheme of former and later two count windows, at first define two BF structures, namely

BFO[1,...,m]

BFN[1,...,m]

The array that each structure is comprised of m integer;

Definition hash function f ₁, f ₂..., f _kK Hash maps for BF; Suppose that w is the aging count value of data stream, BFO represents that last interval is the interior counting mapping of [1, w] scope, and after the BFN representative, an interval is the counting mapping in [w+1,2w] scope;

4) two pointer dynamic control methods: adopt the method for two dynamic pointers, pointer can indicate current count value in the moving section of BFO and BFN, and is current count tag of each distribution of flows, pointer p _iPossess [1, w] between effective count block;

5) new data inserts: when new element x is arranged _iWhen needing to insert, will first respectively carry out k time Hash maps on BFO and BFN, calculate respective value, according to the respective value of new data on BFO and BFN, calculate its corresponding minimum value:

v=min(v ₁,v ₂,...,v _k)

6) judge whether redundancy of new element: according to the minimum value that back calculates, if minimum value v=0 corresponding to BFN, judge that this element did not occur in the corresponding operation interval of BFN; If the minimum value v that BFO is corresponding＜p1 represents that this element did not occur yet in last operation interval; If new element x _iBoth do not occur in BFO, do not occur in BFN yet, and added this new element, otherwise abandon this redundant elements;

7) update data stream counting: be arranged to current pointer counter value for k position corresponding to this data stream in BFN, i.e. b[i]=p2;

8) regular deleting history data: when dynamic pointer p1 reaches critical value w, represent that this data stream has become aging stream, therefore delete BFO, and replacing BFO and BFN, make BFO continue to keep the data stream mapping of a upper operation interval, BFN preserves the data stream mapping in work at present interval, the p of initialization simultaneously _i=1;

9) if after arriving port redundancy optimization without new data, result data flows.

Technical conceive of the present invention is: we eliminate problem at data stream redundancy.Introduce improved Bloom Filter data structure, take into full account the features such as data stream real-time, continuity, single pass, unlimitedness, propose a kind of city mass data flow fast-speed redundancy removing method based on improving Bloom Filter structure.Eliminate problem by the redundancy under the scientific and reasonable solution high amount of traffic environment of the method.

Beneficial effect of the present invention is mainly manifested in: the redundancy that effectively solves under the high amount of traffic environment is eliminated problem.

Description of drawings

Fig. 1 is Bloom Filter structured data storage schematic diagram;

Fig. 2 is Counting Bloom Filter dynamic data set method of counting figure;

Fig. 3 is based on the city mass data flow fast-speed redundancy removing method that improves Bloom Filter structure.

Embodiment

The invention will be further described below in conjunction with accompanying drawing.

With reference to Fig. 1～Fig. 3, a kind of city mass data flow fast-speed redundancy removing method based on improving Bloom Filter structure comprises the following steps:

1) store based on the data set of Bloom Filter structure: tentation data stream can be expressed as S _N=x ₁, x ₂..., x _n, N is the number of element in data stream, N often tends to and infinity in real world.Bloom Filter is the very high random data storage organization of a kind of space efficiency, under the application scenario that can tolerate low error rate, Bloom Filter has exchanged the very big saving of storage space for by few mistake, obtained using very widely in reality.

During original state, Bloom Filter is a bit array that comprises the m position, and each all is set to 0, and is as follows: b[i]=0, i=0...m-1.

In order to express S={x ₁, x ₂..., x _nThe set of such n element, Bloom Filter uses k separate hash function (Hash Function), each element map during they will gathers respectively arrive (1 ..., in scope m).To any one element x, the position of i hash function mapping will be set to 1, i.e. h _i(x)=1,1≤i≤k.Specifically as shown in Figure 1.

2) magnanimity dynamic data set counting storage means: the large data under real urban environment have the features such as real-time, continuity, single pass, unlimitedness.Simple Bloom Filter data store organisation can not satisfy the needs of reality.And Counting Bloom Filter can address this problem.Each of standard Bloom Filter bit array is expanded to a little counter (Counter, be generally each Counter and distribute 4), giving corresponding k(k when inserting element is the hash function number) value of individual Counter adds respectively 1, subtracts respectively 1 for the value of k corresponding Counter when the deletion element.

$h_i\left(x\right)=\left\{\begin{array}{c}\mathrm{counter}+1\\ \mathrm{counter}-1\end{array}\right.$

Counter Bloom Filter method has increased deletion action for Bloom Filter by taking the cost of more storage space.Example is seen accompanying drawing 2.

3) aging element accelerates treatment mechanism: when deleting aging element based on the way of moving window (sliding window) general modfel, need the whole BF structure of real time scan, for fear of the performance cost of bringing with this, this method adopts jump window (jumping window) pattern of former and later two count windows.At first this method defines two BF structures, namely

BFO[1,...,m]

BFN[1,...,m]

The array that each structure is comprised of m integer.

Definition hash function f ₁, f ₂..., f _kK Hash maps for BF.Suppose that w is the aging count value of data stream, BFO represents that last interval is the interior counting mapping of [1, w] scope, and after the BFN representative, an interval is the counting mapping in [w+1,2w] scope.

4) two pointer dynamic control methods: existing data stream redundancy eliminating method is the higher problem of life period complexity all, and for this situation, the present invention has adopted the method for two dynamic pointers.Pointer can indicate current count value in the moving section of BFO and BFN, and is current count tag of each distribution of flows.Pointer p _iPossess [1, w] between effective count block, therefore, but this method formalization representation is the form of accompanying drawing 3.

5) new data inserts: when new element x is arranged _iWhen needing to insert, this method will first respectively be carried out k time Hash maps on BFO and BFN, calculate respective value.According to the respective value of new data on BFO and BFN, calculate its corresponding minimum value:

v=min(v ₁,v ₂,...,v _k)

6) judge whether redundancy of new element: according to the minimum value that back calculates, if minimum value v=0 corresponding to BFN, judge that this element did not occur in the corresponding operation interval of BFN.If the minimum value v that BFO is corresponding＜p1 represents that this element did not occur yet in last operation interval.If new element x _iBoth do not occur in BFO, do not occur in BFN yet, and added this new element, otherwise abandon this redundant elements.

7) update data stream counting: be arranged to current pointer counter value for k position corresponding to this data stream in BFN, i.e. b[i]=p2.

8) regular deleting history data: in order to distinguish historical data and the current data in data stream, this method has been introduced two BF structures, therefore when dynamic pointer p1 reaches critical value w, represents that this data stream has become aging stream.Therefore delete BFO, and change BFO and BFN, the data stream mapping that makes BFO continue to keep a upper operation interval, BFN preserves the data stream mapping in work at present interval.The p of initialization simultaneously _i=1.

9) if after arriving port redundancy optimization without new data, result data flows.

Claims (1)

1. one kind based on the city mass data flow fast-speed redundancy removing method that improves Bloom Filter structure, it is characterized in that: comprise the following steps:

1) store based on the data set of Bloom Filter structure: setting data stream is expressed as

S _N=x ₁, x ₂..., x _n, N is the number of element in data stream, during original state, Bloom Filter is a bit array that comprises the m position, each all is set to 0, and is as follows: b[i]=0, i=0...m-1; Bloom Filter uses k separate hash function, each element map during they will gathers respectively to (1 ..., in scope m), to any one element x, the position that i hash function shines upon will be set to 1, i.e. h _i(x)=1,1≤i≤k;

2) magnanimity dynamic data set counting storage means: each of standard Bloom Filter bit array is expanded to a little counter, add respectively 1 for when inserting element the value of k corresponding Counter, subtract respectively 1 for the value of k corresponding Counter when the deletion element:

3) aging element accelerates treatment mechanism: adopt the jump window scheme of former and later two count windows, at first define two BF structures, namely

BFO[1,...,m]

BFN[1,...,m]

The array that each structure is comprised of m integer;

Definition hash function f ₁, f ₂..., f _kK Hash maps for BF; Suppose that w is the aging count value of data stream, BFO represents that last interval is the interior counting mapping of [1, w] scope, and after the BFN representative, an interval is the counting mapping in [w+1,2w] scope;

4) two pointer dynamic control methods: adopt the method for two dynamic pointers, pointer can indicate current count value in the moving section of BFO and BFN, and is current count tag of each distribution of flows, pointer p _iPossess [1, w] between effective count block;

5) new data inserts: when new element x is arranged _iWhen needing to insert, will first respectively carry out k time Hash maps on BFO and BFN, calculate respective value, according to the respective value of new data on BFO and BFN, calculate its corresponding minimum value:

v=min(v ₁,v ₂,...,v _k)

6) judge whether redundancy of new element: according to the minimum value that back calculates, if minimum value v=0 corresponding to BFN, judge that this element did not occur in the corresponding operation interval of BFN; If the minimum value v that BFO is corresponding＜p1 represents that this element did not occur yet in last operation interval; If new element x _iBoth do not occur in BFO, do not occur in BFN yet, and added this new element, otherwise abandon this redundant elements;

7) update data stream counting: be arranged to current pointer counter value for k position corresponding to this data stream in BFN, i.e. b[i]=p2;

8) regular deleting history data: when dynamic pointer p1 reaches critical value w, represent that this data stream has become aging stream, therefore delete BFO, and replacing BFO and BFN, make BFO continue to keep the data stream mapping of a upper operation interval, BFN preserves the data stream mapping in work at present interval, the p of initialization simultaneously _i=1;

9) if after arriving port redundancy optimization without new data, result data flows.

Images