CN105812802A - Power big data compression transmission method based on sparse coding and decoding - Google Patents

Abstract

The invention relates to a power big data compression transmission method based on sparse coding and decoding. The power big data compression transmission method comprises (1) system starting; (2) data transmission; (3) sparse coding; (4) scalar quantization; (5) self-adaption arithmetic coding; (6) code storage; (7) deciding whether to invoke original data; (8) self-adaption arithmetic decoding; (9) inverse quantization; (10) sparse decoding; (11) data processing. The input power big data can be converted into the sparse coding way for the output. The data coding having the extremely small storage amount can be acquired after the conversion during the data transmission process. The power big data compression transmission method is advantageous in that the data storage amount can be effectively reduced, and at the same time, the inner structure and the substantive characteristic of the data can be found out, and the data backup and the subsequent calculation can be facilitated.

Description

A kind of compression transmitting method of the big data of the electric power based on sparse encoding and decoding

Technical field

The present invention relates to a kind of big data transmission method of electric power, in particular to the compression transmitting method of a kind of big data of the electric power based on sparse encoding and decoding.

Background technology

The Intelligent Fusion direction applied to data message is developed by power information technology, the new stage that its operational mode will be marched toward as service-centric, meanwhile, the value of business data assets is constantly exploited, and the enterprise under Information Condition produces and decision-making will be more intelligent.The concentrated reflection of the development of " big data " technological incorporation type under the new situation just and the intelligent theory of application, there is the Technical Architecture of the reply data characteristicses such as the data scale of construction is huge, data type is various, value density is low and processing speed is fast, and with the application model that the content value-added service of high added value inside and outside industry is target, its core is exactly information resource development and utilization.Electric power big data refer to be collected by various acquisition of information channels such as sensor, smart machine, video monitoring equipment, audio communication device, mobile terminals, magnanimity, structuring, semi-structured, non-structured, and there is the business datum set of incidence relation each other.Popularization and strengthened research along with scientific and technological development and bulk information system, Consumer's Experience demand constantly increases, data accumulated in this process and service also get more and more, and tend to magnanimity, and traditional data base cannot meet so huge call data storage.Under the big data background that current data volume is explosive growth, using efficient compression coding mode to process big data, being effectively reduced memory space is significantly.

Sparse coding can provide effective solution for the storage of the big data of electric power.The concept of sparse coding mostlys come from neurobiology.Biologist proposes, and mammal is in long-term evolution, and generating can be quick, and accurately, low-cost ground represents the ability of the optic nerve aspect of natural image.We are intuitively it is envisioned that the secondary picture that our eyes are often seen is all more than one hundred million pixel, and each sub-picture we all only rebuild by little cost and store, we are called sparse coding it.Sparse coding algorithm is applied in the middle of the big data of electric power just can to solve data storage capacities is limited, data exhibiting scarce capacity and data interactive ability are difficult to meet the problems such as the big data demand of electric power.

Summary of the invention

For the deficiencies in the prior art, the present invention proposes the compression transmitting method of a kind of big data of the electric power based on sparse encoding and decoding, the problem big in order to solve memory data output, the sparse coding method proposed can be greatly saved the memory space of data, reduce the demand to hardware store, save hardware cost, be simultaneously also beneficial to data management easily.

It is an object of the invention to adopt following technical proposals to realize:

The compression transmitting method of a kind of big data of the electric power based on sparse encoding and decoding, it thes improvement is that, described method includes

(1) system is started；

(2) data are transmitted；

(3) sparse coding；

(4) scalar quantization；

(5) adaptive arithmetic code；

(6) coding is preserved；

(7) judge whether to call former data；

(8) self adaptation arithmetic decoding；

(9) inverse quantization；

(10) sparse decoding；

(11) data are processed.

Preferably, described sparse coding is expressed as follows:

$x=\underset{i=1}{\overset{m}{\mathrm{\Σ}}}{a}_i{\mathrm{\φ}}_i,$

Wherein, x ∈ Rⁿ, Ф=[φ₁,φ₂..., φ_m]∈R^n×m；X is the data of input, and Ф is the dictionary for data representation, a=[α₁,α₂,...,α_m]^T∈R^mFor sparse coding coefficient.

Preferably, described step (3) includes data waiting for transmission are carried out sparse coding, it is thus achieved that sparse coding coefficient vector a=[α₁,α₂,...,α_m]^T。

Preferably, described step (4) includes the sparse coding coefficient a=[α to floating number₁,α₂,...,α_m]^TEach coefficient component carry out scalar quantization, generate integer-valued code coefficient d_i, note quantized interval is Δ, and quantitative formula is as follows:

$d_i=\mathrm{round}\left(\frac{{\mathrm{\α}}_i}{\mathrm{\Δ}}\right)$

Wherein, input value is mapped as nearest integer by round ().

Preferably, described step (5) includes the position and the coefficient value that arrange nonzero coefficient after data form record quantifies, and data record carries out lossless adaptive arithmetic coding.

Further, the message coding of whole input is a decimal by described arithmetic coding, the character string that will encode encodes one by one by character, current interval is divided into some subintervals, the length in each subinterval is directly proportional with the probability of the corresponding character being likely to occur under current context, the interval lower limit low every time produced after coding be interval limit after previous character code with previous siding-to-siding block length current_range be multiplied by current character place interval limit Low [current] and, it may be assumed that

Low=Low+current_range*Low [current]；

Upper limit high is the lower limit sum that just newly generated lower limit low and previous siding-to-siding block length are multiplied by current character, it may be assumed that

High=Low+current_range*High [current].

Preferably, described step (6) includes preserving the binary coded data that adaptive arithmetic code generates.

Preferably, described step (8) includes the arithmetic decoding algorithms of application self-adapting and recovers non-zero position sparse greatly and quantized value, arithmetic decoding process is interval according to its place of data judging, and then judge why character, comprising and arrange interval limit, arrange subinterval lower limit and be decoded into character one by one, interval limit is assigned to subinterval lower limit initial value, subinterval length is each character probabilities, loop termination when the interval character of last judgement is end mark " # " time, decoding completes.

Preferably, described step (9) includes carrying out inverse quantization operation, rebuilds the coefficient value α ' of nonzero coefficient_i, α '_i=d_iΔ。

Preferably, described step (10) includes performing sparse decoding operation, nonzero coefficient position is carried out zero padding operation, obtains m and tie up sparse coding vector α ', rebuild, and the data after decoding and rebuilding are:Wherein,_ФIt it is the dictionary of coding side use.

Preferably, described step (3) is obtained by a series of electric power data sample training in advance with the dictionary Ф in (10), and learning model is:

Wherein, Section 1 is the cost value of reconstruct input data, the l of Section 2₁Norm is to ensure that the openness coefficient for decomposition coefficient is punished, λ is regularization parameter, and C is a constant, and when optimizing, we are by α_i(1≤i≤k) and Ф fix one of them, optimize another, so replace, until convergence, it is thus achieved that for the dictionary Ф of electric power data rarefaction representation.

Preferably,

(12.1) gaussian random battle array is adopted to initialize Ф, by string all normalization every in Ф；

(12.2) fixing Ф, updates α_i(1≤i≤k):

(12.3) fixing α_i(1≤i≤k), updates Ф:

(12.3) iterations t=t+1, iteration (12.2) and (12.3) are until restraining.

Compared with the prior art, the invention have the benefit that

The mode that big for the electric power of input data are converted into sparse coding is exported by the present invention.Among the transmission process of data, transmission is no longer gather the data got before, and only minimum simply by converting the amount of storage obtained data encoding.The topmost advantage of this method is just able to effectively reduce the amount of storage of data, also allows for finding out immanent structure and the substitutive characteristics of data simultaneously, is also beneficial to backup and the subsequent arithmetic of data.

Accompanying drawing explanation

Fig. 1 is the compression transmitting method flow chart of a kind of big data of the electric power based on sparse encoding and decoding provided by the invention.

Detailed description of the invention

Below in conjunction with accompanying drawing, the specific embodiment of the present invention is described in further detail.

The compression transmitting method of a kind of big data of the electric power based on sparse encoding and decoding of the present invention, it is proposed that utilize the mode of sparse coding that the big data of power industry are processed.The sparse coding of the big data of electric power refers under the premise obtaining dictionary by known training sample, and electric power data is carried out rarefaction representation.Dictionary owing to obtaining can be regarded as one group of super complete base, and we can represent initial data with the linear combination of one group of base, and specifies to have only to less several base just by data representation out.Obtaining on the basis of sparse coding, construct scalar quantizer, design position and the value of the big coefficient of non-zero after data form record quantifies, and it is carried out lossless adaptive arithmetic code, reduce memory data output further.This method can represent immanent structure and the feature of data fully, greatly reduces the amount of storage of data, and improves the subsequent treatment efficiency of data.

Wherein, sparse coding can be expressed as follows:

$x=\underset{i=1}{\overset{m}{\mathrm{\Σ}}}{a}_i{\mathrm{\φ}}_i,$

Wherein, x ∈ Rⁿ, Ф=[φ₁,φ₂..., φ_m]∈R^n×m.X is the data of input, and Ф is dictionary, a=[α₁,α₂,...,α_m]^T∈R^mIt it is sparse coding coefficient.

Coding side is it can be seen that the data set of input has been broken down into the linear combination of multiple primitive.In actual solution procedure, we usually solve a following cost function:

$\underset{\mathrm{\α}}{\max }{\left|\right|x-\underset{i=1}{\overset{m}{\mathrm{\Σ}}}{a}_i{\mathrm{\φ}}_i\left|\right|}_2^2+\mathrm{\λ}{\left|\right|a\left|\right|}_1$

Above-mentioned Section 1 is the cost value of reconstruct input data, the l of Section 2₁Norm is to ensure that the openness coefficient for decomposition coefficient is punished, λ is regularization parameter.Due to l₁The sparsity constraints of norm, the factor alpha obtained after solving, while ensureing to rebuild former data x as far as possible, containing only the big coefficient of the non-zero having minority, is had laid a good foundation for follow-up coding.

The coding side sparse coding α for asking, does adaptive coding to it further and processes.Sparse coding before comparing, the result obtained after adaptive coding processes greatly reduces again required memory space.Data compression complicated, higher-dimension can be become the form of simple code by twice coding work completely.

The compression transmitting method of a kind of big data of the electric power based on sparse encoding and decoding of the present invention, idiographic flow is as follows:

1) data waiting for transmission are carried out sparse coding, it is thus achieved that sparse coding coefficient vector a=[α₁,α₂,...,α_m]^T。

2) the sparse coding coefficient a=[α to floating number₁,α₂,...,α_m]^TEach coefficient classification α_i(1≤i≤m) carries out scalar quantization, generates integer-valued code coefficient d_i.Note quantized interval is Δ, and quantitative formula is as follows:

$d_i=\mathrm{round}\left(\frac{{\mathrm{\α}}_i}{\mathrm{\Δ}}\right)$

Wherein, input value is mapped as nearest integer by round ().

3) position and the coefficient value of nonzero coefficient after data form record quantifies are set, it is assumed that having K the big coefficient of non-zero after quantization, the position (i.e. atom sequence number) of nonzero coefficient is designated as p_i(1≤i≤K), concrete form is as follows:

p1

.

.

.

EOF

d1

.

.

.

EOF is the coded markings set, it was shown that the beginning of position encoded end and quantized value coding, # is the end mark of whole data record.And this data record is carried out lossless adaptive arithmetic coding.

The message coding of whole input is a number by arithmetic coding, a decimal n meeting (0.0≤n < 1.0).The character string that will encode encodes one by one by character, and current interval is divided into some subintervals by whole process, and the length in each subinterval is directly proportional with the probability of the corresponding character being likely to occur under current context.The interval lower limit (low) every time produced after coding for the interval limit after previous character code and previous siding-to-siding block length current_range be multiplied by current character place interval limit Low [current] and, it may be assumed that

Low=Low+current_range*Low [current]

The upper limit (high) is the lower limit sum that just newly generated lower limit low and previous siding-to-siding block length are multiplied by current character, it may be assumed that

High=Low+current_range*High [current].

The false code of this process is as follows:

Lower_bound=0

Upper_bound=1

whiletherearestillcoefficientstoencode

Currentrange=high low；

High=low+ (current_range × upperboundofnewsymbol)

Low=low+ (current_range × lowerboundofnewsymbol)

endwhile

4) binary coded data that adaptive arithmetic code generates is preserved.

Decoding end, for the data being transmitted through, is decoded work, reconstructs data, and concrete decoding process is as follows:

5) arithmetic decoding algorithms of application self-adapting recovers the position p that non-zero is sparse greatly_i(1≤i≤K) and quantized value d_i(1≤i≤K).Arithmetic decoding process is interval according to its place of data judging, and then judges why character.Comprising and arrange interval limit, arrange subinterval lower limit and be decoded into character one by one, interval limit is assigned to subinterval lower limit initial value, subinterval length is each character probabilities.Loop termination when the interval character of last judgement is end mark " # " time, decoding completes.

The false code of self adaptation arithmetic decoding process is as follows:

Encodedvalue=encodedinput

whilebitstreamisnotfullydecoded

Current_range=High [current]-Low [current]

Encodedvalue=(encodedvalue-Low [current]) ÷ current_range

endwhile

6) carry out inverse quantization operation, rebuild the coefficient value α ' of nonzero coefficient_i, as follows:

α′_i=d_iΔ

7) performing sparse decoding operation, nonzero coefficient position is carried out zero padding operation, obtains m and tie up sparse coding vector α ', and rebuild as follows, the data after decoding and rebuilding are:

$x^{\&prime;}=\mathrm{\&Phi;}{\mathrm{\&alpha;}}^{\&prime;}=\underset{i=1}{\overset{m}{\mathrm{\&Sigma;}}}{a}_i^{\&prime;}{\mathrm{\&phi;}}_i$

Wherein Ф is the dictionary that coding side uses.

8) compression efficiency of electric power data is had direct relation with the dictionary Ф of use in decoding by coding side, and how selecting dictionary Ф is a key issue.This patent, for the characteristic of electric power data, is obtained by a series of electric power data training sample preconditions.The learning model of dictionary Ф is:

C is a constant, and when optimizing, we are by α_i(1≤i≤k) and Ф fix one of them, optimize another, so replace, until convergence.Whole optimization process is as follows:

8.1) gaussian random battle array is adopted to initialize Ф, by string all normalization every in Ф.

Fixing Ф, updates α_i(1≤i≤k):

${\mathrm{\&alpha;}}_i^t=\arg \underset{{\mathrm{\&alpha;}}_i}{\min }{\left|\right|x_i-{\mathrm{\&Phi;\&alpha;}}_i\left|\right|}_2^2+\mathrm{\&lambda;}{\left|\right|{\mathrm{\&alpha;}}_i\left|\right|}_1$

Can solve by the method for linear programming.

8.2) fixing α_i(1≤i≤k), updates Ф:

8.3) this is the quadratic programming problem of a belt restraining, has many optimization methods.

8.4) iterations t=t+1, iteration 8.2) and 8.3) until convergence.

Embodiment

First coding side trains the training set approximate with testing sample, obtains complete dictionary, for input picture is carried out rarefaction representation.Idiographic flow is as follows:

1) for input picture, we input picture is divided into some pieces (this experiment input picture be sized to 128*128, each fritter be sized to 5*5, from left to right, from top to bottom, it is ensured that between adjacent block, have the overlap of 2 pixels.

2) each data block y in input picture is carried out on dictionary Ф sparse coding, obtain code coefficient α, as follows:

$\mathrm{\&alpha;}=\min {\left|\right|y-\mathrm{\&Phi;\&alpha;}\left|\right|}_2^2+\mathrm{\&lambda;}{\left|\right|\mathrm{\&alpha;}\left|\right|}_1$

3), the sparse coding coefficient a=[α to floating number₁,α₂,...,α_m]^TEach coefficient classification α_i(1≤i≤m) carries out scalar quantization, generates integer-valued code coefficient d_i。

4) position and the coefficient value of nonzero coefficient after data form record quantifies are set, and these data are recorded lossless adaptive arithmetic coding.

5) binary coded data that adaptive arithmetic code generates is preserved.

6) arithmetic decoding algorithms of application self-adapting recovers the position p that non-zero is sparse greatly_i(1≤i≤K) and quantized value d_i(1≤i≤K), adaptive arithmetic decoding process is substantially carried out judging interval work, it is simply that interval according to its place of data judging, and then judges why character.

7) carry out inverse quantization operation, rebuild the coefficient value α ' of nonzero coefficient_i, quantification method is as follows:

α′_i=d_iΔ

8) performing sparse decoding operation, nonzero coefficient position is carried out zero padding operation, obtains m and tie up sparse coding vector α ', and rebuild as follows, the data after decoding and rebuilding are:

$x^{\&prime;}={\mathrm{\&Phi;\&alpha;}}^{\&prime;}=\underset{i=1}{\overset{m}{\mathrm{\&Sigma;}}}{a}_i^{\&prime;}{\mathrm{\&phi;}}_i$

Wherein Ф is the dictionary that coding side uses.

The present embodiment is done the allocation of computer tested: the operating system of 64, the internal memory of 16GB, Intel processors, and software runtime environment is MATLABR2012a version.

Finally should be noted that: above example is only in order to illustrate that technical scheme is not intended to limit; the specific embodiment of the present invention still can be modified or equivalent replacement by those of ordinary skill in the field with reference to above-described embodiment; these are without departing from any amendment of spirit and scope of the invention or equivalent replace, within the claims of the present invention all awaited the reply in application.

Claims (12)

1. the compression transmitting method of the big data of the electric power based on sparse encoding and decoding, it is characterised in that described method includes

(1) system is started；

(2) data are transmitted；

(3) sparse coding；

(4) scalar quantization；

(5) adaptive arithmetic code；

(6) coding is preserved；

(7) judge whether to call former data；

(8) self adaptation arithmetic decoding；

(9) inverse quantization；

(10) sparse decoding；

(11) data are processed.

2. the compression transmitting method of a kind of big data of the electric power based on sparse encoding and decoding as claimed in claim 1, it is characterised in that described sparse coding is expressed as follows:

$x=\underset{i=1}{\overset{m}{\mathrm{\&Sigma;}}}{a}_i{\mathrm{\&phi;}}_i,$

Wherein, x ∈ Rⁿ, Φ=[φ₁,φ₂..., φ_m]∈R^n×m；X is the data of input, and Φ is the dictionary for data representation, a=[α₁,α₂,...,α_m]^T∈R^mFor sparse coding coefficient.

3. the compression transmitting method of a kind of big data of the electric power based on sparse encoding and decoding as claimed in claim 1, it is characterised in that described step (3) includes data waiting for transmission are carried out sparse coding, it is thus achieved that sparse coding coefficient vector a=[α₁,α₂,...,α_m]^T。

4. the compression transmitting method of a kind of big data of the electric power based on sparse encoding and decoding as claimed in claim 1, it is characterised in that described step (4) includes the sparse coding coefficient a=[α to floating number₁,α₂,...,α_m]^TEach coefficient component carry out scalar quantization, generate integer-valued code coefficient d_i, note quantized interval is Δ, and quantitative formula is as follows:

$d_i=\mathrm{round}\left(\frac{{\mathrm{\&alpha;}}_i}{\mathrm{\&Delta;}}\right)$

Wherein, input value is mapped as nearest integer by round ().

5. the compression transmitting method of a kind of big data of the electric power based on sparse encoding and decoding as claimed in claim 1, it is characterized in that, described step (5) includes the position and the coefficient value that arrange nonzero coefficient after data form record quantifies, and data record carries out lossless adaptive arithmetic coding.

6. the compression transmitting method of a kind of big data of the electric power based on sparse encoding and decoding as claimed in claim 5, it is characterized in that, the message coding of whole input is a decimal by described arithmetic coding, the character string that will encode encodes one by one by character, current interval is divided into some subintervals, the length in each subinterval is directly proportional with the probability of the corresponding character being likely to occur under current context, the interval lower limit low every time produced after coding be interval limit after previous character code with previous siding-to-siding block length current_range be multiplied by current character place interval limit Low [current] and, that is:

Low=Low+current_range*Low [current]；

Upper limit high is the lower limit sum that just newly generated lower limit low and previous siding-to-siding block length are multiplied by current character, it may be assumed that

High=Low+current_range*High [current].

7. the compression transmitting method of a kind of big data of the electric power based on sparse encoding and decoding as claimed in claim 1, it is characterised in that described step (6) includes preserving the binary coded data that adaptive arithmetic code generates.

8. the compression transmitting method of a kind of big data of the electric power based on sparse encoding and decoding as claimed in claim 1, it is characterized in that, described step (8) includes the arithmetic decoding algorithms of application self-adapting and recovers non-zero position sparse greatly and quantized value, arithmetic decoding process is interval according to its place of data judging, and then judge why character, comprise and interval limit is set, subinterval lower limit is set and is decoded into character one by one, interval limit is assigned to subinterval lower limit initial value, subinterval length is each character probabilities, loop termination when the interval character of last judgement is end mark " # " time, decoding completes.

9. the compression transmitting method of a kind of big data of the electric power based on sparse encoding and decoding as claimed in claim 1, it is characterised in that described step (9) includes carrying out inverse quantization operation, rebuilds the coefficient value α ' of nonzero coefficient_i, α '_i=d_iΔ。

10. the compression transmitting method of a kind of big data of the electric power based on sparse encoding and decoding as claimed in claim 1, it is characterized in that, described step (10) includes performing sparse decoding operation, nonzero coefficient position is carried out zero padding operation, obtain m and tie up sparse coding vector α ', rebuilding, the data after decoding and rebuilding are:Wherein, Φ is the dictionary that coding side uses.

11. the compression transmitting method of a kind of big data of the electric power based on sparse encoding and decoding as claimed in claim 1, it is characterized in that, described step (3) is obtained by a series of electric power data sample training in advance with the dictionary Φ in (10), and learning model is:

Wherein, Section 1 is the cost value of reconstruct input data, the l of Section 2₁Norm is to ensure that the openness coefficient for decomposition coefficient is punished, λ is regularization parameter, and C is a constant, and when optimizing, we are by α_i(1≤i≤k) and Φ fix one of them, optimize another, so replace, until convergence, it is thus achieved that for the dictionary Φ of electric power data rarefaction representation.

12. the compression transmitting method of a kind of big data of the electric power based on sparse encoding and decoding as claimed in claim 11, it is characterised in that

(12.1) gaussian random battle array is adopted to initialize Φ, by string all normalization every in Φ；

(12.2) fixing Φ, updates α_i(1≤i≤k): ${\mathrm{\&alpha;}}_i^t=\arg \underset{{\mathrm{\&alpha;}}_i}{\min }{\left|\right|x_i-\mathrm{\&Phi;}{\mathrm{\&alpha;}}_i\left|\right|}_2^2+\mathrm{\&lambda;}{\left|\right|{\mathrm{\&alpha;}}_i\left|\right|}_1;$

(12.3) fixing α_i(1≤i≤k), updates Φ:

(12.3) iterations t=t+1, iteration (12.2) and (12.3) are until restraining.