CN112182034A - Data compression method and device - Google Patents
Data compression method and device Download PDFInfo
- Publication number
- CN112182034A CN112182034A CN201910596203.4A CN201910596203A CN112182034A CN 112182034 A CN112182034 A CN 112182034A CN 201910596203 A CN201910596203 A CN 201910596203A CN 112182034 A CN112182034 A CN 112182034A
- Authority
- CN
- China
- Prior art keywords
- slope
- data point
- compression
- max
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000013144 data compression Methods 0.000 title claims abstract description 33
- 238000000034 method Methods 0.000 title claims abstract description 29
- 238000007906 compression Methods 0.000 claims abstract description 36
- 230000006835 compression Effects 0.000 claims abstract description 25
- 238000004590 computer program Methods 0.000 claims description 3
- 230000006837 decompression Effects 0.000 abstract description 7
- 238000005516 engineering process Methods 0.000 abstract description 4
- 238000004146 energy storage Methods 0.000 description 3
- 238000010248 power generation Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 1
- 230000008094 contradictory effect Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000002349 favourable effect Effects 0.000 description 1
- 230000000717 retained effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/252—Integrating or interfacing systems involving database management systems between a Database Management System and a front-end application
-
- H—ELECTRICITY
- H03—ELECTRONIC CIRCUITRY
- H03M—CODING; DECODING; CODE CONVERSION IN GENERAL
- H03M7/00—Conversion of a code where information is represented by a given sequence or number of digits to a code where the same, similar or subset of information is represented by a different sequence or number of digits
- H03M7/30—Compression; Expansion; Suppression of unnecessary data, e.g. redundancy reduction
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Compression, Expansion, Code Conversion, And Decoders (AREA)
Abstract
The present invention relates to the field of data compression technologies, and in particular, to a data compression method and apparatus. According to the method, a slope interval range meeting conditions is constructed by setting slope compression deviation, a maximum slope and a minimum slope, selective storage of data points is achieved, data points which do not need to be stored are removed, the data compression rate is improved, the data points meeting the conditions are stored, and better compression and decompression speed performance is guaranteed when data are called.
Description
Technical Field
The present invention relates to the field of data compression technologies, and in particular, to a data compression method and apparatus.
Background
In the data transmission process of the wind-solar-energy-storage combined power generation system, an efficient and high-speed data compression technology is urgently needed. However, data compression techniques face two key issues: firstly, a compression algorithm must be capable of providing a high data compression rate to support the characteristic of mass storage of a real-time database; secondly, the real-time recording and query functions of the real-time database require that the compression algorithm has good speed performance in both the compression process and the decompression process, and particularly in the decompression process, the two problems are mutually contradictory, so that if a data compression technology cannot find out a proper balance point which can give consideration to both the data compression rate and the speed performance in the wind-solar-energy-storage combined power generation system, the wind-solar-energy-storage combined power generation system is not favorable for use.
Disclosure of Invention
The invention aims to provide a data compression method and a data compression device, which are used for solving the problem that the data compression rate and the speed performance cannot be better considered in the conventional data compression process.
In order to achieve the above object, the present invention provides a data compression method, comprising the steps of:
1) acquiring a measured value of a temporary storage data point and a stored value of a previous stored data point, and calculating to obtain a slope K between the temporary storage data point and the previous stored data point;
2) obtaining the maximum slope KmaxAnd minimum slope KminAccording to the maximum slope KmaxMinus a set slopeRate compression deviation to obtain slope K3According to the minimum slope KminAdding the set slope compression deviation to obtain the slope K4；
Wherein, when the previous stored data point is the first data point in the compression process and the temporary stored data point is the second data point in the compression process, the maximum slope K is obtainedmaxAnd minimum slope KminAre all equal to the slope K;
3) obtaining the measured value of the current data point, and calculating to obtain the slope K between the current data point and the previous stored data point1If K is3≤K1≤K4If so, not storing the temporary storage data point;
and, if Kmin>K1Then use K1Updating KminIf K ismin<K1Then K is not updatedmin(ii) a If Kmax<K1Then use K1Updating KmaxIf K ismax>K1Then K is not updatedmax；
4) If K1Out of interval [ K3，K4]And storing the temporary storage data point.
The method has the advantages that the slope interval range meeting the conditions is constructed by setting the slope compression deviation, the maximum slope and the minimum slope, the selective storage of data points is realized, the data points which do not need to be stored are removed, the data compression rate is improved, the data points meeting the conditions are stored, the slope range is gradually increased from one point until the maximum allowable compression range is reached, so that the better compression and decompression speed performance is ensured when the data is called, the method can meet the data compression requirements of data acquisition points under different distribution conditions, the slope range is more accurately positioned, and the data compression rate and the compression and decompression speed performance are improved.
Further, in order to ensure the storage of the temporary storage data point, the temporary storage data point is stored in the buffer area, and if the buffer area is full, the buffer area is switched.
Further, in order to further increase the compression speed, the temporary storage data point is saved to the buffer area, and if the buffer area is full, a background process is called to write the data of the buffer area into the disk.
The invention provides a data compression device, which comprises a memory, a processor and a computer program which is stored in the memory and can run on the processor, wherein the processor executes the program to realize the following steps:
1) acquiring a measured value of a temporary storage data point and a stored value of a previous stored data point, and calculating to obtain a slope K between the temporary storage data point and the previous stored data point;
2) obtaining the maximum slope KmaxAnd minimum slope KminAccording to the maximum slope KmaxSubtracting the set slope compression deviation to obtain the slope K3According to the minimum slope KminAdding the set slope compression deviation to obtain the slope K4；
Wherein, when the previous stored data point is the first data point in the compression process and the temporary stored data point is the second data point in the compression process, the maximum slope K is obtainedmaxAnd minimum slope KminAre all equal to the slope K;
3) obtaining the measured value of the current data point, and calculating to obtain the slope K between the current data point and the previous stored data point1If K is3≤K1≤K4If so, not storing the temporary storage data point;
and, if Kmin>K1Then use K1Updating KminIf K ismin<K1Then K is not updatedmin(ii) a If Kmax<K1Then use K1Updating KmaxIf K ismax>K1Then K is not updatedmax；
4) If K1Out of interval [ K3，K4]And storing the temporary storage data point.
The method has the advantages that the slope interval range meeting the conditions is constructed by setting the slope compression deviation, the maximum slope and the minimum slope, the selective storage of data points is realized, the data points which do not need to be stored are removed, the data compression rate is improved, the data points meeting the conditions are stored, the slope range is gradually increased from one point until the maximum allowable compression range is reached, so that the better compression and decompression speed performance is ensured when the data is called, the method can meet the data compression requirements of data acquisition points under different distribution conditions, the slope range is more accurately positioned, and the data compression rate and the compression and decompression speed performance are improved.
Further, in order to ensure the storage of the temporary data point, the device stores the temporary data point into the buffer area, and if the buffer area is full, the buffer area is switched.
Further, in order to further increase the compression speed, the device stores the temporary data point into the buffer area, and if the buffer area is full, the device calls a background process to write the data in the buffer area into the disk.
Drawings
FIG. 1 is a flow chart of a method of data compression of the present invention;
fig. 2 is a schematic diagram of a data compression method of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings.
The method comprises the following steps:
the invention provides a data compression method, as shown in fig. 1, comprising the following steps:
1) and acquiring a measured value of the temporary storage data point and a stored value of the previous stored data point, and calculating to obtain a slope K between the temporary storage data point and the previous stored data point.
2) Obtaining the maximum slope KmaxAnd minimum slope KminAccording to the maximum slope KmaxSubtracting the set slope compression deviation to obtain the slope K3The slope K is obtained from the minimum slope plus the set slope compression offset YBI4。
When the compression process is carried out until a second data point enters, the compression process is carried out, namely the first data point is a previous storage data point, the temporary storage data point is a second data point, if the second data point enters, the second data point is also a current data point, and at the moment, the maximum slope K adoptedmaxAnd minimumSlope KminAll equal to the slope K.
When only a second data point enters, temporarily storing the data point as a current data point; in other cases, the temporary stored data point is a data point at a time before the current data point.
If the third data point enters, the current data point is the third data point, the temporary storage data point is still the second data point, and the maximum slope K is calculated in the mode of the step 3)maxAnd minimum slope KminAnd (6) updating.
If the fourth data point enters, the current data point is the fourth data point, the temporary storage data point is the third data point, and the maximum slope K is still obtained by the method in step 3)maxAnd minimum slope KminAnd (6) updating. And so on, the compression is realized.
3) Obtaining the measured value of the current data point, and calculating to obtain the slope K between the current data point and the previous stored data point1If K is3≤K1≤K4Then the scratch data point is not stored and if Kmin>K1Then update Kmin＝K1If K ismax<K1Then update Kmax＝K1。
4) If K1Out of interval [ K3，K4]And if the buffer area is full, calling a background process to write the data in the buffer area into the disk.
In this embodiment, the temporary data point, i.e. the data point that needs to be retained, is stored in the buffer area, and as another embodiment, the temporary data point may be directly written into a disk or the like.
In this embodiment, the data in the buffer area is written into the disk only when the buffer area is full, and as another embodiment, the data in the buffer area may be written into the disk when the buffer area is not full.
The embodiment of the device is as follows:
the invention provides a data compression device, which comprises a memory, a processor and a computer program which is stored in the memory and can run on the processor, wherein the steps in the embodiment of the method are realized when the processor executes the program, and detailed description is omitted.
The present invention has been described in relation to particular embodiments thereof, but the invention is not limited to the described embodiments. In the thought given by the present invention, the technical means in the above embodiments are changed, replaced, modified in a manner that is easily imaginable to those skilled in the art, and the functions are basically the same as the corresponding technical means in the present invention, and the purpose of the invention is basically the same, so that the technical scheme formed by fine tuning the above embodiments still falls into the protection scope of the present invention.
Claims (6)
1. A method of data compression, comprising the steps of:
1) acquiring a measured value of a temporary storage data point and a stored value of a previous stored data point, and calculating to obtain a slope K between the temporary storage data point and the previous stored data point;
2) obtaining the maximum slope KmaxAnd minimum slope KminAccording to the maximum slope KmaxSubtracting the set slope compression deviation to obtain the slope K3According to the minimum slope KminAdding the set slope compression deviation to obtain the slope K4；
Wherein, when the previous stored data point is the first data point in the compression process and the temporary stored data point is the second data point in the compression process, the maximum slope K is obtainedmaxAnd minimum slope KminAre all equal to the slope K;
3) obtaining the measured value of the current data point, and calculating to obtain the slope K between the current data point and the previous stored data point1If K is3≤K1≤K4If so, not storing the temporary storage data point;
and, if Kmin>K1Then use K1Updating KminIf K ismin<K1Then K is not updatedmin(ii) a If Kmax<K1Then use K1Updating KmaxIf K ismax>K1Then K is not updatedmax；
4) If K1Out of interval [ K3，K4]And storing the temporary storage data point.
2. A method as claimed in claim 1, wherein the buffered data points are stored in a buffer and the buffer is switched if the buffer is full.
3. A data compression method as claimed in claim 1 or 2, wherein the buffered data points are saved in a buffer, and if the buffer is full, a background process is invoked to write the data in the buffer to the disk.
4. A data compression apparatus comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor when executing the program performs the steps of:
1) acquiring a measured value of a temporary storage data point and a stored value of a previous stored data point, and calculating to obtain a slope K between the temporary storage data point and the previous stored data point;
2) obtaining the maximum slope KmaxAnd minimum slope KminAccording to the maximum slope KmaxSubtracting the set slope compression deviation to obtain the slope K3According to the minimum slope KminAdding the set slope compression deviation to obtain the slope K4；
Wherein, when the previous stored data point is the first data point in the compression process and the temporary stored data point is the second data point in the compression process, the maximum slope K is obtainedmaxAnd minimum slope KminAre all equal to the slope K;
3) obtaining the measured value of the current data point, and calculating to obtain the slope K between the current data point and the previous stored data point1If K is3≤K1≤K4If so, not storing the temporary storage data point;
and, if Kmin>K1Then use K1Updating KminIf K ismin<K1Then K is not updatedmin(ii) a If Kmax<K1Then use K1Updating KmaxIf K ismax>K1Then K is not updatedmax；
4) If K1Out of interval [ K3，K4]And storing the temporary storage data point.
5. The data compression apparatus of claim 4, wherein the buffer is switched if the buffer is full.
6. A data compression apparatus as claimed in claim 4 or 5 wherein a buffer is used to store the buffered data points, and if the buffer is full, a background process is invoked to write the data in the buffer to disk.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910596203.4A CN112182034A (en) | 2019-07-03 | 2019-07-03 | Data compression method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910596203.4A CN112182034A (en) | 2019-07-03 | 2019-07-03 | Data compression method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112182034A true CN112182034A (en) | 2021-01-05 |
Family
ID=73914434
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910596203.4A Pending CN112182034A (en) | 2019-07-03 | 2019-07-03 | Data compression method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112182034A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117194355A (en) * | 2023-11-08 | 2023-12-08 | 本原数据(北京)信息技术有限公司 | Data processing method and device based on database and electronic equipment |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1459743A (en) * | 2002-05-24 | 2003-12-03 | 中国科学院软件研究所 | Self adapting history data compression method |
| CN201993755U (en) * | 2011-01-30 | 2011-09-28 | 上海振华重工(集团)股份有限公司 | Data filtration, compression and storage system of real-time database |
| CN102622367A (en) * | 2011-01-30 | 2012-08-01 | 上海振华重工(集团)股份有限公司 | Method for filtering and compressing process data |
| CN104682962A (en) * | 2015-02-09 | 2015-06-03 | 南京邦耀科技发展有限公司 | Compression method for massive fuel gas data |
-
2019
- 2019-07-03 CN CN201910596203.4A patent/CN112182034A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1459743A (en) * | 2002-05-24 | 2003-12-03 | 中国科学院软件研究所 | Self adapting history data compression method |
| CN201993755U (en) * | 2011-01-30 | 2011-09-28 | 上海振华重工(集团)股份有限公司 | Data filtration, compression and storage system of real-time database |
| CN102622367A (en) * | 2011-01-30 | 2012-08-01 | 上海振华重工(集团)股份有限公司 | Method for filtering and compressing process data |
| CN104682962A (en) * | 2015-02-09 | 2015-06-03 | 南京邦耀科技发展有限公司 | Compression method for massive fuel gas data |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117194355A (en) * | 2023-11-08 | 2023-12-08 | 本原数据(北京)信息技术有限公司 | Data processing method and device based on database and electronic equipment |
| CN117194355B (en) * | 2023-11-08 | 2024-02-13 | 本原数据(北京)信息技术有限公司 | Data processing method and device based on database and electronic equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109802684B (en) | Method and device for data compression | |
| CN108027637A (en) | System and method for carrying out control memory frequency using feedforward compression statistics | |
| CN106649145A (en) | Self-adaptive cache strategy updating method and system | |
| CN103270499A (en) | Log storage method and system | |
| CN112182034A (en) | Data compression method and device | |
| CN105095515A (en) | Bucket dividing method, device and equipment supporting fast query of Map-Reduce output result | |
| CN107153680B (en) | Method and system for on-line node expansion of distributed memory database | |
| US20050256849A1 (en) | Automatic tuning of undo retention | |
| CN107577474B (en) | Processing method and device for upgrading file and electronic equipment | |
| CN109558337B (en) | Dynamic access method, device and storage medium for cache | |
| CN111369302A (en) | Method, system and computer readable storage medium for generating points in real time | |
| CN109033381A (en) | A kind of execution method, device and equipment of service request | |
| CN113434489B (en) | Real-time database online capacity expansion method, system, equipment and storage medium | |
| CN107315806B (en) | Embedded storage method and device based on file system | |
| CN116225630A (en) | Transaction processing method, apparatus, electronic device, storage medium, and program product | |
| GB2612727A (en) | Direct data loading of middleware-generated records | |
| CN113986604A (en) | Data storage method and data storage device | |
| CN112597112A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN106294566A (en) | Cache management system and implementation method | |
| CN106210327A (en) | A kind of method and device of Message Processing | |
| US9256626B2 (en) | Maintaining multiple copy versions of component values in a system | |
| CN111026810A (en) | Data synchronization method, device and storage medium | |
| US11816106B2 (en) | Memory management for KLL sketch | |
| CN110727648B (en) | Efficient data compression method of industrial equipment and data server | |
| CN113034254B (en) | Data processing method and device and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |