CN106648955A - Compression method and relevant device - Google Patents
Compression method and relevant device Download PDFInfo
- Publication number
- CN106648955A CN106648955A CN201611006972.7A CN201611006972A CN106648955A CN 106648955 A CN106648955 A CN 106648955A CN 201611006972 A CN201611006972 A CN 201611006972A CN 106648955 A CN106648955 A CN 106648955A
- Authority
- CN
- China
- Prior art keywords
- data
- initial data
- result
- compression
- decompression
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/08—Error detection or correction by redundancy in data representation, e.g. by using checking codes
- G06F11/10—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's
- G06F11/1004—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's to protect a block of data words, e.g. CRC or checksum
Landscapes
- Engineering & Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Security & Cryptography (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Compression, Expansion, Code Conversion, And Decoders (AREA)
Abstract
The invention relates to the technical field of computers, in particular to the compression technology. A compression method includes the steps of obtaining original data, compressing the original data to obtain a compression result, decompressing the compression result to obtain a decompression result, verifying whether the decompression result is consistent with the original data or not, if yes, outputting the compression result, and if not, outputting the original data. Compared with the prior art, after the original data is compressed, the compression result is decompressed before storage to obtain the decompression result, and whether the decompression result is consistent with the original data or not is verified, if yes, the compression result is output and stored, and if not, it indicates that data obtained after decompression is not consistent with the original data, the compression result is abandoned, and the original data is output and stored. Therefore, the phenomenon that the data obtained after decompression is not consistent with the original data can be avoided.
Description
Technical field
The present invention relates to field of computer technology, more particularly to compress technique.
Background technology
Data flood outburst today, in the face of the drastically expansion of data, storage device (such as disk array) need into
Row data compression come effectively realize data capacity reduce：Assume that memory space is 1TB, but user needs the data of storage to be
1.3TB.Then storage device can be compressed to the initial data of above-mentioned 1.3TB, by compression result (data) storage, so can be with
Memory space is saved, and this compression is for a user transparent；When needing to read data, then need first to compressing number
According to being decompressed.
But in actual application, inventor has found：The data obtained after decompression are possible to initial data not
Unanimously, corrupted data is externally shown as, this may further cause upper layer application exception, such as causes virtual machine, database to be answered
With the machine of delaying.
The content of the invention
It is an object of the invention to provide compression method and relevant apparatus, to solve the above problems.
For achieving the above object, the invention provides following scheme：
On the one hand, embodiments herein provides a kind of compression method, and the compression method is by the controller in storage device
In processor or hardware accelerator card, interact with miscellaneous part and complete.The compression method includes：Obtain initial data；To original
Data are compressed, and obtain compression result；Compression result is decompressed, decompression result is obtained；Verification decompression result
It is whether consistent with initial data；If consistent, output squeezing result；If inconsistent, initial data is exported.Compared to prior art,
The scheme that the present invention is provided, after compress, before storage, can decompress to compression result, be decompressed
Sheepshank fruit, and whether verify decompression result consistent with initial data.If consistent, output squeezing result, namely to compression result
Stored.If inconsistent, the data that obtain and initial data are inconsistent after illustrating to decompress, then abandon compression result, export
Initial data, stores to initial data.The data obtained after decompressing can be so avoided to be possible to differ with initial data
The appearance of the phenomenon of cause.
In a possible design, before being compressed to initial data or in verification decompression result and original number
According to whether it is consistent before, can also first calculate the verification data of initial data.Additionally, being with initial data in verification decompression result
It is no it is consistent before, can also first calculate the verification data of decompression data.And verify the decompression result and the initial data
Consistent step, can specifically be refined as：Relatively whether the verification data of the inspection data of initial data and decompression result
Unanimously.If consistent, output squeezing result；If inconsistent, initial data is exported.Must avoid being obtained after decompression such that it is able to more preferable
Data and the inconsistent problem of initial data.
In a possible design, the verification data of above-mentioned initial data may include the check code of initial data (for example
CRC code) and initial data length at least one.And the verification data of decompression result may include：The school of decompression result
Test at least one of the length of code (such as CRC check code) and decompression result.For ease of distinguishing, can be by the verification of initial data
Code is referred to as the first check code, and the length of initial data is referred to as into the first length, and the check code of decompression result is referred to as into the second school
Code is tested, the length of decompression result is referred to as into the second length.If the verification data of initial data includes the first check code and first
Length, then the verification data of decompression result is accordingly including the second check code and the second length；If the check number of initial data
According to the first check code is only included, then the verification data of decompression result only includes accordingly the second check code；And if initial data
Verification data only include the first length, then the verification data of decompression result accordingly only include the second length.In the present invention
In other embodiment, it is also possible to which hash algorithm respectively obtains the cryptographic Hash of initial data and decompression result, by the Hash of the two
Value is used as check code.The particular content of verification data is present embodiments provided, must avoid being obtained after decompression such that it is able to more preferable
Data and the inconsistent problem of initial data.
In a possible design, also directly byte-by-byte contrast can be carried out to decompression result and initial data, this
The mode of kind is relatively simple.
Another further aspect, embodiments provides a kind of storage device, and the storage device has realizes said method reality
The function of storage device behavior in border.The function can be realized by hardware, it is also possible to perform corresponding software by hardware
Realize.
Another aspect, embodiments provides a kind of computer-readable storage medium, for saving as above-mentioned storage device
Computer software instructions used, it includes the program for performing designed by above-mentioned aspect.
Compared to the scheme that prior art, the present invention are provided, after compress, before storage, can be to compression
As a result decompressed, obtained decompression result, and whether verify decompression result consistent with initial data.If consistent, output
Compression result, namely compression result is stored.If inconsistent, illustrate that the data obtained after decompressing and initial data differ
Cause, then abandon compression result, export initial data, initial data is stored.The number obtained after decompressing can so be avoided
According to the appearance for being possible to the phenomenon inconsistent with initial data.
Description of the drawings
In order to be illustrated more clearly that the embodiment of the present invention or technical scheme of the prior art, below will be to institute in embodiment
The accompanying drawing that needs are used is briefly described, it should be apparent that, drawings in the following description are only some enforcements of the present invention
Example, for those of ordinary skill in the art, without having to pay creative labor, can be with according to these accompanying drawings
Obtain other accompanying drawings.
Fig. 1 is application scenarios schematic diagram provided in an embodiment of the present invention；
Fig. 2 a and Fig. 2 b are the exemplary block diagram of controller provided in an embodiment of the present invention；
Fig. 3 a, Fig. 4 are compression method exemplary process diagram provided in an embodiment of the present invention；
Fig. 3 b are the exemplary block diagram of existing compression method；
Fig. 5 is the topology example figure of compression set provided in an embodiment of the present invention (particularly hardware accelerator card)；
Fig. 6 is the data flow schematic diagram inside hardware accelerator card provided in an embodiment of the present invention.
Specific embodiment
Embodiments provide compression method and relevant apparatus (compression set, storage device/system, hardware-accelerated
Data card).
Above-mentioned storage device/system concretely disk array etc..
Fig. 1 depicts the composition figure of storage device/system provided in an embodiment of the present invention, including at least one controller
101 and hard disk chassis 102.
Controller 101 can be a kind of computing device, such as server, desktop computer.Install on controller 101
There are operating system and other application programs.Controller 101 can receive input and output (I/O) request from applied host machine,
The data carried in storage I/O requests, and write data into hard disk chassis 102.Hard disk chassis 102 then may include that one or more are hard
Disk.
Under an application scenarios, controller 101 can be connected by SAN network with applied host machine (not shown).
Hard disk chassis 102 then may include one or more hard disks.For example, 20 hard disks can be installed.
Fig. 2 a are a kind of topology example figures of controller noted above 101, and as shown in Figure 2 a, controller 101 may include by total
Line connected processor 201, memory 202, communication interface 203 (front end interface card and back end interface card can be further included),
And input equipment 204 and output equipment 205.
Wherein, front end interface card can be used to be communicated with applied host machine, and the processor 201 of controller 101 can pass through front end
Interface card receives the various operational orders of applied host machine.
The processor 201 of a controller 101 possibly central processor CPU, or specific integrated circuit ASIC
(Application Specific Integrated Circuit), or be arranged to implement the one of the embodiment of the present invention
Individual or multiple integrated circuits.In embodiments of the present invention, processor 201 can be used for receive from applied host machine write data please
Either read data request, the request of process write data or read data request are asked, and write data are asked or read
Request of data is sent to other operations such as cache.
Memory 202 can be used for data storage.More specifically, memory 202 can be comprising internal memory, caching (such as in Fig. 2 b
Cache 206).Internal memory may include high-speed RAM memory, it is also possible to also including nonvolatile memory (non-
Volatile memory), for example, at least one magnetic disc store.It is understood that internal memory can be random access memory
(Random-Access Memory, RAM), magnetic disc, hard disk, CD, solid state hard disc (Solid State Disk, SSD) or
Nonvolatile memory etc. is various can be with non-transitory (non-transitory) machine readable media of store program codes.
It should be noted that internal memory also can integrated cache of memory function in embodiment.
Back end interface card, can be used to be communicated with the hard disk in hard disk chassis 102, and the processor 201 of controller 101 can pass through
Back end interface card sends the data to the hard-disc storage in hard disk chassis 102.
If it should be noted that hard disk chassis 102 and controller 101 are commonly mounted on an inframe, after inside is just integrated with
End interface card.
And if hard disk chassis 102 and controller 101 are independently mounted in different frames, back end interface card can not be with other portions
Part is integrated.
Above-mentioned compression set can be applied in above-mentioned storage device/system in the form of software or hardware.
In one example, when compression set is applied in a software form in storage device/system, can be by memory 202
The program for performing technical solution of the present invention is preserved, certainly, memory 202 can also preserve operating system and other application journey
Sequence.Processor 201 can perform the compression method of present invention offer by performing said procedure.
In another example, Fig. 2 b are referred to, compression set can be applied to above-mentioned in the form of hardware accelerator card 207
In storage device/system.More specifically, hardware accelerator card 207 may be installed on the slot of controller 101 (controller chassis).
Certainly, also there is caching in hardware accelerator card 207.
When in use, the processor 201 of controller 101 can future self-application main frame initial data be stored in cache206,
Cache206 may decide when for initial data to be supplied to hardware accelerator card 207, and by hardware accelerator card 207 offer of the present invention is performed
Compression method.
Below by the general character being related to based on invention described above aspect, to the embodiment of the present invention, how storage management is entered
Row is further described.
Fig. 3 a show a kind of exemplary flow of compression method provided in an embodiment of the present invention, to solve to avoid decompression
The data obtained after contracting are possible to the problem inconsistent with initial data.
Method shown in Fig. 3 a is applied in the application scenarios shown in Fig. 1, the processor of the controller 101 by shown in Fig. 2 a
201 interact realization with miscellaneous part, or, the hardware accelerator card 207 of the controller 101 by shown in Fig. 2 b is interacted with miscellaneous part
Realize.
The flow process includes：
300 parts：Obtain initial data.
In one example, perform under the scene of compression method in the processor 201 by controller 101, can be by controller
101 processor 201 performs 300 parts.Specifically, can be received by front end interface clamping by the processor 201 of controller 101 should
The initial data sent with main frame, is deposited into cache 206.When needing to be compressed process, processor 201 reads out again
Initial data in cache 206.
In another example, under the scene that compression method is performed by hardware accelerator card 207, in processor 201 in the future
The initial data of self-application main frame is stored in after cache 206, and initial data is supplied to hardware accelerator card 207 by cache 206.
More specifically, cache 206 is in units of page, initial data to be supplied into hardware accelerator card.Per one page
Page memory capacity is 4kB (Byte).
That is, no matter under certain scene, initial data can be first stored in cache 206 (internal memory).
It should be noted that when there is substantial amounts of initial data to need storage, batch processing can be carried out, obtain every time original
A part for data.
301 parts：Initial data is compressed, compression result is obtained.
Can be compressed using different compress modes, for example, Huffman codings.
In one example, perform under the scene of compression method in the processor 201 by controller 101, can be by controller
101 processor 201 performs 301 parts.
In another example, under the scene that compression method is performed by hardware accelerator card 207, can be by hardware accelerator card
207 perform 301 parts.Multiple compression engines, such as 10 compression engines, such that it is able to same can be included in hardware accelerator card 207
Shi Binghang is compressed process.
In fact, inside hardware accelerator card 207, what it was processed is the data on page.It is aforementioned to refer to, one page internal memory
The amount of storage of page is 4KB, it is assumed that once maximum can process the data of 64KB sizes to hardware accelerator card, then can be in single treatment page 16
Deposit page.
Certainly, the data on this page of page 16 can parallel be compressed process by multiple compression engines.
302 parts：Compression result is decompressed, decompression result is obtained.
The decompression algorithm relative with compression algorithm can be adopted, compression result is decompressed.
In one example, perform under the scene of compression method in the processor 201 by controller 101, can be by controller
101 processor 201 performs 302 parts.
In another example, under the scene that compression method is performed by hardware accelerator card 207, can be by hardware accelerator card
207 perform 302 parts.In hardware accelerator card 207 can include multiple decompression engines, such as 3 decompression engines, such that it is able to
Simultaneously decompression is carried out parallel.
303 parts：Whether verification decompression result is consistent with initial data, if unanimously, into 304 parts, otherwise, enters
305 parts.
Verification mode has various, in one example, directly can carry out byte-by-byte to decompression result and initial data
Contrast, if existing inconsistent, judges that the two is inconsistent, otherwise judges that the two is consistent.
In another example, can using CRC (Cyclic Redundancy Check, CRC) verification or
Hash check etc..Subsequently will be specifically introduced herein.
In one example, perform under the scene of compression method in the processor 201 by controller 101, can be by controller
101 processor 201 performs 303 parts.
In another example, under the scene that compression method is performed by hardware accelerator card 207, can be by hardware accelerator card
207 perform 303 parts.
304 parts：Output squeezing result.
It is consistent with initial data after illustrating above-mentioned compression result decompression if check results are consistent.It is not in solution
The inconsistent situation of the data and initial data that obtain after compression, so output squeezing result.
305 parts：Output initial data.
If inconsistent, the data that obtain and initial data are inconsistent after illustrating to decompress, then abandon compression result, and output is former
Beginning data.
In one example, perform under the scene of compression method in the processor 201 by controller 101, can be by controller
101 processor 201 performs 304 and 305 parts.Under the scene, the processor 201 of controller 101 can be by rear termination
Initial data or compression result are sent to hard disk chassis 102 and are stored by mouth card.
In another example, under the scene that compression method is performed by hardware accelerator card 207, can be by hardware accelerator card
207 perform 304 and 305 parts.Hardware accelerator card 207 exports initial data or compression result to cache 206.Subsequently,
Initial data or compression result are sent to hard disk chassis 102 and are stored by cache 206.
It should be noted that in the present embodiment, due to initial data may finally be exported, it is also possible to which output squeezing is tied
Really.So before output, initial data and compression result are buffered in hardware accelerator card 207.
Fig. 3 b are referred to, in existing compress mode, after initial data is got, it can be compressed, and exported
Compression result.Compared to the scheme that prior art, the present invention are provided, after compress, before storage, can be to compression
As a result decompressed, obtained decompression result, and whether verify decompression result consistent with initial data.If consistent, output
Compression result, namely compression result is stored.If inconsistent, illustrate that the data obtained after decompressing and initial data differ
Cause, then abandon compression result, export initial data, initial data is stored.The number obtained after decompressing can so be avoided
According to being possible to inconsistent with initial data.
Below, technical scheme will be carried out specifically so that hardware accelerator card 207 performs compression method as an example
Introduction.
Fig. 4 shows a kind of interactive examples flow process of compression method provided in an embodiment of the present invention.Method shown in Fig. 4
Apply in the application scenarios shown in Fig. 1, the hardware accelerator card 207 of the controller 101 by shown in Fig. 2 b is interacted with miscellaneous part
Realize.The present embodiment introduces how to verify initial data and decompression result.
The interaction flow includes：
400 parts：Hardware accelerator card 207 receives initial data from cache 206.
400 parts are similar with aforementioned 300 part, and therefore not to repeat here.
401 parts：Hardware accelerator card 207 calculates the verification data of the initial data for getting.
In one example, above-mentioned verification data may include the check code (such as CRC check code) and length of initial data.
CRC (Cyclic Redundancy Check, CRC) algorithm is a kind of simple according to specifying data to produce
A kind of hash function of short fixed digit check code, for detecting or verification data transmission or the mistake that is likely to occur after preserving
By mistake.
In other embodiments of the present invention, also the cryptographic Hash of initial data can be obtained as initial data using hash algorithm
Check code.
Hash algorithm is exactly unidirectional digest algorithm, and by arbitrarily long data fixed-length data is produced, when input data occurs on a small quantity
Change can cause output data significant change.
Certainly, the check code of initial data can be also calculated using other algorithms.
For ease of distinguishing, (CRC) check code of initial data can be referred to as first (CRC) check code, by initial data
Length is referred to as the first length.
In another example, the verification data of initial data also can only include the first (CRC) check code or the first length.
402 parts：Hardware accelerator card 207 is compressed to initial data, obtains compression result.
403 parts：Hardware accelerator card 207 is decompressed to compression result, obtains decompression result.
402-403 parts are similar with aforementioned 301-302 parts, and therefore not to repeat here.
404 parts：Hardware accelerator card 207 calculates the verification data of decompression result.
In one example, consistent with the verification data of initial data, the verification data of decompression result may include：Solution
The check code (such as CRC check code) of compression result and length.
In other embodiments of the present invention, the cryptographic Hash that can also obtain decompression result using hash algorithm is verified as it
Code.
For ease of distinguishing, (CRC) check code of decompression result can be referred to as second (CRC) check code, sheepshank will be decompressed
The length of fruit is referred to as the second length.
Certainly, in another example, the verification data of decompression result also can only include the second (CRC) check code or the
Two length.
If it should be noted that the verification data of initial data includes the first (CRC) check code and the first length, decompressing
The verification data of sheepshank fruit includes accordingly the second (CRC) check code and the second length；If the verification data of initial data is only wrapped
First (CRC) check code is included, then the verification data of decompression result only includes accordingly second (CRC) check code；And if original
The verification data of data only includes the first length, then the verification data of decompression result only includes accordingly the second length.
405 parts：Whether hardware accelerator card 207 compares the verification data of the inspection data of initial data and decompression result
Unanimously, if unanimously, into 406 parts, otherwise, into 407 parts.
For the verification data of initial data includes the first (CRC) check code and the first length, the verification of decompression result
Data need to be respectively compared the first (CRC) check code and the accordingly including the second (CRC) check code and the situation of the second length
Whether two (CRC) check codes are equal, and whether the first length is equal with the second length, when equal, can just obtain verification data one
The result of cause.
And for the verification data of initial data only includes the first (CRC) check code or the first length, decompression result
Verification data only includes accordingly the situation of the second (CRC) check code or the second length, as long as then the first (CRC) check code and the
Two (CRC) check codes are equal, or the first length and the second equal length, so that it may obtain the consistent result of verification data.
406 parts：Hardware accelerator card 207 is to the output squeezing results of cache 206.
407 parts：Hardware accelerator card 207 to cache 206 exports initial data.
406-407 parts are similar with aforesaid 304-305 parts, and therefore not to repeat here.
Fig. 5 shows a kind of possible knot of compression set (particularly hardware accelerator card) involved in above-described embodiment
Structure schematic diagram, including：Module for reading and writing 501, compression module 502, decompression module 503 and correction verification module 504.
Wherein, compression module 502 can be used to be compressed the initial data that module for reading and writing 501 is obtained, and obtain compression knot
Really；Compression module 502 may include multiple compression engines.
Decompression module 503 can be used to decompress the compression result that compression module 502 is obtained, and obtain decompressing sheepshank
Really；Decompression module 503 may include multiple decompression engines.
Correction verification module 504 can be used to verify decompression result and the acquisition of module for reading and writing 501 that decompression module 503 is obtained
Whether initial data is consistent；If consistent, indicate that module for reading and writing 501 exports above-mentioned compression result, if inconsistent, indicate module for reading and writing
The above-mentioned initial data of 501 outputs.
More specifically, refer to Fig. 6, above-mentioned module for reading and writing 501 can for DMA (Direct Memory Access, directly in
Access) engine 5011.
DMA engine 5011 adopts DMA technology transmission data.
DMA engine 5011 further may include dma controller and internal storage (can be RAM).
DMA technology allows the hardware unit of friction speed linking up, without depending on CPU (to refer here to control
The processor 201 of device 101) a large amount of interrupt loads.
Data can be copied to another address space by DMA technology from an address space.Typical example is exactly to move
The block of an external memory is moved to the faster memory field of chip internal.
Before DMA transfer, CPU gives dma controller bus control right, and after DMA transfer is terminated, dma controller
Bus control right should be again handed back to CPU.
Dma controller is obtained after bus control right, and CPU hangs up or only carry out built-in function at once, is exported by dma controller
Read write command, directly control RAM and I/O interfaces carry out DMA transfer.
Under the control of dma controller, data transmission is directly carried out between RAM and external equipment, in transmit process
The participation of CPU is not needed.The list item data (original positions and data length of data) of the data to be transmitted need to be provided during beginning.
This is because, aforementioned be referred to the inter-process of hardware accelerator card 207 is page page by page, each internal memory
Page size is that not all content is all initial data in 4KB, but the storage content of this 4KB.So to provide initial data
List item data (original position and length).Here original position refers to offset address.
Certainly, it is also the list item data for needing to provide write-back during the outside output data of DMA engine (can be described as write-back)
's.
List item data can be via CMMDQ (Controller Memory Management Data Queue, in controller
Deposit management-data queue) module 505 is input into or exports.
CMMDQ modules 505 receive list item data from cache 206 in a serial fashion, and it may be sequentially received a lot
Data, these data can be preserved in the form of queue.
CMMDQ modules 505 can provide the list item data of initial data to DMA engine.In output initial data or compression knot
During fruit, CMMDQ modules 5 can be to the list item data of the outside data for providing output.
Fig. 6 shows that the internal data of hardware accelerator card 207 flows to schematic diagram.It may include following flow process：
601 parts：DMA engine 5011 receives page from cache 206.
More specifically, it is page to be copied on the RAM inside DMA engine 5011 from cache 206.
And initial data is stored in page.
602 parts：DMA engine 5011 obtains the list item data of initial data from CMMDQ modules 505.
603 parts：The list item data of the initial data that DMA engine 5011 is provided according to CMMDQ modules 505, by original number
According to being supplied to compression module 502.
Initial data is stored in page, and the list item data of initial data are used to describe the original position of initial data
And length, after learning original position and length, so that it may initial data is read out from page, initial data is supplied into compression
Module 502.
604 parts：Compression module 502 is compressed to initial data, obtains compression result.
Compression result is stored in the memory of hardware accelerator card.
It is aforementioned to be referred to, there are multiple compression engines to be compressed respectively, each compression engine can produce one or more pressures
Contracting data block, all of compression data block constitutes above-mentioned compression result.Certainly, each compression data block has numbering, in order to
Identification.
605 parts：Decompression module 503 is decompressed to above-mentioned compression result, obtains decompression result.
It should be noted that compression module 502 is obtained after compression result, its work is just completed.So how to make decompression
Contracting module 503 starts decompression workCan be realized by asynchronous queue：After compression is completed, compression module 502 can be different
The numbering of compression data block is put into step queue.
As long as asynchronous queue is not sky, the numbering that decompression module 503 will be in asynchronous queue is to compression data block
Carry out decompression.
606 parts：Correction verification module 504 calculates the inspection data of initial data, and by its check number with decompression result
According to comparing, check results are obtained.
Correlative detail can be found in the part of aforementioned 401,404 and 405.
Check results can be consistent or inconsistent.In practice, can be characterized unanimously by coding 0, be characterized not by coding 1
Unanimously.Certainly, it is possible to use characterizing different check results, therefore not to repeat here for other modes.
607 parts：Correction verification module 504 is to the back-checking result of CMMDQ modules 505.
608 parts：Correction verification module 504 returns the list item data of initial data or compression result to DMA engine 5011.
It should be noted that if check results are consistent, correction verification module 504 to DMA engine 5011 returns compression result
List item data；If inconsistent, correction verification module 504 returns the list item data of initial data to DMA engine 5011.
In other embodiments of the present invention, because correction verification module 504 can be to the back-checking result of CMMDQ modules 505, then
The list item data of initial data or compression result can be returned to DMA engine 5011 from CMMDQ modules 505.
609：DMA engine 5011 exports initial data or compression result according to list item data to cache 206；Meanwhile,
CMMDQ modules 505 outwards export the list item data of (exporting to cache 206) initial data or compression result.
Specifically, if check results are consistent, what DMA engine 5011 was received is the list item data of compression result, then export
Page comprising compression result, and CMMDQ modules 505 output be also compression result list item data.
Conversely, if check results are inconsistent, what DMA engine 5011 was received is the list item data of initial data, then export
Page comprising initial data, and CMMDQ modules 505 output be also initial data list item data.
Additionally, module for reading and writing 501 or dma module 5011 can also carry out 300,304 and 305 of Fig. 3 a illustrated embodiments
Point, 400,406 and 407 parts of embodiment illustrated in fig. 4；
Compression module 502 can also carry out 301 parts of Fig. 3 a illustrated embodiments, 402 parts of embodiment illustrated in fig. 4；
Decompression module 503 can also carry out 302 parts of Fig. 3 a illustrated embodiments, 403 parts of embodiment illustrated in fig. 4；
Correction verification module 504 can also carry out 303 parts of Fig. 3 a illustrated embodiments, 401,404 and of embodiment illustrated in fig. 4
405 parts.
The above-mentioned main angle from interaction between each device is described to scheme provided in an embodiment of the present invention.Can
To be understood by, each device, such as controller 101, hardware accelerator card 207 etc. it comprises and hold to realize above-mentioned functions
The corresponding hardware configuration of each function of row and/or software module.Those skilled in the art should be readily appreciated that, with reference to herein
Disclosed in embodiment description each example unit and algorithm steps, the present invention can be soft with hardware or hardware and computer
The combining form of part is realizing.Certain function depends on actually in the way of hardware or computer software drive hardware performing
In the application-specific and design constraint of technical scheme.Professional and technical personnel can use difference to each specific application
Method is realizing described function, but this realization it is not considered that beyond the scope of this invention.
Above-described specific embodiment, has been carried out further to the purpose of the present invention, technical scheme and beneficial effect
Describe in detail, should be understood that the specific embodiment that the foregoing is only the present invention, be not intended to limit the present invention
Protection domain, all any modification, equivalent substitution and improvements on the basis of technical scheme, done etc. all should
It is included within protection scope of the present invention.
Claims (12)
1. a kind of compression method, it is characterised in that include：
Obtain initial data；
The initial data is compressed, compression result is obtained；
The compression result is decompressed, decompression result is obtained；
Verify the decompression result whether consistent with the initial data；
If consistent, the compression result is exported；
If inconsistent, the initial data is exported.
2. the method for claim 1, it is characterised in that before being compressed to the initial data or in verification solution
Before whether compression result is consistent with initial data, also include：Calculate the verification data of the initial data.
3. method as claimed in claim 2, it is characterised in that whether verifying the decompression result and the initial data
Before consistent, also include：Calculate the verification data of the decompression result.
4. method as claimed in claim 3, it is characterised in that the verification decompression result is with the initial data
It is no unanimously to include：
Whether the inspection data of the comparison initial data is consistent with the verification data of the decompression result.
5. method as claimed in claim 4, it is characterised in that the verification data of the initial data includes：The original number
According to cyclic redundancy check (CRC) codes and the initial data length at least one.
6. method as claimed in claim 4, it is characterised in that the verification data of the decompression result includes：The decompression
At least one of the length of the CRC code of sheepshank fruit and the decompression result.
7. the method for claim 1, it is characterised in that the verification decompression result is with the initial data
It is no unanimously to include：
Whether the byte-by-byte contrast decompression result is consistent with the initial data.
8. the method as described in any one of claim 1-7, it is characterised in that methods described is performed by hardware accelerator card.
9. a kind of compression set, it is characterised in that include：
Compression module, for being compressed to the initial data that module for reading and writing is obtained, obtains compression result；
Decompression module, for decompressing to the compression result, obtains decompression result；
Correction verification module, it is whether consistent with the initial data for verifying the decompression result；If consistent, the read-write is indicated
Module exports the compression result, if inconsistent, indicates that the module for reading and writing exports the initial data.
10. device as claimed in claim 9, it is characterised in that the correction verification module is additionally operable to：
Before being compressed to the initial data or before whether verification decompression result is consistent with initial data, calculate
The verification data of the initial data；
Before whether the verification decompression result is consistent with the initial data, the check number of the decompression result is calculated
According to.
11. devices as claimed in claim 10, it is characterised in that in the verification decompression result and the original number
The consistent aspect according to whether, the correction verification module is used for：The inspection data and the decompression result of the comparison initial data
Verification data it is whether consistent.
12. a kind of storage devices, it is characterised in that include the compression set as described in claim 9-11.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611006972.7A CN106648955B (en) | 2016-11-15 | 2016-11-15 | Compression method and related device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611006972.7A CN106648955B (en) | 2016-11-15 | 2016-11-15 | Compression method and related device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106648955A true CN106648955A (en) | 2017-05-10 |
| CN106648955B CN106648955B (en) | 2020-01-31 |
Family
ID=58805993
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201611006972.7A Active CN106648955B (en) | 2016-11-15 | 2016-11-15 | Compression method and related device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106648955B (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107688505A (en) * | 2017-08-15 | 2018-02-13 | 深圳前海信息技术有限公司 | Data verification method and device based on hardware circuit |
| CN108958966A (en) * | 2018-09-27 | 2018-12-07 | 合肥达博科技有限公司 | A kind of data guard method and device of lossless data compression |
| CN109922049A (en) * | 2019-02-02 | 2019-06-21 | 立旃（上海）科技有限公司 | Verifying device and method based on block chain |
| CN110945792A (en) * | 2018-10-31 | 2020-03-31 | 华为技术有限公司 | Method for compressing data, method for decompressing data and related device |
| WO2020143444A1 (en) * | 2019-01-11 | 2020-07-16 | 京东方科技集团股份有限公司 | Data transmission method and apparatus, and display apparatus |
| CN113360314A (en) * | 2020-03-02 | 2021-09-07 | 芯原微电子（成都）有限公司 | Data processing method, system, equipment and computer readable storage medium |
| CN114077569A (en) * | 2020-08-18 | 2022-02-22 | 富泰华工业（深圳）有限公司 | Method and equipment for compressing data and method and equipment for decompressing data |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1921019A (en) * | 2005-08-24 | 2007-02-28 | 英业达股份有限公司 | Method for testing storage by data compressing treatment |
| US20080240419A1 (en) * | 2007-03-30 | 2008-10-02 | Melanie Jean Sandberg | Apparatus, system, and method for testing data compression and data encryption circuitry |
| CN101478370A (en) * | 2009-01-20 | 2009-07-08 | 中兴通讯股份有限公司 | File compression method and apparatus based on file system |
| CN102129873A (en) * | 2011-03-29 | 2011-07-20 | 西安交通大学 | Data compression device and method for improving last-stage high-speed caching reliability of computer |
| CN103384884A (en) * | 2012-12-11 | 2013-11-06 | 华为技术有限公司 | File compression method and device, file decompression method and device, and server |
-
2016
- 2016-11-15 CN CN201611006972.7A patent/CN106648955B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1921019A (en) * | 2005-08-24 | 2007-02-28 | 英业达股份有限公司 | Method for testing storage by data compressing treatment |
| US20080240419A1 (en) * | 2007-03-30 | 2008-10-02 | Melanie Jean Sandberg | Apparatus, system, and method for testing data compression and data encryption circuitry |
| CN101478370A (en) * | 2009-01-20 | 2009-07-08 | 中兴通讯股份有限公司 | File compression method and apparatus based on file system |
| CN102129873A (en) * | 2011-03-29 | 2011-07-20 | 西安交通大学 | Data compression device and method for improving last-stage high-speed caching reliability of computer |
| CN103384884A (en) * | 2012-12-11 | 2013-11-06 | 华为技术有限公司 | File compression method and device, file decompression method and device, and server |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107688505A (en) * | 2017-08-15 | 2018-02-13 | 深圳前海信息技术有限公司 | Data verification method and device based on hardware circuit |
| CN108958966A (en) * | 2018-09-27 | 2018-12-07 | 合肥达博科技有限公司 | A kind of data guard method and device of lossless data compression |
| CN110945792A (en) * | 2018-10-31 | 2020-03-31 | 华为技术有限公司 | Method for compressing data, method for decompressing data and related device |
| WO2020143444A1 (en) * | 2019-01-11 | 2020-07-16 | 京东方科技集团股份有限公司 | Data transmission method and apparatus, and display apparatus |
| US11322074B2 (en) | 2019-01-11 | 2022-05-03 | Chengdu Boe Optoelectronics Technology Co., Ltd. | Data transmission method and device, display device |
| CN109922049A (en) * | 2019-02-02 | 2019-06-21 | 立旃（上海）科技有限公司 | Verifying device and method based on block chain |
| CN109922049B (en) * | 2019-02-02 | 2021-09-28 | 立旃（上海）科技有限公司 | Verification device and method based on block chain |
| CN113360314A (en) * | 2020-03-02 | 2021-09-07 | 芯原微电子（成都）有限公司 | Data processing method, system, equipment and computer readable storage medium |
| CN114077569A (en) * | 2020-08-18 | 2022-02-22 | 富泰华工业（深圳）有限公司 | Method and equipment for compressing data and method and equipment for decompressing data |
| CN114077569B (en) * | 2020-08-18 | 2023-07-18 | 富泰华工业（深圳）有限公司 | Method and device for compressing data, and method and device for decompressing data |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106648955B (en) | 2020-01-31 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106648955A (en) | Compression method and relevant device | |
| US11334255B2 (en) | Method and device for data replication | |
| CN103761988B (en) | Solid state hard disc and data movement method | |
| US20190073132A1 (en) | Method and system for active persistent storage via a memory bus | |
| US9864538B1 (en) | Data size reduction | |
| CA3021969A1 (en) | System and method for tcp offload for nvme over tcp-ip | |
| US9875153B2 (en) | Validation bits and offsets to represent logical pages split between data containers | |
| CN107256363B (en) | High-speed encryption and decryption device composed of encryption and decryption module array | |
| WO2012050935A2 (en) | Apparatus, system, and method for data transformations within a data storage device | |
| WO2013152660A1 (en) | Operating system startup acceleration method and device, terminal and computer storage medium | |
| US11003606B2 (en) | DMA-scatter and gather operations for non-contiguous memory | |
| US20160098322A1 (en) | Background Initialization for Protection Information Enabled Storage Volumes | |
| CN107450856A (en) | Writing method and reading method of stored data, corresponding devices and terminals | |
| CN106681659A (en) | Data compression method and device | |
| CN108701102A (en) | Direct memory access controller, method for reading data and method for writing data | |
| CN104598409A (en) | Method and device for processing input and output requests | |
| WO2023020136A1 (en) | Data storage method and apparatus in storage system | |
| CN108234552B (en) | Data storage method and device | |
| CN105354107A (en) | Data transmission method and system for NOR Flash | |
| US11194498B1 (en) | Inline compression with small-write compression avoidance | |
| CN114518842B (en) | Multi-stream memory system and method for processing input stream in multi-stream memory system | |
| CN104571940A (en) | Data storage device and computing system with the same | |
| WO2022157510A1 (en) | Data processing systems | |
| CN107341113A (en) | Cache compression method and device | |
| US20240241667A1 (en) | Compression accelerator card and method of simulating hard disk mode to access the compression accelerator card |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20200416 Address after: 518129 Bantian HUAWEI headquarters office building, Longgang District, Guangdong, Shenzhen Patentee after: HUAWEI TECHNOLOGIES Co.,Ltd. Address before: 301, A building, room 3, building 301, foreshore Road, No. 310052, Binjiang District, Zhejiang, Hangzhou Patentee before: Huawei Technologies Co.,Ltd. |
|
| TR01 | Transfer of patent right |