CN114050831A

CN114050831A - Decoding method, system, device and medium based on LZ77

Info

Publication number: CN114050831A
Application number: CN202111343929.0A
Authority: CN
Inventors: 苏建龙; 马恒; 李一鸣
Original assignee: Shandong Yunhai Guochuang Cloud Computing Equipment Industry Innovation Center Co Ltd
Current assignee: Shandong Yunhai Guochuang Cloud Computing Equipment Industry Innovation Center Co Ltd
Priority date: 2021-11-14
Filing date: 2021-11-14
Publication date: 2022-02-15

Abstract

The invention discloses a decoding method based on LZ77, which comprises the following steps: in response to receiving the plurality of characters, judging whether a matching pair exists in the plurality of characters; responding to the existence of the matching pair, and acquiring data corresponding to the matching pair according to the distance parameter and the length parameter in the matching pair; in response to the completion of decoding all matching pairs in a plurality of characters to be decoded, splicing the acquired data with other received characters; shifting the cached data in the buffer by using the spliced data to update the invalid data in the buffer into valid spliced data, and marking the bit number of the valid data in the buffer by using a pointer; and in response to the bit number of the effective data marked by the pointer reaching a threshold value, writing all the effective data in the buffer into the RAM, and clearing the bit number of the effective data marked by the pointer to enable all the effective data in the buffer to be invalid data. The invention also discloses a system, computer equipment and a readable storage medium.

Description

Decoding method, system, device and medium based on LZ77

Technical Field

The invention relates to the field of decoding, in particular to an LZ 77-based decoding method, system, equipment and storage medium.

Background

The Huffman compression algorithm is the most commonly used compression algorithm in the current field of data compression. The algorithm carries out coding by counting the occurrence frequency of characters, wherein the longer the occurrence frequency is, the shorter the code length of the characters is, and otherwise, the longer the code length is. However, due to the diversity of text formats, different texts need to be preliminarily compressed into a uniform format by an algorithm before Huffman compression is carried out. The LZ77 algorithm is used here.

The LZ77 algorithm is a dictionary-based lossless compression algorithm of the "sliding window" type. In the compression process, hardware searches whether the current text content appears in a sliding window, if so, the distance between the current text and the last appearing position and the length of the current repeated text are recorded to form a matching pair to exist at the current position. In the encoding process, LZ77 encoding is considered meaningful only when the length of the repeated text is 3 or more, and encoding is not performed when the length is less than 3.

The size of the sliding window has a great influence on the LZ77 encoding efficiency, theoretically, the compression rate will be larger if the sliding window is larger, but as the sliding window is increased, the compression process will be very time-consuming, and the obtained compression rate is not ideally improved, so that the selection of a suitable sliding window which conforms to the application scene is helpful to the compression efficiency.

The following simple LZ77 encoding flow is illustrated by fig. 1-4:

as shown in fig. 1, the letters in the figure represent the code stream to be encoded, and the range of the sliding window includes a search buffer area and an area to be decoded. Assume in this example that the length of the region to be decoded is 3 and the length of the search buffer is 8. As shown in fig. 2, fig. 2 shows the result of encoding the first character a, which will be encoded in the original text since the search buffer is empty when a is encoded. The sliding window is then moved one byte to the right. As shown in fig. 3, fig. 3 is a schematic diagram of the encoded four bytes of the following BCBC. Here, since the length is at least 3 and the length is 2, it is considered that encoding is meaningless, the second BC is also encoded in the original text.

The encoding of the CBC of the region to be decoded is then started, since the CBC encoding sequence is found to be present in the search buffer. And the first C is at position 3, the length of the sequence overlap being 3. Therefore, the code length at this time is 3, the distance is 3, and the code length is represented by (3, 3) in a matching pair. The sliding window is moved three bytes to the right after encoding is finished. As shown in fig. 4, the ABC sequence to be encoded is encoded in accordance with the previous encoding method, and since the ABC sequence is present in the search encoding region. The symbol of the matching pair of this encoding is (3, 8). I.e. a distance of 8 and a length of 3.

The sequence of the original text becomes ABCBC (3, 3) (3, 8) after encoding by the LZ77 algorithm described above. It can be seen that the compressed data is represented in two formats, one being the textual sequence, such as the ABCBC sequence above. The other is a matching pair, such as (3, 3), (3, 8) above. The first character of the matching pair represents the length, i.e. the length of the sequence repetition. The second character represents the distance, i.e. the distance between the first character of the sequence and the current position in the sliding window.

In practical applications, the size of the sliding window can be up to 32K, so a 32K RAM is required for storing the decoded data during the decoding process. Thus, when decoding, the RAM can be read and written to search the matched characters. But frequent reading and writing of the RAM will degrade the decoding performance.

Disclosure of Invention

In view of the above, in order to overcome at least one aspect of the above problems, an embodiment of the present invention provides an LZ 77-based decoding method, including the following steps:

in response to receiving a plurality of characters, judging whether a matching pair exists in the plurality of characters;

responding to the existence of the matching pair, and acquiring data corresponding to the matching pair according to the distance parameter and the length parameter in the matching pair;

in response to the completion of decoding all the matching pairs in the characters to be decoded, splicing the acquired data with the received other characters;

shifting the cached data in the cache by using the spliced data to update the invalid data in the cache into the valid spliced data, and marking the bit number of the valid data in the cache by using a pointer;

and in response to the bit number of the valid data marked by the pointer reaching a threshold value, writing all the valid data in the buffer into the RAM, and clearing the bit number of the valid data marked by the pointer to enable all the valid data in the buffer to be invalid data.

In some embodiments, in response to the existence of the matching pair, obtaining data corresponding to the matching pair according to the distance parameter and the length parameter in the matching pair, further includes:

and responding to the condition that the distance parameter is smaller than the number of the received characters, and acquiring data corresponding to the matching pair from the received characters according to the distance parameter and the length parameter.

In some embodiments, further comprising:

reading data in the buffer in response to the distance parameter being greater than the number of characters being less than the sum of the threshold and the number of characters;

and acquiring data corresponding to the matching pair from the read data in the buffer and the received characters according to the distance parameter and the length parameter.

In some embodiments, further comprising:

reading data in the buffer and the RAM in response to the distance parameter being greater than the sum of the threshold and the number of characters;

and acquiring data corresponding to the matching pair from the read data in the buffer, the data in the RAM and the received characters according to the distance parameter and the length parameter.

In some embodiments, shifting the buffered data in the buffer with the spliced data to update the invalid data in the buffer to the valid spliced data, further comprises:

determining the number of bits of the valid data which can be written according to the number of bits of the valid data in the cached data of the buffer and the threshold;

determining the bit width of the shift according to the bit number of the effective data which can be written and the size of the spliced data;

and generating a plurality of enabling signals according to the shifted bit width, and further moving the corresponding bit width according to the plurality of enabling signals.

and responding to the fact that the distance parameter in the matching pair is smaller than the length parameter, and continuously writing the same part in the spliced data into the buffer.

In some embodiments, further comprising:

and determining the number of the received characters at each time according to a preset first parameter.

Based on the same inventive concept, according to another aspect of the present invention, an embodiment of the present invention also provides an LZ 77-based decoding system, including:

the receiving module is configured to respond to the receiving of a plurality of characters and judge whether a matching pair exists in the plurality of characters;

the acquisition module is configured to respond to the existence of the matching pair, and acquire data corresponding to the matching pair according to the distance parameter and the length parameter in the matching pair;

the splicing module is configured to splice the acquired data with the received other characters in response to the completion of decoding all the matching pairs in the characters to be decoded;

the updating module is configured to shift the cached data in the cache by using the spliced data so as to update the invalid data in the cache into the valid spliced data, and mark the bit number of the valid data in the cache by using a pointer;

and the writing module is configured to respond to the fact that the bit number of the effective data marked by the pointer reaches a threshold value, write all the effective data in the buffer into the RAM, and clear the bit number of the effective data marked by the pointer to enable all the effective data in the buffer to be invalid data.

Based on the same inventive concept, according to another aspect of the present invention, an embodiment of the present invention further provides a computer apparatus, including:

at least one processor; and

a memory storing a computer program operable on the processor, wherein the processor executes the program to perform any of the steps of the LZ 77-based decoding method described above.

Based on the same inventive concept, according to another aspect of the present invention, an embodiment of the present invention also provides a computer-readable storage medium storing a computer program which, when executed by a processor, performs the steps of any of the LZ 77-based decoding methods described above.

The invention has one of the following beneficial technical effects: the scheme provided by the invention can enable the control end to realize remote control on the controlled end only by logging in the mailbox and sending the mail without installing any program.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other embodiments can be obtained by using the drawings without creative efforts.

FIGS. 1-4 are schematic diagrams of an encoding flow based on LZ 77;

FIG. 5 is a flowchart illustrating an LZ 77-based decoding method according to an embodiment of the present invention;

FIG. 6 is a diagram of a hardware architecture for implementing an LZ 77-based decoding method according to an embodiment of the present invention;

fig. 7 is a schematic structural diagram of an external handshake module according to an embodiment of the present invention;

FIG. 8 is a schematic structural diagram of a control module according to an embodiment of the present invention;

FIG. 9 is a diagram illustrating decoding of matching pairs at different distance parameters according to an embodiment of the present invention;

FIG. 10 is a diagram illustrating shifting bits by bit widths of a shifter module according to an embodiment of the present invention under different enable signals;

FIG. 11 is a schematic diagram of an LZ 77-based decoding system according to an embodiment of the present invention;

FIG. 12 is a schematic structural diagram of a computer device provided in an embodiment of the present invention;

fig. 13 is a schematic structural diagram of a computer-readable storage medium according to an embodiment of the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the present invention more apparent, the following embodiments of the present invention are described in further detail with reference to the accompanying drawings.

It should be noted that all expressions using "first" and "second" in the embodiments of the present invention are used for distinguishing two entities with the same name but different names or different parameters, and it should be noted that "first" and "second" are merely for convenience of description and should not be construed as limitations of the embodiments of the present invention, and they are not described in any more detail in the following embodiments.

According to an aspect of the present invention, an embodiment of the present invention proposes an LZ 77-based decoding method, as shown in fig. 5, which may include the steps of:

s1, responding to the received characters, judging whether a matching pair exists in the characters;

s2, responding to the existence of the matching pair, and acquiring data corresponding to the matching pair according to the distance parameter and the length parameter in the matching pair;

s3, in response to the fact that all matching pairs in the characters to be decoded are decoded, splicing the acquired data with other received characters;

s4, shifting the cached data in the buffer by using the spliced data to update the invalid data in the buffer into the valid spliced data, and marking the bit number of the valid data in the buffer by using a pointer;

and S5, in response to the bit number of the valid data marked by the pointer reaching a threshold value, writing all the valid data in the buffer into the RAM, and clearing the bit number of the valid data marked by the pointer to enable all the valid data in the buffer to be invalid data.

The scheme provided by the invention can realize parallel decoding of the matching pair and the original text, and can directly acquire the original text data without accessing the RAM when decoding short matching pair, thereby effectively reducing the frequency of reading data in the RAM, greatly improving the decoding capability of hardware and meeting the throughput rate requirement of an upstream engine.

In some embodiments, fig. 6 shows a schematic diagram of a hardware architecture for implementing the LZ 77-based decoding method of the present invention, which mainly includes five modules, namely, an external interface handshake module, a control module, a shifter module, a data buffer module, and a 32KB sliding window RAM. The decoding method based on LZ77 proposed by the present invention is described in detail below with reference to the hardware architecture diagram shown in FIG. 6.

In some embodiments, in step S1, in response to receiving the plurality of characters, it is determined whether there is a matching pair in the plurality of characters, and specifically, as shown in fig. 7, the receiving of the characters may be implemented by an external interface handshake module. The module mainly completes two functions, one is to perform handshake with an upstream Huffman decoding engine, send out Ready signal after internal decoding is finished, and obtain the next symbol in LZ77 format to be decoded from the Huffman decoding engine. Another function is to select data of different paths according to application scenarios, in the normal compression and decompression process, data is input to the module of the present invention in the form of LZ77, but some application scenarios may not be subjected to compression and decompression, but the original text data is directly input to the module. At this time, data is still required to be written into the sliding window RAM in a certain format and output.

In some embodiments, in step S2, in response to the existence of the matching pair, data corresponding to the matching pair is obtained according to the distance parameter and the length parameter in the matching pair, and specifically, the control module may be used to implement control decoding. The control module controls decoding through the state machine, and when the characters to be decoded are obtained, the control module can calculate the position of the matching pair needing to be decoded currently and whether the RAM needs to be accessed. In the process of shift splicing, the module calculates the length of shift and address information to be accessed next time.

In some embodiments, FIG. 8 illustrates a transition diagram of the state machine in the control module, initialized first before the system decodes. The purpose of initialization is to restore the scene at the last end so that the data coming this time can be correctly decoded. After initialization is completed, the state ST _ RCV _ WAIT is entered, and the handshake module generates Rdy signals to acquire a certain number of symbols to be decoded in LZ77 format from an upstream Huffman decoding engine. According to different obtained symbols, the state machine jumps to different states for decoding. And if all the obtained symbols to be decoded are literal, the state machine enters an ST _ RCV _ LIT state, and the original text existing in the data buffer are directly spliced. If the obtained symbol to be decoded has a matching pair, the state machine enters ST _ RCV _ MCH to start to decode the first matching pair, if the matching pair exists after the first matching pair is finished, the decoding is continued in the state, and if the rest is the terrestrial, the state machine directly enters the ST _ RCV _ LIT state to perform data splicing. When the ST _ RCV _ WAIT decodes a matching pair, the position pointed by the current matching pair is judged. And determining the position of the acquired data according to the position pointed by the matching mid-range distance parameter.

In some embodiments, S2, in response to the existence of the matching pair, acquiring data corresponding to the matching pair according to the distance parameter and the length parameter in the matching pair, further includes:

In some embodiments, further comprising:

Specifically, if the distance parameter in the matching pair is smaller than the number of the received characters, the data corresponding to the matching pair can be directly obtained from the received characters; if the distance parameter is larger than the sum of the number of the characters and the threshold value, the data in the buffer is required to be read; and if the distance parameter is greater than the sum of the threshold value and the number of the characters, reading data in the buffer and the RAM.

For example, as shown in fig. 9, the number of the currently received characters is 5, where the first 4 characters are all original text, and the 5 th character is a matching pair. If the distance parameter D in the matching pair is less than 5, the matching pair can be decoded directly according to the first 4 characters, if the distance parameter in the matching pair is greater than 5 and less than 16, the matching pair needs to be decoded according to the first 4 characters and the data in the buffer, and if the distance parameter in the matching pair is greater than 16, the buffer and the sliding window RAM need to be accessed, so as to decode the matching pair according to the first 4 characters, the data in the buffer and the data in the sliding window RAM.

Judging whether data needs to be read from the buffer or the RAM or not according to the length parameters of the matching pairs so as to obtain data corresponding to the matching pairs, and after all the matching pairs are decoded, performing step S3, namely responding to the completion of decoding all the matching pairs in the characters to be decoded, splicing the obtained data with other received characters so as to obtain decoded data corresponding to the received characters.

Specifically, the data buffered in the buffer includes invalid data already written into the RAM and valid data not yet written into the RAM, that is, the valid data is data that has not yet been written into the sliding window after data splicing. It should be noted that the data splicing here refers to splicing the spliced data obtained in step S3 with the current valid data in the buffer.

When the bit number of the effective data marked by the pointer in the buffer reaches a threshold value, writing the data in the buffer into the RAM, and clearing the bit number of the effective data marked by the pointer to enable all the effective data in the buffer to be invalid data. In this way, when the spliced data is written into the buffer, a shift operation can be performed, so that invalid data with the same number of bits is replaced by spliced valid data, and the pointer is updated.

In some embodiments, as shown in fig. 10, the function of the shifter module is to calculate how much data in the data buffer needs to be shifted for splicing according to the current configuration and the control information. The invention adopts a barrel type shift circuit which can realize configurable shift operation. The bit width of the shift and the direction of the shift can be selected by real-time configuration data. The control information, i.e., the enable signal, is generated by the controller module. In the present invention, the shift operation may be always left shift, so the value of the right direct is always 0. The shifter module triggers different enable signals according to different shift bit widths, so as to complete the shift operation of the designated bit width. Splicing data in the buffer needs to look at two places, one is the amount of data to be written upstream, and the other is the amount of valid data that can be written in the data buffer. The minimum value of the two is taken as the number of the Byte of the shift operation.

For example, if the concatenated data obtained in step S3 is 6 bits and the number of valid data that can be written in the buffer is 5 bits, the enable signal 1 and the enable signal 3 may be generated so as to be shifted by 5 bits. After shifting by 5 bits, the number of bits of valid data in the buffer has reached a threshold value, a write signal to the RAM is generated, and data is written into the sliding window. Meanwhile, the data is output as effective data. At this time, the data in the buffer is invalid data, and the spliced data is written with only 5 bits, so that it is necessary to generate an enable signal 1, move 1 bit in the buffer again, write the last bit of the spliced data into the buffer, and update the value of the pointer marking the number of valid data bits to 1, indicating that 1-bit valid data exists in the buffer at this time.

Specifically, according to the above description, when splicing data in the buffer, the shift bit width of the shifter is the minimum value of the upstream and downstream data amounts. If the length of one matching pair is greater than the distance, for example, the length is 50, and the distance is 1, then the same data is continuously filled into the downstream data buffer. Similar processing can be done for the case where the distance values are 1, 2, 3, 4.

In some embodiments, further comprising:

Specifically, the number of input characters can be changed by configuring the input parameters, thereby improving the decoding efficiency.

The scheme provided by the invention can realize parallel decoding of the matching pair and the original text, and buffer the latest decoded character through the data buffer, and the short matching pair can directly access the data buffer without reading the RAM. Therefore, when short matching time is decoded, the original text data can be directly acquired without accessing the RAM, the frequency of reading data in the RAM is effectively reduced, the decoding capability of hardware is improved to a great extent, and the throughput rate requirement of an upstream engine is met.

Based on the same inventive concept, according to another aspect of the present invention, an embodiment of the present invention further provides an LZ 77-based decoding system 400, as shown in fig. 11, including:

a receiving module 401, configured to, in response to receiving a plurality of characters, determine whether a matching pair exists in the plurality of characters;

an obtaining module 402, configured to, in response to the existence of the matching pair, obtain data corresponding to the matching pair according to a distance parameter and a length parameter in the matching pair;

the splicing module 403 is configured to splice the acquired data with the received other characters in response to that all matching pairs in the characters to be decoded are decoded;

an updating module 404 configured to shift the buffered data in the buffer by using the spliced data to update the invalid data in the buffer into the valid spliced data, and mark the number of bits of the valid data in the buffer by using a pointer;

a writing module 405 configured to, in response to the number of bits of the valid data of the pointer flag reaching a threshold, write all valid data in the buffer into the RAM, and clear the number of bits of the valid data of the pointer flag to make all valid data in the buffer invalid data.

In some embodiments, the acquisition module 402 is further configured to:

In some embodiments, the update module 404 is further configured to:

In some embodiments, further comprising a parameter module configured to:

Based on the same inventive concept, according to another aspect of the present invention, as shown in fig. 12, an embodiment of the present invention further provides a computer apparatus 501, including:

at least one processor 520; and

memory 510, memory 510 storing a computer program 511 executable on the processor, processor 520 executing the program to perform the steps of any of the LZ 77-based decoding methods described above.

Based on the same inventive concept, according to another aspect of the present invention, as shown in fig. 13, an embodiment of the present invention further provides a computer-readable storage medium 601, where the computer-readable storage medium 601 stores computer program instructions 610, and the computer program instructions 610, when executed by a processor, perform the steps of any of the LZ 77-based decoding methods as described above.

Finally, it should be noted that, as will be understood by those skilled in the art, all or part of the processes of the methods of the above embodiments may be implemented by a computer program, which may be stored in a computer-readable storage medium, and when executed, may include the processes of the embodiments of the methods described above.

Further, it should be appreciated that the computer-readable storage media (e.g., memory) herein can be either volatile memory or nonvolatile memory, or can include both volatile and nonvolatile memory.

Those of skill would further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as software or hardware depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the disclosed embodiments of the present invention.

The foregoing is an exemplary embodiment of the present disclosure, but it should be noted that various changes and modifications could be made herein without departing from the scope of the present disclosure as defined by the appended claims. The functions, steps and/or actions of the method claims in accordance with the disclosed embodiments described herein need not be performed in any particular order. Furthermore, although elements of the disclosed embodiments of the invention may be described or claimed in the singular, the plural is contemplated unless limitation to the singular is explicitly stated.

It should be understood that, as used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly supports the exception. It should also be understood that "and/or" as used herein is meant to include any and all possible combinations of one or more of the associated listed items.

The numbers of the embodiments disclosed in the embodiments of the present invention are merely for description, and do not represent the merits of the embodiments.

It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.

Those of ordinary skill in the art will understand that: the discussion of any embodiment above is meant to be exemplary only, and is not intended to intimate that the scope of the disclosure, including the claims, of embodiments of the invention is limited to these examples; within the idea of an embodiment of the invention, also technical features in the above embodiment or in different embodiments may be combined and there are many other variations of the different aspects of the embodiments of the invention as described above, which are not provided in detail for the sake of brevity. Therefore, any omissions, modifications, substitutions, improvements, and the like that may be made without departing from the spirit and principles of the embodiments of the present invention are intended to be included within the scope of the embodiments of the present invention.

Claims

1. An LZ 77-based decoding method, comprising the steps of:

shifting the cached data in the buffer by using the spliced data to update the invalid data in the buffer into valid spliced data, and marking the bit number of the valid data in the buffer by using a pointer;

and in response to the bit number of the effective data marked by the pointer reaching a threshold value, writing all the effective data in the buffer into a RAM, and clearing the bit number of the effective data marked by the pointer to enable all the effective data in the buffer to be invalid data.

2. The method of claim 1, wherein in response to the existence of the matching pair, obtaining data corresponding to the matching pair based on a distance parameter and a length parameter in the matching pair, further comprising:

3. The method of claim 2, further comprising:

4. The method of claim 3, further comprising:

5. The method of claim 1, wherein the spliced data is used to shift the buffered data in a buffer to update invalid data in the buffer to valid spliced data, further comprising:

6. The method of claim 1, wherein the spliced data is used to shift the buffered data in a buffer to update invalid data in the buffer to valid spliced data, further comprising:

7. The method of claim 1, further comprising:

8. An LZ 77-based decoding system, comprising:

9. A computer device, comprising:

at least one processor; and

memory storing a computer program operable on the processor, wherein the processor executes the program to perform the steps of the method according to any of claims 1-7.

10. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, is adapted to carry out the steps of the method according to any one of claims 1 to 7.