CN117119066A - Mobile terminal high-frequency service data dumping method and system adopting message queue - Google Patents
Mobile terminal high-frequency service data dumping method and system adopting message queue Download PDFInfo
- Publication number
- CN117119066A CN117119066A CN202311360210.7A CN202311360210A CN117119066A CN 117119066 A CN117119066 A CN 117119066A CN 202311360210 A CN202311360210 A CN 202311360210A CN 117119066 A CN117119066 A CN 117119066A
- Authority
- CN
- China
- Prior art keywords
- data
- compression
- message queue
- mobile terminal
- service data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 42
- 230000006835 compression Effects 0.000 claims abstract description 76
- 238000007906 compression Methods 0.000 claims abstract description 76
- 230000005540 biological transmission Effects 0.000 claims abstract description 30
- 230000004044 response Effects 0.000 claims abstract description 9
- 230000009191 jumping Effects 0.000 claims description 5
- 230000008569 process Effects 0.000 description 10
- 238000010586 diagram Methods 0.000 description 3
- 238000013507 mapping Methods 0.000 description 3
- 238000004891 communication Methods 0.000 description 2
- 238000007796 conventional method Methods 0.000 description 2
- 238000012795 verification Methods 0.000 description 2
- 230000005856 abnormality Effects 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000006837 decompression Effects 0.000 description 1
- 238000006073 displacement reaction Methods 0.000 description 1
- 230000008676 import Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000006855 networking Effects 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 239000004576 sand Substances 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/04—Protocols for data compression, e.g. ROHC
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1448—Management of the data involved in backup or backup restore
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L1/00—Arrangements for detecting or preventing errors in the information received
- H04L1/12—Arrangements for detecting or preventing errors in the information received by using return channel
- H04L1/16—Arrangements for detecting or preventing errors in the information received by using return channel in which the return channel carries supervisory signals, e.g. repetition request signals
- H04L1/18—Automatic repetition systems, e.g. Van Duuren systems
- H04L1/1829—Arrangements specially adapted for the receiver end
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L1/00—Arrangements for detecting or preventing errors in the information received
- H04L1/12—Arrangements for detecting or preventing errors in the information received by using return channel
- H04L1/16—Arrangements for detecting or preventing errors in the information received by using return channel in which the return channel carries supervisory signals, e.g. repetition request signals
- H04L1/18—Automatic repetition systems, e.g. Van Duuren systems
- H04L1/1867—Arrangements specially adapted for the transmitter end
- H04L1/1874—Buffer management
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/50—Queue scheduling
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/548—Queue
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Quality & Reliability (AREA)
- Environmental & Geological Engineering (AREA)
- Computer Security & Cryptography (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a mobile terminal high-frequency service data dump method and a system adopting a message queue. The method comprises the following steps: the mobile terminal compresses service data by using an improved compression algorithm ZSTD, calculates Hash data and establishes a unified data table; judging the current network environment, and if the current network environment is reliable, sending the data table to a message queue server; if the network is overtime, the data table is stored in the local end, and the transmission is automatically and periodically attempted in the background; after receiving the data table, the message queue server judges whether the transmission is normal or not by verifying the correctness of the Hash data; and sending the data table to a plurality of redundant backup database servers through the distributed high concurrency network interface. The invention greatly reduces response time under the condition of ensuring stable data, can save waiting time of operators at the mobile end of the power grid, improves working efficiency and ensures stable digital level of field operation equipment.
Description
Technical Field
The invention relates to the technical field of a data quick dump method of a database and a message queue, in particular to a mobile terminal high-frequency service data dump method and a system adopting the message queue.
Background
For the field operation high-frequency service in the power grid industry, the automatic recording and automatic archiving of new and old equipment information can be realized at the mobile phone end by depending on the marketing field operation APP, so that the charging errors caused by manual recording are reduced, the paper bill carrying is reduced, and the repeated recording of internal service data is avoided. However, the environment on which the power grid equipment depends is mostly an outdoor or indoor narrow space, the situation that the network condition is poor usually occurs, the situations of server delay, errors and the like caused by various unknown network problems are faced, if data are directly written into a database server at a mobile terminal, the data loss and database errors can be caused due to transmission interruption, and the waiting time of power grid operators is increased. Meanwhile, in order to ensure data security and reliability, equipment information loss caused by errors of a single database server is avoided, and a redundant backup server is usually required to be used.
Disclosure of Invention
The invention aims to provide a mobile terminal high-frequency service data dump method and a system adopting a message queue, so as to reduce the response time of a database as much as possible and rapidly and accurately write complex contents into the database.
In order to achieve the purpose of the invention, the technical scheme provided by the invention is as follows:
first aspect
The invention provides a mobile terminal high frequency service data dump method adopting a message queue, which comprises the following steps:
step 1: after receiving service data input by operators, the mobile terminal compresses the service data by using an improved compression algorithm ZSTD, calculates Hash data and establishes a unified data table;
step 2: the mobile terminal judges the current network environment, and if the current network environment is reliable, the mobile terminal sends the data table to a message queue server; if the network is overtime, the data table is stored in the local end, and the transmission is automatically and periodically attempted in the background;
step 3: after receiving the data table, the message queue server judges whether the transmission is normal or not by verifying the correctness of the Hash data; if the Hash data is incorrect, a data submitting request is sent to the mobile terminal again, and if the Hash data is correct, a response is sent to the mobile terminal to finish transmission;
step 4: the message queue server sends the data table to a plurality of redundant backup database servers through a distributed high concurrency network interface;
step 5: and after the redundant backup database server receives the data table, archiving is completed.
In step 2, the data table is sent to the message queue server through the SSL encrypted data transmission channel.
In step 1, the improved compression algorithm ZSTD is a ZSTD algorithm based on multiple Hash tables and concurrent flows.
In step 1, the improved compression algorithm ZSTD uses finite state entropy as an entropy encoding mode.
In step 1, the improved compression algorithm ZSTD is used to compress service data, which specifically includes:
step 1.1: dividing service data into a plurality of data blocks, reading each data block into a cache in sequence, and initializing a hash table for LZ77 compression, wherein the size of each data block is not more than 128 KB;
step 1.2: inputting the data block in the cache into an LZ77 core for LZ77 compression, and outputting two parts of contents in the compression stage: the first portion of content is an unsuccessful matching text literal and a series of triples (ll, of, ml), where ll represents the length of the unsuccessful matching text, of represents an offset from the matching text, ml represents the length of the successful matching text, and the second portion of content is a successful matching text literal;
step 1.3: distributing the first part of content output after LZ77 kernel compression to a Hough kernel, carrying out Hough compression according to the triplet and the literal quantity, and further carrying out FSE compression; inputting the second part of content directly into the FSE core for FSE compression;
step 1.4: selecting an optimal compression result from the results output by the FSE core to perform BLOCK reconstruction combination operation;
step 1.5: and judging whether the data is the last BLOCK, if not, jumping to the step 1.1 to circulate until the compression of all the service data is completed.
Second aspect
Correspondingly to the method, the invention also provides a mobile terminal high-frequency service data dump system adopting the message queue, which comprises a mobile terminal, a message queue server and a redundant backup database server;
the mobile terminal is used for compressing service data by using an improved compression algorithm ZSTD after receiving the service data input by an operator, calculating Hash data and establishing a unified data table; the method is also used for judging the current network environment, and if the current network environment is reliable, the data table is sent to a message queue server; if the network is overtime, the data table is stored in the local end, and the transmission is automatically and periodically attempted in the background;
the message queue server is used for judging whether transmission is normal or not by verifying the correctness of the Hash data after receiving the data table; if the Hash data is incorrect, a data submitting request is sent to the mobile terminal again, and if the Hash data is correct, a response is sent to the mobile terminal to finish transmission; the system is also used for sending the data table to a plurality of redundant backup database servers through a distributed high concurrency network interface;
and after the redundant backup database server receives the data table, archiving is completed.
And sending the data table to a message queue server through the SSL encrypted data transmission channel.
The improved compression algorithm ZSTD is a ZSTD algorithm based on a plurality of Hash tables and concurrent flows.
The improved compression algorithm ZSTD uses finite state entropy as an entropy coding mode.
The method for compressing the service data by using the improved compression algorithm ZSTD specifically comprises the following steps:
step 1.1: dividing service data into a plurality of data blocks, reading each data block into a cache in sequence, and initializing a hash table for LZ77 compression, wherein the size of each data block is not more than 128 KB;
step 1.2: inputting the data block in the cache into an LZ77 core for LZ77 compression, and outputting two parts of contents in the compression stage: the first portion of content is an unsuccessful matching text literal and a series of triples (ll, of, ml), where ll represents the length of the unsuccessful matching text, of represents an offset from the matching text, ml represents the length of the successful matching text, and the second portion of content is a successful matching text literal;
step 1.3: distributing the first part of content output after LZ77 kernel compression to a Hough kernel, carrying out Hough compression according to the triplet and the literal quantity, and further carrying out FSE compression; inputting the second part of content directly into the FSE core for FSE compression;
step 1.4: selecting an optimal compression result from the results output by the FSE core to perform BLOCK reconstruction combination operation;
step 1.5: and judging whether the data is the last BLOCK, if not, jumping to the step 1.1 to circulate until the compression of all the service data is completed.
Compared with the prior art, the technical scheme provided by the invention is as follows:
according to the invention, the service data is compressed by using the improved compression algorithm ZSTD at the mobile terminal, and the Hash data is calculated, so that under the condition that the data reliability is further ensured, the data is cached and detected through the message queue, and then the data is distributed to the redundant backup server, the response time is greatly reduced under the condition that the data stability is ensured, the waiting time of operators at the mobile terminal of a power grid can be saved, the working efficiency is improved, and the digital level stability of on-site operation equipment is ensured.
Drawings
FIG. 1 is a schematic diagram of a method according to an embodiment of the present invention compared with a conventional method;
FIG. 2 is a schematic flow chart of a method according to an embodiment of the present invention;
fig. 3 is a flow chart of a method for compressing service data by using a modified compression algorithm ZSTD according to an embodiment of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is evident that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
FIG. 1 is a schematic diagram of a method according to an embodiment of the present invention compared with a conventional method. In the traditional method, the data uploaded by operators are directly sent to a database server, if the network environment is not good, the data can be directly lost, and due to the fact that the database is crowded, time delay can occur when a plurality of operators upload at the same time. At the same time, since the connection is established directly between the equipment used by the operator and the database server, this approach results in a significant increase in latency if multiple redundant servers are used simultaneously.
In the improved method provided by the invention, the message queue is adopted as a first entry of data, and the data preprocessing is performed at the mobile terminal to ensure the reliability of the data, relieve the storage pressure of the database, improve the dump speed, and conveniently connect a plurality of redundant backup servers as an intermediate distribution server through the message queue server to improve the reliability of the whole data.
Fig. 2 is a schematic flow chart of a method provided by an embodiment of the present invention, which specifically includes the following steps:
step 1: after receiving service data input by operators, the mobile terminal compresses the service data by using an improved compression algorithm ZSTD, calculates Hash data and establishes a unified data table;
step 2: the mobile terminal judges the current network environment, and if the current network environment is reliable, the mobile terminal sends the data table to a message queue server; if the network is overtime, the data table is stored in the local end, and the transmission is automatically and periodically attempted in the background; in the step, the mobile terminal is decoupled from the communication, the subsequent operation is completed by the message queue server, and the information loss caused by network abnormality is avoided by a mode of automatic uploading after local cache networking.
Step 3: after receiving the data table, the message queue server judges whether the transmission is normal or not by verifying the correctness of the Hash data; if the Hash data is incorrect, a data submitting request is sent to the mobile terminal again, and if the Hash data is correct, a response is sent to the mobile terminal to finish transmission;
step 4: the message queue server sends the data table to a plurality of redundant backup database servers through a distributed high concurrency network interface;
step 5: and after the redundant backup database server receives the data table, archiving is completed.
According to the method, an improved ZSTD algorithm compression data packet is designed at a mobile terminal, hash data of the data packet is calculated for subsequent verification, and a unified data table is established. And then judging the current network environment, if the network environment is good, directly transmitting data to a message queue server through an SSL encrypted data transmission channel, and if the network environment is bad, firstly locally storing the data and then attempting to continuously transmit. After receiving the data, the message queue server firstly calculates the Hash data and compares the Hash data with the data in the data packet, and after verification, the message queue server sends a successful data transmission signal to the mobile terminal to complete data transmission. And then the message queue server sends data to a plurality of redundant backup database servers through a distributed high concurrency network interface, and the redundant backup database servers archive the data after receiving the data, thereby completing the rapid dumping of the mobile terminal high-frequency service data.
By adopting the mode, operators only need to store the data after the data acquisition is completed, and the worry that the data cannot be written into the database due to the network problem is avoided, and a message queue method of decoupling communication and the database is used.
Preferably, the improved compression algorithm ZSTD in step 1 is a ZSTD algorithm based on multiple Hash tables concurrent streams.
It should be noted that ZSTD is an efficient compression algorithm that uses a range of algorithms and techniques to achieve compression and decompression of data. Specifically, ZSTD employs dictionary matching, dynamic encoding, entropy encoding, and the like to achieve high compression ratio and fast compression speed.
The improved compression algorithm ZSTD in the scheme of the invention uses finite state entropy (Finite State Entropy) as an entropy coding mode, and the principle is as follows:
for the alphabetCharacter string onSThe information entropy can be expressed as: />；
Wherein,p s representing character stringsSChinese characterProbability of occurrence.
The goal of entropy coding is by encoding stringsSIs of the alphabet of (a)Encoding is performed such that the required storage space is close to the information entropy.
Use stateTo represent the pair stringSWhereinxIs a positive integer. Mapping the encoding process as a function:
representing the currentCharacter stringSIs encoded asIn the case of (2) and the next character issTime-new coding.
Mapping the encoding process into a functionI.e. the current code isxIn the case of (a), the last character is decoded assAnd is also provided withsThe character string which has not been decoded before is encoded as +.>。
The above equation constitutes an inverse function, the encoding process is forward encoding from front to back, and the decoding process is backward decoding from back to front.
To speed up the computation, all probabilities are normalized toThe accuracy of (a), namely the following formula:
this can lose some accuracy by selecting an appropriate onekTo ensure that the accuracy is not lost too much.
At the same time, the coding process is also from a single onexBecomes as followsIs a tuple of (1), wherein-> ，Is a bit sequence. The decoding process also becomes from->To new->Is mapped to the mapping of (a).
The coding process is shown in the following formula, and the whole coding process can be obtained by repeating the following steps:
wherein the method comprises the steps ofIs thatxMove to the righti s Bit, adding it to the previous +.>To obtain a new bit sequence. These calculations are done by a displacement operation and are therefore very fast.
For the decoding process, for any oneIt is necessary to obtain the +.>Only the inverse operation of the coding process is needed. The last step xObtained by the formula:
wherein the method comprises the steps ofRepresents->Last of (3)i s The bit, i.e. the last step can be obtained by performing the inverse of the right shift operationx。
Fig. 3 is a schematic diagram of a modified ZSTD compression algorithm scheme, which uses finite state entropy as the entropy encoder, i.e. the FSE kernel part of the figure. The encoder has the characteristics of high speed and high compression rate, and is suitable for mobile terminal service.
The improved compression algorithm ZSTD is used for compressing service data, and the method specifically comprises the following steps:
step 1.1: dividing service data into a plurality of data blocks, reading each data block into a cache in sequence, and initializing a hash table for LZ77 compression, wherein the size of each data block is not more than 128 KB;
step 1.2: inputting the data block in the cache into an LZ77 core for LZ77 compression, and outputting two parts of contents in the compression stage: the first part of content is the literal amount of unsuccessfully matched text, and a series of triples (ll, of, ml), where ll represents the length of unsuccessfully matched text, of represents the offset from the matched text, ml represents the length of successfully matched text, and the second part is the literal amount of successfully matched;
step 1.3: distributing the first part of content output after LZ77 kernel compression to a Hough kernel, carrying out Hough compression according to the triplet and the literal quantity, and further carrying out FSE compression; inputting the second part of content directly into the FSE core for FSE compression;
step 1.4: selecting an optimal compression result from the results output by the FSE core to perform BLOCK reconstruction combination operation;
step 1.5: and judging whether the data is the last BLOCK, if not, jumping to the step 1.1 to circulate until the compression of all the service data is completed.
Correspondingly to the method, the invention also provides a mobile terminal high-frequency service data dump system adopting the message queue, which comprises a mobile terminal, a message queue server and a redundant backup database server;
the mobile terminal is used for compressing service data by using an improved compression algorithm ZSTD after receiving the service data input by an operator, calculating Hash data and establishing a unified data table; the method is also used for judging the current network environment, and if the current network environment is reliable, the data table is sent to a message queue server; if the network is overtime, the data table is stored in the local end, and the transmission is automatically and periodically attempted in the background;
the message queue server is used for judging whether transmission is normal or not by verifying the correctness of the Hash data after receiving the data table; if the Hash data is incorrect, a data submitting request is sent to the mobile terminal again, and if the Hash data is correct, a response is sent to the mobile terminal to finish transmission; the system is also used for sending the data table to a plurality of redundant backup database servers through a distributed high concurrency network interface;
and after the redundant backup database server receives the data table, archiving is completed.
The mobile terminal runs corresponding functional software to realize corresponding functions. The message queue server needs to use a separate cloud server to construct a zeroMQ message queue service, and after receiving data, the message queue server firstly completes Hash generation and comparison so as to verify the transmission accuracy of the data, then generates an import task, and sends the data to a plurality of redundant backup database servers through a distributed high concurrency network interface to complete archiving. In order to ensure the safety of the database, SSL is used as an encryption algorithm to complete data transmission, and a redundant server formed by a plurality of servers is used to ensure the safety and reliability of stored data.
Finally, it should be noted that: the above-described embodiments are provided for illustration and description of the present invention only and are not intended to limit the invention to the embodiments described. In addition, it will be understood by those skilled in the art that the present invention is not limited to the embodiments described above, and that many variations and modifications may be made in accordance with the teachings of the present invention, which fall within the scope of the claimed invention.
Claims (10)
1. A mobile terminal high frequency service data dump method adopting message queue is characterized by comprising the following steps:
step 1: after receiving service data input by operators, the mobile terminal compresses the service data by using an improved compression algorithm ZSTD, calculates Hash data and establishes a unified data table;
step 2: the mobile terminal judges the current network environment, and if the current network environment is reliable, the mobile terminal sends the data table to a message queue server; if the network is overtime, the data table is stored in the local end, and the transmission is automatically and periodically attempted in the background;
step 3: after receiving the data table, the message queue server judges whether the transmission is normal or not by verifying the correctness of the Hash data; if the Hash data is incorrect, a data submitting request is sent to the mobile terminal again, and if the Hash data is correct, a response is sent to the mobile terminal to finish transmission;
step 4: the message queue server sends the data table to a plurality of redundant backup database servers through a distributed high concurrency network interface;
step 5: and after the redundant backup database server receives the data table, archiving is completed.
2. The method for data dumping of mobile side high frequency services using message queues according to claim 1, wherein in step 2, the data table is sent to the message queue server through SSL encrypted data transmission channel.
3. The method for dumping data of high frequency service at mobile end by using message queue as set forth in claim 1, wherein in step 1, said improved compression algorithm ZSTD is ZSTD algorithm based on multiple Hash table concurrent streams.
4. The method for dumping data of mobile terminal high frequency service by using message queue as claimed in claim 1, wherein in step 1, the improved compression algorithm ZSTD uses finite state entropy as entropy coding mode.
5. The method for dumping high frequency service data at mobile terminal by using message queue as set forth in claim 4, wherein in step 1, the improved compression algorithm ZSTD is used to compress service data, specifically including:
step 1.1: dividing service data into a plurality of data blocks, reading each data block into a cache in sequence, and initializing a hash table for LZ77 compression, wherein the size of each data block is not more than 128 KB;
step 1.2: inputting the data block in the cache into an LZ77 core for LZ77 compression, and outputting two parts of contents in the compression stage: the first portion of content is an unsuccessful matching text literal and a series of triples (ll, of, ml), where ll represents the length of the unsuccessful matching text, of represents an offset from the matching text, ml represents the length of the successful matching text, and the second portion of content is a successful matching text literal;
step 1.3: distributing the first part of content output after LZ77 kernel compression to a Hough kernel, carrying out Hough compression according to the triplet and the literal quantity, and further carrying out FSE compression; inputting the second part of content directly into the FSE core for FSE compression;
step 1.4: selecting an optimal compression result from the results output by the FSE core to perform BLOCK reconstruction combination operation;
step 1.5: and judging whether the data is the last BLOCK, if not, jumping to the step 1.1 to circulate until the compression of all the service data is completed.
6. A mobile terminal high frequency service data dump system adopting a message queue is characterized by comprising a mobile terminal, a message queue server and a redundant backup database server;
the mobile terminal is used for compressing service data by using an improved compression algorithm ZSTD after receiving the service data input by an operator, calculating Hash data and establishing a unified data table; the method is also used for judging the current network environment, and if the current network environment is reliable, the data table is sent to a message queue server; if the network is overtime, the data table is stored in the local end, and the transmission is automatically and periodically attempted in the background;
the message queue server is used for judging whether transmission is normal or not by verifying the correctness of the Hash data after receiving the data table; if the Hash data is incorrect, a data submitting request is sent to the mobile terminal again, and if the Hash data is correct, a response is sent to the mobile terminal to finish transmission; the system is also used for sending the data table to a plurality of redundant backup database servers through a distributed high concurrency network interface;
and after the redundant backup database server receives the data table, archiving is completed.
7. The system for mobile side high frequency traffic data dump using message queues according to claim 6, wherein the data table is sent to the message queue server via SSL encrypted data transmission channels.
8. The mobile side high frequency service data dump system using message queues according to claim 6, wherein the improved compression algorithm ZSTD is a ZSTD algorithm based on multiple Hash tables concurrent streams.
9. The system for dumping high frequency service data at mobile end by using message queue as set forth in claim 6, wherein said improved compression algorithm ZSTD uses finite state entropy as entropy coding mode.
10. The system for dumping high frequency service data at mobile end using message queue as claimed in claim 9, wherein said compressing service data by improved compression algorithm ZSTD comprises:
step 1.1: dividing service data into a plurality of data blocks, reading each data block into a cache in sequence, and initializing a hash table for LZ77 compression, wherein the size of each data block is not more than 128 KB;
step 1.2: inputting the data block in the cache into an LZ77 core for LZ77 compression, and outputting two parts of contents in the compression stage: the first portion of content is an unsuccessful matching text literal and a series of triples (ll, of, ml), where ll represents the length of the unsuccessful matching text, of represents an offset from the matching text, ml represents the length of the successful matching text, and the second portion of content is a successful matching text literal;
step 1.3: distributing the first part of content output after LZ77 kernel compression to a Hough kernel, carrying out Hough compression according to the triplet and the literal quantity, and further carrying out FSE compression; inputting the second part of content directly into the FSE core for FSE compression;
step 1.4: selecting an optimal compression result from the results output by the FSE core to perform BLOCK reconstruction combination operation;
step 1.5: and judging whether the data is the last BLOCK, if not, jumping to the step 1.1 to circulate until the compression of all the service data is completed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311360210.7A CN117119066B (en) | 2023-10-20 | 2023-10-20 | Mobile terminal high-frequency service data dumping method and system adopting message queue |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311360210.7A CN117119066B (en) | 2023-10-20 | 2023-10-20 | Mobile terminal high-frequency service data dumping method and system adopting message queue |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117119066A true CN117119066A (en) | 2023-11-24 |
| CN117119066B CN117119066B (en) | 2024-07-02 |
Family
ID=88813196
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311360210.7A Active CN117119066B (en) | 2023-10-20 | 2023-10-20 | Mobile terminal high-frequency service data dumping method and system adopting message queue |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117119066B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070136200A1 (en) * | 2005-12-09 | 2007-06-14 | Microsoft Corporation | Backup broker for private, integral and affordable distributed storage |
| US20160335283A1 (en) * | 2015-05-11 | 2016-11-17 | Vmware, Inc. | Prioritizing backup of endpoint devices in enterprise network environments |
| US20200326910A1 (en) * | 2020-06-23 | 2020-10-15 | Intel Corporation | Normalized probability determination for character encoding |
| CN113204556A (en) * | 2021-05-21 | 2021-08-03 | 山东英信计算机技术有限公司 | Storage method, system, terminal and storage medium based on low-order hash label |
| CN114301468A (en) * | 2021-12-23 | 2022-04-08 | 山东云海国创云计算装备产业创新中心有限公司 | FSE encoding method, device, equipment and storage medium |
-
2023
- 2023-10-20 CN CN202311360210.7A patent/CN117119066B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070136200A1 (en) * | 2005-12-09 | 2007-06-14 | Microsoft Corporation | Backup broker for private, integral and affordable distributed storage |
| US20160335283A1 (en) * | 2015-05-11 | 2016-11-17 | Vmware, Inc. | Prioritizing backup of endpoint devices in enterprise network environments |
| US20200326910A1 (en) * | 2020-06-23 | 2020-10-15 | Intel Corporation | Normalized probability determination for character encoding |
| CN113204556A (en) * | 2021-05-21 | 2021-08-03 | 山东英信计算机技术有限公司 | Storage method, system, terminal and storage medium based on low-order hash label |
| CN114301468A (en) * | 2021-12-23 | 2022-04-08 | 山东云海国创云计算装备产业创新中心有限公司 | FSE encoding method, device, equipment and storage medium |
Non-Patent Citations (2)
| Title |
|---|
| WALKER LOGAN A: "A Comparison of Lossless Compression Methods in Microscopy Data Storage Applications", 《 BIORXIV : THE PREPRINT SERVER FOR BIOLOGY》, 25 January 2023 (2023-01-25) * |
| 曾国庆: "分布式存储中数据压缩研究与实现", 《中国优秀硕士学位论文全文数据库 信息科技辑》, 15 August 2018 (2018-08-15) * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117119066B (en) | 2024-07-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US6711709B1 (en) | Integrated block checking system for rapid file transfer of compressed data | |
| US7307552B2 (en) | Method and apparatus for efficient hardware based deflate | |
| US8456332B2 (en) | Systems and methods for compression of logical data objects for storage | |
| US8924591B2 (en) | Method and device for data segmentation in data compression | |
| US8959155B1 (en) | Data compression through redundancy removal in an application acceleration environment | |
| US11743001B2 (en) | Indexing-based feedback codes and methods of use | |
| RU2005104018A (en) | DATA COMPRESSION | |
| CN103684680A (en) | Decoding a coded data block | |
| US11023412B2 (en) | RDMA data sending and receiving methods, electronic device, and readable storage medium | |
| WO2012155683A1 (en) | Low-density parity check method and device as the same | |
| CN111262876A (en) | Data processing method, device and equipment based on block chain and storage medium | |
| US20180278268A1 (en) | Method for generating a sequence for a pola code and medium therefor and method and apparatus for transmitting data using thereof | |
| CN112131609A (en) | Merkle tree-based electric energy quality data exchange format file integrity verification method and system | |
| US11550756B2 (en) | System and method for error-resilient data reduction | |
| US6788224B2 (en) | Method for numeric compression and decompression of binary data | |
| CN112399479A (en) | Method, electronic device and storage medium for data transmission | |
| CN108347300B (en) | Method and device for adjusting Polar code and coding and decoding device | |
| US20220005229A1 (en) | Point cloud attribute encoding method and device, and point cloud attribute decoding method and devcie | |
| CN117240409B (en) | Data processing method for smart phone and smart wearable device | |
| CN117119066B (en) | Mobile terminal high-frequency service data dumping method and system adopting message queue | |
| CN108900621B (en) | Fog computing mode-based difference cloud synchronization method | |
| CN111464267B (en) | Communication data checking method and device, computer equipment and storage medium | |
| CN112910568A (en) | Data communication method, data sending method and data receiving method | |
| US20090030922A1 (en) | Method and Apparatus for Constructing Efficient Slepian-Wolf Codes With Mismatched Decoding | |
| CN113986853A (en) | Block chain data storage and sharing method, system, equipment and terminal |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |