CN111610970A - Method for asynchronously submitting large data scale list package - Google Patents
Method for asynchronously submitting large data scale list package Download PDFInfo
- Publication number
- CN111610970A CN111610970A CN201910132636.4A CN201910132636A CN111610970A CN 111610970 A CN111610970 A CN 111610970A CN 201910132636 A CN201910132636 A CN 201910132636A CN 111610970 A CN111610970 A CN 111610970A
- Authority
- CN
- China
- Prior art keywords
- data
- submitted
- data set
- value pair
- server
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 28
- 238000004806 packaging method and process Methods 0.000 claims abstract description 32
- 238000012545 processing Methods 0.000 claims abstract description 8
- 230000006835 compression Effects 0.000 claims abstract description 4
- 238000007906 compression Methods 0.000 claims abstract description 4
- 238000005538 encapsulation Methods 0.000 claims description 15
- 125000004122 cyclic group Chemical group 0.000 claims description 5
- 235000014276 Diplazium esculentum Nutrition 0.000 claims description 4
- 244000108321 Diplazium esculentum Species 0.000 claims description 4
- 230000006837 decompression Effects 0.000 claims description 4
- 230000004931 aggregating effect Effects 0.000 claims description 2
- 238000012360 testing method Methods 0.000 description 6
- 238000006243 chemical reaction Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/30—Creation or generation of source code
- G06F8/38—Creation or generation of source code for implementing user interfaces
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention discloses a large data sheet list packaging asynchronous submission method, when a user submits form data, firstly, the form data is subjected to collective subpackage, and then, the data which cannot be subjected to subpackage is subjected to front-end compression, so that the data amount which must be submitted at one time can be reduced, and the pressure of a server is reduced; when data is received in the background, the compressed data is decompressed according to a fixed set mode, so that the pressure of a server can be avoided, and finally, the problems of data confusion and dirty reading easily generated under the condition of submitting a large-data-volume form are solved by adding a thread synchronization mode to each function processing process.
Description
Technical Field
The invention belongs to the technical field of internet data processing, and particularly relates to a large data volume encapsulation asynchronous submission method.
Background
The web page development often encounters the problem of submitting a large amount of data forms, and a common method for submitting a large amount of form data online is to synchronously submit the form data at one time by using a post method and a get method, so that the problem of dirty data reading is often caused, the system is tired, the operation time is long, the system response is slow, the white page phenomenon is more likely to occur in the uploading process, and the system is not responded.
Disclosure of Invention
The invention aims to reduce the bandwidth pressure of a server in the process of submitting a large-data-volume form, avoid the problems of overlong time of function operation calculation and real-time concurrency caused by large data volume of a single submitted form, and effectively avoid the problem of dirty reading of data in the process of submitting the data. To achieve the above object, the present invention provides a method for large data table single encapsulation asynchronous submission, comprising the following steps:
s1, performing set packaging on the form data to be submitted to obtain n data sets;
s2, after the encapsulation is finished, putting the encapsulation key value pair data sets of all the data sets into a form to be submitted;
s3, asynchronously submitting the encapsulation key value pair data set of the data set meeting the numerical value requirement to a server according to the size of the form data to be submitted in the step S2;
and S4, the server receives the packaging key value pair data set submitted in the step S3, creates a timer when the data set is submitted for the first time, judges whether the data are received or not by acquiring the time of the timer, returns a result to wait or verify the integrity of the data, and decompresses the compressed data set.
Preferably, the aggregate package in step S1 includes the following steps:
s1.1: carrying out cyclic packaging on the classification of the form data to be submitted to obtain an array set;
s1.2: after the circulation is finished, the server calls a comparison data set of the same type as the array set in the database and puts the comparison data set into an encapsulation key value pair data set of the array set named by the form data type.
Preferably, the cyclic encapsulation described in step S1.1 includes the following steps:
s1.1.1: packaging the form data to be submitted into n groups and packaging one by one according to the unit quantity that a certain specific quantity is a group;
s1.1.2: and all the form data to be submitted are completely packaged to obtain n data sets.
Preferably, when the form data to be submitted is greater than 1MB and less than or equal to 2MB, the asynchronous submission further comprises the following steps:
s3.1: circularly submitting the encapsulated key-value pair data set of the array set to a server in advance;
s3.2: and after the packaging key value pair data set of the array set is submitted, submitting the contrast data set key value pair data set to a server.
Preferably, when the form data to be submitted is greater than 2MB, the asynchronous submission further includes:
s3.3: judging the size of the packed key value pair compressed data set of the array set;
s3.4: submitting the packed key value pair compressed data set of the array set smaller than or equal to 2MB to a server; and (4) aggregating and packaging the real operation question type answer key value pair compressed data set larger than 2MB again, compressing the packed data set, and asynchronously submitting the packed data set to the server.
Preferably, wherein verifying data integrity and decompressing the compressed data set comprises the steps of:
s4.1, judging whether the submitted compressed data is complete or not by the server, if the compressed data is incomplete, quitting the submitting step, if the compressed data is complete, decompressing the submitted compressed data, if the decompression fails, pushing out the submitting step, re-executing the step S1, and if the decompression succeeds, generating a new data set;
and S4.2, creating a new processing flow for the new data set, circularly locking, and carrying out the next data processing.
Preferably, the key-value pair data types include, but are not limited to, json key-value pair data types.
Preferably, the compression method includes, but is not limited to, the gzip method of pako.
Advantageous effects
The implementation of the invention has the following beneficial effects:
when a system user submits form data, the form data is firstly subjected to collective subpackage, and then the data which cannot be subjected to subpackage is subjected to front-end compression, so that the data volume submitted once is reduced, and the bandwidth pressure of a server is greatly reduced; when data are received at the background, compressed data are decompressed according to a fixed set mode, the pressure of a server is avoided, and finally, the mode of adding thread synchronization to the process of processing each function solves the problem of data confusion easily generated under the condition of submitting a large-data-volume form, and meanwhile, the problem of dirty reading of data in the data submission process can be effectively avoided, and the problem of no response of a system caused by the phenomenon of white pages in the data uploading process is solved.
Drawings
FIG. 1 is a schematic data processing flow diagram according to the present invention;
FIG. 2 is a schematic flow chart of an embodiment of the present invention;
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the scope of protection of the present invention.
Example one
According to the examination ID of the examination paper submitted by the examinee: 20160001, obtaining the question information and question type information of the test paper as the brief answer.
According to the question type short-answer questions and the number of the questions 3, carrying out set packaging on answer answers of examinees to obtain an answer packaging key value pair data set of the short-answer questions;
judging the specific accounting step of each question according to the professional requirements of the financial accounting specialty;
acquiring worksheet information adopted in each accounting step;
and circularly converting the data types of the worksheets adopted in each billing step into json key value pair data types according to the number of the worksheets adopted in each billing step.
The cyclic conversion comprises the following steps:
a unit quantity in a group of a specific quantity according to the difference of the quantity of worksheets in the billing step;
converting the data types of a group of worksheets according to the sequence of the worksheets;
after the conversion of the data types of one group of worksheets is finished, converting the data types of the next group of worksheets;
step S2.2.3.3 is repeated until all the data types of the worksheets have been converted for this billing step.
And after the circulation is finished, compressing the data set by using all the worksheet key values obtained in the step S2.2.3 by using a pako.
Compressing data sets by using a pako.js method for the worksheet key-value pairs adopted in each obtained accounting step, and respectively putting the data sets into answer key-value pairs named in each accounting step;
and putting the key value pair data set of the answers obtained in all the accounting steps into an answer packaging key value pair data set named by the real exercise question.
After the packaging is finished, packaging key value pair data sets of answer answers of all question types into a form to be submitted;
and asynchronously submitting the answering answer packaging key value pair data set of each question type meeting the numerical value requirement to a server according to the size of the form data to be submitted.
When the submitted form data is 6MB, the asynchronous submission further comprises one or more of the following steps:
according to the sequence of the frequently-used question types of the test paper submitted by the examinee, circularly submitting the frequently-used question type answer key value pair data set to a server in advance;
after the submission of the data set of the key value of the answer of each frequently-used question type is finished, compressing the data set of the key value of the answer of the real exercise question type to obtain a compressed data set of the key value of the answer of the real exercise question type;
judging that the size of the real operation question type answer key value pair compressed data set is 1.5M;
and submitting the real operation question type answer key value pair compressed data set to a server.
Example two
According to the examination ID of the examination paper submitted by the examinee: 20160002, the question information and question type information of the test paper are obtained as multiple choice questions.
According to the question type multi-choice questions and the number of the questions 5, carrying out set packaging on answer answers of examinees to obtain an answer packaging key value pair data set of the short-answer questions;
the aggregate packaging comprises the following steps:
circularly packaging the answering answers according to the number of questions contained in the test paper multi-choice questions to obtain an array answering answer set;
the cyclic packaging comprises the following steps:
according to the different quantity of the questions of the multiple test paper, taking a certain specific quantity as a unit quantity of a group;
packaging answer answers of a specific number of questions into a group according to the sequence of the questions;
packaging the answer of the next group of questions after the packaging of one group of answers is finished;
until all answers to all questions are packaged.
After circulation is finished, all answer sets of a certain common question type are put into an answer packaging key value pair data set named by the common question type.
After the packaging is finished, putting the key value pair data set packaged by the answer into a form to be submitted;
and asynchronously submitting the answering answer packaging key value pair data set meeting the numerical value requirement to a server according to the size of the form data to be submitted.
When the form data to be submitted is 1.5MB, the asynchronous submission further comprises the following steps:
according to the sequence of the frequently-used question types of the test paper submitted by the examinee, circularly submitting the frequently-used question type answer key value pair data set to a server in advance;
after the data set of the key value pairs of the answers of the frequently-used questions is submitted, the data set of the key value pairs of the answers of the actual exercise questions is submitted to the server.
The above embodiments are only for illustrating the invention and are not to be construed as limiting the invention, and those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention, so that all equivalent technical solutions also belong to the scope of the invention, and the scope of the invention is defined by the claims.
Claims (8)
1. A method for single package asynchronous submission of a large data volume table comprises the following steps:
s1, performing set packaging on the form data to be submitted to obtain n data sets;
s2, after the encapsulation is finished, putting the encapsulation key value pair data sets of all the data sets into a form to be submitted;
s3, asynchronously submitting the encapsulation key value pair data set of the data set meeting the numerical value requirement to a server according to the size of the form data to be submitted in the step S2;
and S4, the server receives the packaging key value pair data set submitted in the step S3, creates a timer when the data set is submitted for the first time, judges whether the data are received or not by acquiring the time of the timer, returns a result to wait or verify the integrity of the data, and decompresses the compressed data set.
2. The method for encapsulating single packages of large data volumes asynchronously committed as claimed in claim 1, wherein said aggregate encapsulation of step S1 includes the steps of:
s1.1: carrying out cyclic packaging on the classification of the form data to be submitted to obtain an array set;
s1.2: after the circulation is finished, the server calls a comparison data set of the same type as the array set in the database and puts the comparison data set into an encapsulation key value pair data set of the array set named by the form data type.
3. The method for single encapsulation asynchronous submission of large data volumes as recited in claim 2, wherein the circular encapsulation at step S1.1 includes the steps of:
s1.1.1: packaging the form data to be submitted into n groups and packaging one by one according to the unit quantity that a certain specific quantity is a group;
s1.1.2: and all the form data to be submitted are completely packaged to obtain n data sets.
4. The method of claim 2, wherein when the form data to be submitted is greater than 1MB and less than or equal to 2MB, the asynchronous submission further comprises the steps of:
s3.1: circularly submitting the encapsulated key-value pair data set of the array set to a server in advance;
s3.2: and after the packaging key value pair data set of the array set is submitted, submitting the contrast data set key value pair data set to a server.
5. The method of claim 2, wherein when the form data to be submitted is greater than 2MB, the asynchronously submitting further comprises:
s3.3: judging the size of the packed key value pair compressed data set of the array set;
s3.4: submitting the packed key value pair compressed data set of the array set smaller than or equal to 2MB to a server; and (4) aggregating and packaging the real operation question type answer key value pair compressed data set larger than 2MB again, compressing the packed data set, and asynchronously submitting the packed data set to the server.
6. The method of large data volume singleton envelope asynchronous submission of claim 1, wherein verifying data integrity and decompressing the compressed data set includes the steps of:
s4.1, judging whether the submitted compressed data is complete or not by the server, if the compressed data is incomplete, quitting the submitting step, if the compressed data is complete, decompressing the submitted compressed data, if the decompression fails, pushing out the submitting step, re-executing the step S1, and if the decompression succeeds, generating a new data set;
and S4.2, creating a new processing flow for the new data set, circularly locking, and carrying out the next data processing.
7. The method of claim 1, wherein the key-value pair data type includes but is not limited to a json key-value pair data type.
8. A method for single encapsulation asynchronous submission of large data tables as recited in any of claims 1-6, wherein the compression method includes but is not limited to the gzip method of pako.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910132636.4A CN111610970A (en) | 2019-02-22 | 2019-02-22 | Method for asynchronously submitting large data scale list package |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910132636.4A CN111610970A (en) | 2019-02-22 | 2019-02-22 | Method for asynchronously submitting large data scale list package |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111610970A true CN111610970A (en) | 2020-09-01 |
Family
ID=72193752
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910132636.4A Pending CN111610970A (en) | 2019-02-22 | 2019-02-22 | Method for asynchronously submitting large data scale list package |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111610970A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112148675A (en) * | 2020-11-24 | 2020-12-29 | 树根互联技术有限公司 | Data transmission method and device, electronic equipment and storage medium |
| CN113126970A (en) * | 2021-03-12 | 2021-07-16 | 广州八爪鱼科技有限公司 | Method and device for designing open interface of arbitration case service system |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110238859A1 (en) * | 2010-03-25 | 2011-09-29 | Canon Kabushiki Kaisha | Information processing apparatus and document data producing method |
| CN102523119A (en) * | 2011-12-16 | 2012-06-27 | 广东东研网络科技有限公司 | EPON network management system data transmission method based on SNMP protocol |
| CN105224257A (en) * | 2015-10-16 | 2016-01-06 | 浪潮(北京)电子信息产业有限公司 | The disposal route of large files and system in a kind of cloud storage system |
| CN107172174A (en) * | 2017-06-01 | 2017-09-15 | 重庆高略启达智慧城市信息技术有限公司 | The quick upload of file is realized between clients and servers |
| CN108234539A (en) * | 2016-12-14 | 2018-06-29 | 北京金山云网络技术有限公司 | A kind of file upload, download, transmission method and device |
| CN108614888A (en) * | 2018-05-04 | 2018-10-02 | 珠海横琴盛达兆业科技投资有限公司 | A method of based on WEB realize disparate databases between data upload with it is synchronous |
-
2019
- 2019-02-22 CN CN201910132636.4A patent/CN111610970A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110238859A1 (en) * | 2010-03-25 | 2011-09-29 | Canon Kabushiki Kaisha | Information processing apparatus and document data producing method |
| CN102523119A (en) * | 2011-12-16 | 2012-06-27 | 广东东研网络科技有限公司 | EPON network management system data transmission method based on SNMP protocol |
| CN105224257A (en) * | 2015-10-16 | 2016-01-06 | 浪潮(北京)电子信息产业有限公司 | The disposal route of large files and system in a kind of cloud storage system |
| CN108234539A (en) * | 2016-12-14 | 2018-06-29 | 北京金山云网络技术有限公司 | A kind of file upload, download, transmission method and device |
| CN107172174A (en) * | 2017-06-01 | 2017-09-15 | 重庆高略启达智慧城市信息技术有限公司 | The quick upload of file is realized between clients and servers |
| CN108614888A (en) * | 2018-05-04 | 2018-10-02 | 珠海横琴盛达兆业科技投资有限公司 | A method of based on WEB realize disparate databases between data upload with it is synchronous |
Non-Patent Citations (1)
| Title |
|---|
| 肖刚;: "基于B/S结构的考试***性能优化", no. 22 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112148675A (en) * | 2020-11-24 | 2020-12-29 | 树根互联技术有限公司 | Data transmission method and device, electronic equipment and storage medium |
| CN113126970A (en) * | 2021-03-12 | 2021-07-16 | 广州八爪鱼科技有限公司 | Method and device for designing open interface of arbitration case service system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110784419B (en) | Method and system for visualizing professional railway electric service data | |
| EP3859644A1 (en) | Block chain-based data check system and method, computing device, and storage medium | |
| CN108304309B (en) | Method and device for automatically comparing test data | |
| CN111610970A (en) | Method for asynchronously submitting large data scale list package | |
| CN104166568B (en) | A kind of method of the picture file of electrical power system loaded in parallel based on CIM/G | |
| CN109582952A (en) | Poem generation method, device, computer equipment and medium | |
| EP4167194A1 (en) | Key point detection method and apparatus, model training method and apparatus, device and storage medium | |
| CN109246219A (en) | A kind of working method and system of IoT data collection system | |
| CN114300084A (en) | Method, device and equipment for automatically generating clinical test case report table | |
| CN109543772A (en) | Data set automatic matching method, device, equipment and computer readable storage medium | |
| CN107256586A (en) | Multimedia platform evaluation method and device and computer storage medium | |
| CN110019662B (en) | Label reconstruction method and device | |
| CN101888451A (en) | Telephone voice system test information generation method, server and equipment | |
| CN110245341B (en) | Identification code batch generation method and device | |
| CN111611511A (en) | Method for synchronously submitting large data scale list in packaging manner | |
| CN106709256A (en) | ECG data processing system and method thereof | |
| CN113920497B (en) | Nameplate recognition model training method, nameplate recognition method and related devices | |
| TWI769466B (en) | Neural network system and method of operating the same | |
| CN110888874B (en) | Device and method for creating form through Excel and storage medium | |
| CN114896161A (en) | File construction method and device based on artificial intelligence, computer equipment and medium | |
| Ivanitskaia | The impact of digitalization on unemployment: the case of the Nordic countries | |
| CN113948187A (en) | Method and device for calculating nutrient components of food formula, computer and storage medium | |
| CN112463611A (en) | Test report generation method and device based on HTML5 and storage medium | |
| CN110837464A (en) | Test case processing method and device | |
| CN201196783Y (en) | Examination device for Word operation test |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20200901 |