CN106484522A - Queue processing method and device - Google Patents
Queue processing method and device Download PDFInfo
- Publication number
- CN106484522A CN106484522A CN201610919579.0A CN201610919579A CN106484522A CN 106484522 A CN106484522 A CN 106484522A CN 201610919579 A CN201610919579 A CN 201610919579A CN 106484522 A CN106484522 A CN 106484522A
- Authority
- CN
- China
- Prior art keywords
- data
- queue
- redis
- redis queue
- hash table
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4812—Task transfer initiation or dispatching by interrupt, e.g. masked
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a kind of queue processing method and device.Included according to the queue processing method that the present invention provides：Judge that in REDIS queue, running total is according to amount；If described total amount of data is more than or equal to predetermined threshold, after waiting the first scheduled duration, rejudge the total amount of data in REDIS queue；If described total amount of data is less than predetermined threshold, extracts the data of predetermined quantity from queue database table, be filled in REDIS queue.The technical scheme being provided according to the present invention, can control the ephemeral data amount in REDIS internal memory, mass data will not be occurred to pour in the situation that internal memory leads to overflow, effectively limit the I/O frequency with data base simultaneously.
Description
Technical field
The present invention relates to the communications field, in particular to a kind of queue processing method and device.
Background technology
With the continuous extension of the business of reselling, the data volume that system is processed increases severely, and increasing service is using queue
Processing mode, allows the data that high concurrent produces enter in enqueue, orderly process.
However, the overstocked treatment effeciency also resulting in program of data is affected, traditional queue mode also can cause
Interactively enter output (Input OutputInterface, abbreviation IO) the excessive problem of pressure with data base.
Therefore, a kind of efficient and controlled queue processing mechanism of offer is provided badly at present, to solve IO pressure greatly, inefficiency
Problem.
Content of the invention
Present invention is primarily targeted at disclosing a kind of call limitation method and device, at least to solve in correlation technique
Queue mode also can cause to interactively enter the excessive problem of output IO pressure with data base.
According to an aspect of the invention, it is provided a kind of queue processing method.
Included according to the queue processing method of the present invention：Judge that in REDIS queue, running total is according to amount；If described sum
It is more than or equal to predetermined threshold according to amount, then, after waiting the first scheduled duration, rejudge the total amount of data in REDIS queue；
If described total amount of data is less than predetermined threshold, extracts the data of predetermined quantity from queue database table, be filled into REDIS
In queue.
Preferably, extract data from queue database table, be filled into REDIS queue and include：Data base is taken out
Data, is filled in described REDIS queue one by one, wherein, using Data Identification as the key of Hash table, data content json is gone here and there
As the value of Hash table, it is saved in together in REDIS Hash table.
Preferably, said method also includes：When extracting data from described REDIS queue, judge that described REDIS queue is
No for sky；If described REDIS queue is sky, after waiting the second scheduled duration, again attempt to extract described REDIS queue
In data；If described REDIS queue is not empty, taken out from Hash table in pending data according to Data Identification
Hold.
Preferably, after pending data content being taken out from Hash table according to Data Identification, also include：According to business
Demand processes described data content, returns the step that next data is extracted in execution from described REDIS queue.
According to a further aspect in the invention, there is provided a kind of queue processing device.
Included according to the queue processing device of the present invention：First judge module, for judging running total in REDIS queue
According to amount；First processing module, for described total amount of data be more than or equal to predetermined threshold when, wait the first scheduled duration it
Afterwards, rejudge the total amount of data in REDIS queue；Second processing module, for being less than predetermined threshold in described total amount of data
When, extract data from queue database table, be filled in REDIS queue.
Compared with prior art, the embodiment of the present invention at least has advantages below：Can control interim in REDIS internal memory
Data volume, will not occur mass data to pour in the situation that internal memory leads to overflow, and effectively limit the IO frequency with data base simultaneously
Rate.
Brief description
Fig. 1 is the flow chart of queue processing method according to embodiments of the present invention；
Fig. 2 is that extraction data according to the preferred embodiment of the invention is filled into the flow chart in REDIS queue；
Fig. 3 is to fetch data the flow chart of process the queue from REDIS according to the preferred embodiment of the invention；
Fig. 4 is the structured flowchart of queue processing device according to embodiments of the present invention；
Fig. 5 is the structured flowchart of queue processing device according to the preferred embodiment of the invention.
Specific embodiment
With reference to Figure of description, the specific implementation of the present invention is made a detailed description.
Fig. 1 is the flow chart of queue processing method according to embodiments of the present invention.As shown in figure 1, this queue processing method
Including：
Step S101：Judge that in REDIS queue, running total is according to amount；
Step S103：If above-mentioned total amount of data is more than or equal to predetermined threshold, after waiting the first scheduled duration, weight
The new total amount of data judging in REDIS queue；
Step S105：If above-mentioned total amount of data is less than predetermined threshold, extract predetermined number from queue database table
According to amount, it is filled in REDIS queue.
Using the method shown in Fig. 1, when in REDIS queue, running total data volume is more than or equal to predetermined threshold, then etc.
After treating the first scheduled duration, rejudge the total amount of data in REDIS queue afterwards；When running total evidence in REDIS queue
From queue database table, when amount is less than predetermined threshold, extract the data of predetermined quantity, be filled in REDIS queue.Can control
Ephemeral data amount in REDIS internal memory processed, will not occur mass data to pour in the situation that internal memory leads to overflow, effectively limit simultaneously
Make the I/O frequency with data base.
Wherein, above-mentioned predetermined threshold and predetermined quantity are the value that dynamically can arrange according to practical situation, both
Collective effect can control the ephemeral data amount in REDIS internal memory, and mass data will not be occurred to pour in the feelings that internal memory leads to overflow
Condition, effectively limits the I/O frequency with data base simultaneously.Furthermore, it is possible to according to the efficiency of handled data volume and requirement,
Adjust treatment progress quantity on demand.
Preferably, extract data from queue database table, be filled in REDIS queue and may further include：By number
According to storehouse take out data, be filled into one by one in above-mentioned REDIS queue, wherein, using Data Identification as Hash table key, by data
The value that content json is gone here and there as Hash table, is saved in REDIS Hash table together.
Preferably, above-mentioned queue processing method can also include：When extracting data from above-mentioned REDIS queue, in judgement
State whether REDIS queue is empty；If above-mentioned REDIS queue is sky, after waiting the second scheduled duration, again attempt to extract
Data in above-mentioned REDIS queue；If above-mentioned REDIS queue is not empty, is taken out from Hash table according to Data Identification and treat
The data content processing.
After pending data content is taken out from Hash table according to Data Identification, following process can also be included：
Process above-mentioned data content according to business demand, return the step that next data is extracted in execution from above-mentioned REDIS queue.
Further describe above-mentioned preferred implementation below in conjunction with Fig. 2 and Fig. 3.
Fig. 2 is that extraction data according to the preferred embodiment of the invention is filled into the flow chart in REDIS queue.As Fig. 2 institute
Show, mainly include the following steps that：
Step S201：Judge running total in REDIS queue according to amount；If less than predetermined threshold, execution step S203,
Otherwise, execution step S207.
Step S203：The data of predetermined quantity is extracted from data base is to list.
Step S205：The data that data base is taken out, one by one in typing REDIS queue.Wherein, data ID is saved in
In REDIS queue；The value that data ID is gone here and there as Hash table as the key of Hash table, data content json, is saved in REDIS Hash
In table.After the completion of this batch data is processed, return execution step S201.
Step S207：Wait 1 second, be then back to execution step S201.
Fig. 3 is to fetch data the flow chart of process the queue from REDIS according to the preferred embodiment of the invention.As shown in figure 3,
Mainly include the following steps that：
Step S301：Extract the data in REDIS queue (processing one by one)
Step S303：Judge whether the data in this REDIS queue is empty, if it is, execution step S305, otherwise, return
Return execution step S309.
Step S305：As there being data ID to take out in queue, according to queue ID, take out from Hash table in pending data
Hold.
Step S307:According to business demand process take out data, after the completion of return execution step S301.
Step S309:As no data in queue, process waits 1 second, returns execution step S301, that is, again attempt to read
REDIS queuing data.
Fig. 4 is the structured flowchart of queue processing device according to embodiments of the present invention.As shown in figure 4, this queue processing dress
Put including：First judge module 40, for judging that in REDIS queue, running total is according to amount；First processing module 42, for
When stating total amount of data more than or equal to predetermined threshold, after waiting the first scheduled duration, rejudge the sum in REDIS queue
According to amount；Second processing module 44, for when above-mentioned total amount of data is less than predetermined threshold, extracting number from queue database table
According to being filled in REDIS queue.
When in REDIS queue, running total data volume is more than or equal to predetermined threshold, then first processing module 42 waits the
After one scheduled duration, rejudge the total amount of data in REDIS queue afterwards；When in REDIS queue, running total is little according to measuring
When predetermined threshold, Second processing module 44 extracts the data of predetermined quantity from queue database table, is filled into REDIS queue
In.The ephemeral data amount in REDIS internal memory can be controlled using this device, mass data will not be occurred to pour in internal memory and lead to overflow
Situation, effectively limit the I/O frequency with data base simultaneously.
Preferably, as shown in figure 5, above-mentioned Second processing module 44 may further include：Fills unit 440, for inciting somebody to action
Data base take out data, be filled into one by one in above-mentioned REDIS queue, wherein, using Data Identification as Hash table key, by number
According to content json string as the value of Hash table, it is saved in together in REDIS Hash table.
Preferably, as shown in figure 5, said apparatus can also include：Second judge module 46, for from above-mentioned REDIS
When data is extracted in queue, judge whether above-mentioned REDIS queue is empty；First extraction module 48, in above-mentioned REDIS queue being
Space-time, after waiting the second scheduled duration, again attempts to extract the data in above-mentioned REDIS queue；Second extraction module 50, uses
In not being space-time in above-mentioned REDIS queue, pending data content is taken out from Hash table according to Data Identification.
Preferably, as shown in figure 5, said apparatus can also include：3rd processing module 52, at according to business demand
Manage above-mentioned data content, return the step that next data is extracted in execution from above-mentioned REDIS queue.
It should be noted that modules in above-mentioned queue processing device, unit be combined with each other enforcement queue at
Reason flow process may refer to the description of Fig. 1 to Fig. 3, and here is omitted.
In sum, the above-described embodiment providing by the present invention, on the one hand can control the nonce in REDIS internal memory
According to amount, mass data will not be occurred to pour in the situation that internal memory leads to overflow, effectively limit the I/O frequency with data base simultaneously,
Ensure that data base is stable；On the other hand, can be with the quantity of distributed in demand treatment progress so that whole ranker be stable high
Effect, and freely controlled.
The several specific embodiments being only the present invention disclosed above, but, the present invention is not limited to this, any ability
What the technical staff in domain can think change all should fall into protection scope of the present invention.
Claims (8)
1. a kind of queue processing method is it is characterised in that include：
Judge that in REDIS queue, running total is according to amount；
If described total amount of data is more than or equal to predetermined threshold, after waiting the first scheduled duration, rejudge REDIS team
Total amount of data in row；
If described total amount of data is less than predetermined threshold, extracts the data of predetermined quantity from queue database table, be filled into
In REDIS queue.
2. method according to claim 1, it is characterised in that extracting data from queue database table, is filled into REDIS
Queue includes：
The data that data base is taken out, is filled in described REDIS queue, one by one wherein, using Data Identification as Hash table
Key, using data content json string as the value of Hash table, is saved in REDIS Hash table together.
3. method according to claim 1 and 2 is it is characterised in that also include：
When extracting data from described REDIS queue, judge whether described REDIS queue is empty；
If described REDIS queue is sky, after waiting the second scheduled duration, again attempt to extract in described REDIS queue
Data；
If described REDIS queue is not empty, pending data content is taken out from Hash table according to Data Identification.
4. method according to claim 3 is it is characterised in that take out pending number according to Data Identification from Hash table
After content, also include：
Process described data content according to business demand, return the step that next data is extracted in execution from described REDIS queue
Suddenly.
5. a kind of queue processing device is it is characterised in that include：
First judge module, for judging that in REDIS queue, running total is according to amount；
First processing module, for described total amount of data be more than or equal to predetermined threshold when, wait the first scheduled duration after,
Rejudge the total amount of data in REDIS queue；
Second processing module, for when described total amount of data is less than predetermined threshold, extracting data, filling out from queue database table
It is charged in REDIS queue.
6. device according to claim 5 is it is characterised in that described Second processing module includes：
Fills unit, for the data taking out data base, is filled in described REDIS queue, one by one wherein, by Data Identification
As the key of Hash table, using data content json string as the value of Hash table, it is saved in together in REDIS Hash table.
7. the device according to claim 5 or 6 is it is characterised in that also include：
Second judge module, for when extracting data from described REDIS queue, judging whether described REDIS queue is empty；
First extraction module, for being space-time in described REDIS queue, after waiting the second scheduled duration, again attempts to extract
Data in described REDIS queue；
Second extraction module, for not being space-time in described REDIS queue, takes out pending from Hash table according to Data Identification
Data content.
8. device according to claim 7 is it is characterised in that also include：
3rd processing module, for processing described data content according to business demand, returns execution and carries from described REDIS queue
The step taking off a data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610919579.0A CN106484522A (en) | 2016-10-21 | 2016-10-21 | Queue processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610919579.0A CN106484522A (en) | 2016-10-21 | 2016-10-21 | Queue processing method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN106484522A true CN106484522A (en) | 2017-03-08 |
Family
ID=58270494
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610919579.0A Withdrawn CN106484522A (en) | 2016-10-21 | 2016-10-21 | Queue processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106484522A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107608858A (en) * | 2017-09-28 | 2018-01-19 | 北京京东尚科信息技术有限公司 | Information processing system, method and apparatus |
| CN109409412A (en) * | 2018-09-28 | 2019-03-01 | 新华三大数据技术有限公司 | Image processing method and device |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103390049A (en) * | 2013-07-23 | 2013-11-13 | 南京联创科技集团股份有限公司 | Method for processing high-speed message queue overflow based on memory database cache |
| CN103927305A (en) * | 2013-01-11 | 2014-07-16 | ***通信集团山东有限公司 | Method and device for controlling memory overflow |
| CN104252458A (en) * | 2013-06-25 | 2014-12-31 | 博雅网络游戏开发（深圳）有限公司 | Data analysis method and device |
-
2016
- 2016-10-21 CN CN201610919579.0A patent/CN106484522A/en not_active Withdrawn
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103927305A (en) * | 2013-01-11 | 2014-07-16 | ***通信集团山东有限公司 | Method and device for controlling memory overflow |
| CN104252458A (en) * | 2013-06-25 | 2014-12-31 | 博雅网络游戏开发（深圳）有限公司 | Data analysis method and device |
| CN103390049A (en) * | 2013-07-23 | 2013-11-13 | 南京联创科技集团股份有限公司 | Method for processing high-speed message queue overflow based on memory database cache |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107608858A (en) * | 2017-09-28 | 2018-01-19 | 北京京东尚科信息技术有限公司 | Information processing system, method and apparatus |
| CN109409412A (en) * | 2018-09-28 | 2019-03-01 | 新华三大数据技术有限公司 | Image processing method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107766160B (en) | queue message processing method and terminal equipment | |
| CN104077420B (en) | Method and device for importing data into HBase database | |
| WO2014161431A2 (en) | System optimization method and device | |
| CN103841052A (en) | Bandwidth resource distribution system and method | |
| CN105306277A (en) | Message scheduling method and message scheduling device for message queues | |
| CN107547629A (en) | Method, apparatus, electronic equipment and the computer-readable recording medium that client static resource is downloaded | |
| CN102123084B (en) | Resource scheduling method and system in cloud computing operating system | |
| CN106484522A (en) | Queue processing method and device | |
| CN104182279A (en) | Task scheduling method, device and system | |
| CN102945185B (en) | Task scheduling method and device | |
| CN104283643A (en) | Message speed limiting method and device | |
| CN104866339A (en) | Distributed persistent management method, system and device of FOTA data | |
| CN106201665A (en) | A kind of dynamic queue's method for arranging and equipment | |
| CN107657434A (en) | The Notification Method, device and gateway device of payment result are supplemented in game with money | |
| CN106776072A (en) | Information push method and system | |
| CN108074085A (en) | A kind of bill withholds method and apparatus | |
| CN107239325A (en) | Document data processing method and device | |
| CN103870331B (en) | A kind of method and electronic equipment of dynamically distributes processor cores | |
| CN104778125B (en) | A kind of EMS memory management process and system | |
| CN105808324A (en) | Method for improving fluency of system and mobile terminal | |
| CN103916316A (en) | Linear speed capturing method of network data packages | |
| CN106503543A (en) | A kind of method and apparatus of management application program | |
| CN104375883B (en) | CFS scheduler | |
| CN109840597B (en) | Model prediction method and device, electronic equipment and storage medium | |
| CN102647293B (en) | The automatic filling mining method and device of telecom network management system north orientation FTP performance files |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20170308 |
|
| WW01 | Invention patent application withdrawn after publication |