CN106227782A - A kind of method being inserted into data base based on multi-data source - Google Patents
A kind of method being inserted into data base based on multi-data source Download PDFInfo
- Publication number
- CN106227782A CN106227782A CN201610566067.0A CN201610566067A CN106227782A CN 106227782 A CN106227782 A CN 106227782A CN 201610566067 A CN201610566067 A CN 201610566067A CN 106227782 A CN106227782 A CN 106227782A
- Authority
- CN
- China
- Prior art keywords
- data
- cdr
- thread
- queue
- correspondence
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/548—Queue
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Computing Systems (AREA)
- Data Mining & Analysis (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a kind of method being inserted into data base based on multi-data source.The method by being divided into multiple data source to business demand, and the message queue of each data source correspondence establishment one not fixed size, and each queue creates the thread of a timing acquisition queuing data, and last each thread creates corresponding thread pool again；When CDR produce data time call insert CDR data interface carry out data be stored in correspondence data source message queue；Carrying out being grouped in the thread pool being stored in correspondence by the thread-data of timing acquisition queue, thread pool calls the task of oneself in the case of idle thread at once, the CDR data being divided into group is carried out disposable batch be submitted to the data base of correspondence in task.CDR data batch real-time can be reached by the method to be inserted into data base, reduce data base's pressure, promote database performance.
Description
Technical field
The present invention relates to Internet technical field, particularly relate to a kind of method being inserted into data base based on multi-data source.
Background technology
In voice telephone traffic system, CDR (calling detail records) is an important data parameters, each time
The operating systems such as calling, call, on-hook all can have more CDR data, and the CDR data of these all generations will be real-time
It is inserted into data base, in order on the system page, be capable of inquiry display.
The technical scheme great majority realized in existing system are all to be inserted into data base one by one, produce in system every time
Article one, cdr logging, calls the method being inserted into data base the most at once, then etc. to be returned, finally shows process on the page
Result.
Use the scheme being inserted into data base one by one, be to meet requirement in the case of certain telephone traffic is little
, but when telephone traffic is big when, bigger problem can be produced.When CDR data produce much when, a data
The when of insertion, response speed can be the most slack-off, and owing to the response speed put in storage each time is slack-off, user has not on the page
Good Consumer's Experience, and a data inserts and also data base can be produced the biggest performance impact, and then other are to data
The operation in storehouse also can be slack-off.
Except such scheme, the processing scheme also having is to use asynchronous mode, when system produces CDR data, calls
The method inserting data base, is now stored in this operation in queue, more periodically from queue, taking-up data are inserted into
Data base.
The mode using queue can solve user's problem of low-response on interface, but periodically takes out from queue
Data carry out data base the most one by one and insert or can there is insertion speed slowly, affect database performance, can not be effectively
Solve basic problem.
Summary of the invention
The invention aims to overcome the defect of prior art, it is provided that one is inserted into data base based on multi-data source
Method, the method can be inserted into mass data in real time data base rapidly, alleviate a data base and frequently insert institute
The pressure brought, promotes database performance.
For achieving the above object, the invention provides a kind of method being inserted into data base based on multi-data source, the method
Comprise the following steps:
Step 1: by business demand being divided into multiple data source, and each data source correspondence establishment one is solid
The message queue of sizing, each queue creates the thread of a timing acquisition queuing data, and last each thread creates phase again
The thread pool answered.
Step 2: system provides an interface inserting CDR data, calls when CDR produces data and inserts CDR data
Interface carries out data and is stored in the data source message queue of correspondence.
Step 3: carry out being grouped in the thread pool being stored in correspondence by the thread-data of timing acquisition queue.
Step 4: thread pool calls the task of oneself in the case of idle thread at once, the CDR being divided into group in task
Data carry out disposable batch and are submitted to the data base of correspondence.
Further, the interface of CDR data is that the business in the CDR data come into according to called side indicates, to data
Divide, data are stored in the message queue in corresponding data source.
Further, in the thread of each timing acquisition queue, just at most obtain in corresponding queue every 100 milliseconds
Article 700, then different types of CDR data are grouped by CDR data, then are stored in the thread pool of oneself and carry out task and hold
OK.Wherein time interval 100 milliseconds and the most most these 2 parameters of bar number 700 obtaining CDR data will be according to CDR data
The situation of calling of interface is adjusted, to reach the effect of optimum.
Further, additionally it is possible to the quantity produced according to CDR carries out the distributed deployment serviced more, in order to provide CDR number
Quantity and speed according to warehouse-in.
The beneficial effect that technical solution of the present invention is brought:
First, use the present invention rapidly the CDR batch data simultaneously produced in a large number can be inserted into data in real time
Storehouse.
Second, use the mode of multi-data source can alleviate a data base and frequently insert brought pressure, make data
Storehouse also can quickly respond other operation while inserting CDR data, promotes database performance.
Accompanying drawing explanation
In order to be illustrated more clearly that the embodiment of the present invention or technical scheme of the prior art, below will be to embodiment or existing
In having technology to describe, the required accompanying drawing used is briefly described, it should be apparent that, the accompanying drawing in describing below is only this
Some embodiments of invention, for those of ordinary skill in the art, on the premise of not paying creative work, it is also possible to
Other accompanying drawing is obtained according to these accompanying drawings.
Fig. 1 is the method flow diagram of the present invention.
Detailed description of the invention
Below in conjunction with the accompanying drawing in the embodiment of the present invention, the technical scheme in the embodiment of the present invention is carried out clear, complete
Describe, it is clear that described embodiment is only a part of embodiment of the present invention rather than whole embodiments wholely.Based on
Embodiment in the present invention, it is every other that those of ordinary skill in the art are obtained under not making creative work premise
Embodiment, broadly falls into the scope of protection of the invention.
In the use of call platform, can produce substantial amounts of CDR data all the time, these data will insert data
Storehouse, inserts data base item by item and can produce the appearance of the problem such as time delay, database blockage.In order to solve this problem, the present invention
Propose a kind of method being rapidly inserted into data base based on multi-data source, CDR data can be reached real-time by the method
Batch is inserted into data base, reduces data base's pressure, promotes database performance.
It is illustrated in figure 1 the method flow diagram of the present invention, it is an object of the invention to the feelings produced at the concurrent and a large amount of CDR of height
It is quickly inserted into data base under condition, and reduces the pressure of data base to a certain extent, be implemented as follows:
Step 1: system divides multiple data sources, and each data source pair according to business demand initialized time
Should create the message queue of a not fixed size, each queue creates the thread of a timing acquisition queuing data, the most often
Individual thread creates corresponding thread pool again.
Step 2: system provides an interface inserting CDR data, when CDR produces data, calls this interface.This interface
The function that realizes be that the business in the CDR data come into according to called side indicates, data are divided, data are stored in
In the message queue in corresponding data source.
Step 3: in the thread of each timing acquisition queue, just at most obtains 700 every 100 milliseconds in corresponding queue
Then different types of CDR data are grouped by bar CDR data, then are stored in the thread pool of oneself and carry out tasks carrying.
Wherein time interval 100 milliseconds and the most most these 2 parameters of bar number 700 obtaining CDR data will be according to CDR data-interfaces
Situation of calling be adjusted, to reach optimum effect.
Step 4: at once call the task of oneself in the thread pool that each thread is corresponding in the case of idle thread, appointing
The CDR data being divided into group in business carry out disposably batch and are submitted in the data base of oneself correspondence.
Step 5: can also carry out, according to the quantity that CDR produces, the distributed deployment that services, in order to provide CDR data to enter more
The quantity in storehouse and speed.
Above the embodiment of the present invention is described in detail, specific case used herein to the principle of the present invention and
Embodiment is set forth, and the explanation of above example is only intended to help to understand method and the core concept thereof of the present invention；
Simultaneously for one of ordinary skill in the art, according to the thought of the present invention, the most all can
Change part, and in sum, this specification content should not be construed as limitation of the present invention.
Claims (4)
1. the method being inserted into data base based on multi-data source, it is characterised in that the method comprises the following steps:
Step 1: by business demand being divided into multiple data source, and each data source correspondence establishment one is fixing big
Little message queue, each queue creates the thread of a timing acquisition queuing data, and last each thread creates accordingly again
Thread pool；
Step 2: system provides an interface inserting CDR data, calls the interface inserting CDR data when CDR produces data
Carry out data and be stored in the data source message queue of correspondence；
Step 3: carry out being grouped in the thread pool being stored in correspondence by the thread-data of timing acquisition queue；
Step 4: thread pool calls the task of oneself in the case of idle thread at once, the CDR data being divided into group in task
Carry out disposable batch and be submitted to the data base of correspondence.
Method the most according to claim 1, it is characterised in that the interface of CDR data is the CDR come into according to called side
Business in data indicates, and divides data, data are stored in the message queue in corresponding data source.
Method the most according to claim 1, it is characterised in that in the thread of each timing acquisition queue, every 100 milliseconds
In corresponding queue, just at most obtain 700 CDR data, then different types of CDR data are grouped, then are stored in
The thread pool of oneself carries out tasks carrying, wherein time interval 100 milliseconds and the most most bar number 700 obtaining CDR data
These 2 parameters of bar to be adjusted according to the situation of calling of CDR data-interface, to reach the effect of optimum.
Method the most according to claim 1, it is characterised in that can also carry out, according to the quantity that CDR produces, the distribution that services more
Formula is disposed, in order to provide quantity and the speed of CDR data loading.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610566067.0A CN106227782A (en) | 2016-07-15 | 2016-07-15 | A kind of method being inserted into data base based on multi-data source |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610566067.0A CN106227782A (en) | 2016-07-15 | 2016-07-15 | A kind of method being inserted into data base based on multi-data source |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN106227782A true CN106227782A (en) | 2016-12-14 |
Family
ID=57530861
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610566067.0A Pending CN106227782A (en) | 2016-07-15 | 2016-07-15 | A kind of method being inserted into data base based on multi-data source |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106227782A (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107562538A (en) * | 2017-08-22 | 2018-01-09 | 中国铁道科学研究院电子计算技术研究所 | Data pick-up multitask management process and system in railway traffic statistics |
| CN108829578A (en) * | 2018-04-27 | 2018-11-16 | 北京中创信测科技股份有限公司 | A kind of CDR association backfill accuracy automated testing method and system |
| CN109614506A (en) * | 2018-11-05 | 2019-04-12 | 厦门市美亚柏科信息股份有限公司 | A kind of method, apparatus and storage medium importing picture library based on rocksdb re-scheduling data |
| CN110275918A (en) * | 2019-06-17 | 2019-09-24 | 浙江百应科技有限公司 | A kind of million rank excel data quick and stable import systems |
| CN110727615A (en) * | 2019-10-11 | 2020-01-24 | 福建天晴数码有限公司 | Method and system for realizing acceleration of interface data request |
| CN110888865A (en) * | 2019-12-04 | 2020-03-17 | 北京明略软件***有限公司 | Data processing method and device based on one-way linked list |
| CN113204443A (en) * | 2021-06-03 | 2021-08-03 | 京东科技控股股份有限公司 | Data processing method, equipment, medium and product based on federal learning framework |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102096709A (en) * | 2011-01-19 | 2011-06-15 | 北京沃泰丰通信技术有限公司 | Call detail record statistical method and system |
| CN102622426A (en) * | 2012-02-27 | 2012-08-01 | 杭州闪亮科技有限公司 | Database writing system and database writing method |
| US20130132372A1 (en) * | 2011-11-17 | 2013-05-23 | Benefitfocus.Com | Systems and methods for dynamic service integration |
| CN103810223A (en) * | 2012-11-15 | 2014-05-21 | 中国科学院软件研究所 | Data grouping-based memory data organization query method |
| CN104158757A (en) * | 2014-08-21 | 2014-11-19 | 福建星海通信科技有限公司 | GPS (Global Positioning System) data loading and processing method and system |
| CN105162863A (en) * | 2015-09-01 | 2015-12-16 | 北京皮尔布莱尼软件有限公司 | Picture uploading device and method and computing equipment |
-
2016
- 2016-07-15 CN CN201610566067.0A patent/CN106227782A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102096709A (en) * | 2011-01-19 | 2011-06-15 | 北京沃泰丰通信技术有限公司 | Call detail record statistical method and system |
| US20130132372A1 (en) * | 2011-11-17 | 2013-05-23 | Benefitfocus.Com | Systems and methods for dynamic service integration |
| CN102622426A (en) * | 2012-02-27 | 2012-08-01 | 杭州闪亮科技有限公司 | Database writing system and database writing method |
| CN103810223A (en) * | 2012-11-15 | 2014-05-21 | 中国科学院软件研究所 | Data grouping-based memory data organization query method |
| CN104158757A (en) * | 2014-08-21 | 2014-11-19 | 福建星海通信科技有限公司 | GPS (Global Positioning System) data loading and processing method and system |
| CN105162863A (en) * | 2015-09-01 | 2015-12-16 | 北京皮尔布莱尼软件有限公司 | Picture uploading device and method and computing equipment |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107562538A (en) * | 2017-08-22 | 2018-01-09 | 中国铁道科学研究院电子计算技术研究所 | Data pick-up multitask management process and system in railway traffic statistics |
| CN107562538B (en) * | 2017-08-22 | 2020-01-07 | 中国铁道科学研究院电子计算技术研究所 | Data extraction multitask management method and system in railway transportation statistics |
| CN108829578A (en) * | 2018-04-27 | 2018-11-16 | 北京中创信测科技股份有限公司 | A kind of CDR association backfill accuracy automated testing method and system |
| CN109614506A (en) * | 2018-11-05 | 2019-04-12 | 厦门市美亚柏科信息股份有限公司 | A kind of method, apparatus and storage medium importing picture library based on rocksdb re-scheduling data |
| CN110275918A (en) * | 2019-06-17 | 2019-09-24 | 浙江百应科技有限公司 | A kind of million rank excel data quick and stable import systems |
| CN110727615A (en) * | 2019-10-11 | 2020-01-24 | 福建天晴数码有限公司 | Method and system for realizing acceleration of interface data request |
| CN110727615B (en) * | 2019-10-11 | 2020-12-25 | 福建天晴数码有限公司 | Method and system for realizing acceleration of interface data request |
| CN110888865A (en) * | 2019-12-04 | 2020-03-17 | 北京明略软件***有限公司 | Data processing method and device based on one-way linked list |
| CN113204443A (en) * | 2021-06-03 | 2021-08-03 | 京东科技控股股份有限公司 | Data processing method, equipment, medium and product based on federal learning framework |
| CN113204443B (en) * | 2021-06-03 | 2024-04-16 | 京东科技控股股份有限公司 | Data processing method, device, medium and product based on federal learning framework |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106227782A (en) | A kind of method being inserted into data base based on multi-data source | |
| CN101834786B (en) | Queue scheduling method and device | |
| CN109901926A (en) | Method, server and storage medium based on big data behavior scheduling application task | |
| CN102325148B (en) | WebService service calling method | |
| CN107426448A (en) | A kind of automatic calling allocation method being lined up based on dynamic priority | |
| US9374475B1 (en) | System for processing customer records | |
| CN104639775A (en) | Method and device for queuing routes in call center | |
| CN102223454A (en) | Method and apparatus for seat arrangement of call center | |
| CN102929929A (en) | Method and device for data summarization | |
| CN102855173A (en) | Method and device for testing software performance | |
| CN105975277A (en) | Template-based mixed cloud elastic telescoping set building method | |
| CN105610992A (en) | Task allocation load balancing method for distributed stream computing system | |
| CN106549786B (en) | Configuration method and device for service provisioning resources | |
| CN102546423B (en) | Method and device for queue scheduling and network device | |
| CN102402416A (en) | Method and device for setting dequeueing sequence of element | |
| CN109710623A (en) | A kind of intelligence order synchronous method | |
| CN101694661B (en) | Method and device for generating statistical form and statistical server | |
| CN109474902A (en) | Message message accounting, method, terminal and system | |
| CN105740397A (en) | Big data parallel operation-based voice mail business data analysis method | |
| CN102083045B (en) | Method and device for processing service charging | |
| CN105471707B (en) | Buddy list sort method and system | |
| CN109375998B (en) | Processing method in high-concurrency request environment | |
| CN106886452A (en) | A kind of method of simplified cloud system task scheduling | |
| CN106790368A (en) | Resource regulating method and device in a kind of distributed system | |
| CN106487889A (en) | A kind of task of facing cloud data center and data aggregate dispositions method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20161214 |