CN104834558B - A kind of method and system of data processing - Google Patents
A kind of method and system of data processing Download PDFInfo
- Publication number
- CN104834558B CN104834558B CN201510254277.1A CN201510254277A CN104834558B CN 104834558 B CN104834558 B CN 104834558B CN 201510254277 A CN201510254277 A CN 201510254277A CN 104834558 B CN104834558 B CN 104834558B
- Authority
- CN
- China
- Prior art keywords
- data
- terminal device
- processing
- client terminal
- data processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present invention provides a kind of method and system of data processing, can efficiently, rapidly process data.This method includes：Service terminal device distributes the data after grouping to multigroup client terminal device；Client terminal device receives the data after the grouping, and the data are stored；Client terminal device reads the data of storage, then to the data fragmentation of reading, then makees single thread processing respectively to every sheet data.Technical solution using the present invention, solve the problems, such as single kafka clients delay machine when data receiver and processing and multithreading update update is caused to be lost simultaneously, and it can realize the pause of manual control data processing task and restart, simultaneously avoid data overstock or processing failure caused by loss of data, data processing reliability is improved, ensure that the accuracy and real-time of data.
Description
Technical field
The present invention relates to technical field of the computer network, a kind of particularly method and system of data processing.
Background technology
With the extensive use of computer network and the fast development of e-commerce industry, more and more people pay close attention to number
According to importance, particularly with digitization operation for, processing and analysis to data, while cutting operating costs, also
The accuracy in operation direction can be greatly improved, wherein real time data is even more focus of attention.But for magnanimity real time data
Processing, is always a difficult point.
In existing technology, Distributed Message Queue is generally selected for the processing of magnanimity real time data, as kafka comes
Carry out the transmitting-receiving and management of message.Kafka Distributed Message Queues carry out message management with keyword (topic), each
Topic is divided into as multiple groupings (partition), the least unit for being shifted, storing and being read using partition as data.
The message of each topic to being evenly dispersed to after data by kafka Distributed Message Queues received server-side
On multiple partition.Kafka message queues client is according to topic the and partition num specified into line number
According to reception, therefore every group of Kafka message queues client can only correspond to the data for receiving difference partition under a topic,
And each partition can only be received by a Kafka message queues client.Kafka message queues client is from data team
After row receive data, by data persistence storage into database, database passes through more according to corresponding service logic afterwards
The mode of thread parallel processing handles data, if handle successfully, updates the data flag bit, identifies and completed data processing；
If data processing fails, untreatment data is read from database, re-starts processing.
The present inventor has found that the prior art has the following disadvantages by the practical experience in the industry for many years：
1st, in use, if surprisingly delayed machine using kafka clients, it may result in the client and correspond to what is received
Data are overstock, and can not be handled, and influence the real-time of data；
2nd, in existing scheme, data processing task can only be dispatched by thread oneself, can not realize data processing pause and
Restart；
3rd, since existing scheme is that multithreading handles data simultaneously, a plurality of thread may handle a number simultaneously
According to, be just present with system resource waste and update lose situation, can not realize high concurrent, high performance processing message；
4th, in existing scheme, re-started, it is necessary to read untreated message from database when data processing fails
Processing, if overstock, excessive data are untreated, and the I/O performances of database will be greatly reduced, so as to cause treatment effeciency
It reduces.
The content of the invention
In view of this, the present invention provides a kind of method and system of data processing, solves kafka clients and surprisingly delays machine
When data receiver and processing, multithreading update simultaneously the problem of update is caused to be lost, and can realize at manual control data
The pause of reason task and restart and reduce and directly interacted with the data message of database, so as to improve reliability, function is steady
Fixed and process performance is high.
To achieve the above object, according to an aspect of the invention, there is provided a kind of method of data processing.
A kind of method of data processing of the present invention includes：Service terminal device fills the data after grouping to multigroup client
Put distribution；Client terminal device receives the data after the grouping, and the data are stored；Client terminal device reads storage
The data then to the data fragmentation of reading, then make single thread processing respectively to every sheet data.
Optionally, the service terminal device includes kafka Message Queuing servers end device and task coordinate device, and institute
It states client terminal device and is provided with kafka message queue client terminal devices.
Optionally, the step of which being stored includes：Data described in persistent storage and cache the data；
And the step of data of client terminal device reading storage, includes：Client terminal device reads the data of caching.
Optionally, client terminal device reads the data of storage, then to the data fragmentation of reading, then to every sheet data
The step of making single thread processing respectively includes：Data are read in file distributing unit timing in client terminal device, and according to data
Service fields to data carry out burst, be then distributed to data processing unit different in client terminal device；Each data processing
The data that the processing of unit single thread receives.
Optionally, it is described single thread processing is made respectively to every sheet data when, this method further includes：In needs pause or again
In the case of opening data processing, the file distributing unit will lock or unlock instruction be distributed to each data processing unit with
Suspend or restart data processing.
According to another aspect of the present invention, a kind of system of data processing is provided.
A kind of system of data processing of the present invention includes service terminal device and client terminal device, wherein：The server-side
Device is used for the data after being grouped and distributes to multigroup client terminal device；The client terminal device includes：Receiving module, for connecing
The data after the grouping are received, and the data are stored；Processing module is then right for reading the data of storage
The data fragmentation of reading, then single thread processing is made respectively to every sheet data.
Optionally, the service terminal device includes kafka Message Queuing servers end device and task coordinate device, and institute
It states and kafka message queue client terminal devices is provided in client terminal device.
Optionally, the client terminal device is additionally operable to：Data described in persistent storage and cache the data；And it reads
Take the data of caching.
Optionally, the processing module includes file distributing unit and multiple data processing units, wherein：The data point
Bill member carries out burst according to the service fields of data for periodically reading data to data, is then distributed to different institutes
State data processing unit；The data that the data processing unit receives for single thread processing.
Optionally, the processing module is additionally operable to：In the case where needing to suspend or restart data processing, the data point
Bill member will lock or unlock instruction is distributed to each data processing unit to suspend or restart data processing.
Technique according to the invention scheme, service terminal device distribute the data after grouping to multigroup client terminal device, this
Sample avoids the data that one group of client is surprisingly delayed caused by machine and overstocks, so as to ensure that the real-time of data and processing are imitated
Rate；Client terminal device receives the data after the grouping, and the data are carried out persistent storage and caching, reads data every time
When read from caching, a large amount of untreated numbers can be carried out with hard disc data storehouse to avoid in the case of data processing failure
According to information exchange, database performance is caused to decline, so as to discharge hard drive space, improve process performance；Client terminal device reading is deposited
The data of storage, then to the data fragmentation of reading, then make single thread processing to every sheet data, had so both improved place respectively
Speed is managed, and multithreading can be avoided the occurrence of and handle the problem of data cause update to be lost simultaneously, so as to save memory source,
Improve process performance；In addition, invention increases a memory database, wherein locking instruction and unlock instruction are stored with,
When needing to carry out the pause of data processing and restart, the instruction that the file distributing unit of processing module sends user is distributed to
Each data processing unit carries out corresponding operating, it is achieved thereby that manual control data handles the pause of task and restarts.
Description of the drawings
Attached drawing does not form inappropriate limitation of the present invention for more fully understanding the present invention.Wherein：
Fig. 1 is a kind of schematic diagram of the key step of the method for data processing according to embodiments of the present invention.
Fig. 2 is a kind of detailed step flow chart of the method for data processing according to embodiments of the present invention.
Fig. 3 is that a kind of kafka Distributed Message Queues work of the method for data processing according to embodiments of the present invention is former
Manage schematic diagram.
Fig. 4 is a kind of main modular schematic diagram of the system of data processing according to embodiments of the present invention.
Specific embodiment
It explains below in conjunction with attached drawing to the exemplary embodiment of the present invention, including the various of the embodiment of the present invention
Details should think them only exemplary to help understanding.Therefore, those of ordinary skill in the art should recognize
It arrives, various changes and modifications can be made to the embodiments described herein, without departing from scope and spirit of the present invention.Together
For clarity and conciseness, the description to known function and structure is omitted in sample in following description.
Fig. 1 is a kind of schematic diagram of the key step of the method for data processing according to embodiments of the present invention.Such as Fig. 1 institutes
Show, a kind of method of data processing of the embodiment of the present invention mainly includes steps S11 to step S13.
Step S11：Service terminal device distributes the data after grouping to multigroup client terminal device.With reference to the preferred of the present invention
Embodiment, service terminal device are provided with for the server-side device and task coordinate device, client terminal device of kafka message queues
The client terminal device of kafka message queues is stored with carrying out the reception of data.
It is well known that the efficiency that can greatly improve data processing is received and stored to the classification of data, then, in data
It is exactly a kind of good processing mode to receive and Distributed Message Queue is introduced when storing.At present, there is task processing most of
System in can all be related to message queue, by message queue come decoupling each system, reduce the dependence between each system,
Improve stability, such as activemq, kafka etc..
It is illustrated in the preferred embodiment of the present invention by taking kafka as an example.For example, when there is data request information,
Kafka server ends can classify the real time data received according to different keywords (topic), sorted data
It can be averaged in the multiple groupings (partition) being assigned under this topic, be carried out by minimum data unit of partition
The distribution and reading of data.In this way, the mass data received simultaneously can be different according to its own feature carried out it is detailed
Grouping.Afterwards, every group of data are distributed to different clients and carry out reception storage.The reception of data and storing process will be rear
Text is described in detail, and details are not described herein again.
Service terminal device also includes task coordinate device, when there is client terminal device addition or situation about leaving,
Task coordinate device meeting starting load equalization algorithm, the data balancing after kafka server-side devices are grouped are assigned to accordingly
Client terminal device.Herein, Zookeeper or other task coordinate devices can be used to realize in task coordinate device.
In a preferred embodiment of the invention, in order to solve the data product that one group of kafka client is surprisingly delayed caused by machine
Pressure problem, takes the structure of distributed more machines, introduces multigroup kafka client terminal devices, and kafka is changed by dynamic
The Group ID of client terminal device and " duplicate removal " mechanism is introduced to realize data receiver.Number is being received due to working as kafka clients
According to when, it is necessary to specify topic, and the data of each topic can only correspond to one group of kafka client of a Group ID
The Group ID of every group of kafka client are dynamically changed by introducing multigroup kafka clients in end, therefore, the present invention, so that
The data of each topic can simultaneously be received by multigroup kafka clients.Every group of kafka client is when receiving data
Data can be checked, judge whether it had been received by other kafka clients, carried out if having other clients
It receives and discard processing then is carried out to corresponding data, otherwise carry out reception storage.
After scheme as described above, even if surprisingly the machine of delaying can not receive data to one group of kafka client, nor affect on
The real-time reception of data, this scheme can solve single client data caused by machine of delaying and overstock and to solve server-side large quantities of
The problem of client receiving efficiency is low when measuring propelling data.
Step S12：Client terminal device receives the data after the grouping, and the data are stored.Wherein, this is counted
Include according to the step of storage：Data described in persistent storage and cache the data.
In a preferred embodiment of the invention, after client terminal device receives data, the data are distinguished into persistent storage
To database and it is cached to a very high database of reading performance.After increasing the caching, when carrying out data processing, directly from
Information is read in the caching, reading speed greatly improves, and after the completion of data processing, corresponding data are deleted during this is cached, and
Handling result is fed back into database to carry out data mode change.When data processing fails, not feedback information, from caching
Untreatment data is read to continue.In this way, need the data interaction amount with database that will greatly reduce in data processing, it can
To discharge big quantity space, so as to ensure that the I/O performances of database, treatment effeciency is improved, advantageously ensures that the real-time of data
Property.
Step S13：Client terminal device reads the data of storage, then to the data fragmentation of reading, then to every the piece number
According to making single thread processing respectively.
In a preferred embodiment of the invention, step S13 can be carried out in accordance with the following steps：
Step S131：Data are read in file distributing unit timing in client terminal device, and according to the service fields of data
Burst is carried out to data, is then distributed to data processing unit different in client terminal device；
Step S132：The data that each data processing unit single thread processing receives.
It can be seen from step S131 and step S132 after pending data storage is into caching, in client terminal device
File distributing unit read data from caching, and burst is carried out to data according to the service fields of data, is then distributed to
Different data processing units.Herein, the service fields of data can be configured as needed.
There is individual data queue inside each data processing unit, handled by individual thread, thus shape
Into the mechanism of single thread successively sequence processing data, so as to both ensure that process performance, in turn avoid multithreading and update simultaneously
Caused update is lost and memory source waste problem.
It, can be by the corresponding data of database if data processing success after each data processing unit handles data
Status Change is processed, while deletes the data in caching；If data processing fails, state is not changed, continues with number
According to the lower a data in queue.File distributing unit can read untreated data in the slave caching of timing, and according to data
Service fields to data carry out burst, be then distributed and handle again, so as to ensure that the accuracy of data processing, avoid
Data overstock or data processing failure caused by loss of data.
In addition in embodiment of the present invention, can also increase a memory database for store locking and unlock instruction, with
It realizes the pause of manual control data processing and restarts.In the case where needing to suspend or restart data processing, the data point
Bill member will lock or unlock instruction is distributed to each data processing unit to suspend or restart data processing.For example, user
It needs to suspend data processing in daily fixed time to carry out other operations, can be set at this time in the memory database
One timer module is controlled.
When needing to suspend data processing, the timer in memory database can send one and lock instruction to data distribution
Unit.Instruction can be distributed to all data processing units by file distributing unit after receiving locking instruction.Data processing unit is received
Then etc. to after instruction, its memory queue is emptied first, treats Chu Liwan current datas, its data queue becomes blocking shape afterwards
State continues to execute after waiting unlock.
When needing to restart data processing, the timer in memory database can send a unlock instruction to data distribution
Unit.File distributing unit receives and instruction is distributed to all data processing units after unlock instruction.Data processing unit receives
After instruction, the data queue for waking up obstruction continues data processing, so as to achieve the effect that restart.
Fig. 2 is a kind of detailed step flow chart of the method for data processing according to embodiments of the present invention.
Flow chart as shown in Figure 2, after service terminal device receives real time data, according to disappearing for kafka message queues
Breath management rule is grouped data, and multigroup client terminal device (step is distributed under the coordination of task conditioning unit
S21).Client terminal device is cached and also may be used according to receiving rule reception data and being saved in database and caching LevelDB herein
Select the database (step S22) that other readwrite performances are high.After storing successfully, number is read in file distributing unit timing from caching
According to, and it is distributed to different data processing units (step S23) after carrying out burst to data according to the service fields of setting.Each number
According to there is individual data queue inside processing unit, handled (step S24) by individual thread.When data processing success
When, it is processed by corresponding data Status Change in database, while deletes the data in caching LevelDB；If processing is lost
It loses, does not change the data mode in database.File distributing unit periodically can read untreated data from caching LevelDB
Again handle, ensure that the accuracy of data processing.
When needing to carry out the pause of data processing and restart, the locking and unlock that are stored in memory database Redis refer to
Order can be distributed to each data processing unit by file distributing unit, to be handled accordingly.Here memory database
Other high performance cachings or database can be selected.
Fig. 3 is that a kind of kafka Distributed Message Queues work of the method for data processing according to embodiments of the present invention is former
Manage schematic diagram.
When there is data request information, kafka server ends can be by the real time data received according to different keywords
(topic) classify, sorted data, which can be averaged, to be assigned in such multiple groupings (partition).
Kafka clients carry out data receiver according to the topic specified, and every group of kafka client has the Group of oneself
ID, the different kafka clients in every group correspond to the data for receiving corresponding partition.
As shown in figure 3, the data of one group of same keyword (topic) are divided into multiple groupings (partition)：Grouping 1,
Grouping 2, grouping 3 ....Corresponding client terminal device has 3 groups, kafka customer end As, kafka customer end Bs and kafka clients
C.Under the scheduling of task conditioning unit Zookeeper, 3 groups of kafka clients receive multiple groupings of the topic simultaneously respectively
The data of partition are simultaneously checked with duplicate removal data.When the kafka customer end As machine of delaying leaves, Zookeeper's is negative
It carries equalization algorithm to start, the data of remaining grouping partition is re-started and distribute to kafka customer end Bs and kafka visitors
Family end C is received.
This scheme can solve single group client delay data caused by machine overstock and solve server-side high-volume push number
According to when client receiving efficiency it is low the problem of.
Fig. 4 is a kind of main modular schematic diagram of the system of data processing according to embodiments of the present invention.As shown in figure 4,
Data handling system 4 in the embodiment of the present invention mainly includes service terminal device 41 and client terminal device 42, wherein, client dress
42 are put again including receiving module 421 and processing module 422.
Service terminal device 41 is used for the data after being grouped and distributes to multigroup client terminal device；Client terminal device 42 includes：
Receiving module 421 for receiving the data after the grouping, and the data is stored；Processing module 422, for reading
The data of storage, then to the data fragmentation of reading, then make single thread processing respectively to every sheet data.
It is introduced below in conjunction with the preferred embodiment of the present invention.The service terminal device 41 includes kafka message queues
Server-side device and task coordinate device, and the client terminal device be provided with kafka message queues client terminal device with into
Row data receiver and storage.
Data persistence is stored to database and is cached to after client terminal device receiving module 421 receives data
In LevelDB.Afterwards, the file distributing unit of client terminal device processing module 422 is read with LevelDB into row data communication
Information in LevelDB simultaneously carries out burst according to the service fields of data to data, is then distributed to different data processing lists
It is handled in member.There is individual queue in each data processing unit, handled by individual thread.Data processing
After success, the corresponding data mode in hard disc data storehouse can be changed to processed, while delete the data in LevelDB, if
Processing failure does not change data mode.
Client terminal device processing module 422 can be also used for when needing to suspend or restart data processing, data distribution list
Member will lock or unlock instruction is distributed to each data processing unit to carry out corresponding operating.Wherein, lock and unlock instruction preserves
In increased memory database Redis, reading, access speed are all quickly.
When needing to suspend data processing, the locking instruction in memory database Redis is distributed to by file distributing unit
All data processing units.Then etc. after data processing unit receives instruction, its memory queue is emptied first, treats that Chu Liwan is current
Data, afterwards its data queue become blocked state, continued to execute after waiting unlock.
When needing to restart data processing, the unlock instruction in memory database Redis is distributed to by file distributing unit
All data processing units.After data processing unit receives instruction, the data queue for waking up obstruction continues data processing, so as to reach
To the effect restarted.
Technical solution according to embodiments of the present invention, service terminal device divide the data after grouping to multigroup client terminal device
Hair, so avoids the data that one group of client is surprisingly delayed caused by machine and overstocks, so as to ensure that the real-time of data and place
Manage efficiency；Client terminal device receives the data after the grouping, and the data are carried out persistent storage and caching, reads every time
It reads, can be carried out to avoid in the case of data processing failure with hard disc data storehouse a large amount of untreated from caching during data
Data information exchange, database performance is caused to decline, so as to discharge hard drive space, improves process performance；Client terminal device is read
The data of storage are taken, then to the data fragmentation of reading, then single thread processing is made respectively to every sheet data, so both improves
Processing speed, and multithreading can be avoided the occurrence of and handle the problem of data cause update to be lost simultaneously, so as to saving memory
Resource improves process performance；In addition, invention increases a memory database, wherein being stored with locking instruction and unlock
Instruction, when needing to carry out the pause of data processing and when restarting, instruction that the file distributing unit of processing module sends user
It is distributed to each data processing unit and carries out corresponding operating, it is achieved thereby that manual control data handles the pause of task and restarts.
Above-mentioned specific embodiment, does not form limiting the scope of the invention.Those skilled in the art should be bright
It is white, depending on design requirement and other factors, various modifications, combination, sub-portfolio and replacement can occur.It is any
Modifications, equivalent substitutions and improvements made within the spirit and principles in the present invention etc., should be included in the scope of the present invention
Within.
Claims (10)
- A kind of 1. method of data processing, which is characterized in that including：Service terminal device distributes the data after grouping to multigroup client terminal device；Client terminal device receives the data after the grouping, and the data are stored；Client terminal device reads the data of storage, then to the data fragmentation of reading, then makees single line respectively to every sheet data Journey processing.
- 2. according to the method described in claim 1, it is characterized in that, the service terminal device includes kafka Message Queuing Services Device end device and task coordinate device, and the client terminal device is provided with kafka message queue client terminal devices.
- 3. according to the method described in claim 1, it is characterized in that,The step of data are stored includes：Data described in persistent storage and cache the data；And the step of data of client terminal device reading storage, includes：Client terminal device reads the data of caching.
- 4. according to the method described in claim 1, it is characterized in that, the data of client terminal device reading storage, then right The data fragmentation of reading, then the step of making single thread processing respectively to every sheet data, include：Data are read in file distributing unit timing in client terminal device, and data are divided according to the service fields of data Then piece is distributed to data processing unit different in client terminal device；The data that each data processing unit single thread processing receives.
- 5. according to the method described in claim 4, it is characterized in that, it is described single thread processing is made respectively to every sheet data when, This method further includes：In the case where needing to suspend data processing, the file distributing unit is distributed to each data processing by instruction is locked Unit is to suspend data processing；In the case where needing to restart data processing, unlock instruction is distributed to each data processing by the file distributing unit Unit is to restart data processing.
- 6. a kind of system of data processing, which is characterized in that including service terminal device and client terminal device, wherein：The service terminal device is used for the data after being grouped and distributes to multigroup client terminal device；The client terminal device includes：Receiving module for receiving the data after the grouping, and the data is stored；Processing module for reading the data of storage, then to the data fragmentation of reading, then makees list respectively to every sheet data Thread process.
- 7. system according to claim 6, which is characterized in that the service terminal device includes kafka Message Queuing Services Device end device and task coordinate device, and kafka message queue client terminal devices are provided in the client terminal device.
- 8. system according to claim 6, which is characterized in that the client terminal device is additionally operable to：Described in persistent storage Data and cache the data；And read the data of caching.
- 9. system according to claim 6, which is characterized in that the processing module includes file distributing unit and multiple numbers According to processing unit, wherein：The file distributing unit carries out burst according to the service fields of data for periodically reading data to data, then It is distributed to the different data processing units；The data that the data processing unit receives for single thread processing.
- 10. system according to claim 9, which is characterized in that the processing module is additionally operable to：In the case where needing to suspend data processing, the file distributing unit is distributed to each data processing by instruction is locked Unit is to suspend data processing；In the case where needing to restart data processing, unlock instruction is distributed to each data processing by the file distributing unit Unit is to restart data processing.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510254277.1A CN104834558B (en) | 2015-05-19 | 2015-05-19 | A kind of method and system of data processing |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510254277.1A CN104834558B (en) | 2015-05-19 | 2015-05-19 | A kind of method and system of data processing |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104834558A CN104834558A (en) | 2015-08-12 |
| CN104834558B true CN104834558B (en) | 2018-06-01 |
Family
ID=53812466
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510254277.1A Active CN104834558B (en) | 2015-05-19 | 2015-05-19 | A kind of method and system of data processing |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104834558B (en) |
Families Citing this family (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105183858B (en) * | 2015-09-10 | 2018-12-21 | 国家计算机网络与信息安全管理中心 | A kind of distributed data real-time repetition removal method based on message queue |
| CN106302385B (en) * | 2016-07-26 | 2019-11-15 | 努比亚技术有限公司 | A kind of message distribution device and method |
| CN106817295B (en) * | 2016-12-08 | 2020-06-05 | 上海有孚网络股份有限公司 | Message processing device and method |
| CN106775989B (en) * | 2016-12-31 | 2019-07-02 | 北京神州绿盟信息安全科技股份有限公司 | A kind of job control method and device |
| CN106844682B (en) * | 2017-01-25 | 2019-08-16 | 北京百分点信息科技有限公司 | Method for interchanging data, apparatus and system |
| CN107273542B (en) * | 2017-07-06 | 2020-11-27 | 华泰证券股份有限公司 | High-concurrency data synchronization method and system |
| CN107423145A (en) * | 2017-07-11 | 2017-12-01 | 北京潘达互娱科技有限公司 | A kind of method and apparatus for avoiding information drop-out |
| CN110019310A (en) * | 2017-12-29 | 2019-07-16 | 北京京东尚科信息技术有限公司 | Data processing method and system, computer system, computer readable storage medium |
| CN110647477B (en) * | 2018-06-27 | 2022-02-11 | 阿里巴巴（中国）有限公司 | Data caching method, device, terminal and computer readable storage medium |
| CN108984285B (en) * | 2018-06-28 | 2021-10-15 | 上海数据交易中心有限公司 | Data collision flow analysis method and device, storage medium and terminal |
| CN111104412B (en) * | 2018-10-25 | 2023-05-30 | 阿里巴巴集团控股有限公司 | Concurrent control method, device and system based on single thread |
| CN110706019A (en) * | 2019-09-03 | 2020-01-17 | 苏宁云计算有限公司 | Effective price tag pushing method and device, computer equipment and storage medium |
| CN110765143B (en) * | 2019-10-10 | 2022-08-02 | 腾讯科技（深圳）有限公司 | Data processing method, device, server and storage medium |
| CN111010403A (en) * | 2019-12-26 | 2020-04-14 | 紫光云(南京)数字技术有限公司 | Method and device for automatically generating SASL authentication file and computer storage medium |
| CN111158610B (en) * | 2019-12-31 | 2022-02-22 | 苏州浪潮智能科技有限公司 | Method, device and equipment for synchronously setting cache acceleration and readable medium |
| CN112560660A (en) * | 2020-12-10 | 2021-03-26 | 杭州宇泛智能科技有限公司 | Face recognition system and preset method thereof |
| CN113760782A (en) * | 2021-08-23 | 2021-12-07 | 南京森根科技股份有限公司 | Dynamically adjustable annular cache system and control method thereof |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6134594A (en) * | 1997-10-28 | 2000-10-17 | Microsoft Corporation | Multi-user, multiple tier distributed application architecture with single-user access control of middle tier objects |
| CN102833336A (en) * | 2012-08-31 | 2012-12-19 | 河海大学 | Data sub-packet processing method in separate distributed information acquisition and concurrent processing system |
| CN103502943A (en) * | 2011-12-31 | 2014-01-08 | 华为技术有限公司 | Distributed task processing method, device and system based on message queue |
| CN103838552A (en) * | 2014-03-18 | 2014-06-04 | 北京邮电大学 | System and method for processing multi-core parallel assembly line signals of 4G broadband communication system |
| CN104506950A (en) * | 2014-12-29 | 2015-04-08 | 珠海全志科技股份有限公司 | Multithread download method and download device in network streaming media play, and download equipment |
-
2015
- 2015-05-19 CN CN201510254277.1A patent/CN104834558B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6134594A (en) * | 1997-10-28 | 2000-10-17 | Microsoft Corporation | Multi-user, multiple tier distributed application architecture with single-user access control of middle tier objects |
| CN103502943A (en) * | 2011-12-31 | 2014-01-08 | 华为技术有限公司 | Distributed task processing method, device and system based on message queue |
| CN102833336A (en) * | 2012-08-31 | 2012-12-19 | 河海大学 | Data sub-packet processing method in separate distributed information acquisition and concurrent processing system |
| CN103838552A (en) * | 2014-03-18 | 2014-06-04 | 北京邮电大学 | System and method for processing multi-core parallel assembly line signals of 4G broadband communication system |
| CN104506950A (en) * | 2014-12-29 | 2015-04-08 | 珠海全志科技股份有限公司 | Multithread download method and download device in network streaming media play, and download equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104834558A (en) | 2015-08-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104834558B (en) | A kind of method and system of data processing | |
| US11983198B2 (en) | Multi-cluster warehouse | |
| US7680848B2 (en) | Reliable and scalable multi-tenant asynchronous processing | |
| EP3770774B1 (en) | Control method for household appliance, and household appliance | |
| JP3944154B2 (en) | Method and system for dynamically adjusting a thread pool in a multi-threaded server | |
| Page et al. | Framework for task scheduling in heterogeneous distributed computing using genetic algorithms | |
| Fang et al. | Parallel stream processing against workload skewness and variance | |
| CN103309738B (en) | User job dispatching method and device | |
| US9979647B2 (en) | Periodic advertisements of host capabilities in virtual cloud computing infrastructure | |
| CN100375093C (en) | Processing of multiroute processing element data | |
| CN104754036A (en) | Message processing system and processing method based on kafka | |
| JP2004213625A (en) | Response-time basis workload distribution technique based on program | |
| CN105160570A (en) | Concurrent on-line transaction processing method capable of horizontal expansion | |
| CN103491024B (en) | Job scheduling method and device for streaming data | |
| CN109804354A (en) | Message cache management for message queue | |
| US20190129771A1 (en) | Balanced message distribution in distributed message handling systems | |
| JP3788697B2 (en) | Message control unit | |
| CN110058940B (en) | Data processing method and device in multi-thread environment | |
| US10114866B2 (en) | Memory-constrained aggregation using intra-operator pipelining | |
| CN110768829B (en) | Method for realizing linear increase of traffic analysis service performance based on DPDK | |
| CN103793273A (en) | Distributed type queue scheduling method and device based on Redis | |
| CN106790552B (en) | A kind of content providing system based on content distributing network | |
| CN109617710B (en) | Large data transmission bandwidth scheduling method with deadline constraint between data centers | |
| CN109218385B (en) | Method and device for processing data | |
| JP2004192047A (en) | Message processing device, message processing method, and message processing program |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| EXSB | Decision made by sipo to initiate substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |