CN112486655A - High-concurrency data processing system and method - Google Patents
High-concurrency data processing system and method Download PDFInfo
- Publication number
- CN112486655A CN112486655A CN202011423413.2A CN202011423413A CN112486655A CN 112486655 A CN112486655 A CN 112486655A CN 202011423413 A CN202011423413 A CN 202011423413A CN 112486655 A CN112486655 A CN 112486655A
- Authority
- CN
- China
- Prior art keywords
- server
- message queue
- load balancing
- data
- processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012545 processing Methods 0.000 title claims abstract description 72
- 238000000034 method Methods 0.000 title claims abstract description 21
- 238000005516 engineering process Methods 0.000 claims abstract description 9
- 238000013507 mapping Methods 0.000 claims description 14
- 230000008569 process Effects 0.000 claims description 10
- 238000003672 processing method Methods 0.000 claims description 4
- 238000010586 diagram Methods 0.000 description 2
- 230000002776 aggregation Effects 0.000 description 1
- 238000004220 aggregation Methods 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5083—Techniques for rebalancing the load in a distributed system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/50—Queue scheduling
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
- H04L67/1004—Server selection for load balancing
- H04L67/1023—Server selection for load balancing based on a hash applied to IP addresses or costs
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/60—Scheduling or organising the servicing of application requests, e.g. requests for application data transmissions using the analysis and optimisation of the required network resources
- H04L67/63—Routing a service request depending on the request content or context
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a high-concurrency data processing system and a method, wherein the system comprises a database, a load balancing server, a plurality of message queue servers and a plurality of stream processing servers, wherein the stream processing servers are arranged in one-to-one correspondence with the message queue servers; each message queue server is provided with a subject channel composed of message queues and used for temporarily storing one type of data requests, and the subject channel is used for temporarily storing the data requests distributed by the load balancing server; each stream processing server is used for processing the data request in the theme channel of the message queue server correspondingly arranged by adopting the stream processing technology and storing the processing result into the database. By adopting the technical scheme of the invention, a large amount of high-concurrency data processing requirements can be met.
Description
Technical Field
The invention relates to the field of servers, in particular to a high-concurrency data processing system and a high-concurrency data processing method.
Background
With the rapid development of the industrial internet, the number of industrial devices is increasing, device data needs to be reported and stored, the frequency of data reporting is increasing, great pressure is applied to a server, and meanwhile, real-time service processing is also accompanied in the data reporting process. In order to improve the processing capability of the server and process and store data in real time in the face of high concurrent reporting and processing of multiple devices, a problem to be solved is first of all, and therefore an architecture with high concurrent processing capability is needed.
Disclosure of Invention
The invention aims to solve the technical problem that the server pressure is high due to the fact that the output processing amount is increased rapidly in the prior art, and provides a high-concurrency data processing system and method.
In an embodiment of the present invention, a highly concurrent data processing system is provided, which includes a database, a load balancing server, a plurality of message queue servers, and a plurality of stream processing servers arranged in one-to-one correspondence with the message queue servers,
the load balancing server is used for distributing an external data request to the corresponding message queue server according to a pre-configured IP mapping relation and a scheduling algorithm;
each message queue server is provided with a subject channel composed of message queues and used for temporarily storing one type of data requests, and the subject channel is used for temporarily storing the data requests distributed by the load balancing server;
each stream processing server is used for processing the data request in the theme channel of the message queue server correspondingly arranged by adopting the stream processing technology and storing the processing result into the database.
In the embodiment of the invention, the load balancing server adopts an Nginx server.
In the embodiment of the invention, the message queue server adopts a Kafka server.
In the embodiment of the invention, the stream processing server adopts a Flink server.
In the embodiment of the invention, the load balancing server sets the message queue servers corresponding to different types of data requests and the IP addresses of the message queue servers in the pre-configured IP mapping relation and scheduling algorithm.
In an embodiment of the present invention, a method for processing high-concurrency data is further provided, where the method includes:
the load balancing server distributes the external data request to the corresponding message queue server according to the pre-configured IP mapping relation and the scheduling algorithm;
the message queue server temporarily stores the data requests distributed by the load balancing server in a subject channel for temporarily storing the data requests of the same type;
the stream processing server obtains the data request in the topic channel of the corresponding message queue server, processes the data request by adopting a stream processing technology, and stores the processing result into a database.
Compared with the prior art, the high-concurrency data processing system and the method thereof are adopted, and the load balancing server distributes the external data request to the corresponding message queue server according to the pre-configured IP mapping relation and the scheduling algorithm; the message queue server temporarily stores the data requests distributed by the load balancing server in a subject channel for temporarily storing the data requests of the same type; the stream processing server obtains the data request in the subject channel of the corresponding message queue server, processes the data request by adopting a stream processing technology, and stores the processing result into the database.
Drawings
FIG. 1 is a block diagram of a highly concurrent data processing system according to an embodiment of the present invention.
FIG. 2 is a schematic diagram of a data processing flow of a highly concurrent data processing system according to an embodiment of the present invention.
Detailed Description
As shown in fig. 1, in the embodiment of the present invention, a highly concurrent data processing system is provided, which includes a database, a load balancing server, a plurality of message queue servers, and a plurality of stream processing servers arranged in one-to-one correspondence with the plurality of message queue servers. The following description will be made separately.
The database is used for storing data. In the embodiment of the invention, the database adopts a MySQL database.
And the load balancing server is used for distributing the external data request to the corresponding message queue server according to the pre-configured IP mapping relation and the scheduling algorithm.
It should be noted that, in the embodiment of the present invention, the load balancing server employs an Nginx server. The Nginx server has the advantages of high speed, good expansibility and high reliability. In the embodiment of the invention, the Nginx server is adopted as the load balancing server, and the Nginx server is used for carrying out the distribution operation of the reverse proxy on the data request by utilizing the high reliability special of the Nginx server. In the pre-configured IP mapping relationship and scheduling algorithm, message queue servers corresponding to different types of data requests and IP addresses of the message queue servers are set, so that the load aggregation server can send the data requests to the corresponding message queue servers according to the types of the data requests.
Each message queue server is provided with a subject channel composed of message queues and used for temporarily storing one type of data requests, and the subject channel is used for temporarily storing the data requests distributed by the load balancing server.
It should be noted that, since the highly concurrent data processing system needs to process a large number of highly concurrent data requests, a plurality of message queue servers need to be provided. In the embodiment of the invention, the message queue server adopts a Kafka server. The Kafka server is a high-throughput, distributed, publish-subscribe schema-based messaging system. In the embodiment of the invention, the Kafka server cluster is arranged to process the high-concurrency data request. The Kafka server serves as a producer to establish topic (topic) channels, each topic channel is used for temporarily storing one type of data request, and the data requests distributed by the load balancing server are temporarily stored in the topic channel.
Each stream processing server is used for processing the data request in the theme channel of the message queue server correspondingly arranged by adopting the stream processing technology and storing the processing result into the database.
It should be noted that, in the embodiment of the present invention, the stream processing server employs a Flink server. The plurality of Flink servers form a Flink server cluster. The Flink servers can not only respond to high-concurrency requests, but also reduce the coupling degree among programs and perform independent business processing on consumed data, each Flink server in the Flink server cluster serves as a consumer, data requests of corresponding main body channels in the Kafka server are appointed for consumption, high-concurrency logic processing is performed according to business requirements, and processed results are stored in the MySQL database to provide subsequent API requirements.
The configuration process of the high concurrency data processing system is as follows:
the first step is as follows: building a Kafka server cluster, selecting a plurality of computers to install Kafka server software, and configuring the IP of each Kafka server;
the second step is that: building a Nginx server, selecting a machine to install Nginx server software, modifying a configuration file, configuring a mapping relation according to the IP of each Kafka server, and distributing a request to the corresponding Kafka server according to the mapping relation and a scheduling algorithm when the Nginx server obtains the request sent by the equipment;
the third step: building a Flink server cluster, selecting a plurality of computers to build the Flink servers, configuring the stream processing service in each Flink server, and enabling each stream processing service to point to a corresponding theme channel in one Kafka server;
and step four, installing a MySQL database, and associating each Flink server with the MySQL database to realize data transmission.
As shown in fig. 2, the data processing procedure of the highly concurrent data processing system includes:
the load balancing server acquires an external data request;
the load balancing server distributes the external data request to the corresponding message queue server according to the pre-configured IP mapping relation and the scheduling algorithm;
the message queue server temporarily stores the data requests distributed by the load balancing server in a subject channel for temporarily storing the data requests of the same type;
the stream processing server obtains the data request in the topic channel of the corresponding message queue server, processes the data request by adopting a stream processing technology, and stores the processing result into a database.
In summary, with the high-concurrency data processing system and method of the present invention, the load balancing server distributes the external data request to the corresponding message queue server according to the pre-configured IP mapping relationship and scheduling algorithm; the message queue server temporarily stores the data requests distributed by the load balancing server in a subject channel for temporarily storing the data requests of the same type; the stream processing server obtains the data request in the subject channel of the corresponding message queue server, processes the data request by adopting a stream processing technology, and stores the processing result into the database.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents and improvements made within the spirit and principle of the present invention are intended to be included within the scope of the present invention.
Claims (10)
1. A high concurrency data processing system is characterized by comprising a database, a load balancing server, a plurality of message queue servers and a plurality of stream processing servers which are arranged in one-to-one correspondence with the message queue servers,
the load balancing server is used for distributing an external data request to the corresponding message queue server according to a pre-configured IP mapping relation and a scheduling algorithm;
each message queue server is provided with a subject channel composed of message queues and used for temporarily storing one type of data requests, and the subject channel is used for temporarily storing the data requests distributed by the load balancing server;
each stream processing server is used for processing the data request in the theme channel of the message queue server correspondingly arranged by adopting the stream processing technology and storing the processing result into the database.
2. The highly concurrent data processing system according to claim 1, wherein the load balancing server employs a Nginx server.
3. The highly concurrent data processing system according to claim 1, wherein the message queue server employs a Kafka server.
4. The highly concurrent data processing system according to claim 1, wherein the stream processing server employs a Flink server.
5. The system according to claim 1, wherein the load balancing server sets message queue servers corresponding to different types of data requests and IP addresses of the message queue servers in the pre-configured IP mapping relationship and scheduling algorithm.
6. A high concurrency data processing method is characterized by comprising the following steps:
the load balancing server distributes the external data request to the corresponding message queue server according to the pre-configured IP mapping relation and the scheduling algorithm;
the message queue server temporarily stores the data requests distributed by the load balancing server in a subject channel for temporarily storing the data requests of the same type;
the stream processing server obtains the data request in the topic channel of the corresponding message queue server, processes the data request by adopting a stream processing technology, and stores the processing result into a database.
7. The method for highly concurrent data processing according to claim 6, wherein the load balancing server employs a Nginx server.
8. The highly concurrent data processing method according to claim 6, wherein the message queue server employs a Kafka server.
9. The highly concurrent data processing method according to claim 6, wherein the stream processing server employs a Flink server.
10. The method according to claim 6, wherein the load balancing server sets message queue servers corresponding to different types of data requests and IP addresses of the message queue servers in the pre-configured IP mapping relationship and scheduling algorithm.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011423413.2A CN112486655A (en) | 2020-12-08 | 2020-12-08 | High-concurrency data processing system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011423413.2A CN112486655A (en) | 2020-12-08 | 2020-12-08 | High-concurrency data processing system and method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112486655A true CN112486655A (en) | 2021-03-12 |
Family
ID=74940582
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011423413.2A Pending CN112486655A (en) | 2020-12-08 | 2020-12-08 | High-concurrency data processing system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112486655A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113691611A (en) * | 2021-08-23 | 2021-11-23 | 湖南大学 | Block chain distributed high-concurrency transaction processing method, system, equipment and storage medium |
| CN114095502A (en) * | 2021-10-08 | 2022-02-25 | 浙江吉利控股集团有限公司 | Service processing method, system, device and medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105450618A (en) * | 2014-09-26 | 2016-03-30 | Tcl集团股份有限公司 | Operation method and operation system of big data process through API (Application Programming Interface) server |
| US20190095992A1 (en) * | 2017-09-24 | 2019-03-28 | Annie Mafotsing Soh | Method and system to facilitate decentralized money services software as a service |
| CN110365644A (en) * | 2019-06-05 | 2019-10-22 | 华南理工大学 | A method of building internet of things equipment high-performance monitoring platform |
-
2020
- 2020-12-08 CN CN202011423413.2A patent/CN112486655A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105450618A (en) * | 2014-09-26 | 2016-03-30 | Tcl集团股份有限公司 | Operation method and operation system of big data process through API (Application Programming Interface) server |
| US20190095992A1 (en) * | 2017-09-24 | 2019-03-28 | Annie Mafotsing Soh | Method and system to facilitate decentralized money services software as a service |
| CN110365644A (en) * | 2019-06-05 | 2019-10-22 | 华南理工大学 | A method of building internet of things equipment high-performance monitoring platform |
Non-Patent Citations (1)
| Title |
|---|
| 余宣杰等: "《银行大数据应用》", 31 August 2019, 机械工业出版社, pages: 59 - 60 * |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113691611A (en) * | 2021-08-23 | 2021-11-23 | 湖南大学 | Block chain distributed high-concurrency transaction processing method, system, equipment and storage medium |
| CN114095502A (en) * | 2021-10-08 | 2022-02-25 | 浙江吉利控股集团有限公司 | Service processing method, system, device and medium |
| CN114095502B (en) * | 2021-10-08 | 2023-11-03 | 浙江吉利控股集团有限公司 | Service processing method, system, device and medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110677277B (en) | Data processing method, device, server and computer readable storage medium | |
| CN112486648A (en) | Task scheduling method, device, system, electronic equipment and storage medium | |
| CN114270344A (en) | Messaging platform for delivering real-time messages | |
| CN112486655A (en) | High-concurrency data processing system and method | |
| CN106155811B (en) | Resource service device, resource scheduling method and device | |
| CN105450618A (en) | Operation method and operation system of big data process through API (Application Programming Interface) server | |
| CN107172187A (en) | A kind of SiteServer LBS and method | |
| TWI786527B (en) | User code operation method of programming platform, electronic equipment and computer-readable storage medium | |
| CN105183299A (en) | Human-computer interface service processing system and method | |
| CN105471700B (en) | A kind of methods, devices and systems of Message Processing | |
| CN108304236B (en) | User interface refreshing method based on message subscription under cloud platform | |
| CN110300188A (en) | Data transmission system, method and apparatus | |
| US20150358249A1 (en) | Communicating data in flows between first and second computers over a network | |
| CN103338156B (en) | A kind of name pipeline server concurrent communication method based on thread pool | |
| Guo | Aggregating uncertain incast transfers in BCube-like data centers | |
| CN108259605B (en) | Data calling system and method based on multiple data centers | |
| CN111711675B (en) | Solution for concurrent message transmission in local area network | |
| TWI433517B (en) | Method and apparatus of processing sip messages based on multiple cores | |
| CN115378937A (en) | Distributed concurrency method, device and equipment for tasks and readable storage medium | |
| CN111683140B (en) | Method and apparatus for distributing messages | |
| CN110018782A (en) | A kind of data read/write method and relevant apparatus | |
| Aksyonov et al. | Perspectives of modeling in metallurgical production (WIP) | |
| CN102999390A (en) | Rear-end resource control method and device under cloud computing environment | |
| CN114490026A (en) | Message consumption optimization method and terminal | |
| CN105323225B (en) | The interactive correspondence protocol fitting method and system of cross-terminal |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |