CN106487902A - A kind of method of data capture based on message-oriented middleware and system - Google Patents
A kind of method of data capture based on message-oriented middleware and system Download PDFInfo
- Publication number
- CN106487902A CN106487902A CN201610912318.6A CN201610912318A CN106487902A CN 106487902 A CN106487902 A CN 106487902A CN 201610912318 A CN201610912318 A CN 201610912318A CN 106487902 A CN106487902 A CN 106487902A
- Authority
- CN
- China
- Prior art keywords
- message
- data
- proxy server
- oriented middleware
- expansible
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/55—Push-based network services
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
- H04L67/562—Brokering proxy services
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
- H04L67/568—Storing data temporarily at an intermediate stage, e.g. caching
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computer And Data Communications (AREA)
Abstract
The present invention relates to a kind of method of data capture based on message-oriented middleware, methods described includes：Message data storage request is sent to proxy server, proxy server returned packet data-pushing success status are to data-pushing end；By proxy server, multiple web application servers are carried out assembling and set up the first expansible cluster, receive message data using the first expansible cluster simultaneously, realize the real-time caching of message data；And the second expansible cluster using message-oriented middleware and by multiple web application servers are assembled and set up, the message data completing to merge disposably puts in storage.
Description
Technical field
The present invention relates to computer application field, and more particularly, to a kind of data collection based on message-oriented middleware
Method and system.
Background technology
With the development of IT technology, the allegro business demand of the modern enterprise and competitive pressure real-time to data processing
Put forward higher requirement with effectiveness, the speed that data produces is too fast, and data volume to be processed is too big.Existing information is received
Collection typically adopts two ways, and one kind is file mode transmission, and that is, data message, by way of generating file, is fixed on certain
To service receiving terminal, after service receiving terminal receives message file, concentration puts Treatment Analysis in storage to time period concentration of transmissions；One kind is logical
Cross the transmission of message string message mode, that is, data message is sent to server receiving terminal in time by real-time messages crossfire literary composition, service
End interface application program is implemented to process Receive message data, completes in-stockroom operation.
Exploding with data volume, both the above data collection process mode is analyzing demand in the face of current real-time processing
And big data quantity data processing when, all can not realize systemic-function well.Mode one obviously cannot meet very well in real time
Data transfer demands, although mode two meets real-time Transmission demand, single real-time processing warehouse-in performance is in the little feelings of data volume
Also can manage it under condition, but if data volume breaks through certain magnitude, just have mass data and be deposited in transmitting terminal and cannot send in time,
Single warehouse-in performance will become bottleneck.Therefore, otherwise mass data cannot real-time Transmission, or implement transmission when, in a large number
Data congestion is it is impossible to timely transmission process, performance and efficiency also dramatic decrease that system is processed, or even cannot process operation.
Content of the invention
In order to solve the problems referred to above of background technology presence, the present invention provides a kind of data collection based on message-oriented middleware
Method is it is characterised in that methods described includes：
Message data storage request is sent to proxy server, proxy server returned packet data-pushing success status
To data-pushing end；
By proxy server, multiple web application servers are carried out assembling and set up the first expansible cluster, profit
Receive message data with the first expansible cluster simultaneously, realize the real-time caching of message data；And
Second can expand using message-oriented middleware and by what multiple web application servers were assembled and set up
Exhibition cluster, the message data completing to merge disposably is stored in data base.
Preferably, described proxy server can support high parallel connection number.
Preferably, described proxy server is Nginx.
Preferably, described message-oriented middleware is Apache ActiveMQ.
Preferably, the plurality of web application server is all Tomcat server.
According to a further aspect in the invention, the present invention provides a kind of data gathering system based on message-oriented middleware, and it is special
Levy and be, described system includes：
Data-pushing end, it is used for for message data storage request being sent to proxy server, and Receiving Agent server
The message data returning pushes success status；
Proxy server, it is used for the message data storage request that real-time reception data-pushing end pushes, and returned packet
Data-pushing success status are to data-pushing end, and multiple Web level of application servers are assembled can to set up first
Extension cluster；
First expansible cluster, it is used for receiving message data simultaneously, realizes the real-time caching of data message；
Second expansible cluster, it is set up by assembling to multiple Web level of application servers, for complete
The message data merging is become disposably to be stored in data base；And
Message-oriented middleware, it is used for providing message communicating for the second expansible cluster.
Preferably, described proxy server can support high parallel connection number.
Preferably, described proxy server is Nginx.
Preferably, described message-oriented middleware is Apache ActiveMQ.
Preferably, the plurality of web application server is all Tomcat server.
In the present invention, using data message form pushed data in real time, ensure that the reality of data transfer most possibly
Shi Xing, meanwhile, the present invention, by the way of real time data caching, does not first carry out in-stockroom operation to data, and returned packet at once
Pushing success status to data-pushing end, thus significantly reducing the response time of each message request, improving rapidly number
The efficiency of network transmission when factually.Meanwhile, locate in the case of the request of high-volume high concurrent for single web application server
Reason is limited in one's ability, by multiple Web level of application servers are carried out with cluster, realizes processing request simultaneously, concurrently please improve
The disposal ability asked.And in terms of message data warehouse-in, the waste data once put in storage for each message data of prior art
The operation of base resource, the present invention carries out the processing mode of an in-stockroom operation again using multiple message datas after merging, thus pole
Improve greatly data loading performance.
Brief description
By reference to the following drawings, the illustrative embodiments of the present invention can be more fully understood by：
The flow chart that Fig. 1 shows the method for the data collection based on middleware of the specific embodiment of the invention；And
Fig. 2 shows the structure chart of the system of the data collection based on middleware of the specific embodiment of the invention.
Specific embodiment
With reference now to accompanying drawing, introduce the illustrative embodiments of the present invention, however, the present invention can be with many different shapes
Formula is implementing, and is not limited to embodiment described herein, provides these embodiments to be at large and fully disclose
The present invention, and fully pass on the scope of the present invention to person of ordinary skill in the field.For showing of being illustrated in the accompanying drawings
Term in example property embodiment is not limitation of the invention.In the accompanying drawings, identical cells/elements use identical attached
Icon is remembered.
Unless otherwise stated, term (inclusion scientific and technical terminology) used herein has to person of ordinary skill in the field
Common understand implication.Further it will be understood that the term being limited with the dictionary that is usually used is it should be understood to and it
The linguistic context of association area has consistent implication, and is not construed as Utopian or excessively formal meaning.
Fig. 1 is the flow chart of the method for the data collection based on middleware of the specific embodiment of the invention.Based on centre
The method 100 of the data collection of part is from the beginning of step 101.
In step 101, message data storage request is sent to proxy server, proxy server returned packet data pushes away
Send success status to data-pushing end.
In step 102, by proxy server, multiple web application servers are carried out assembling and set up first and can expand
Exhibition cluster, receives message data using the first expansible cluster simultaneously, realizes the real-time caching of message data.
In step 103, set up using message-oriented middleware and by being assembled to multiple web application servers
The second expansible cluster, complete merge message data be disposably stored in data base.
Preferably, described proxy server can support high parallel connection number.
Preferably, described proxy server is Nginx.
Preferably, described message-oriented middleware is Apache ActiveMQ.
Preferably, the plurality of web application server is all Tomcat server.
Fig. 2 is the structure chart of the system of the data collection based on middleware of the specific embodiment of the invention.As Fig. 2 institute
Show, described system 200 includes data-pushing end 201, proxy server 202, the first expansible cluster 203, the second expansible collection
Group 204, data base 205 and message-oriented middleware 206.
Data-pushing end 201, it is used for for message data storage request being sent to proxy server 202, and Receiving Agent
The message data that server 202 returns pushes success status；
Proxy server 202, it is used for the message data storage request that real-time reception data-pushing end 201 pushes, and returns
The civilian data-pushing success status of return are assembled to data-pushing end 201, and to multiple Web level of application servers to build
Vertical first expansible cluster 203；
First expansible cluster 203, it is used for receiving message data simultaneously, realizes the real-time caching of data message；
Second expansible cluster 204, it is set up by assembling to multiple Web level of application servers, uses
Disposably it is stored in data base 205 in the message data completing to merge；And
Message-oriented middleware 206, it is used for providing message communicating for the second expansible cluster 204.
Preferably, described proxy server 202 can support high parallel connection number.
Preferably, described proxy server 202 is Nginx.Nginx is a high performance HTTP and reverse proxy service
Device.In the case of high connection concurrently, Nginx can support up to 50, the response of 000 concurrent connection number, and internal memory, CPU
Deng system resources consumption but very low, run highly stable.
Preferably, described message-oriented middleware 206 is Apache ActiveMQ.Apache ActiveMQ is that one kind is increased income
It is achieved that JMS1.1 specification, the middleware of message-oriented (MOM), provide efficient, extendible, steady for application program
Fixed and safety enterprise-level message communicating.
Preferably, the plurality of web application server is all Tomcat server.
By above embodiments describing the present invention.However, it is known in those skilled in the art, as subsidiary
Patent right requirement limited, except present invention others disclosed above embodiments equally fall in the scope of the present invention
Interior.
Normally, all terms using in the claims are all solved in the usual implication of technical field according to them
Release, unless in addition clearly defined wherein.All of reference " one/described/should【Device, assembly etc.】" all it is opened ground
It is construed at least one of described device, assembly etc. example, unless otherwise expressly specified.Any method disclosed herein
Step all need not be run with disclosed accurate order, unless explicitly stated otherwise.
Claims (10)
1. a kind of method of data capture based on message-oriented middleware is it is characterised in that methods described includes：
Message data storage request is sent to proxy server, proxy server returned packet data-pushing success status are to number
According to push end；
By proxy server, multiple web application servers are carried out assembling and set up the first expansible cluster, utilize
One expansible cluster receives message data simultaneously, realizes the real-time caching of message data；And
The second expansible collection using message-oriented middleware and by multiple web application servers are assembled and set up
Group, the message data completing to merge disposably is stored in data base.
2. method according to claim 1 is it is characterised in that described proxy server can support high parallel connection number.
3. method according to claim 2 is it is characterised in that described proxy server is Nginx.
4. method according to claim 1 is it is characterised in that described message-oriented middleware is ApacheActiveMQ.
5. method according to claim 1 is it is characterised in that the plurality of web application server is all Tomcat
Server.
6. a kind of data gathering system based on message-oriented middleware is it is characterised in that described system includes：
Data-pushing end, it is used for for message data storage request being sent to proxy server, and Receiving Agent server returns
Message data push success status；
Proxy server, it is used for the message data storage request that real-time reception data-pushing end pushes, and returned packet data
Push success status to data-pushing end, and multiple Web level of application servers are assembled expansible to set up first
Cluster；
First expansible cluster, it is used for receiving message data simultaneously, realizes the real-time caching of data message；
Second expansible cluster, it is set up by assembling to multiple Web level of application servers, is used for completing to close
And message data be disposably stored in data base；And
Message-oriented middleware, it is used for providing message communicating for the second expansible cluster.
7. system according to claim 6 is it is characterised in that described proxy server can support high parallel connection number.
8. system according to claim 7 is it is characterised in that described proxy server is Nginx.
9. system according to claim 6 is it is characterised in that described message-oriented middleware is ApacheActiveMQ.
10. system according to claim 6 is it is characterised in that the plurality of web application server is all Tomcat
Server.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610912318.6A CN106487902A (en) | 2016-10-19 | 2016-10-19 | A kind of method of data capture based on message-oriented middleware and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610912318.6A CN106487902A (en) | 2016-10-19 | 2016-10-19 | A kind of method of data capture based on message-oriented middleware and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN106487902A true CN106487902A (en) | 2017-03-08 |
Family
ID=58270179
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610912318.6A Pending CN106487902A (en) | 2016-10-19 | 2016-10-19 | A kind of method of data capture based on message-oriented middleware and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106487902A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109873904A (en) * | 2019-03-05 | 2019-06-11 | 平安科技（深圳）有限公司 | User of message status report process method, device and the storage medium of message push |
| CN113055483A (en) * | 2021-03-19 | 2021-06-29 | 中国工商银行股份有限公司 | Message middleware data processing method, device and system |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101515291A (en) * | 2009-03-26 | 2009-08-26 | 北京泰合佳通信息技术有限公司 | Method for leading data into database in a batch way and system thereof |
| CN103577551A (en) * | 2013-10-16 | 2014-02-12 | 青岛海信传媒网络技术有限公司 | Method and device for submitting data to database in batch |
| CN103595800A (en) * | 2013-11-18 | 2014-02-19 | 中国联合网络通信集团有限公司 | Real-time data distributing and processing method and device |
| CN103631539A (en) * | 2013-12-13 | 2014-03-12 | 百度在线网络技术（北京）有限公司 | Distributed storage system and distributed storage method based on erasure coding mechanism |
| CN105761039A (en) * | 2016-02-17 | 2016-07-13 | 华迪计算机集团有限公司 | Method for processing express delivery information big data |
-
2016
- 2016-10-19 CN CN201610912318.6A patent/CN106487902A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101515291A (en) * | 2009-03-26 | 2009-08-26 | 北京泰合佳通信息技术有限公司 | Method for leading data into database in a batch way and system thereof |
| CN103577551A (en) * | 2013-10-16 | 2014-02-12 | 青岛海信传媒网络技术有限公司 | Method and device for submitting data to database in batch |
| CN103595800A (en) * | 2013-11-18 | 2014-02-19 | 中国联合网络通信集团有限公司 | Real-time data distributing and processing method and device |
| CN103631539A (en) * | 2013-12-13 | 2014-03-12 | 百度在线网络技术（北京）有限公司 | Distributed storage system and distributed storage method based on erasure coding mechanism |
| CN105761039A (en) * | 2016-02-17 | 2016-07-13 | 华迪计算机集团有限公司 | Method for processing express delivery information big data |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109873904A (en) * | 2019-03-05 | 2019-06-11 | 平安科技（深圳）有限公司 | User of message status report process method, device and the storage medium of message push |
| CN113055483A (en) * | 2021-03-19 | 2021-06-29 | 中国工商银行股份有限公司 | Message middleware data processing method, device and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20080177872A1 (en) | Managing aggregation and sending of communications | |
| US20060026169A1 (en) | Communication method with reduced response time in a distributed data processing system | |
| CN107256180B (en) | Data processing method, device and terminal | |
| US20130254261A1 (en) | System and Method of Managing Servers for Streaming Desktop Applications | |
| CN111277629A (en) | High-availability-based web high-concurrency system and method | |
| CN102387206A (en) | Synthesis method and system of concurrent request of Web service | |
| CN105900391A (en) | Real-time communications using RESTLIKE API | |
| CN112118315A (en) | Data processing system, method, device, electronic equipment and storage medium | |
| CN112236760B (en) | Graph data updating method, system, computer readable storage medium and equipment | |
| US7818752B2 (en) | Interface for application components | |
| CN111221793A (en) | Data mining method, platform, computer equipment and storage medium | |
| CN112121413A (en) | Response method, system, device, terminal and medium of function service | |
| CN106487902A (en) | A kind of method of data capture based on message-oriented middleware and system | |
| US20150249625A1 (en) | Using analytics to optimize performance of a messaging system via topic migration to alternate delivery methods | |
| CN114866528A (en) | Data communication method based on MQTT and Websocket | |
| CN113014608A (en) | Flow distribution control method and device, electronic equipment and storage medium | |
| CN116743754A (en) | Information processing method, device, server cluster, electronic equipment and medium | |
| Zhang et al. | A locally distributed mobile computing framework for DNN based android applications | |
| CN108076111B (en) | System and method for distributing data in big data platform | |
| CN108810070B (en) | Resource sharing method and device, intelligent equipment and storage medium | |
| CN103312808A (en) | Cloud operating system bus and communication method thereof | |
| CN111327511B (en) | Instant messaging method, system, terminal equipment and storage medium | |
| CN114301980A (en) | Method, device and system for scheduling container cluster and computer readable medium | |
| CN110380991A (en) | A kind of IOCP mechanism and the Internet of Things Network Communication acceleration system based on eFPGA and IOCP | |
| CN115037631B (en) | Cluster-based network simulation method, device and network simulation system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20170308 |
|
| RJ01 | Rejection of invention patent application after publication |