CN103634411A - Real-time market data broadcasting system and real-time market data broadcasting method with state consistency - Google Patents
Real-time market data broadcasting system and real-time market data broadcasting method with state consistency Download PDFInfo
- Publication number
- CN103634411A CN103634411A CN201310689144.8A CN201310689144A CN103634411A CN 103634411 A CN103634411 A CN 103634411A CN 201310689144 A CN201310689144 A CN 201310689144A CN 103634411 A CN103634411 A CN 103634411A
- Authority
- CN
- China
- Prior art keywords
- data
- broadcast
- module
- state
- client
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Landscapes
- Computer And Data Communications (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The invention relates to the field of real-time network data broadcasting, and in particular relates to a real-time market data broadcasting system and a real-time market data broadcasting method with state consistency. The real-time market data broadcasting system is simultaneously arranged a host computer or a backup computer or arranged on a host computer and a plurality of backup computers; all original service processing data are stored in a log area in a working page memory of the host computer; a broadcasting module is used for scanning market data of the data area of the working page memory in real time, used for establishing indexes and used for ordering data streams; according to the situation that whether a present server is the host computer or a backup computer and whether the computer is in the recovery state or the active state, and the like, a state computer is used for tracking service state and used for controlling whether the broadcasting data can be issued to the market, so that the broadcasting module can efficiently process memory data. Compared with the prior art, the system has the advantages that rapid positioned data and a cache structure of the state can be established at a server, low response time delay is ensured while handling capacity is ensured, single-point network bottleneck is avoided, and the market stability is improved.
Description
[technical field]
The present invention relates to network data real-time broadcast field, relate in particular to balancing technique and data issued state consistency correlation technique between high Concurrency Access and low delay response, specifically a kind of marketing data real time broadcasting system and method with state consistency.
[background technology]
When processing request, real-time Radio Data System not only wants to bear high Concurrency Access amount, also want to respond low delay, the data object of broadcast system issue has two kinds of public message and private messages, public message is published to field, the whole city, private message only can send to relevant several clients, therefore,, when producing a broadcast data, the object that it need to be sent out has comprised public message that field, whole city client all needs and the private message of the business side of participating in directly.When a broadcast is generated, it need to be published to market with lower time delay, and in the moment of issue, network sends the time demand of all public messages and private message, at the time point of issuing, levels off to 0.Therefore, when market scale progressively increases, server data pressure to be released was infinitely amplified in moment, when breaking through the ability to bear of network, just can cause congested, even make server that service can not be externally provided, this belongs to common single node network bottleneck, and its caused problem is as finance, to have in the market of high-stability requirement, not allow generation.
In prior art, be to use multidrop network transmission technology to solve the problem of single node network bottleneck, adopt the mode cooperating with each other between a plurality of publisher servers to come the pressure of separate data issue to a plurality of publishing point, for each publisher server, increase along with load, still have the bottleneck problem that potential single node network moment produces, but scale that can expansion servers is released the pressure.When publishing point becomes increasing, between server, synchronously can become and become increasingly complex with the management cooperating, management can significantly increase with difficulty, the cost safeguarded.
In some field, such as financial market, Radio Data System also needs to guarantee that the data of issuing have state consistency, and data must be complete continuous, but also can not repeat and even recall phenomenon.Because business service system is externally service of real-time continuous, this just requires broadcast system also to want can be by market business data real-time release, and broadcast system must provide continuous service ability.Common hot standby mode can guarantee the continuous service ability of server, and in this manner, the message of persistence or real-time network synchronously can guarantee the data consistency between active and standby machine.For broadcast system, in addition, it also needs to possess state consistency, after fault recovery, guarantees that specific sensitive data can not occur out of order state in client.
In order to make broadcast system possess state consistency, a kind of common way is at server end, to set up slip condition database to preserve every data corresponding to the issued state of each client, when distributing data, according to the state in database, control data and the client of transmission, the scale of this database is data volume and number clients object cartesian product, but this database can be a mass data in the large system of traffic carrying capacity, and inquiry can increase the response time delay of server greatly frequently.
[summary of the invention]
Object of the present invention is exactly in order to solve when market scale progressively increases, server meets with single node network bottleneck, thereby cause having the market of high-stability requirement congested, the technical problem that server can not externally be served, while realizing real-time data broadcast system processing request, can not only bear high Concurrency Access amount can also respond low delay, and guaranteeing that the data that broadcast data is issued in client have state consistency, data integrity is continuous, there will not be out of order state.
To achieve these goals, a kind of marketing data real time broadcasting system with state consistency is provided, described real time broadcasting system is deployed on main frame and standby host simultaneously or is deployed on main many standby hosts, in host work page internal memory, all Business Processing initial data have been deposited in log area, these data are produced on standby host by certain synchronization mechanism, the data field of working page internal memory is the data that Business Processing produces, the data that broadcast system need to be issued are also in wherein, broadcast data on active and standby machine is managed by state machine, for the broadcast data producing on standby host, by it, control not externally issue, for broadcast module, it is responsible for the marketing data in real time scan working page internal storage data district, and set up index, to data flow sequencing, state machine is main frame or standby host according to current server, current in returning to form or the information such as active state, follow-up service state, and control broadcast data and whether can issue to market, on each main frame, there is a sending module that broadcast data is published to market, it is independent by the network data transmission work that expends performance, thereby allow broadcast module can process more efficiently internal storage data.
Described real time broadcasting system comprises data directory module, request processing module and data-pushing module, the data to be released that described data directory module produces operation system are set up index, according to condition, get data to be released rapidly, the deposit data to be released that operation system produces is in a slice shared drive, be referred to as working page internal memory, data directory module exists as a backstage asynchronous process, its work is driven by a timer, periodic point in each timing wake-up starts scanning work page internal storage data district, data directory module is not carried out any modification to original data, its major responsibility is that these data are set up to the index that can be used for inquiry, and to its sequencing, simultaneously, it is divided into a plurality of data flow according to data type by data, and correspondingly set up a plurality of index areas, index area is based upon in system shared drive, thereby when data directory module breaks down while restarting, index area can't be lost, it can continue the result of last time and down work, when setting up index, same data flow can be numbered by natural number incremental order, form a continuous sequence, to guarantee that data can sequentially be issued, or own data of losing can be checked oneself and find to client, in the process of scanning, it is unit that data directory module be take the affairs of Business Processing, handle the data of affairs and just can process the next one, guarantee that the data that market receives are complete, but not one of them fragment of certain business procedure, described request processing module is processed the active obtaining request of data from client, according to data flow, navigate to the content in index area, for public message, they are by client active obtaining, for avoiding producing unnecessary flow and guarantee that data integrity is continuous, client all can be safeguarded self a state, records the broadcast data in each data flow self having got, and client can be inquired about maximum to determine whether need to obtain data to server, described data-pushing module has the message of high priority for server active push, data directory module is when processing data to be released, when running into this class message, can trigger in real time pushing module, for avoiding interference the efficiency that client active request is processed, this class message adopts special-purpose route lines between disparate modules, consider that propelling data is also via Buffer Pool for the high concurrent pressure of issuing of broadcast data, more externally issued by data transmission process independently.
Described real time broadcasting system also comprises state machine module, state machine module is the conforming important guarantee of data issue, it is deployed on active and standby machine simultaneously, mainly solve and when main frame breaks down switching, obtain marketing data state and consistency relevant control, state machine has comprised initialization, recover, active three kinds of states, and only under active state, broadcast module is external distributing data, on main frame, the state of state machine circulation is by the Determines of Business Processing, when business module is when recovering the stage, broadcast state machine is equally in returning to form, when business module finishes to recur the external service of beginning, can produce specific message, broadcast module makes state machine enter active state while processing this message, no matter be process failure or server failure, state machine is all according to such changing rules state, therefore, the high-priority message producing in convalescence can't be pushed to client, guarantee that data issued state is consistent.
Described data directory module is driven by a timer, and it is processes data in units that data directory module be take the affairs of Business Processing, thereby guarantees the transactional that its data is processed, and each treatment cycle mainly comprises the following steps:
A. the progress situation of data directory module treatment cycle from state machine reads, as processed in current data stream to which affairs;
B. scanning work page internal storage data district reads the data of next Business Processing affairs;
C. the data that read are categorized into corresponding broadcasting stream according to type of message, for every new data distributes numbering, and in shared drive index area, safeguard that a numbering is to the index relative of data itself;
If d. handled record is not a PUSH message, skips to f step, otherwise continue;
E. judge that whether state machine is in active state, be the queue to be released to main frame external interface by message push, otherwise do not send;
F. upgrade state machine, record the progress of this processing, i.e. the processed transaction number arriving, if the maximum transaction in state machine is number lower, is updated to currency;
G. when broadcast is when recovering the stage, if work as the transaction number of pre-treatment, keep abreast with the transaction number that state machine is safeguarded, the recovery that completes historical data has been described, state machine has been switched to active state, allowed follow-up data issue;
H. a treatment cycle finishes, and process enters resting state, waits for that timer wakes up next time.
During described client active obtaining data, according to following method request broadcast system, issue marketing data:
A. client initial start, now client does not have any data, and the process of obtaining data is as follows:
(1) client is initiated request and to broadcast system, is inquired about and obtain the maximum numbering of current certain broadcasting stream, is designated as N;
(2) client is initiated request and is pulled the interval <1 of being of this broadcasting stream sequence number, all broadcast datas between N> to broadcast system;
(3) broadcast system drawing-die piece is received and is asked and verify reasonability, by asked data packing, puts into outgoing queue and replys successfully response of a request;
(4) broadcast system drawing-die piece triggers host interface process with asynchronous system and issues broadcast data;
B. client is obtained broadcast data continuously in the real-time deal stage
Under this scene, main flow process is identical with initial start, and difference is that asked scope is that the current maximum numbering M that obtains of client and the maximum of broadcast system data available are numbered the data between N, i.e. <M, N>;
C. abnormal conditions, client part broadcast data is lost
Now, the broadcast data of client because deleted by mistake or other accident and become discontinuous, thereby on subsequent transaction, decision-making has produced impact, now client need to send repeat requests and pull this segment data.
Described data directory module is when work for the treatment of page data in EMS memory records, data that can directive sending for some, can carry out pushing away under active, although the data that push away only can send to limited participant in the market, but because broadcast system is served ten million client, in transaction peak period, may produce the Data Storm of certain scale, therefore to take buffer-type transmit mechanism, the same during with client active obtaining, data-pushing is to be just performed when state machine indicates active state equally, and idiographic flow is as follows:
(1) data directory module sends a message to broadcast pushing module by asynchronous process communication mechanism;
(2) broadcast pushing module is received message, by the mode of unblock, message is added to queue to be sent;
(3) the execution main line of the pushing module message in processing queue one by one, adds message to send message body;
(4) if when transmission data volume exceeds more greatly the capacity of message body, split into a plurality of transmission data-messages;
(5) the host interface queue to be sent data-message to be sent being added;
(6) queue to be sent is as the intermediary of broadcast system and host interface module, by host interface module, with certain timing frequency, processed and propelling data to market.
Described broadcast system fault handling method is as follows:
(1) fault scenes one: Business Processing process breaks down and restarts:
When Business Processing process breaks down and restarts, all shared drive data can not lost, because the generation of broadcast data is based on affairs, so business process does not need to regenerate after recovering, meanwhile, do not have new business datum processed, just can not produce new broadcast data yet, broadcast module can stop issuing data, until business process is switched to normal mode of operation;
(2) fault scenes two: broadcast system is restarted;
When broadcast system has occurred to restart, because broadcast module index area and memory state machine all guarantee without affairs, first data directory module can remove index area Reset Status machine after restarting, then data directory module is set up index to working page internal storage data again, state machine experiences initialization, recovery, active state again, thereby guarantees data consistent;
(3) fault scenes three: main frame breaks down, and standby host is taken over:
The speed of processing due to standby host is less than main processing, therefore switching moment, standby processor still some data need to be processed and not need and sends, data directory module on standby processor is after handling this part data, state machine could be switched to active state, could notify pushing module to send data, effectively guarantee that broadcast data does not reflux;
(4) fault scenes four: after main frame failure, carry out active and standby switching, the main frame of only depositing is afterwards restarted:
Main frame restart after state machine enter and return to form, the neither transmission of all broadcast datas generating in this stage, until switch to active state, thereby guarantees that data do not reflux, and reaches consistency.
A kind of employing has the broadcasting method of the marketing data real time broadcasting system of state consistency, the published method of public message and private message is distinguished, the mode that public message is regularly pulled by customer end adopted, private message is real time propelling movement, now, public message sends to the time point of client to be randomly dispersed in the timing cycle tf of client, and the transmitting time of private data is tg, for a broadcast, the data number V ' to be released of unit interval meets the following conditions:
V’=(n+g)/Max(tg,tf)
In a definite system, tf and g are constant, and the value of g is very little, and in securities exchange system, g is generally both parties, and 2, now, the value of tg is almost 0, much smaller than tf, therefore has:
V’=(n+g)/tf
In each system, tf is a constant that can be configured according to system pressure, and value is larger, and the pressure of broadcast system distributing data is less, and the retractility of systematic function is stronger.
Compared with the existing technology, its advantage is in the present invention:
1. at server end, setting up can rapidly locating and the buffer structure of state, at himself state of client maintenance, improves the ability of quick response, when ensureing higher throughput, can also guarantee lower response time delay;
2. by weakening the data-pushing ability of server end, strengthen the active obtaining ability of client, thereby reasonably balanced separate data issue and the pressure obtaining between server and client, to avoid single node network bottleneck, improves the stability in market;
3. at server end, set up broadcast state machine and at client maintenance its data state, by a broadcast state machine, control the sequential that data produce, recover, issue, thereby guarantee the state consistency of data issue after fault recovery;
4. can be used for the standby environment of a master one, can also expand in the distributed environment with multiple host based on transversally cutting.
[accompanying drawing explanation]
Fig. 1 is real time broadcasting system deployed environment schematic diagram;
Fig. 2 is real time broadcasting system high-level schematic functional block diagram;
Fig. 3 is that Indexer index module is set up index flow chart;
Fig. 4 is that client data pulls flow chart;
Fig. 5 is broadcast system active push data flowchart;
Fig. 6 is that broadcast system Business Processing process breaks down and restarts scene schematic diagram;
Fig. 7 is broadcast system switch between master and slave machines scene schematic diagram;
Specify Fig. 1 as Figure of abstract of the present invention.
[embodiment]
Below in conjunction with accompanying drawing, the invention will be further described, and the structure of this system and principle are very clearly concerning this professional people.Should be appreciated that specific embodiment described herein, only in order to explain the present invention, is not intended to limit the present invention.
Real time broadcasting system of the present invention not only can provide higher throughput, can also guarantee lower response time delay, wherein state machine module is the conforming important guarantee of data issue, it is deployed on active and standby machine simultaneously, mainly solves and when main frame breaks down switching, obtains marketing data state and consistency relevant control.
Embodiment 1
As shown in Figure 1, Fig. 1 is the deployed environment of real time broadcasting system in the present invention, and this broadcast system is deployed on main frame and standby host simultaneously, in the extra high environment of reliability requirement, even can be deployed in a master how standby upper.In host work page internal memory in figure, all Business Processing initial data have been deposited in log area, these data are produced on standby host by certain synchronization mechanism, the data field of working page internal memory is the data that Business Processing produces, and the data that broadcast system need to be issued are also in wherein.Broadcast data on active and standby machine is managed by state machine, for the broadcast data producing on standby host, by its empty system, externally do not issued, for broadcast module, it is responsible for the marketing data in real time scan working page internal storage data district, and set up index, to data flow sequencing.State machine is main frame or standby host according to current server, current in returning to form or the information such as active state, follow-up service state, and control broadcast data whether can be to market issue etc.On each main frame, have a sending module that broadcast data is published to market, it is independent by the network data transmission work that expends performance, thereby allows broadcast module can process more efficiently internal storage data.
Embodiment 2
Fig. 2 is each functional module of real time broadcasting system proposed by the invention, Indexer(data directory module) be the source of broadcast data issue, the new data that its regular work for the treatment of page internal memory produces, and these Data classifications are arrived to different data flow, and carry out sequencing numbering, in the process of processing, if run into the data that need propelling movement, it also will inform data-pushing module in the mode of interprocess communication by the index of data, to add its task queue.For fear of interprocess communication, block the progress of work, interprocess communication has adopted task queue to add the mode of asynchronous notifications.Data-pushing module is processed from Indexer(data directory module in the mode of unblock) message, and they are put into task queue, so that subsequent treatment, data-pushing task originally can't be because of processing Indexer(data directory module) message and interrupted.Main frame external interface is an interface layer outside broadcast module and transaction platform, it receives all kinds of requests of client and produces corresponding task and is put in the queue of drawing-die piece, therefore, request processing module is just freed from network connects, and only needs to be responsible for the task in processing queue.
State machine is being safeguarded a Indexer(data directory module on the one hand) each latest transaction numbering of processing, this numbering judges whether one of sign of recurrence end when recurring for fault; It has reacted the state of whole broadcast system on the other hand, and its state judges as broadcast module the foundation whether data can send.
No matter be data-pushing module or request module, it is all through the unified issue of main frame external interface that their data send, and by data processing and data are sent to separate coming, Each performs its own functions, greatly improved the ability of issue.
Embodiment 3
Fig. 3 is the groundwork flow process of the Indexer module (data directory module) in the present invention, Indexer(data directory module) by a timer, driven, Indexer(data directory module) take the affairs of Business Processing is processes data in units, thereby guarantee the transactional that its data is processed, each treatment cycle mainly comprises the following steps:
(1) Indexer(data directory module) the progress situation of a treatment cycle from state machine reads, as processed in current data stream to which affairs;
(2) scanning work page internal storage data district reads the data of next Business Processing affairs;
(3) data that read are categorized into corresponding broadcasting stream according to type of message, for every new data distributes numbering, and in shared drive index area, safeguard that a numbering is to the index relative of data itself;
(4) if handled record is not a PUSH message, jump to the 6th) step, otherwise continue;
(5) judge that whether state machine is in active state, be the queue to be released to main frame external interface by message push, otherwise do not send;
(6) upgrade state machine, record the progress of this processing, i.e. the processed transaction number arriving.If the maximum transaction in state machine is number lower, be updated to currency;
(7) when broadcast is when recovering the stage, if work as the transaction number of pre-treatment, keep abreast with the transaction number that state machine is safeguarded, the recovery that completes historical data has been described, state machine has been switched to active state, allowed follow-up data issue;
(8) treatment cycles finish, and process enters resting state, waits for that timer wakes up next time.
Embodiment 4
Fig. 4 is the once flow chart of complete interaction sequence that client pulls broadcast data, and for taking out the data circulation logic between nucleus module, host interface and response routing module all omit.During client active obtaining data, can, according to self asking broadcast system to issue marketing data, mainly contain following scene:
A. client initial start, now client does not have any data, and the process of obtaining data is as follows:
(1) client is initiated request and to broadcast system, is inquired about and obtain the maximum numbering of current certain broadcasting stream, is designated as N;
(2) client is initiated request and is pulled the interval <1 of being of this broadcasting stream sequence number, all broadcast datas between N> to broadcast system;
(3) broadcast system drawing-die piece is received and is asked and verify reasonability, by asked data packing, puts into outgoing queue and replys successfully response of a request;
(4) broadcast system drawing-die piece triggers host interface process with asynchronous system and issues broadcast data;
B. client is obtained broadcast data continuously in the real-time deal stage
Under this scene, main flow process is identical with initial start, and difference is that asked scope is that the current maximum numbering M that obtains of client and the maximum of broadcast system data available are numbered the data between N, i.e. <M, N>;
C. abnormal conditions, client part broadcast data is lost
Now, the broadcast data of client because deleted by mistake or other accident and become discontinuous, thereby on subsequent transaction, decision-making has produced impact.
As shown in table 1, the broadcast sequence <8 of client, 10> loses, and now client need to send repeat requests and pull this segment data.
Table 1 client broadcast data sequence numbering signal table
| 1 | 2 | 3 | 4 | 5 | 6 | 7 | 11 | 12 | 13 |
Embodiment 5
Fig. 5 is the flow chart of broadcast system active push marketing data, Indexer(data directory module) when work for the treatment of page data in EMS memory records, data that can directive sending for some, can carry out active push, although one the data of propelling movement only can send to limited participant in the market, but because broadcast system is served ten million client, in transaction peak period, may produce the Data Storm of certain scale, therefore to take buffer-type transmit mechanism, the same during with client active obtaining, data-pushing is to be just performed when state machine indicates active state equally, idiographic flow is as follows:
(1) Indexer(data directory module) by asynchronous process communication mechanism, send a message to broadcast pushing module
(2) broadcast pushing module is received message, by the mode of unblock, message is added to queue to be sent
(3) the execution main line of the pushing module message in processing queue one by one, adds message to send message body
(4) if when transmission data volume exceeds more greatly the capacity of message body, split into a plurality of transmission data-messages
(5) the host interface queue to be sent data-message to be sent being added
(6) queue to be sent is as the intermediary of broadcast system and host interface module, by host interface module, with certain timing frequency, processed and propelling data to market
Embodiment 6
The core that can keep consistency after broadcast system fault is state machine, and it has used a series of technology and means to guarantee the normal issue of marketing data, and its synchronize design is mainly for following four fault application scenarioss:
Fault scenes one: Business Processing process breaks down and restarts
When Business Processing process breaks down and restarts, all shared drive data can not lost, such as working page internal memory, broadcast index area, state machine state data etc., because the generation of broadcast data is based on affairs, thus business process after recovering, do not need to regenerate, meanwhile, do not have new business datum processed, also just can not produce new broadcast data, broadcast module can stop issuing data, until business process is switched to normal mode of operation.
Fault scenes two: broadcast system is restarted
When broadcast system has occurred to restart, because broadcast module index area and memory state machine all guarantee without affairs, first Indexer module (data directory module) can remove index area Reset Status machine after restarting, then Indexer(data directory module) again working page internal storage data is set up to index, state machine experiences initialization, recovery, active state again, thereby guarantees data consistent.
Fault scenes three: main frame breaks down, standby host is taken over
Generally, the speed of processing due to standby host is less than main processing, is therefore switching moment, and standby processor still some data need to be processed and not need transmission, as shown in Figure 7.Data directory module I ndexer on standby processor, after handling this part data, could switch to active state by state machine, could notify pushing module to send data, has effectively guaranteed broadcast data not " backflow ".
Fault scenes four: carry out active and standby switching after main frame failure, the main frame of only depositing is afterwards restarted
Main frame restart after state machine enter and return to form, the neither transmission of all broadcast datas generating in this stage, until switch to active state, thereby guarantees data not " backflow ", reaches consistency.
Claims (8)
1. a marketing data real time broadcasting system with state consistency, it is characterized in that described real time broadcasting system is deployed on main frame and standby host simultaneously or is deployed on main many standby hosts, in host work page internal memory, all Business Processing initial data have been deposited in log area, these data are produced on standby host by certain synchronization mechanism, the data field of working page internal memory is the data that Business Processing produces, the data that broadcast system need to be issued are also in wherein, broadcast data on active and standby machine is managed by state machine, for the broadcast data producing on standby host, by it, control not externally issue, for broadcast module, it is responsible for the marketing data in real time scan working page internal storage data district, and set up index, to data flow sequencing, state machine is main frame or standby host according to current server, current in returning to form or the information such as active state, follow-up service state, and control broadcast data and whether can issue to market, on each main frame, there is a sending module that broadcast data is published to market, it is independent by the network data transmission work that expends performance, thereby allow broadcast module can process more efficiently internal storage data.
2. a kind of marketing data real time broadcasting system with state consistency as claimed in claim 1, it is characterized in that described real time broadcasting system comprises data directory module, request processing module and data-pushing module, the data to be released that described data directory module produces operation system are set up index, according to condition, get data to be released rapidly, the deposit data to be released that operation system produces is in a slice shared drive, be referred to as working page internal memory, data directory module exists as a backstage asynchronous process, its work is driven by a timer, periodic point in each timing wake-up starts scanning work page internal storage data district, data directory module is not carried out any modification to original data, its major responsibility is that these data are set up to the index that can be used for inquiry, and to its sequencing, simultaneously, it is divided into a plurality of data flow according to data type by data, and correspondingly set up a plurality of index areas, index area is based upon in system shared drive, thereby when data directory module breaks down while restarting, index area can't be lost, it can continue the result of last time and down work, when setting up index, same data flow can be numbered by natural number incremental order, form a continuous sequence, to guarantee that data can sequentially be issued, or own data of losing can be checked oneself and find to client, in the process of scanning, it is unit that data directory module be take the affairs of Business Processing, handle the data of affairs and just can process the next one, guarantee that the data that market receives are complete, but not one of them fragment of certain business procedure, described request processing module is processed the active obtaining request of data from client, according to data flow, navigate to the content in index area, for public message, they are by client active obtaining, for avoiding producing unnecessary flow and guarantee that data integrity is continuous, client all can be safeguarded self a state, records the broadcast data in each data flow self having got, and client can be inquired about maximum to determine whether need to obtain data to server, described data-pushing module has the message of high priority for server active push, data directory module is when processing data to be released, when running into this class message, can trigger in real time pushing module, for avoiding interference the efficiency that client active request is processed, this class message adopts special-purpose route lines between disparate modules, consider that propelling data is also via Buffer Pool for the high concurrent pressure of issuing of broadcast data, more externally issued by data transmission process independently.
3. a kind of marketing data real time broadcasting system with state consistency as claimed in claim 1, it is characterized in that described real time broadcasting system also comprises state machine module, state machine module is the conforming important guarantee of data issue, it is deployed on active and standby machine simultaneously, mainly solve and when main frame breaks down switching, obtain marketing data state and consistency relevant control, state machine has comprised initialization, recover, active three kinds of states, and only under active state, broadcast module is external distributing data, on main frame, the state of state machine circulation is by the Determines of Business Processing, when business module is when recovering the stage, broadcast state machine is equally in returning to form, when business module finishes to recur the external service of beginning, can produce specific message, broadcast module makes state machine enter active state while processing this message, no matter be process failure or server failure, state machine is all according to such changing rules state, therefore, the high-priority message producing in convalescence can't be pushed to client, guarantee that data issued state is consistent.
4. a kind of marketing data real time broadcasting system with state consistency as claimed in claim 2, it is characterized in that described data directory module is driven by a timer, it is processes data in units that data directory module be take the affairs of Business Processing, thereby guarantee the transactional that its data is processed, each treatment cycle mainly comprises the following steps:
A. the progress situation of data directory module treatment cycle from state machine reads, as processed in current data stream to which affairs;
B. scanning work page internal storage data district reads the data of next Business Processing affairs;
C. the data that read are categorized into corresponding broadcasting stream according to type of message, for every new data distributes numbering, and in shared drive index area, safeguard that a numbering is to the index relative of data itself;
If d. handled record is not a PUSH message, skips to f step, otherwise continue;
E. judge that whether state machine is in active state, be the queue to be released to main frame external interface by message push, otherwise do not send;
F. upgrade state machine, record the progress of this processing, i.e. the processed transaction number arriving, if the maximum transaction in state machine is number lower, is updated to currency;
G. when broadcast is when recovering the stage, if work as the transaction number of pre-treatment, keep abreast with the transaction number that state machine is safeguarded, the recovery that completes historical data has been described, state machine has been switched to active state, allowed follow-up data issue;
H. a treatment cycle finishes, and process enters resting state, waits for that timer wakes up next time.
5. a kind of marketing data real time broadcasting system with state consistency as claimed in claim 2, issues marketing data according to following method request broadcast system while it is characterized in that described client active obtaining data:
A. client initial start, now client does not have any data, and the process of obtaining data is as follows:
(1) client is initiated request and to broadcast system, is inquired about and obtain the maximum numbering of current certain broadcasting stream, is designated as N;
(2) client is initiated request and is pulled the interval <1 of being of this broadcasting stream sequence number, all broadcast datas between N> to broadcast system;
(3) broadcast system drawing-die piece is received and is asked and verify reasonability, by asked data packing, puts into outgoing queue and replys successfully response of a request;
(4) broadcast system drawing-die piece triggers host interface process with asynchronous system and issues broadcast data;
B. client is obtained broadcast data continuously in the real-time deal stage
Under this scene, main flow process is identical with initial start, and difference is that asked scope is that the current maximum numbering M that obtains of client and the maximum of broadcast system data available are numbered the data between N, i.e. <M, N>;
C. abnormal conditions, client part broadcast data is lost
Now, the broadcast data of client because deleted by mistake or other accident and become discontinuous, thereby on subsequent transaction, decision-making has produced impact, now client need to send repeat requests and pull this segment data.
6. a kind of marketing data real time broadcasting system with state consistency as claimed in claim 2, it is characterized in that described data directory module is when work for the treatment of page data in EMS memory records, data that can directive sending for some, can carry out pushing away under active, although the data that push away only can send to limited participant in the market, but because broadcast system is served ten million client, in transaction peak period, may produce the Data Storm of certain scale, therefore to take buffer-type transmit mechanism, the same during with client active obtaining, data-pushing is to be just performed when state machine indicates active state equally, idiographic flow is as follows:
(1) data directory module sends a message to broadcast pushing module by asynchronous process communication mechanism;
(2) broadcast pushing module is received message, by the mode of unblock, message is added to queue to be sent;
(3) the execution main line of the pushing module message in processing queue one by one, adds message to send message body;
(4) if when transmission data volume exceeds more greatly the capacity of message body, split into a plurality of transmission data-messages;
(5) the host interface queue to be sent data-message to be sent being added;
(6) queue to be sent is as the intermediary of broadcast system and host interface module, by host interface module, with certain timing frequency, processed and propelling data to market.
7. a kind of marketing data real time broadcasting system with state consistency as claimed in claim 1 or 2, is characterized in that described broadcast system fault handling method is as follows:
(1) fault scenes one: Business Processing process breaks down and restarts:
When Business Processing process breaks down and restarts, all shared drive data can not lost, because the generation of broadcast data is based on affairs, so business process does not need to regenerate after recovering, meanwhile, do not have new business datum processed, just can not produce new broadcast data yet, broadcast module can stop issuing data, until business process is switched to normal mode of operation;
(2) fault scenes two: broadcast system is restarted;
When broadcast system has occurred to restart, because broadcast module index area and memory state machine all guarantee without affairs, first data directory module can remove index area Reset Status machine after restarting, then data directory module is set up index to working page internal storage data again, state machine experiences initialization, recovery, active state again, thereby guarantees data consistent;
(3) fault scenes three: main frame breaks down, and standby host is taken over:
The speed of processing due to standby host is less than main processing, therefore switching moment, standby processor still some data need to be processed and not need and sends, data directory module on standby processor is after handling this part data, state machine could be switched to active state, could notify pushing module to send data, effectively guarantee that broadcast data does not reflux;
(4) fault scenes four: after main frame failure, carry out active and standby switching, the main frame of only depositing is afterwards restarted:
Main frame restart after state machine enter and return to form, the neither transmission of all broadcast datas generating in this stage, until switch to active state, thereby guarantees that data do not reflux, and reaches consistency.
8. a broadcasting method with the marketing data real time broadcasting system of state consistency as claimed in claim 1, it is characterized in that the published method of public message and private message to be distinguished, the mode that public message is regularly pulled by customer end adopted, private message is real time propelling movement, now, public message sends to the time point of client to be randomly dispersed in the timing cycle tf of client, and the transmitting time of private data is tg, for a broadcast, the data number V ' to be released of unit interval meets the following conditions:
V’=(n+g)/Max(tg,tf)
In a definite system, tf and g are constant, and the value of g is very little, and in securities exchange system, g is generally both parties, and 2, now, the value of tg is almost 0, much smaller than tf, therefore has:
V’=(n+g)/tf
In each system, tf is a constant that can be configured according to system pressure, and value is larger, and the pressure of broadcast system distributing data is less, and the retractility of systematic function is stronger.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310689144.8A CN103634411B (en) | 2013-12-16 | 2013-12-16 | A kind of marketing data real time broadcasting system and method with state consistency |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310689144.8A CN103634411B (en) | 2013-12-16 | 2013-12-16 | A kind of marketing data real time broadcasting system and method with state consistency |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103634411A true CN103634411A (en) | 2014-03-12 |
| CN103634411B CN103634411B (en) | 2017-06-06 |
Family
ID=50215027
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201310689144.8A Active CN103634411B (en) | 2013-12-16 | 2013-12-16 | A kind of marketing data real time broadcasting system and method with state consistency |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103634411B (en) |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105242979A (en) * | 2015-09-09 | 2016-01-13 | 高胜法 | Backward recovery error tolerance method with forward recovery feature |
| CN110688427A (en) * | 2019-09-11 | 2020-01-14 | 北京控制工程研究所 | Asynchronous data synchronization method of four-machine hot backup real-time system |
| CN110955509A (en) * | 2019-12-11 | 2020-04-03 | 深圳迅策科技有限公司 | Finance concurrent transaction processing apparatus |
| WO2020119699A1 (en) * | 2018-12-11 | 2020-06-18 | Oppo广东移动通信有限公司 | Resource publishing method and apparatus in internet of things, device, and storage medium |
| CN111884767A (en) * | 2020-03-25 | 2020-11-03 | 上交所技术有限责任公司 | Method and system for solving 1-hop cost time full-order multicast segmentation problem |
| CN112039931A (en) * | 2019-06-03 | 2020-12-04 | 厦门本能管家科技有限公司 | Elastic broadcasting method and system for P2P network |
| CN112685199A (en) * | 2020-12-30 | 2021-04-20 | 平安普惠企业管理有限公司 | Message queue repairing method and device, computer equipment and storage medium |
| CN113301069A (en) * | 2020-04-07 | 2021-08-24 | 阿里巴巴集团控股有限公司 | Serverless management system, calling method thereof and cloud native application platform |
| CN114650301A (en) * | 2022-03-17 | 2022-06-21 | 郑州郑大信息技术有限公司 | Message queuing method based on transaction system |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20060120396A1 (en) * | 2004-12-02 | 2006-06-08 | Kddi Corporation | Communication system, delay insertion server, backup server and communication control apparatus |
| CN101212458A (en) * | 2006-12-25 | 2008-07-02 | 中兴通讯股份有限公司 | Database system backup method |
| CN101282229A (en) * | 2007-04-06 | 2008-10-08 | 中兴通讯股份有限公司 | Method for implementing broadcasting service by PPPoE |
| CN101917263A (en) * | 2010-08-16 | 2010-12-15 | 深圳市赫迪威信息技术有限公司 | Multi-machine hot standby method and system |
-
2013
- 2013-12-16 CN CN201310689144.8A patent/CN103634411B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20060120396A1 (en) * | 2004-12-02 | 2006-06-08 | Kddi Corporation | Communication system, delay insertion server, backup server and communication control apparatus |
| CN101212458A (en) * | 2006-12-25 | 2008-07-02 | 中兴通讯股份有限公司 | Database system backup method |
| CN101282229A (en) * | 2007-04-06 | 2008-10-08 | 中兴通讯股份有限公司 | Method for implementing broadcasting service by PPPoE |
| CN101917263A (en) * | 2010-08-16 | 2010-12-15 | 深圳市赫迪威信息技术有限公司 | Multi-machine hot standby method and system |
Cited By (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105242979A (en) * | 2015-09-09 | 2016-01-13 | 高胜法 | Backward recovery error tolerance method with forward recovery feature |
| CN105242979B (en) * | 2015-09-09 | 2017-12-12 | 高胜法 | It is a kind of that there is the preceding backward recovery fault-tolerance approach to recovery feature |
| US11463376B2 (en) | 2018-12-11 | 2022-10-04 | Guangdong Oppo Mobile Telecommunications Corp., Ltd. | Resource distribution method and apparatus in Internet of Things, device, and storage medium |
| WO2020119699A1 (en) * | 2018-12-11 | 2020-06-18 | Oppo广东移动通信有限公司 | Resource publishing method and apparatus in internet of things, device, and storage medium |
| CN111314394A (en) * | 2018-12-11 | 2020-06-19 | Oppo广东移动通信有限公司 | Resource publishing method, device, equipment and storage medium of Internet of things |
| CN112039931A (en) * | 2019-06-03 | 2020-12-04 | 厦门本能管家科技有限公司 | Elastic broadcasting method and system for P2P network |
| CN112039931B (en) * | 2019-06-03 | 2023-04-28 | 本无链科技(深圳)有限公司 | Elastic broadcasting method and system for P2P network |
| CN110688427B (en) * | 2019-09-11 | 2022-03-04 | 北京控制工程研究所 | Asynchronous data synchronization method of four-machine hot backup real-time system |
| CN110688427A (en) * | 2019-09-11 | 2020-01-14 | 北京控制工程研究所 | Asynchronous data synchronization method of four-machine hot backup real-time system |
| CN110955509A (en) * | 2019-12-11 | 2020-04-03 | 深圳迅策科技有限公司 | Finance concurrent transaction processing apparatus |
| CN111884767A (en) * | 2020-03-25 | 2020-11-03 | 上交所技术有限责任公司 | Method and system for solving 1-hop cost time full-order multicast segmentation problem |
| CN111884767B (en) * | 2020-03-25 | 2023-01-20 | 上交所技术有限责任公司 | Method and system for solving 1-hop cost time full-order multicast segmentation problem |
| CN113301069A (en) * | 2020-04-07 | 2021-08-24 | 阿里巴巴集团控股有限公司 | Serverless management system, calling method thereof and cloud native application platform |
| CN112685199A (en) * | 2020-12-30 | 2021-04-20 | 平安普惠企业管理有限公司 | Message queue repairing method and device, computer equipment and storage medium |
| CN112685199B (en) * | 2020-12-30 | 2023-10-20 | 董小君 | Message queue repairing method and device, computer equipment and storage medium |
| CN114650301A (en) * | 2022-03-17 | 2022-06-21 | 郑州郑大信息技术有限公司 | Message queuing method based on transaction system |
| CN114650301B (en) * | 2022-03-17 | 2024-05-24 | 郑州郑大信息技术有限公司 | Message queuing method based on transaction system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN103634411B (en) | 2017-06-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103634411A (en) | Real-time market data broadcasting system and real-time market data broadcasting method with state consistency | |
| US7440977B2 (en) | Recovery method using extendible hashing-based cluster logs in shared-nothing spatial database cluster | |
| CN103116661B (en) | A kind of data processing method of database | |
| US8856091B2 (en) | Method and apparatus for sequencing transactions globally in distributed database cluster | |
| CN107832138B (en) | Method for realizing flattened high-availability namenode model | |
| CN113010496B (en) | Data migration method, device, equipment and storage medium | |
| CN107919977B (en) | Online capacity expansion and online capacity reduction method and device based on Paxos protocol | |
| CN111338773A (en) | Distributed timed task scheduling method, scheduling system and server cluster | |
| JP6225262B2 (en) | System and method for supporting partition level journaling to synchronize data in a distributed data grid | |
| CN101388759A (en) | Method and system for implementing from asynchronous copy to synchronous copy by data | |
| CN103763155A (en) | Multi-service heartbeat monitoring method for distributed type cloud storage system | |
| US9152491B2 (en) | Job continuation management apparatus, job continuation management method and job continuation management program | |
| WO2019223599A1 (en) | Data acquisition system and method, node device and storage medium | |
| CN102938705A (en) | Method for managing and switching high availability multi-machine backup routing table | |
| CN102307206A (en) | Caching system and caching method for rapidly accessing virtual machine images based on cloud storage | |
| CN107623703B (en) | Synchronization method, device and system for Global Transaction Identifier (GTID) | |
| US20120278429A1 (en) | Cluster system, synchronization controlling method, server, and synchronization controlling program | |
| KR20100099319A (en) | Node system, server switching method, server device, and data transfer method | |
| CN101834904A (en) | Method and equipment for database backup | |
| US11729133B2 (en) | Software defined network publish-subscribe system and method | |
| CN108390919A (en) | A kind of message synchronization system and method for highly reliable two-node cluster hot backup | |
| JP5331050B2 (en) | Data synchronization system, data synchronization method, information processing apparatus, information processing method, and program | |
| CN108259569A (en) | It is a kind of based on IPSAN share storage without acting on behalf of continuous data protection method | |
| CN106776151A (en) | SAMBA cluster TDB data-base recordings backup method, apparatus and system | |
| US10140183B2 (en) | Efficient state tracking for clusters |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |