CN107800521A - A kind of data acquisition storage method of highly reliable free of losses - Google Patents
A kind of data acquisition storage method of highly reliable free of losses Download PDFInfo
- Publication number
- CN107800521A CN107800521A CN201710869224.XA CN201710869224A CN107800521A CN 107800521 A CN107800521 A CN 107800521A CN 201710869224 A CN201710869224 A CN 201710869224A CN 107800521 A CN107800521 A CN 107800521A
- Authority
- CN
- China
- Prior art keywords
- data
- server
- sent
- biography
- queue
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L1/00—Arrangements for detecting or preventing errors in the information received
- H04L1/22—Arrangements for detecting or preventing errors in the information received using redundant apparatus to increase reliability
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/16—Error detection or correction of the data by redundancy in hardware
- G06F11/20—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
- G06F11/202—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where processing functionality is redundant
- G06F11/2023—Failover techniques
- G06F11/2028—Failover techniques eliminating a faulty processor or activating a spare
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/16—Error detection or correction of the data by redundancy in hardware
- G06F11/20—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
- G06F11/2053—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where persistent mass storage functionality or persistent mass storage control functionality is redundant
- G06F11/2094—Redundant storage or storage space
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a kind of data acquisition storage method of highly reliable free of losses, after server receives the network data that client is sent, data are inserted into message transmit queue tail of the queue first, and notification database storage thread, then network data is stored network data according to mass memory expense cache algorithm；Database purchase thread reads data from message transmit queue team head, and data are inserted into background data base；Main adopt passes server and from adopting the viability that passes and heartbeat message is periodically sent between server and has detected master server.The present invention can be in the environment of high flow capacity and the huge flow of burst, by data high stable, highly reliably it is sent to server end module, ensure simultaneously under abnormal conditions in server end primary module and server end slave module handoff procedure, by gathered data it is lossless be forwarded to background data base.
Description
Technical field
The present invention relates to information gathering and analysis technical field, the data acquisition storage of specifically a kind of highly reliable free of losses
Method.
Background technology
In data collecting system running, it is necessary to by under the information records such as the parameter of system operation, state, for will
To analyze the state of system.Military's data collecting system runs on field airborne equipment, mainly runs field, extraneous ring
Border is severe, not only needs to face the various complex environments of various natures, and because system equipment is in movement environment, it is also necessary to
In face of the various emergency situations such as jolt.
Ensure the high reliability of data acquisition using the technology of dual-host backup at present, dual-host backup includes gatherer process
Backup and the backup of back-end data, current background DB Backup technology is ripe, only need to utilize rational database skill
Art, it is possible to reach the backup purpose of database engine and database purchase；And gatherer process backup is due to each system application
Purpose is different, and design at present is relatively fewer using the framework of double collection hot backup systems, is especially also not carried out in double copies
In acquisition system, method that system acquisition data can nondestructively switch.
Chinese patent CN201610959821.7 discloses a kind of integrated highly reliable high-precision Long-term Monitoring Systems of design,
Including two changes of current gather and edit unit, two integrated controllers, interchanger, two monitoring computers, solve system equipment composition
Complicated, the problem of precision is low, long-term reliability is insufficient, the function of highly reliable system, high accuracy and long term monitoring is realized, should
Invention reaches highly reliable purpose, but the system mainly reaches highly reliable using hardware backup using dual-host backup on hardware
Purpose, do not solve the data loss problem during Two-channel switching.
The content of the invention
It is an object of the invention to provide a kind of data acquisition storage side of high, good reliability the highly reliable free of losses of precision
Method, to solve the problems mentioned in the above background technology.
To achieve the above object, the present invention provides following technical scheme：
A kind of data acquisition storage method of highly reliable free of losses, is comprised the following steps that：
(1) activation system, after server receives the network data that client is sent, data are inserted send data first
Buffer queue tail of the queue, and notification database storage thread, then by network data according to mass memory expense cache algorithm by net
Network data storage；
(2) and then notice data transmission blocks are sent, and data transmission blocks are that Infinite Cyclic sends processing thread, and connection is worked as
Preceding master adopts biography server, and network data is sent using asynchronous non-blocking fashion；
(3) when send have data in data buffer storage queue when, read transmittable data from data buffer storage queue team head is sent,
The asynchronous call Basic API function provided by operating system, gives data to underlying operating system, meanwhile, by number after transmission
Bottom asynchronous call function is passed to according to processing module, data block is sent to server by underlying operating system；It will then send
Data block insertion is sending data block queue, and underlying operating system will be called after sending at data after data are sent completely
Manage module；
(4) when send there is no data in data buffer storage queue when, then thread be in obstruction wait state；
(5) after the completion of data are sent, data processing module after sending is called by underlying operating system, according to operating system
Result to sending network data, makes different responses：When sending data success, returned from data queue is sent
The data block is received, and from data record buffer queue, marks the machine data to be successfully sent；When sending, data are unsuccessful
When, data are placed in and send data buffer storage queue team head, notify data transmission blocks, it is reconnected from adopting biography server,
And data are resend；
(6) data record buffer queue is detected, those have been sent into success, and send data of the time more than 5 minutes and delay
Deposit the recovery of message memory headroom.
As the further scheme of the present invention：Data processing module is call back function module after the transmission.
As the further scheme of the present invention：Server in the step (1) includes main adopt and passes server and passed from adopting
Server, when system starts, while start main adopt and pass server and from biography server is adopted, wherein main biography server of adopting provides data
Service is received, forwarded and inquired about, semidormancy state is then in from biography server is adopted.
As the further scheme of the present invention：When system starts, other services have been detected whether by heartbeat message first
Device system is present, and current server system is that current server system is in semidormancy state, no from biography server is adopted if having
Then, biography server is adopted based on system, there is provided normal service function；After system normal operation, serviced if main biography of adopting can not be received
The heartbeat message of device system, then start from data receiver, forwarding and the inquiry service function for adopting biography server, meanwhile, open
It is dynamic from adopt pass server function before, first detection is main to be adopted and passes server processes and whether there is, and main biography clothes are adopted if being killed in the presence of if
Be engaged in device process, make from adopt pass server switch to it is main adopt biography server, send warning message, notify attendant to exclude main to adopt biography clothes
The failure of business device；After attendant fixes a breakdown, restart former master and adopt biography server, original master adopts biography startup of server will be according to upper
State system start method startup, it is former to pass server still as from adopting biography server from adopting, and it is in standby semidormancy state.
As the further scheme of the present invention：The semidormancy state is to send heartbeat per 20s between principal and subordinate's server to disappear
Breath, the viability of detection service device.
As further scheme of the invention：When transmission data are unsuccessful in the step (5), main biography server of adopting is delayed
Machine, from adopt pass server enable, client and from adopt pass server establish connection after, from adopt pass server obtained from data base querying
Last stored data is obtained, and returns to client message, client resends and finally deposited from data record buffer queue
Enter all data after database message.
Compared with prior art, the beneficial effects of the invention are as follows：
The present invention by data high stable, can highly reliably be sent in the environment of high flow capacity and the huge flow of burst
To server end module, while ensure under the abnormal conditions in server end primary module and server end slave module handoff procedure,
By gathered data it is lossless be forwarded to background data base.
Brief description of the drawings
Fig. 1 is the method schematic diagram of the present invention.
Embodiment
The technical scheme of this patent is described in more detail with reference to embodiment.
Referring to Fig. 1, a kind of data acquisition storage method of highly reliable free of losses, is comprised the following steps that：
(1) activation system, after server receives the network data that client is sent, data are inserted send data first
Buffer queue tail of the queue, and notification database storage thread, then by network data according to mass memory expense cache algorithm by net
Network data storage；
(2) and then notice data transmission blocks are sent, and data transmission blocks are that Infinite Cyclic sends processing thread, and connection is worked as
Preceding master adopts biography server, and network data is sent using asynchronous non-blocking fashion；
(3) when send have data in data buffer storage queue when, read transmittable data from data buffer storage queue team head is sent,
The asynchronous call Basic API function provided by operating system, gives data to underlying operating system, meanwhile, by number after transmission
Bottom asynchronous call function is passed to according to processing module, data block is sent to server by underlying operating system；It will then send
Data block insertion is sending data block queue, and underlying operating system will be called after sending at data after data are sent completely
Manage module；
(4) when send there is no data in data buffer storage queue when, then thread be in obstruction wait state；
(5) after the completion of data are sent, data processing module after sending is called by underlying operating system, according to operating system
Result to sending network data, makes different responses：When sending data success, returned from data queue is sent
The data block is received, and from data record buffer queue, marks the machine data to be successfully sent；When sending, data are unsuccessful
When, data are placed in and send data buffer storage queue team head, notify data transmission blocks, it is reconnected from adopting biography server,
And data are resend；
(6) data record buffer queue is detected, those have been sent into success, and send data of the time more than 5 minutes and delay
Deposit the recovery of message memory headroom.All message data of server buffer in the present invention, according to data cached offer efficiently inspection
Rope and random challenge function.
Data processing module is call back function module after the transmission.
To ensure the stability and high reliability of system, using the hot backup system mechanism of master-slave mode, in the step (1)
Server include main adopt and pass server and from adopting biography server, when system starts, while start main adopt and pass server and from adopting
Server is passed, wherein master, which adopts, passes server offer data receiver, forwarding and inquiry service, semidormancy is then in from biography server is adopted
State.
System starts mainly comprising the preliminary starting algorithm of system and by switching to master server from server semidormancy state
Algorithm two parts, are described as follows respectively：
The starting algorithm of server：After system starts, other server systems have been detected whether by heartbeat message first
In the presence of current server system is that current server system is in semidormancy state, otherwise, is from biography server is adopted if having
Biography server is adopted based on system, there is provided normal service function.After starting as main system, start reading database thread, retrieval
The message data of database has been inserted, has built the efficient data block storage organization based on 4 dimension chained lists.
Dormancy switchs to master server algorithm from server：After system starts, if can not receive main adopt passes server system
Heartbeat message, then start from data receiver, forwarding and the inquiry service function for adopting biography server, meanwhile, to prevent master from adopting
Pass server be in pseudo- resting state (not sending heartbeat message, other parts function is in normal condition), start from adopt biography
Before the function of server, the main biography server processes of adopting of first detection whether there is, and biography server processes are adopted if killing master in the presence of if,
Make from adopt pass server switch to it is main adopt biography server, start reading database thread, the message data of database has been inserted in retrieval,
The efficient data block storage organization based on 4 dimension chained lists is built, sends warning message, notifies attendant to exclude master and adopt biography server
Failure；After attendant fixes a breakdown, restart former master and adopt biography server, original master adopts biography startup of server will be according to above-mentioned system
The startup method of uniting starts, former to pass server still as from adopting biography server from adopting, and is in standby semidormancy state.
The semidormancy state is to send heartbeat message, the viability of detection service device between principal and subordinate's server per 20s.By
Heartbeat message frequency is 20s between principal and subordinate's server, and the switching of principal and subordinate's server can not possibly be set more than 2 minutes for this
The data before being cached 5 minutes in buffer queue after transmission are counted, therefore above-mentioned mechanism ensure that, server is in principal and subordinate's server
The data lost in handoff procedure can be resend.
Send in the step (5) data it is unsuccessful when, main adopt passes server and delayed machine, and calls this after Server Restart
Module, after master server is delayed, it is understood that there may be received by master server, but because the machine of delaying is without being stored into database clothes
The data of business device, at this moment from Server Restart, client and after server establishes connection, are obtained from server from data base querying
Last stored data is obtained, and returns to client message, client resends and finally deposited from data record buffer queue
Enter all data after database message.
The present invention by data high stable, can highly reliably be sent in the environment of high flow capacity and the huge flow of burst
To server end module, while ensure under the abnormal conditions in server end primary module and server end slave module handoff procedure,
By gathered data it is lossless be forwarded to background data base.
The better embodiment of this patent is explained in detail above, but this patent is not limited to above-mentioned embodiment party
Formula, can also be on the premise of this patent objective not be departed from one skilled in the relevant art's possessed knowledge
Various changes can be made.
Claims (6)
1. a kind of data acquisition storage method of highly reliable free of losses, it is characterised in that comprise the following steps that：
(1) activation system, after server receives the network data that client is sent, data are inserted send data buffer storage first
Queue tail of the queue, and notification database storage thread, then by network data according to mass memory expense cache algorithm by network number
According to storage；
(2) and then notice data transmission blocks are sent, and data transmission blocks are that Infinite Cyclic sends processing thread, are connected current main
Biography server is adopted, network data is sent using asynchronous non-blocking fashion；
(3) when send have data in data buffer storage queue when, read transmittable data from data buffer storage queue team head is sent, pass through
The asynchronous call Basic API function that operating system provides, gives data to underlying operating system, meanwhile, at data after transmission
Reason module passes to bottom asynchronous call function, and data block is sent into server by underlying operating system；Data will then be sent
Block insertion is sending data block queue, and underlying operating system will call data processing mould after sending after data are sent completely
Block；
(4) when send there is no data in data buffer storage queue when, then thread be in obstruction wait state；
(5) after the completion of data are sent, data processing module after sending is called by underlying operating system, according to operating system to hair
The result of SCN Space Cable Network data, make different responses：, should from data queue's recovery is sent when sending data success
Data block, and from data record buffer queue, mark the machine data to be successfully sent；, will when transmission data are unsuccessful
Data, which are placed in, sends data buffer storage queue team head, notifies data transmission blocks, it is reconnected from adopting biography server, and by number
According to resending；
(6) data record buffer queue is detected, those have been sent into success, and send data buffer storage of the time more than 5 minutes and disappear
Cease memory headroom recovery.
2. the data acquisition storage method of highly reliable free of losses according to claim 1, it is characterised in that after the transmission
Data processing module is call back function module.
3. the data acquisition storage method of highly reliable free of losses according to claim 1, it is characterised in that the step
(1) server in includes main adopt and passes server and from adopting biography server, when system starts, while start it is main adopt pass server and
From biography server is adopted, wherein main adopt passes server offer data receiver, forwarding and inquiry service, server is passed then in partly from adopting
Resting state.
4. the data acquisition storage method of highly reliable free of losses according to claim 3, it is characterised in that system starts
When, detected whether first by heartbeat message with the presence of other server systems, current server system is to pass from adopting if having
Server, current server system are in semidormancy state, otherwise, based on system adopt biography server, there is provided normally service work(
Energy；After system normal operation, if the main heartbeat message adopted and pass server system can not be received, start from the number for adopting biography server
According to reception, forwarding and inquiry service function, meanwhile, start before the function of adopting biography server, first detection master adopts biography clothes
Business device process whether there is, if in the presence of if kill master adopt biography server processes, make from adopt pass server switch to master adopt biography server,
Warning message is sent, notifies attendant to exclude the main failure adopted and pass server；After attendant fixes a breakdown, former master is restarted
Adopt biography server, original is main to adopt and pass startup of server and will start method startup according to said system, former from adopting biography server still conduct
From adopting biography server, and it is in standby semidormancy state.
5. the data acquisition storage method of highly reliable free of losses according to claim 3, it is characterised in that the semidormancy
State is to send heartbeat message, the viability of detection service device between principal and subordinate's server per 20s.
6. the data acquisition storage method of highly reliable free of losses according to claim 1, it is characterised in that the step
(5) sent in data it is unsuccessful when, main adopt passes server and delayed machine, passes server from adopting and enables, client and from adopting biography server
After establishing connection, last stored data is obtained from data base querying from biography server is adopted, and returns to client message, client
End resends all data after last deposit database message from data record buffer queue.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710869224.XA CN107800521A (en) | 2017-09-23 | 2017-09-23 | A kind of data acquisition storage method of highly reliable free of losses |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710869224.XA CN107800521A (en) | 2017-09-23 | 2017-09-23 | A kind of data acquisition storage method of highly reliable free of losses |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN107800521A true CN107800521A (en) | 2018-03-13 |
Family
ID=61531945
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710869224.XA Pending CN107800521A (en) | 2017-09-23 | 2017-09-23 | A kind of data acquisition storage method of highly reliable free of losses |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107800521A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2019200671A1 (en) * | 2018-04-20 | 2019-10-24 | 北京全路通信信号研究设计院集团有限公司 | Master/standby machine data synchronization method for railway centralized traffic control system |
| CN112685008A (en) * | 2020-11-30 | 2021-04-20 | 上海赫千电子科技有限公司 | Service failure control method adopting service-oriented architecture based on AUTOSAR |
| CN112788113A (en) * | 2020-12-30 | 2021-05-11 | 成都中嵌自动化工程有限公司 | Data communication system and method capable of breakpoint continuous transmission |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1964560A (en) * | 2005-11-07 | 2007-05-16 | 华为技术有限公司 | A method to prevent data loss in hand-over of cell and its device |
| CN101776711A (en) * | 2010-01-19 | 2010-07-14 | 华北电网有限公司 | Electric energy metering system substation |
| CN102868560A (en) * | 2012-09-28 | 2013-01-09 | 南京恩瑞特实业有限公司 | System and method for realizing hot standby of servers |
| CN204993390U (en) * | 2015-10-07 | 2016-01-20 | 杭州锐达数字技术有限公司 | A two transmission system for data collection station |
| CN206100062U (en) * | 2016-08-31 | 2017-04-12 | 常熟米豆智能设备科技有限公司 | Based on incessant communication device of car networking systems real -time data |
-
2017
- 2017-09-23 CN CN201710869224.XA patent/CN107800521A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1964560A (en) * | 2005-11-07 | 2007-05-16 | 华为技术有限公司 | A method to prevent data loss in hand-over of cell and its device |
| CN101776711A (en) * | 2010-01-19 | 2010-07-14 | 华北电网有限公司 | Electric energy metering system substation |
| CN102868560A (en) * | 2012-09-28 | 2013-01-09 | 南京恩瑞特实业有限公司 | System and method for realizing hot standby of servers |
| CN204993390U (en) * | 2015-10-07 | 2016-01-20 | 杭州锐达数字技术有限公司 | A two transmission system for data collection station |
| CN206100062U (en) * | 2016-08-31 | 2017-04-12 | 常熟米豆智能设备科技有限公司 | Based on incessant communication device of car networking systems real -time data |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2019200671A1 (en) * | 2018-04-20 | 2019-10-24 | 北京全路通信信号研究设计院集团有限公司 | Master/standby machine data synchronization method for railway centralized traffic control system |
| CN112685008A (en) * | 2020-11-30 | 2021-04-20 | 上海赫千电子科技有限公司 | Service failure control method adopting service-oriented architecture based on AUTOSAR |
| CN112788113A (en) * | 2020-12-30 | 2021-05-11 | 成都中嵌自动化工程有限公司 | Data communication system and method capable of breakpoint continuous transmission |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| KR101494143B1 (en) | Radio resource optimization method, device and system | |
| CN105991412B (en) | Information push method and device | |
| CN100417087C (en) | Alarm managing method for SNMP protocol network managing system | |
| CN1901568B (en) | Method for realizing historical property collection in network managing system | |
| CN106202075A (en) | A kind of method and device of data base's active-standby switch | |
| CN107479514B (en) | A kind of industry big data process control data acquisition and Transmission system and implementation method | |
| US20090177743A1 (en) | Device, Method and Computer Program Product for Cluster Based Conferencing | |
| US11792098B2 (en) | Link detection method and system | |
| CN105391573A (en) | Data acquisition system, data analysis system and monitoring analysis system based on intelligent terminal | |
| CN108347463A (en) | A kind of terminal document method for uploading | |
| CN102137415A (en) | RLF (radio link failure) data transmitting method and device | |
| CN107800521A (en) | A kind of data acquisition storage method of highly reliable free of losses | |
| CN108063832B (en) | Cloud storage system and storage method thereof | |
| CN110661705B (en) | Hardware network switching engine and network fault processing system and method | |
| WO2017220021A1 (en) | Short message processing method and apparatus | |
| CN104639613B (en) | Implementation method based on the mobile data synchronization middleware for improving procotol | |
| CN113472858A (en) | Buried point data processing method and device and electronic equipment | |
| CN103096253A (en) | System and method preventing incoming call request or message request from being lost | |
| CN106357478B (en) | A kind of server cluster monitoring method and system | |
| US20140004813A1 (en) | Emergency Call Processing Method and Related Apparatus and System | |
| CN102780570A (en) | Achieving method and system for management of cloud computing devices | |
| CN111787039A (en) | System and method for monitoring pathological parameters | |
| CN103281209B (en) | The processing method of a kind of warning information and equipment | |
| CN113485176A (en) | Vehicle data acquisition, caching and retransmitting method and remote monitoring terminal | |
| CN106685697B (en) | Method and system for recovering and processing abnormal marginal message data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20180313 |
|
| WD01 | Invention patent application deemed withdrawn after publication |