CN111104233A - Data acquisition method and device, computer equipment and computer readable storage medium - Google Patents
Data acquisition method and device, computer equipment and computer readable storage medium Download PDFInfo
- Publication number
- CN111104233A CN111104233A CN201911146802.2A CN201911146802A CN111104233A CN 111104233 A CN111104233 A CN 111104233A CN 201911146802 A CN201911146802 A CN 201911146802A CN 111104233 A CN111104233 A CN 111104233A
- Authority
- CN
- China
- Prior art keywords
- data
- target system
- preset
- memory
- data acquisition
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/545—Interprogram communication where tasks reside in different layers, e.g. user- and kernel-space
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1847—File system types specifically adapted to static storage, e.g. adapted to flash memory or SSD
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/252—Integrating or interfacing systems involving database management systems between a Database Management System and a front-end application
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/548—Queue
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Hardware Redundancy (AREA)
- Debugging And Monitoring (AREA)
Abstract
The embodiment of the invention provides a data acquisition method, a data acquisition device, computer equipment and a computer readable storage medium. The method comprises the steps of obtaining data of an external data source, storing the data into a memory, judging whether a target system corresponding to a data receiver meets a preset survival condition, if the preset survival condition is met, forwarding the data to the target system, if the preset survival condition is not met, forwarding the data to a preset storage for temporary storage, and forwarding the data when a subsequent target system is available so as to achieve data acquisition. By the method that the data are forwarded to the preset memory for temporary storage when the target system is judged to be unavailable, and the data are forwarded to the target system when the subsequent target system is available, high availability can be achieved under the condition that high-performance data collection is guaranteed, and particularly for collection of mass data such as big data, the high performance, the high availability and the stability of the whole data collection link are improved.
Description
Technical Field
The present invention relates to the field of data acquisition technologies, and in particular, to a data acquisition method and apparatus, a computer device, and a computer-readable storage medium.
Background
In a conventional data acquisition, for example, in a scheme of acquiring mass data such as particularly large data, a core component includes a real-time acquirer, the real-time acquirer includes a data reading component and a data transmitting component, and the data transmitting component is responsible for transmitting data to a target system, for example, a file system, a NoSql database, a message queue, and other target systems corresponding to respective data receivers. Once the target system is unavailable, the data sending component may not find the sending target, causing the real-time collector to be unavailable, resulting in data extrusion, loss, and unavailability of the entire collection link. Therefore, in the conventional technology, the real-time collector of the data collection link cannot achieve the really high availability.
Disclosure of Invention
The embodiment of the invention provides a data acquisition method, a data acquisition device, computer equipment and a computer readable storage medium, which can solve the problem that a real-time acquisition device in the prior art is low in high availability.
In a first aspect, an embodiment of the present invention provides a data acquisition method, including: acquiring data of an external data source, and storing the data into a memory; judging whether a target system corresponding to a data receiver meets a preset survival condition, wherein the target system is a preset component for receiving the acquired data and is included in a data acquisition system, and the target system survival refers to a state that the target system normally receives the data; if the target system meets the preset survival condition, reading the data from the memory, and forwarding the data to the target system to realize data acquisition; and if the target system does not meet the preset survival condition, reading the data from the memory, forwarding the data to a preset storage for temporary storage, and forwarding the data to the target system when the subsequent target system is available so as to realize the data acquisition.
In a second aspect, an embodiment of the present invention further provides a data acquisition apparatus, including: the device comprises an acquisition unit, a storage unit and a processing unit, wherein the acquisition unit is used for acquiring data of an external data source and storing the data into the memory; the system comprises a first judging unit, a second judging unit and a third judging unit, wherein the first judging unit is used for judging whether a target system corresponding to a data receiver meets a preset survival condition, the target system refers to a preset component contained in a data acquisition system and used for receiving acquired data, and the target system survival refers to a state that the target system is in a normal data receiving state; the first forwarding unit is used for reading the data from the memory and forwarding the data to the target system to realize data acquisition if the target system meets the preset survival condition; and the temporary storage unit is used for reading the data from the memory and forwarding the data to a preset storage for temporary storage if the target system does not meet the preset survival condition, and forwarding the data to the target system when the subsequent target system is available so as to realize data acquisition.
In a third aspect, an embodiment of the present invention further provides a computer device, which includes a memory and a processor, where the memory stores a computer program, and the processor implements the data acquisition method when executing the computer program.
In a fourth aspect, the embodiment of the present invention further provides a computer-readable storage medium, where a computer program is stored, and when the computer program is executed by a processor, the computer program causes the processor to execute the data acquisition method.
The embodiment of the invention provides a data acquisition method, a data acquisition device, computer equipment and a computer readable storage medium. When data acquisition is realized, data of an external data source is acquired, the data is stored in a memory, whether a target system corresponding to a data receiver meets a preset survival condition or not is judged, if the target system meets the preset survival condition, the data is read from the memory and forwarded to the target system to realize the data acquisition, if the target system does not meet the preset survival condition, the data is read from the memory and forwarded to a preset storage for temporary storage, and when the target system is available subsequently, the data is forwarded to the target system to realize the data acquisition. The data are forwarded to the preset storage for temporary storage by the data real-time collector when the target system is judged to be unavailable, and the data are forwarded to the target system when the subsequent target system is available, so that the data of the whole acquisition link cannot be extruded, lost and unavailable due to the fact that the target system is unavailable and the transmission target cannot be found by the real-time collector, the read data are stored in the memory, high availability can be achieved under the condition that high-performance acquisition of the data is guaranteed, and particularly, the high performance, high availability and stability of the whole data acquisition link are improved for the acquisition of mass data such as big data.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
Fig. 1 is a schematic flow chart of a data acquisition method according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of an application environment architecture of a data acquisition method according to an embodiment of the present invention;
FIG. 3 is another schematic flow chart of a data acquisition method according to an embodiment of the present invention;
FIG. 4 is a schematic block diagram of a data acquisition device provided by an embodiment of the present invention; and
FIG. 5 is a schematic block diagram of a computer device provided by an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It will be understood that the terms "comprises" and/or "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
Referring to fig. 1 and fig. 2, fig. 1 is a schematic flowchart of a data acquisition method according to an embodiment of the present invention, and fig. 2 is a schematic diagram of an application environment architecture of the data acquisition method according to the embodiment of the present invention. As shown in fig. 1, the method comprises the following steps S101-S104:
s101, obtaining data of an external data source, and storing the data into a memory.
Specifically, in data acquisition, there are two general ways to acquire data from an external data source, one way is to actively go to the external data source for reading, and the other way is to receive data sent from the external data source, that is, to read data from the external data source or to receive the data sent from the external data source. In data acquisition, a core component includes a data real-time acquirer, please refer to fig. 2, as shown in fig. 2, taking reading data of an external data source as an example, in an embodiment of the present invention, the data real-time acquirer includes a data reading component (i.e., a Reader component), an intermediate Storage component (i.e., a Storage component), and a sending component (i.e., a Sender component), and in addition to the data real-time acquirer, the data acquisition system further includes a preset Storage (e.g., a disk file), where the Reader component is responsible for reading data, and a data reading mode may be reading a file, reading data in a message queue, and writing the data into the Storage component, and the Storage stores data using a memory queue. The data real-time acquisition unit reads data from an external data source according to a certain period, the data is stored in the memory queue after being read, namely the data of the external data source is read and stored in the memory.
S102, judging whether a target system corresponding to a data receiver meets a preset survival condition, wherein the target system refers to a preset component contained in a data acquisition system and used for receiving acquired data, and the target system survival refers to a state that the target system normally receives data.
Specifically, the real-time data collector determines, through a sending component (i.e., a Sender component) and a target system corresponding to the data receiver, for example, a downstream system in the data collection system, to determine whether the target system meets a preset survival condition, so as to determine whether the target system is alive, that is, determine whether the target system is in a normal connection state and can normally receive data. The target system survival refers to a state that the target system normally receives data, wherein the target system refers to a preset component included in a data acquisition system for receiving the acquired data, and for example, the target system includes a file system, a NoSql database, a message queue and other data receivers.
S103, if the target system meets the preset survival condition, reading the data from the memory, and forwarding the data to the target system to realize data acquisition.
Specifically, if the real-time data collector judges that the target system meets the preset survival condition, that is, the target system can normally receive data, the data is read from the memory, and the data is forwarded to the target system, so as to realize data collection. Referring to fig. 2, as shown in fig. 2, the external data read by the real-time data collector is forwarded to the target system through the path 1 described by the black line, so as to achieve the data collection, and particularly for the collection of mass data such as big data, the data is stored in the memory, so that the high performance of the real-time data collector during the data collection processing can be achieved.
And S104, if the target system does not meet the preset survival condition, reading the data from the memory, forwarding the data to a preset storage for temporary storage, and forwarding the data to the target system when the subsequent target system is available so as to realize data acquisition.
In particular, in order to ensure high performance of the data real-time collector, the Storage component stores data in a memory queue, if the Sender component does not send out the data in time, the memory will burst due to excessive data, causing the memory to overflow, therefore, once the real-time data collector finds that the target system is unavailable, the Sender component writes the data acquired from the memory into the preset memory, for example, a disk file, and switches the current state of the Sender component in the data real-time collector to be unavailable for the target system, namely, if the real-time data collector judges that the target system does not meet the preset survival condition, namely, the target system cannot normally receive data, the data is read from the memory, and forwarding the data to a preset memory for temporary storage, and forwarding the data to a target system to realize data acquisition when a subsequent target system is available. Referring to fig. 2, as shown in fig. 2, the data real-time collector determines that the target system does not satisfy the preset survival condition, and after the read external data passes through the path 1 described by the black line to the data sending component, the data sending component forwards the read external data to a preset memory (e.g., a disk file) for temporary storage through the path 2 described by the dotted line, and when the subsequent target system is available, the data is forwarded to the target system from the preset memory through the path 3(1) described by the dotted line, so as to implement the data collection, thereby implementing the high availability of the data real-time collector.
In a general big data real-time acquisition scheme, a real-time acquisition device can not guarantee zero loss of data on the premise of guaranteeing high performance, the acquired data can be selected to be cached in a memory or fall to the disk, if the data is cached in the memory, the high performance of data acquisition and transmission can be guaranteed, however, once the situation that a target system is unavailable is met, the acquisition device makes mistakes, so that the data is lost, and if the data falls to the local disk, the data can be guaranteed not to be lost when a program makes mistakes or a server crashes, but once the data falls to the disk, the performance is inevitably reduced. In the embodiment of the invention, the data is cached in the memory and combined with the preset memory corresponding to the disk, when the target system can normally receive the data, the high performance of data acquisition and transmission is ensured by caching in the memory, and when the target system can not normally receive the data, the data is temporarily stored in the preset memory for temporary storage, so that the high availability of the data acquisition and transmission can be ensured, the data loss is avoided, and the high performance and the high availability of the data acquisition and transmission are realized.
When data acquisition is realized, data of an external data source is acquired, the data is stored in a memory, whether a target system corresponding to a data receiver meets a preset survival condition or not is judged, if the target system meets the preset survival condition, the data is read from the memory and forwarded to the target system to realize the data acquisition, if the target system does not meet the preset survival condition, the data is read from the memory and forwarded to a preset storage for temporary storage, and when the target system is available subsequently, the data is forwarded to the target system to realize the data acquisition. The data are forwarded to the preset storage for temporary storage by the data real-time collector when the target system is judged to be unavailable, and the data are forwarded to the target system when the subsequent target system is available, so that the data of the whole acquisition link cannot be extruded, lost and unavailable due to the fact that the target system is unavailable and the transmission target cannot be found by the real-time collector, the read data are stored in the memory, high availability can be achieved under the condition that high-performance acquisition of the data is guaranteed, and particularly, the high performance, high availability and stability of the whole data acquisition link are improved for the acquisition of mass data such as big data.
In an embodiment, the step of determining whether the target system corresponding to the data receiver satisfies the preset survival condition includes:
according to a preset period, confirming a target system corresponding to a data receiver in a heartbeat mode to judge whether the target system is alive or not;
if the connection with the target system is judged to be abnormal according to the heartbeat feedback, judging that the target system does not survive;
and if the target system is judged not to survive and the times of confirmation with the target system exceed the preset times, judging that the target system does not meet the preset survival condition.
The heartbeat mechanism is a mechanism that regularly sends a self-defined structure (heartbeat packet) to let the other party know that the other party is still alive to ensure the validity of the connection, for example, the heartbeat mechanism in the TCP protocol, the TCP protocol sends an empty data packet (KeepAlive probe) with an ACK flag to the other party, and after the other party receives the ACK packet, if all connections are normal, an ACK should be replied; if the connection is wrong (for example, the opposite side is restarted, the connection state is lost), a RST should be replied; if the opposite side does not reply, the server sends ACK again at intervals, and if a plurality of continuous packets are ignored, the connection is disconnected.
Specifically, a data sending component in the real-time data collector confirms with a target system corresponding to a data receiver in a heartbeat manner according to a preset period to judge whether the target system is alive or not, if the target system is normally connected according to heartbeat feedback judgment, the target system is judged to be alive, the data can be continuously sent to the target system so that the target system can normally receive the data, if the connection with the target system is judged to be abnormal according to the heartbeat feedback, the target system is judged not to survive, the target system cannot normally receive data, and further, if it is determined that the target system does not survive, and the number of times of confirmation with the target system exceeds a preset number of times, the target system is judged not to meet the preset survival condition, and the connection is disconnected and data cannot be sent to the target system. Referring to fig. 2, the Sender module will periodically perform heartbeat confirmation with the target system to determine whether the target system is alive, retry after failing to send data, and after repeating for n times, fail to send the data to the target system, and determine that the target system is unavailable. And determining the state of the target system by repeated confirmation, so that the accuracy of judging the state of the target system is improved.
Referring to fig. 3, fig. 3 is another schematic flow chart of a data acquisition method according to an embodiment of the present invention. Steps S301 to S304 in fig. 3 correspond to steps S101 to S104 in fig. 1, respectively, and are not described again here. As shown in fig. 3, in this embodiment, after the step of reading the data from the memory and forwarding the data to a preset memory for temporary storage, the method further includes:
s305, continuously judging whether the target system meets the preset survival condition or not according to the preset period;
s306, if the target system meets the preset survival condition, reading the data from the preset storage, sending the data to the target system, and starting to continuously add the data contained in the memory to the preset storage.
Specifically, a data sending component in the real-time data collector continuously judges whether the target system meets the preset survival condition according to the preset period, if the target system meets the preset survival condition, the data is read from a preset storage (for example, a magnetic disk), the data is sent to the target system, the data contained in the memory is continuously added to the preset storage, and if the target system does not meet the preset survival condition, the step of continuously judging whether the target system meets the preset survival condition according to the preset period is returned to be executed, so that the state of the target system is continuously monitored. Referring to fig. 2, the Sender module will continue to periodically send heartbeats to the target system, try to connect, if the target system is found to be available, switch the current state of the Sender module to the target system, and the Sender module will switch from reading data from the memory to storing in the preset storage via path 2 to reading data in the disk file from the disk, and send the data in the disk file to the target system via path 3 (1). At this time, the Sender component reads data from the disk file and sends the read data to the target system, but does not read data from the memory (i.e. Storage component), and meanwhile, the memory also receives data read by the Reader component from an external data source in real time, so as to avoid the memory from being burst due to excessive data and causing memory overflow, an additional component (i.e. repeater) is started to read data in a memory queue from the memory and add the data to the disk through a path 3(2), so that data acquisition is realized through a data acquisition flow correspondingly described by paths 3(2) and 3(1), and thus, the data acquisition and transmission are high in performance and high in availability, if the target system still does not meet the preset survival condition, the target system still is not available, and the step of continuously judging whether the target system meets the preset survival condition according to the preset period is returned to be executed, and continuing to monitor the state of the target system until the target system meets the preset survival condition or the data acquisition system finishes the service.
Further, the method further comprises: the available memory space of a preset size is maintained.
The step of maintaining the available memory space of the preset size includes: and keeping the preset proportion of the data in the memory in the available memory.
Specifically, when the repeater component is started to read data in the memory queue and add the data to the preset memory, a certain spare memory is reserved in the memory to avoid data loss caused by memory overflow, so as to ensure the safety of the data, for example, the data in the memory is kept in 90% of the available memory, so that the accumulated temporary storage data is sent on the premise of ensuring the orderliness of the data.
Further, the step of sending the data to the target system comprises:
and sending the data to the target system in a zero copy mode.
The Zero-copy (Zero-copy) technique refers to that when a computer executes an operation, a CPU does not need to copy data from a memory to another specific area. This technique is typically used to save CPU cycles and memory bandwidth when transferring files over a network.
Specifically, please refer to fig. 2, if the Sender module in the data real-time collector determines that the target system is available, the current state of the Sender module is switched to the target system, and the Sender module switches to read the data in the disk file and sends the data to the target system with high performance through a zero-copy technology, so that the speed of adding the data from the memory to the preset storage is lower than the speed of reading the data from the preset storage by the data sending module, the efficiency of sending the data in the disk to the target system by the Sender module is improved, and the backlog data in the disk file can be forwarded as soon as possible.
In one embodiment, after the steps of reading the data from the preset storage, sending the data to the target system, and starting to continuously add the data contained in the memory to the preset storage, the method further includes:
s307, judging whether the data contained in the preset memory is sent to the target system or not;
s308, if the data contained in the preset memory is sent to the target system, the data read from the preset memory is switched to the data read from the memory, and the data read from the memory is forwarded to the target system.
Specifically, a sending component in the real-time collector judges whether sending of data contained in the preset storage to the target system is completed, if the sending of the data contained in the preset storage to the target system is completed, the data read from the preset storage is switched to the data read from the memory, the data read from the memory are forwarded to the target system, especially for collection of mass data such as big data, high-performance transmission of the data is achieved, and if the sending of the data stored in the preset storage to the target system is not completed, the step of judging whether sending of the data stored in the preset storage to the target system is completed is returned to be executed, so that the sending state of the data stored in the preset storage is continuously monitored. For example, the Replicator component is started to read data in the memory queue and add the data to the disk, and the data in the memory is kept in 90% of the available memory, and the backlog data is efficiently sent on the premise of ensuring the data order, and once the data in the disk file is read and sent, the Sender component is switched from reading the disk file to reading the memory data to continue reading the data from the memory through the path 1, so that the high performance of the real-time data collector is realized, and the high-performance transmission of the data is realized.
It should be noted that, the data acquisition methods described in the above embodiments may recombine the technical features included in different embodiments as needed to obtain a combined implementation, but all of them are within the protection scope of the present invention.
Referring to fig. 4, fig. 4 is a schematic block diagram of a data acquisition device according to an embodiment of the present invention. Corresponding to the data acquisition method, the embodiment of the invention also provides a data acquisition device. As shown in fig. 4, the data acquisition apparatus includes a unit for executing the data acquisition method, and the apparatus may be configured in a computer device such as a server. Specifically, referring to fig. 4, the data acquisition apparatus 400 includes an obtaining unit 401, a first determining unit 402, a first forwarding unit 403, and a temporary storage unit 404.
The acquiring unit 401 is configured to acquire data of an external data source and store the data in a memory;
a first determining unit 402, configured to determine whether a target system corresponding to a data receiving party meets a preset survival condition, where the target system refers to a preset component included in a data acquisition system for receiving acquired data, and the target system survival refers to a state where the target system is in a normal data receiving state;
a first forwarding unit 403, configured to, if the target system meets the preset survival condition, read the data from the memory, and forward the data to the target system, so as to implement the data acquisition;
a temporary storage unit 403, configured to read the data from the memory and forward the data to a preset storage for temporary storage if the target system does not meet the preset survival condition, and forward the data to the target system when the subsequent target system is available, so as to implement the data acquisition.
In one embodiment, the first judging unit 402 includes:
the judging subunit is used for confirming the target system corresponding to the data receiver in a heartbeat mode according to a preset period so as to judge whether the target system is alive or not;
the first judgment subunit is used for judging that the target system does not survive if the connection with the target system is judged to be abnormal according to the heartbeat feedback;
and the second judging subunit is used for judging that the target system does not meet the preset survival condition if the target system is judged not to survive and the times of confirmation with the target system exceed the preset times.
In one embodiment, the data acquisition apparatus 400 further comprises:
a second judging unit, configured to continuously judge whether the target system meets the preset survival condition according to the preset period;
and the sending unit is used for reading the data from the preset storage if the target system meets the preset survival condition, sending the data to the target system, and starting to continuously add the data contained in the memory to the preset storage.
In one embodiment, the data acquisition apparatus 400 further comprises:
a third judging unit, configured to judge whether transmission of the data included in the preset memory to the target system is completed;
and the switching unit is used for switching the reading of the data from the preset memory to the reading of the data from the memory and forwarding the data read from the memory to the target system if the data contained in the preset memory is sent to the target system completely.
In an embodiment, the first forwarding unit 403 is specifically configured to send the data to the target system in a zero-copy manner.
It should be noted that, as can be clearly understood by those skilled in the art, the specific implementation processes of the data acquisition device and each unit may refer to the corresponding descriptions in the foregoing method embodiments, and for convenience and brevity of description, no further description is provided herein.
Meanwhile, the division and connection modes of the units in the data acquisition device are only used for illustration, in other embodiments, the data acquisition device may be divided into different units as required, and the units in the data acquisition device may also adopt different connection sequences and modes to complete all or part of the functions of the data acquisition device.
The data acquisition means may be implemented in the form of a computer program which may be run on a computer device as shown in fig. 5.
Referring to fig. 5, fig. 5 is a schematic block diagram of a computer device according to an embodiment of the present invention. The computer device 500 may be a computer device such as a desktop computer or a server, or may be a component or part of another device.
Referring to fig. 5, the computer device 500 includes a processor 502, memory, and a network interface 505 connected by a system bus 501, where the memory may include a non-volatile storage medium 503 and an internal memory 504.
The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. The computer program 5032, when executed, causes the processor 502 to perform one of the data acquisition methods described above.
The processor 502 is used to provide computing and control capabilities to support the operation of the overall computer device 500.
The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503, and when the computer program 5032 is executed by the processor 502, the processor 502 can be enabled to perform a data collection method as described above.
The network interface 505 is used for network communication with other devices. Those skilled in the art will appreciate that the configuration shown in fig. 5 is a block diagram of only a portion of the configuration associated with aspects of the present invention and is not intended to limit the computing device 500 to which aspects of the present invention may be applied, and that a particular computing device 500 may include more or less components than those shown, or may combine certain components, or have a different arrangement of components. For example, in some embodiments, the computer device may only include a memory and a processor, and in such embodiments, the structures and functions of the memory and the processor are consistent with those of the embodiment shown in fig. 5, and are not described herein again.
Wherein the processor 502 is configured to run the computer program 5032 stored in the memory to implement the following steps: acquiring data of an external data source, and storing the data into a memory; judging whether a target system corresponding to a data receiver meets a preset survival condition, wherein the target system is a preset component for receiving the acquired data and is included in a data acquisition system, and the target system survival refers to a state that the target system normally receives the data; if the target system meets the preset survival condition, reading the data from the memory, and forwarding the data to the target system to realize data acquisition; and if the target system does not meet the preset survival condition, reading the data from the memory, forwarding the data to a preset storage for temporary storage, and forwarding the data to the target system when the subsequent target system is available so as to realize the data acquisition.
In an embodiment, when the processor 502 performs the step of determining whether the target system corresponding to the data receiver meets the preset survival condition, the following steps are specifically performed:
according to a preset period, confirming a target system corresponding to a data receiver in a heartbeat mode to judge whether the target system is alive or not;
if the connection with the target system is judged to be abnormal according to the heartbeat feedback, judging that the target system does not survive;
and if the target system is judged not to survive and the times of confirmation with the target system exceed the preset times, judging that the target system does not meet the preset survival condition.
In an embodiment, after the step of reading the data from the memory and forwarding the data to a preset memory for temporary storage is implemented by the processor 502, the following steps are further implemented:
continuously judging whether the target system meets the preset survival condition or not according to the preset period;
and if the target system meets the preset survival condition, reading the data from the preset storage, sending the data to the target system, and starting to continuously add the data contained in the memory to the preset storage.
In an embodiment, after the processor 502 performs the steps of reading the data from the preset storage, sending the data to the target system, and starting to continuously add the data contained in the memory to the preset storage, the following steps are further performed:
judging whether the data contained in the preset memory is sent to the target system or not;
and if the data contained in the preset memory is sent to the target system, switching the reading of the data from the preset memory to the reading of the data from the memory, and forwarding the data read from the memory to the target system.
In an embodiment, when the processor 502 implements the step of sending the data to the target system, the following steps are specifically implemented:
and sending the data to the target system in a zero copy mode.
It should be understood that, in the embodiment of the present invention, the Processor 502 may be a Central Processing Unit (CPU), and the Processor 502 may also be other general purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable gate arrays (FPGAs) or other Programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and the like. Wherein a general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
It will be understood by those skilled in the art that all or part of the processes in the method for implementing the above embodiments may be implemented by a computer program, and the computer program may be stored in a computer readable storage medium. The computer program is executed by at least one processor in the computer system to implement the flow steps of the embodiments of the method described above.
Accordingly, the present invention also provides a computer-readable storage medium. The computer readable storage medium may be a non-volatile computer readable storage medium, the computer readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform the steps of:
a computer program product which, when run on a computer, causes the computer to perform the steps of the data acquisition method described in the embodiments above.
The computer readable storage medium may be an internal storage unit of the aforementioned device, such as a hard disk or a memory of the device. The computer readable storage medium may also be an external storage device of the device, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), etc. provided on the device. Further, the computer-readable storage medium may also include both an internal storage unit and an external storage device of the apparatus.
It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described apparatuses, devices and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
The storage medium is an entity and non-transitory storage medium, and may be various entity storage media capable of storing computer programs, such as a usb disk, a removable hard disk, a Read-only memory (ROM), a magnetic disk, or an optical disk.
Those of ordinary skill in the art will appreciate that the elements and algorithm steps of the examples described in connection with the embodiments disclosed herein may be embodied in electronic hardware, computer software, or combinations of both, and that the components and steps of the examples have been described in a functional general in the foregoing description for the purpose of illustrating clearly the interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
In the embodiments provided in the present invention, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative. For example, the division of each unit is only one logic function division, and there may be another division manner in actual implementation. For example, various elements or components may be combined or may be integrated into another system, or some features may be omitted, or not implemented.
The steps in the method of the embodiment of the invention can be sequentially adjusted, combined and deleted according to actual needs. The units in the device of the embodiment of the invention can be merged, divided and deleted according to actual needs. In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a storage medium. Based on such understanding, the technical solution of the present invention essentially or partially contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing an electronic device (which may be a personal computer, a terminal, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention.
While the invention has been described with reference to specific embodiments, the invention is not limited thereto, and various equivalent modifications and substitutions can be easily made by those skilled in the art within the technical scope of the invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (10)
1. A method of data acquisition, comprising:
acquiring data of an external data source, and storing the data into a memory;
judging whether a target system corresponding to a data receiver meets a preset survival condition, wherein the target system is a preset component for receiving the acquired data and is included in a data acquisition system, and the target system survival refers to a state that the target system normally receives the data;
if the target system meets the preset survival condition, reading the data from the memory, and forwarding the data to the target system to realize data acquisition;
and if the target system does not meet the preset survival condition, reading the data from the memory, forwarding the data to a preset storage for temporary storage, and forwarding the data to the target system when the subsequent target system is available so as to realize the data acquisition.
2. The data collection method of claim 1, wherein the step of determining whether the target system corresponding to the data receiver satisfies a predetermined survival condition comprises:
according to a preset period, confirming a target system corresponding to a data receiver in a heartbeat mode to judge whether the target system is alive or not;
if the connection with the target system is judged to be abnormal according to the heartbeat feedback, judging that the target system does not survive;
and if the target system is judged not to survive and the times of confirmation with the target system exceed the preset times, judging that the target system does not meet the preset survival condition.
3. The data acquisition method according to claim 1 or 2, wherein after the step of reading the data from the memory and forwarding the data to a preset memory for temporary storage, the method further comprises:
continuously judging whether the target system meets the preset survival condition or not according to the preset period;
and if the target system meets the preset survival condition, reading the data from the preset storage, sending the data to the target system, and starting to continuously add the data contained in the memory to the preset storage.
4. The data acquisition method according to claim 3, wherein after the steps of reading the data from the preset memory, sending the data to the target system, and starting to continuously add the data contained in the memory to the preset memory, the method further comprises:
judging whether the data contained in the preset memory is sent to the target system or not;
and if the data contained in the preset memory is sent to the target system, switching the reading of the data from the preset memory to the reading of the data from the memory, and forwarding the data read from the memory to the target system.
5. The data acquisition method of claim 3, wherein the step of sending the data to the target system comprises:
and sending the data to the target system in a zero copy mode.
6. A data acquisition device, comprising:
the device comprises an acquisition unit, a storage unit and a processing unit, wherein the acquisition unit is used for acquiring data of an external data source and storing the data into the memory;
the system comprises a first judging unit, a second judging unit and a third judging unit, wherein the first judging unit is used for judging whether a target system corresponding to a data receiver meets a preset survival condition, the target system refers to a preset component contained in a data acquisition system and used for receiving acquired data, and the target system survival refers to a state that the target system is in a normal data receiving state;
the first forwarding unit is used for reading the data from the memory and forwarding the data to the target system to realize data acquisition if the target system meets the preset survival condition;
and the temporary storage unit is used for reading the data from the memory and forwarding the data to a preset storage for temporary storage if the target system does not meet the preset survival condition, and forwarding the data to the target system when the subsequent target system is available so as to realize data acquisition.
7. The data acquisition device according to claim 6, wherein the first judgment unit includes:
the judging subunit is used for confirming the target system corresponding to the data receiver in a heartbeat mode according to a preset period so as to judge whether the target system is alive or not;
the first judgment subunit is used for judging that the target system does not survive if the connection with the target system is judged to be abnormal according to the heartbeat feedback;
and the second judging subunit is used for judging that the target system does not meet the preset survival condition if the target system is judged not to survive and the times of confirmation with the target system exceed the preset times.
8. The data acquisition device of claim 6 or 7, wherein the mass data acquisition device further comprises:
a second judging unit, configured to continuously judge whether the target system meets the preset survival condition according to the preset period;
and the sending unit is used for reading the data from the preset storage if the target system meets the preset survival condition, sending the data to the target system, and starting to continuously add the data contained in the memory to the preset storage.
9. A computer device, comprising a memory and a processor coupled to the memory; the memory is used for storing a computer program; the processor is adapted to run a computer program stored in the memory to perform the steps of the data acquisition method according to any one of claims 1-5.
10. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program which, when executed by a processor, causes the processor to carry out the steps of the data acquisition method according to any one of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911146802.2A CN111104233B (en) | 2019-11-21 | 2019-11-21 | Data acquisition method, device, computer equipment and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911146802.2A CN111104233B (en) | 2019-11-21 | 2019-11-21 | Data acquisition method, device, computer equipment and computer readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111104233A true CN111104233A (en) | 2020-05-05 |
| CN111104233B CN111104233B (en) | 2023-09-26 |
Family
ID=70421674
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911146802.2A Active CN111104233B (en) | 2019-11-21 | 2019-11-21 | Data acquisition method, device, computer equipment and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111104233B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113783939A (en) * | 2021-08-20 | 2021-12-10 | 奇安信科技集团股份有限公司 | File transmission method and device, electronic equipment and storage medium |
| CN116541448A (en) * | 2023-05-10 | 2023-08-04 | 百应科技有限公司 | Data integration processing method and device based on SaaS |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107797825A (en) * | 2017-10-24 | 2018-03-13 | 努比亚技术有限公司 | A kind of data transmission method for uplink, terminal and computer-readable recording medium |
| CN109739921A (en) * | 2019-01-07 | 2019-05-10 | 北京云基数技术有限公司 | A kind of big data acquisition system and method |
| CN209373598U (en) * | 2019-02-21 | 2019-09-10 | 深圳市甲易科技有限公司 | Real time data and external data association process system |
| CN110362450A (en) * | 2019-07-16 | 2019-10-22 | 深圳市网心科技有限公司 | A kind of log data acquisition method, device and computer readable storage medium |
-
2019
- 2019-11-21 CN CN201911146802.2A patent/CN111104233B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107797825A (en) * | 2017-10-24 | 2018-03-13 | 努比亚技术有限公司 | A kind of data transmission method for uplink, terminal and computer-readable recording medium |
| CN109739921A (en) * | 2019-01-07 | 2019-05-10 | 北京云基数技术有限公司 | A kind of big data acquisition system and method |
| CN209373598U (en) * | 2019-02-21 | 2019-09-10 | 深圳市甲易科技有限公司 | Real time data and external data association process system |
| CN110362450A (en) * | 2019-07-16 | 2019-10-22 | 深圳市网心科技有限公司 | A kind of log data acquisition method, device and computer readable storage medium |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113783939A (en) * | 2021-08-20 | 2021-12-10 | 奇安信科技集团股份有限公司 | File transmission method and device, electronic equipment and storage medium |
| CN116541448A (en) * | 2023-05-10 | 2023-08-04 | 百应科技有限公司 | Data integration processing method and device based on SaaS |
| CN116541448B (en) * | 2023-05-10 | 2023-12-08 | 汉友科技有限公司 | Data integration processing method and device based on SaaS |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111104233B (en) | 2023-09-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109496401B (en) | Service takeover method, storage device and service takeover device | |
| CN111104233A (en) | Data acquisition method and device, computer equipment and computer readable storage medium | |
| CN109962827B (en) | Equipment link detection method, device, equipment and readable storage medium | |
| CN111031135B (en) | Message transmission method and device and electronic equipment | |
| CN109361625B (en) | Method, device and controller for checking forwarding table item | |
| CN112272291A (en) | Video storage method, device, management equipment and readable storage medium | |
| CN105487937A (en) | RDMA (Remote Direct Memory Access) implementation method and device | |
| CN108418859B (en) | Method and device for writing data | |
| CN115390546A (en) | Vehicle diagnostic communication method, device, equipment and medium | |
| CN111679925A (en) | Database fault processing method and device, computing equipment and storage medium | |
| CN113626235A (en) | Fault tolerance processing method and device and server | |
| CN112039835B (en) | Adaptive polling time estimation method, device, equipment and storage medium | |
| CN116670636A (en) | Data access method, device and storage medium | |
| CN110445666B (en) | Network quality detection method and device and server | |
| CN111970497A (en) | Video stream processing method and device, SDN controller and storage medium | |
| CN117014514A (en) | Scheduling method, device, equipment and medium of vehicle cloud communication gateway | |
| CN114095462B (en) | Fault-tolerant method and system for SRIO communication system of radar processor | |
| CN108121496B (en) | Data storage method, device and system | |
| JP6743724B2 (en) | Communication network and communication terminal | |
| CN114125064A (en) | Method and device for ensuring equipment positioning reliability, computer equipment and storage medium | |
| CN115514698A (en) | Protocol calculation method, switch, cross-device link aggregation system and storage medium | |
| CN111314432B (en) | Message processing method and device | |
| CN108874530B (en) | Method and device for expanding and shrinking service board of message forwarding equipment | |
| EP1988469A1 (en) | Error control device | |
| US20210103549A1 (en) | Information processing system and information processing method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |