CN111858589A - Method, device and system for acquiring and transmitting time sequence data - Google Patents

Method, device and system for acquiring and transmitting time sequence data Download PDF

Info

Publication number
CN111858589A
CN111858589A CN202010687638.2A CN202010687638A CN111858589A CN 111858589 A CN111858589 A CN 111858589A CN 202010687638 A CN202010687638 A CN 202010687638A CN 111858589 A CN111858589 A CN 111858589A
Authority
CN
China
Prior art keywords
data
index data
time sequence
index
module
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202010687638.2A
Other languages
Chinese (zh)
Inventor
张立
杨爽
党国利
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Jinher Software Co Ltd
Original Assignee
Beijing Jinher Software Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Jinher Software Co Ltd filed Critical Beijing Jinher Software Co Ltd
Priority to CN202010687638.2A priority Critical patent/CN111858589A/en
Publication of CN111858589A publication Critical patent/CN111858589A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/22Indexing; Data structures therefor; Storage structures
    • G06F16/2228Indexing structures
    • G06F16/2255Hash tables
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/25Integrating or interfacing systems involving database management systems

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The invention discloses a method, a device and a system for acquiring and transmitting time sequence data, which belong to the field of time sequence data transmission, and are characterized in that after original data of equipment are acquired, index data of the equipment corresponding to the original data are calculated according to the original data; then storing index data of a preset number of devices into a first storage module; and finally, transmitting the index data in the first storage module into the time sequence database. The original data transmitted into the time data is changed into index data, and the data volume needing to be transmitted is greatly reduced. Meanwhile, index data of a plurality of devices are stored in the same first storage module, and finally, the connection with the time sequence database is greatly reduced. By the method, the data volume needing to be transmitted is small, the network connection concurrency pressure is small, the time sequence data transmission capability is improved, and the time required by the time sequence data transmission is short.

Description

Method, device and system for acquiring and transmitting time sequence data
Technical Field
The present invention relates to a time series data transmission technology, and in particular, to a method, an apparatus, and a system for acquiring and transmitting time series data.
Background
The time series data is a series of data based on time. The data points are connected into a line in a time coordinate to form a multi-dimensional historical report, the trend and the rule of data indexes are shown, and big data analysis, machine learning, prediction and early warning can be performed. The time sequence database is a database for storing time sequence data, and needs to support basic functions of quick writing, persistence, multidimensional aggregation query and the like of the time sequence data. The time sequence database is designed for solving the massive data scene, and becomes a very important service in the direction of the internet of things.
The current mainstream method of the time sequence database is that the time sequence database directly samples the index data of a derivation acquisition module at an equipment end regularly, but the more the equipment is, the larger the network connection concurrency quantity is, the larger the acquired data quantity is, the continuous high concurrency report is, and the network can be stressed greatly; meanwhile, a certain time is spent for establishing connection between each export acquisition module and the time sequence database, and when more equipment is used, the time required for time sequence data transmission is short.
Disclosure of Invention
In order to overcome the defects of the prior art, the invention provides a method, a device and a system for acquiring and transmitting time sequence data, which have low network concurrency pressure and short time sequence data transmission time.
The technical scheme adopted by the invention for solving the technical problems is as follows:
in a first aspect,
a method for acquiring and transmitting time series data comprises the following steps:
acquiring original data of any equipment;
calculating index data of equipment corresponding to the original data according to the original data;
storing index data of a preset number of devices in a first storage module;
transmitting the metric data into the timing database.
Further, transmitting the metric data into the timing database comprises:
Aggregating and storing the index data in the preset number of first storage modules to a second storage module at least once;
establishing connection between the second storage module and the time sequence database;
transmitting the indicator data of the second storage module into the timing database.
Further, aggregating and storing the index data in the preset number of first storage modules to the second storage module includes:
and storing the index data in the first storage module into the second storage module in a table form.
Further, the table adopts a hash table, and a key in the hash table is a device id; value is index data of the corresponding device.
Further, the transmitting the metric data of the second storage module into the timing database includes:
uploading the index data of the second storage module to the time sequence database at regular time; or,
and the time sequence database periodically reads the index data of the second storage module.
In a second aspect of the present invention,
an acquisition and transmission device of time sequence data, comprising:
the device comprises an original data acquisition unit, a data processing unit and a data processing unit, wherein the original data acquisition unit is used for acquiring original data of any equipment;
the index data calculation unit is used for calculating the index data of the equipment corresponding to the original data according to the original data;
The index data storage unit is used for storing the index data of a preset number of devices into the first storage module;
and the index data transmission unit is used for transmitting the index data into the time sequence database.
In a third aspect,
a system for acquiring and transmitting time series data, comprising:
a device for generating time series data;
the device comprises an original data collection module, a data processing module and a data processing module, wherein the original data collection module is used for collecting original data of one device;
the index data calculation module is used for calculating the original data collected by the original data collection modules in preset quantity and obtaining the index data of each device;
and the time sequence database is used for storing the index data of each device obtained by the index data calculation module.
Further, still include: and the index data aggregation module is used for storing the index data obtained by the preset number of index data calculation modules in a table form, and is connected with the time sequence database to transmit the stored index data to the time sequence database.
Further, the table adopts a hash table, and a key in the hash table is a device id; value is index data of the corresponding device.
Further, the step of connecting the time-series database and transmitting the stored index data to the time-series database is:
Uploading the index data of the index data aggregation module to the time sequence database at regular time; or,
and the time sequence database periodically reads the index data of the index data aggregation module.
This application adopts above technical scheme, possesses following beneficial effect at least:
the technical scheme of the invention discloses a method, a device and a system for acquiring and transmitting time sequence data, wherein after the original data of equipment is acquired, index data of the equipment corresponding to the original data is calculated according to the original data; then storing index data of a preset number of devices into a first storage module; and finally, transmitting the index data in the first storage module into the time sequence database. The original data transmitted into the time data is changed into index data, and the data volume needing to be transmitted is greatly reduced. Meanwhile, index data of a plurality of devices are stored in the same first storage module, and finally, the connection with the time sequence database is greatly reduced. By the method, the data volume needing to be transmitted is small, the network connection concurrency pressure is small, the time sequence data transmission capability is improved, and the time required by the time sequence data transmission is short.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a flowchart of a method for acquiring and transmitting time series data according to an embodiment of the present invention;
fig. 2 is a flowchart of another method for acquiring and transmitting time series data according to an embodiment of the present invention;
fig. 3 is a schematic structural diagram of a device for acquiring and transmitting time series data according to an embodiment of the present invention;
fig. 4 is a schematic structural diagram of a system for acquiring and transmitting time series data according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the following detailed description of the technical solutions of the present invention is provided with reference to the accompanying drawings and examples. It is to be understood that the embodiments described are only a few embodiments of the present application and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the examples given herein without making any creative effort, shall fall within the protection scope of the present application.
Referring to fig. 1, an embodiment of the present invention provides a method for acquiring and transmitting time series data, including the following steps:
acquiring original data of any equipment;
calculating index data of equipment corresponding to the original data according to the original data;
Storing index data of a preset number of devices in a first storage module;
the index data is transmitted into the temporal database.
According to the method for acquiring and transmitting the time sequence data, provided by the embodiment of the invention, after the original data of the equipment is acquired, the index data of the equipment corresponding to the original data is calculated according to the original data; then storing index data of a preset number of devices into a first storage module; and finally, transmitting the index data in the first storage module into the time sequence database. The original data transmitted into the time data is changed into index data, and the data volume needing to be transmitted is greatly reduced. Meanwhile, index data of a plurality of devices are stored in the same first storage module, and finally, the connection with the time sequence database is greatly reduced. By the method, the data volume needing to be transmitted is small, the network connection concurrency pressure is small, the time sequence data transmission capability is improved, and the time required by the time sequence data transmission is short.
As a supplementary description of the above embodiment, an embodiment of the present invention further provides another method for acquiring and transmitting time series data, as shown in fig. 2, including the following steps:
acquiring original data of any equipment;
Calculating index data of equipment corresponding to the original data according to the original data;
it should be noted that the raw data and the index data are data generated by the device itself, and specific states of the device, such as a fault, a normal operation, and other operating states, can be determined through the data, so that the raw data may be voltage, current, power, and the like of the device. And the index data is a representative index of a specific state. If h represents too high temperature, it can be understood that the actual index data is calculated and not as simple as the above example.
Storing index data of a preset number of devices in a first storage module;
aggregating and storing the index data in the preset number of first storage modules to a second storage module at least once;
establishing connection between the second storage module and the time sequence database;
and transmitting the index data of the second storage module into the time sequence database.
The aggregating and storing of the index data in the preset number of first storage modules to the second storage module comprises:
and storing the index data in the first storage module into the second storage module in a table form.
In actual use, the table adopts a hash table, and a key in the hash table is a device id; value is index data of the corresponding device.
As an optional implementation manner of the embodiment of the present invention, transmitting the index data of the second storage module into the temporal database includes:
and uploading the index data of the second storage module to a time sequence database at regular time.
In another optional implementation, the transmitting the index data of the second storage module into the time sequence database includes:
and the time sequence database periodically reads the index data of the second storage module.
According to another method for acquiring and transmitting time series data, which is provided by the embodiment of the invention, the technical advantages of the method are that the sub-modules reduce the load pressure of mass network connection and the calculation pressure of data aggregation, the traditional time series database is directly connected with hundreds of millions of devices to periodically sample the data of the hundreds of millions of devices, and the time and performance consumption is large when the traditional time series database is connected with the hundreds of millions of devices to establish network connection and process requests.
In an embodiment, the present invention provides a device for acquiring and transmitting time series data, as shown in fig. 3, including:
an original data acquisition unit 301 configured to acquire original data of any device;
an index data calculation unit 302, configured to calculate, according to the original data, index data of a device corresponding to the original data;
an index data storage unit 303, configured to store index data of a preset number of devices in a first storage module;
and an index data transmission unit 304, configured to transmit the index data into the temporal database.
In some embodiments, optionally, the method further comprises: an index data aggregation unit 305, configured to aggregate and store the index data in a preset number of first storage modules to a second storage module at least once; establishing connection between the second storage module and the time sequence database; and transmitting the index data of the second storage module into the time sequence database.
According to the time sequence data acquisition and transmission device provided by the embodiment of the invention, the original data acquisition unit acquires the original data of any equipment; the index data calculation unit calculates index data of equipment corresponding to the original data according to the original data; the index data storage unit stores index data of a preset number of devices to a first storage module; the index data aggregation unit aggregates the index data in the preset number of first storage modules at least once and stores the aggregated index data in the second storage module, and the index data transmission unit transmits the index data to the time sequence database. The number of simultaneous connections to the time series database is reduced while the amount of data transferred to the database is reduced.
In an embodiment, the present invention further provides a system for acquiring and transmitting time series data, as shown in fig. 4, including:
a device 401 for generating time series data;
a raw data collection module 402, configured to collect raw data of a device;
an index data calculation module 403, configured to calculate raw data collected by a preset number of raw data collection modules and obtain index data of each device;
and the time sequence database 404 is used for storing the index data of each device obtained by the index data calculation module.
And the index data aggregation module 405 is configured to store the index data obtained by the preset number of index data calculation modules in a table form, and connect with the time sequence database to transmit the stored index data to the time sequence database.
The original data are collected from the equipment and then input into the original data collection module, one equipment corresponds to one original data collection module, and each piece of original data is used as a message and uploaded to the index data calculation module in real time in a network communication mode.
The original data are transmitted into the index data calculation module and then calculated to obtain index data, the index data are transmitted to the index data aggregation module through a network, each index data calculation module undertakes data processing of thousands of original data collection modules and even hundreds of thousands of original data collection modules, and each calculated index data is taken as a message and uploaded to the index data aggregation module through the network in real time.
The index data are aggregated and stored in a hash table of the index data after being transmitted into the index data aggregation module. The key in the hash table is the device id, and the value in the hash table is the latest index data of the current device. Each index data aggregation module is responsible for processing hundreds of index data calculation modules, and uploads the latest data of all the devices in the index data hash table to the time sequence database at regular time through a network, or the time sequence database periodically samples the latest data of each device in the index data hash table.
According to the time sequence data acquisition and transmission system provided by the embodiment of the invention, the original data acquisition module acquires original data for calculating data indexes from equipment. And the index data calculation module is used for calculating an index data result from the original data received by the original data collection module. And the index data aggregation module is used for aggregating all the index data received from the index data calculation module into a group of latest index data of all the equipment of the current module. The three modules adopt a distributed structure to carry out data communication between the modules by using network connection, and the problem of writing pressure caused by time sequence data of mass equipment is solved by a mode of sharing network connection concurrent pressure and data aggregation calculation pressure.
It is understood that the same or similar parts in the above embodiments may be mutually referred to, and the same or similar parts in other embodiments may be referred to for the content which is not described in detail in some embodiments.
It should be noted that, in the description of the present application, the terms "first", "second", etc. are used for descriptive purposes only and are not to be construed as indicating or implying relative importance. Further, in the description of the present application, the meaning of "a plurality" means at least two unless otherwise specified.
Any process or method descriptions in flow charts or otherwise described herein may be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or steps of the process, and the scope of the preferred embodiments of the present application includes other implementations in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those reasonably skilled in the art of the present application.
It should be understood that portions of the present application may be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the various steps or methods may be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one or combination of the following techniques, which are known in the art, may be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application specific integrated circuit having an appropriate combinational logic gate circuit, a Programmable Gate Array (PGA), a Field Programmable Gate Array (FPGA), or the like.
It will be understood by those skilled in the art that all or part of the steps carried by the method for implementing the above embodiments may be implemented by hardware related to instructions of a program, which may be stored in a computer readable storage medium, and when the program is executed, the program includes one or a combination of the steps of the method embodiments.
In addition, functional units in the embodiments of the present application may be integrated into one processing module, or each unit may exist alone physically, or two or more units are integrated into one module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode. The integrated module, if implemented in the form of a software functional module and sold or used as a stand-alone product, may also be stored in a computer readable storage medium.
The storage medium mentioned above may be a read-only memory, a magnetic or optical disk, etc.
In the description herein, reference to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the application. In this specification, the schematic representations of the terms used above do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
Although embodiments of the present application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present application, and that variations, modifications, substitutions and alterations may be made to the above embodiments by those of ordinary skill in the art within the scope of the present application.

Claims (10)

1. A method for acquiring and transmitting time series data is characterized by comprising the following steps:
acquiring original data of any equipment;
calculating index data of equipment corresponding to the original data according to the original data;
storing index data of a preset number of devices in a first storage module;
transmitting the metric data into the timing database.
2. The method of claim 1, wherein: transmitting the metric data into the timing database comprises:
aggregating and storing the index data in the preset number of first storage modules to a second storage module at least once;
establishing connection between the second storage module and the time sequence database;
transmitting the indicator data of the second storage module into the timing database.
3. The method of claim 2, wherein: the step of aggregating and storing the index data in the preset number of first storage modules to the second storage module comprises the following steps:
And storing the index data in the first storage module into the second storage module in a table form.
4. The method of claim 3, wherein: the table adopts a hash table, and keys in the hash table are equipment ids; value is index data of the corresponding device.
5. The method of claim 2, wherein: the transmitting the indicator data of the second storage module into the timing database comprises:
uploading the index data of the second storage module to the time sequence database at regular time; or,
and the time sequence database periodically reads the index data of the second storage module.
6. A device for acquiring and transmitting time series data is characterized by comprising:
the device comprises an original data acquisition unit, a data processing unit and a data processing unit, wherein the original data acquisition unit is used for acquiring original data of any equipment;
the index data calculation unit is used for calculating the index data of the equipment corresponding to the original data according to the original data;
the index data storage unit is used for storing the index data of a preset number of devices into the first storage module;
and the index data transmission unit is used for transmitting the index data into the time sequence database.
7. A system for acquiring and transmitting time series data, comprising:
A device for generating time series data;
the device comprises an original data collection module, a data processing module and a data processing module, wherein the original data collection module is used for collecting original data of one device;
the index data calculation module is used for calculating the original data collected by the original data collection modules in preset quantity and obtaining the index data of each device;
and the time sequence database is used for storing the index data of each device obtained by the index data calculation module.
8. The system of claim 7, further comprising: and the index data aggregation module is used for storing the index data obtained by the preset number of index data calculation modules in a table form, and is connected with the time sequence database to transmit the stored index data to the time sequence database.
9. The system of claim 8, wherein: the table adopts a hash table, and keys in the hash table are equipment ids; value is index data of the corresponding device.
10. The system of claim 8, wherein: the step of connecting with the time sequence database and transmitting the stored index data to the time sequence database is that:
uploading the index data of the index data aggregation module to the time sequence database at regular time; or,
And the time sequence database periodically reads the index data of the index data aggregation module.
CN202010687638.2A 2020-07-16 2020-07-16 Method, device and system for acquiring and transmitting time sequence data Pending CN111858589A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010687638.2A CN111858589A (en) 2020-07-16 2020-07-16 Method, device and system for acquiring and transmitting time sequence data

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010687638.2A CN111858589A (en) 2020-07-16 2020-07-16 Method, device and system for acquiring and transmitting time sequence data

Publications (1)

Publication Number Publication Date
CN111858589A true CN111858589A (en) 2020-10-30

Family

ID=72984097

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010687638.2A Pending CN111858589A (en) 2020-07-16 2020-07-16 Method, device and system for acquiring and transmitting time sequence data

Country Status (1)

Country Link
CN (1) CN111858589A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114401461A (en) * 2022-03-22 2022-04-26 深圳市讯禾实业有限公司 Multichannel data acquisition and storage system based on internet of things technology

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140040276A1 (en) * 2012-07-31 2014-02-06 International Business Machines Corporation Method and apparatus for processing time series data
CN110389967A (en) * 2019-07-26 2019-10-29 深圳市腾讯计算机***有限公司 Date storage method, device, server and storage medium
CN110909062A (en) * 2019-11-29 2020-03-24 迈普通信技术股份有限公司 Data processing method and device, electronic equipment and readable storage medium
CN111209285A (en) * 2020-04-23 2020-05-29 成都四方伟业软件股份有限公司 Statistical index storage method and device based on time sequence data
CN111274256A (en) * 2020-01-20 2020-06-12 远景智能国际私人投资有限公司 Resource control method, device, equipment and storage medium based on time sequence database

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140040276A1 (en) * 2012-07-31 2014-02-06 International Business Machines Corporation Method and apparatus for processing time series data
CN110389967A (en) * 2019-07-26 2019-10-29 深圳市腾讯计算机***有限公司 Date storage method, device, server and storage medium
CN110909062A (en) * 2019-11-29 2020-03-24 迈普通信技术股份有限公司 Data processing method and device, electronic equipment and readable storage medium
CN111274256A (en) * 2020-01-20 2020-06-12 远景智能国际私人投资有限公司 Resource control method, device, equipment and storage medium based on time sequence database
CN111209285A (en) * 2020-04-23 2020-05-29 成都四方伟业软件股份有限公司 Statistical index storage method and device based on time sequence data

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114401461A (en) * 2022-03-22 2022-04-26 深圳市讯禾实业有限公司 Multichannel data acquisition and storage system based on internet of things technology

Similar Documents

Publication Publication Date Title
US10176208B2 (en) Processing time series data from multiple sensors
JP2019511054A (en) Distributed cluster training method and apparatus
CN111078755A (en) Time sequence data storage query method and device, server and storage medium
CN111585840B (en) Service resource monitoring method, device and equipment
CN111740860A (en) Log data transmission link monitoring method and device
CN111190792A (en) Log storage method and device, electronic equipment and readable storage medium
CN111858589A (en) Method, device and system for acquiring and transmitting time sequence data
CN117632897A (en) Dynamic capacity expansion and contraction method and device
WO2021147319A1 (en) Data processing method, apparatus, device, and medium
EP4092989A1 (en) Measuring metrics of a computer network
CN110727564A (en) Real-time monitoring method for test data of server performance pressure test
CN112069166B (en) Power station equipment data acquisition method and system
CN117762968A (en) Data aggregation method, system, equipment and storage medium
CN116471214A (en) Power communication data monitoring method and system
Viharos et al. ” Big Data” Initiative as an IT Solution for Improved Operation and Maintenance of Wind Turbines
CN113434604B (en) Time sequence database synchronization method, system, equipment and storage medium
CN111625517B (en) New energy real-time data processing method and device based on change storage
JP2010271998A (en) Database management method, database device, and program
CN109408466B (en) Agricultural Internet of things redundant data processing method and device
CN113254253A (en) Data processing method, system and equipment
CN116743618B (en) Data acquisition and analysis method, equipment and medium of station remote equipment
CN110858341A (en) Index monitoring method, device, equipment and medium based on distributed storage system
CN114666193B (en) Abnormality detection method for CDN, computing device and storage medium
CN117743473B (en) Data management synchronization performance monitoring system
CN116032849B (en) Data exchange method, device, system and electronic equipment

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination