CN115981951A - Monitoring data storage method based on OpenTSDB and HBase, computer device and storage medium - Google Patents
Monitoring data storage method based on OpenTSDB and HBase, computer device and storage medium Download PDFInfo
- Publication number
- CN115981951A CN115981951A CN202211565310.9A CN202211565310A CN115981951A CN 115981951 A CN115981951 A CN 115981951A CN 202211565310 A CN202211565310 A CN 202211565310A CN 115981951 A CN115981951 A CN 115981951A
- Authority
- CN
- China
- Prior art keywords
- monitoring data
- opentsdb
- hbase
- value
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a monitoring data storage method, a system, a computer device and a storage medium based on OpenTSDB and HBase, which comprises the steps of obtaining an ID value corresponding to monitoring data when the data type of the monitoring data is text type data, determining a keyword corresponding to the monitoring data according to the ID value, writing the monitoring data into an HBase database according to the keyword, writing the data type information and the ID value of the monitoring data into OpenTSDB and the like. The invention saves the monitoring data belonging to the text type to the HBase database, and can solve the problem that OpenTSDB can not save the text type data without introducing a new component, thereby simultaneously storing the numerical value type monitoring data and the text type monitoring data and reducing the complexity of the system; by setting a uniform data processing program to execute each step, the programming complexity is reduced for the client. The invention is widely applied to the technical field of data processing.
Description
Technical Field
The invention relates to the technical field of data processing, in particular to a monitoring data storage method, a monitoring data storage system, a computer device and a storage medium based on OpenTSDB and HBase.
Background
The monitoring data is essentially time series data, which refers to a series of numerical data points that are some indicator of a period of time. OpenTSDB is a time series database that uniquely identifies an index value (value) by an index (metric), a tag (tag), and a time stamp (timestamp), and the index data has indelibility. OpenTSDB typically implements horizontal extension of the cluster by multiple TSDs (service processes of OpenTSDB), each TSD instance receiving write and read requests for time-series data, thereby implementing storage of monitoring data.
The OpenTSDB supports second-level data acquisition metrics, supports permanent storage, can be used for capacity planning, and is easily accessed to the existing alarm system. The OpenTSDB can obtain corresponding metrics from a large-scale cluster (including network devices, operating systems, and application programs in the cluster), and store, index, and serve the data, and is also easy to perform processing such as web-based processing and graphical processing, so that the data is more easily understood.
Currently, the related technology of OpenTSDB can only store numerical type monitoring data, and cannot store text type monitoring data generated by a monitoring system. The disadvantages of the OpenTSDB related art can be overcome by writing and querying the value type monitoring data and the text type monitoring data separately, but this increases the complexity of data processing.
Disclosure of Invention
The invention aims to provide a monitoring data storage method, a monitoring data storage system, a computer device and a storage medium based on OpenTSDB and HBase, aiming at the technical problem that only numerical type monitoring data can be stored but text type monitoring data cannot be stored in the prior OpenTSDB related technology.
In one aspect, an embodiment of the present invention includes a monitoring data storage system based on OpenTSDB and HBase, including:
acquiring monitoring data;
when the data type of the monitoring data is text type data, obtaining an ID value corresponding to the monitoring data, determining a keyword corresponding to the monitoring data according to the ID value, writing the monitoring data into an HBase database according to the keyword, and writing the data type information and the ID value of the monitoring data into OpenTSDB.
Further, the monitoring data storage method based on OpenTSDB and HBase further includes:
and when the monitoring data belongs to numerical data, writing the data type information of the monitoring data and the numerical value of the monitoring data into the OpenTSDB.
Further, the obtaining an ID value corresponding to the monitoring data includes:
processing according to the monitoring data by using a snowflake algorithm to generate a globally unique long integer numerical value;
and taking the long integer value as the ID value.
Further, the determining the keyword corresponding to the monitoring data includes:
taking a plurality of characters positioned at the last in the ID value;
turning over the extracted characters;
forming a character string by taking the turned characters as the integral prefix of the ID value;
and taking the character string as the keyword.
Further, the monitoring data storage method based on OpenTSDB and HBase further includes:
establishing a pm _ text table in the HBase database;
setting a plurality of pre-partitions of the HBase database through the pm _ text table; each pre-partition is respectively provided with the keywords of the specific prefix corresponding to the pre-partition, and the pre-partition is used for storing the monitoring data corresponding to the keywords.
Further, the taking the last characters in the ID value includes:
determining the number of the characters according to the number of the pre-partitions in the HBase database; the number of combinations formed by the characters is greater than the number of pre-partitions.
Further, the monitoring data storage method based on OpenTSDB and HBase further includes:
acquiring a query request;
creating a query request object according to the query request;
sending the query request object to an OpenTSDB;
acquiring the monitoring data or the ID value returned by the OpenTSDB in response to the query request object;
and when the OpenTSDB returns the ID value, determining a keyword corresponding to the monitoring data according to the ID value, and inquiring the HBase database according to the keyword to obtain the monitoring data.
Further, the monitoring data storage method based on OpenTSDB and HBase further includes:
setting a delete field of a request object to true, and requesting/api/query of OpenTSDB;
or alternatively
Querying the OpenTSDB to obtain the keywords;
deleting the corresponding monitoring data in the HBase database according to the keywords;
requesting the/api/query of the OpenTSDB to delete the keyword reserved in the OpenTSDB.
In another aspect, an embodiment of the present invention further includes a computer apparatus, including a memory and a processor, where the memory is configured to store at least one program, and the processor is configured to load the at least one program to execute the monitoring data storage method based on OpenTSDB and HBase in the embodiment.
In another aspect, an embodiment of the present invention further includes a storage medium in which a processor-executable program is stored, where the processor-executable program is used to execute the OpenTSDB and HBase-based monitoring data storage method in the embodiment when executed by a processor.
The beneficial effects of the invention are: in the monitoring data storage method based on the OpenTSDB and the HBase in the embodiment, the monitoring data belonging to the text type is stored in the HBase database, the OpenTSDB directly stores the monitoring data belonging to the numerical type, and the problem that the OpenTSDB cannot store the text type data can be solved without introducing a new component, so that the numerical type monitoring data and the text type monitoring data are stored at the same time, and the complexity of the system is reduced; by setting a uniform data processing program to execute each step, for the client, no matter the monitoring data is of a numerical type or a text type, the data can be saved or inquired through the same programming interface, and the programming complexity is reduced.
Drawings
FIG. 1 is a diagram illustrating steps of a monitoring data storage method based on OpenTSDB and HBase in an embodiment;
FIG. 2 is a schematic system diagram illustrating an embodiment of a monitoring data storage method based on OpenTSDB and HBase;
FIG. 3 is a diagram illustrating region information of a pm _ text table in an embodiment;
FIG. 4 is a diagram illustrating a process of writing monitor data according to an embodiment;
FIG. 5 is a diagram illustrating a query process of monitoring data according to an embodiment;
FIG. 6 is a schematic flow chart of querying a keyword rowkey of text type monitoring data in the embodiment;
FIG. 7 is a flowchart illustrating loading of text type monitoring data according to a keyword rowkey in the embodiment;
fig. 8 is a schematic diagram of a process of deleting monitoring data in the embodiment.
Detailed Description
In this embodiment, referring to fig. 1, a monitoring data storage method based on OpenTSDB and HBase includes the following steps:
s1, acquiring monitoring data;
s2, when the data type of the monitoring data is text data, obtaining an ID value corresponding to the monitoring data, determining a keyword corresponding to the monitoring data according to the ID value, writing the monitoring data into an HBase database according to the keyword, and writing the data type information and the ID value of the monitoring data into an OpenTSDB;
and S3, when the monitoring data belong to numerical data, writing the data type information of the monitoring data and the numerical value of the monitoring data into OpenTSDB.
In this embodiment, referring to fig. 2, a monitoring data storage method based on OpenTSDB and HBase relates to three parts, namely, a data processing program, an OpenTSDB database, and an HBase database, where the data processing program is an entry for reading and writing monitoring data, and may perform functions of writing, querying, deleting, and the like of the monitoring data. The data processing program, the OpenTSDB and the Hbase database may be run by the same computer device, or may be run by a plurality of different computer devices, respectively.
In this embodiment, the HBase is responsible for storing all data, including OpenTSDB data and text-type monitoring data. In this embodiment, openTSDB creates 4 tables, which are tsdb, tsdb-uid, tsdb-meta, and tsdb-tree, where the tsdb-uid table stores a string of an index name, an index value, a tag name, and a tag value, and the tsdb table stores numerical data of performance data and an index name, a tag name, and a tag value number of the piece of data. In this embodiment, a newly created HBase table pm _ text is used to store the value of the text type monitoring data, the pm _ text table column cluster is c, the data retention time is 1 year, the compression algorithm is Snappy, and 5 pre-partitions (regions) are created. The region information of the pm _ text table is shown in fig. 3.
In this embodiment, the OpenTSDB is responsible for reading and writing the numerical type monitoring data. And sending JSON data to a request path/api/put of the OpenTSDB in a POST mode to realize data writing. Wherein, the metric field of the request JSON represents an index of the monitoring data, the timestamp field represents a timestamp of the data, and the value field represents a numerical value of the numerical monitoring data or a rowkey of the text monitoring data. the tags field indicates the tags of the data, a plurality of tags can be set for one piece of data, but at least one tag is set, and the tag name and the tag value can be customized.
In this embodiment, the OpenTSDB specifies different values for the dataType tag according to different data types of the monitoring data. If the monitored data is data of a value type, the value of dataType is set to 1 and the value field is the value of the monitored data. If the monitoring data is text type data, the dataType value is set to 2 and the value field is the rowkey information of the monitoring data in HBase.
In this embodiment, steps S1 to S3 and the like are executed by a data processing program. Steps S1 to S3 are the write-in process of the monitoring data, and one specific execution flow is shown in fig. 4.
In step S1, the monitoring device may generate monitoring data, the monitoring device sends a write request to the data processing program, and the data processing program receives the monitoring data and the write request.
After step S1, in response to the write request, the data processing program first creates an OpenTSDB data write request object, sets fields such as metric, timestamp, and tags, then checks the data type of the monitoring data, and determines whether the data type of the monitoring data is text-type data or numeric-type data.
In the case where the monitoring data belongs to the text type data, since the OpenTSDB itself cannot directly write the text type monitoring data, the data processing program performs step S2.
In step S2, the data processing program first obtains an ID value corresponding to the monitoring data. In this embodiment, the data processing program may use a snowflake algorithm to process according to the monitoring data to generate a globally unique integer value, and the integer value is used as the ID value corresponding to the monitoring data. In the case where there are a plurality of monitoring data, the different monitoring data can be distinguished by the ID value corresponding to each monitoring data.
Referring to fig. 4, in step S2, after obtaining the ID value corresponding to the monitoring data, the data processing program determines a keyword corresponding to the monitoring data according to the ID value. In this embodiment, the data processing program may take a plurality of characters located at the last in the ID value, turn over the plurality of characters taken out, form a character string with the plurality of characters after turning over as a prefix of the whole ID value, and take the character string as a keyword (rowkey).
For example, if a monitored data has an ID value of "12345678", the last 3 bits of the ID value of "678" may be taken to flip the characters of "678", for example, to disorder the order of the characters therein. One of the possible results of the character flipping is "876", prefixing "876" with the ID value as a whole "12345678", forming a character string "87612345678", with the character string "87612345678" as the key (rowkey).
In this embodiment, in the HBase database, a pm _ text table is established, a plurality of pre-partitions (regions) (for example, 5 pre-partitions) in the HBase database are set through the pm _ text table, if a piece of monitoring data belongs to text type data, the piece of monitoring data is written into the HBase database, and specifically written into which pre-partition of the HBase database, the piece of monitoring data is related to a prefix of a keyword (rowkey) of the piece of monitoring data. For example, if the prefix of the keyword (rowkey) corresponding to one piece of monitoring data is 000 to 200, the monitoring data is written into the first pre-partition in the HBase database; if the prefix of a key word (rowkey) corresponding to the monitoring data is 200 to 400, the monitoring data is written into a second pre-partition in the HBase database; if the prefix of a key word (rowkey) corresponding to one piece of monitoring data is 400 to 600, the monitoring data is written into a third pre-partition in the HBase database; if the prefix of a keyword (rowkey) corresponding to monitoring data is 600 to 800, the monitoring data is written into a fourth pre-partition in the HBase database; if the prefix of a keyword (rowkey) corresponding to monitoring data is 800 to 999, the monitoring data is written into a fifth pre-partition in the HBase database, wherein the fifth pre-partition is 8230
Under the mechanism of setting a plurality of pre-partitions, because the ID values of the monitoring data generated by the monitoring equipment are generally monotonically increased, in a plurality of monitoring data generated in the same period of time, the keywords rowkey generated after the last characters of the corresponding ID values are turned over can hit different pre-partitions, and when the monitoring data are written into the HBase database because of belonging to text data, the monitoring data are more likely to be respectively uniformly and dispersedly written into different pre-partitions, so that the effect similar to polling region writing is realized, and the purpose of load balancing is achieved.
In this embodiment, specifically how many characters in one ID value are used for flipping, which may be determined according to the number of pre-partitions in the HBase database. For example, the number of combinations formed by a plurality of characters in the ID value may be greater than the number of all pre-partitions, so that when there are a plurality of monitoring data, the keyword (rowkey) generated by the monitoring data may cover all pre-partitions in the HBase database, that is, the monitoring data may be more likely to be written into different pre-partitions uniformly and dispersedly, which is beneficial to load balancing of reading and writing of the HBase cluster.
Referring to fig. 4, in step S2, after obtaining a keyword rowkey corresponding to the monitoring data, the data processing program writes the keyword rowkey as a rowkey of an HBase database write request, writes text content of the monitoring data as a value into a pm _ text table of the HBase database, and thereby writes the monitoring data into a corresponding pre-partition in the HBase database; after the text type monitoring data is successfully written into the HBase database, setting the value of a tag of a dataType of an OpenTSDB data writing request object as 2, setting the value field as ID, then sending a request to the OpenTSDB, writing the data type information and the ID value of the monitoring data into the OpenTSDB, completing the storage of the text type monitoring data, and ending the step S2.
In the case where the monitoring data belongs to the numerical type data, since OpenTSDB itself can directly write the numerical type monitoring data, the data processing program executes step S3.
In step S3, the data processing program sets the value of dataType in the OpenTSDB data write request object to be 1, and the value of the value field is the value of the monitoring data, and then sends a request to the OpenTSDB, so as to write the data type information of the monitoring data and the value of the monitoring data into the OpenTSDB, until the storing of the numerical type monitoring data is completed, and then step S3 is executed.
In this embodiment, the principle of executing steps S1 to S3 includes:
1. the OpenTSDB is realized based on the HBase, and the HBase is suitable for storing mass data, so that the monitoring data belonging to the text type is stored in the HBase database, the OpenTSDB can directly store the monitoring data belonging to the numerical value type, and the problem that the OpenTSDB cannot store the text type data can be solved without introducing a new component;
2. the unified data processing program is set to execute each step, and for the client, no matter the monitoring data is numerical value type monitoring data or text type monitoring data, the data can be saved or inquired through the same programming interface, so that the programming complexity is reduced.
In this embodiment, on the basis of executing steps S1 to S3, the following steps may be executed:
s4, acquiring a query request;
s5, creating an inquiry request object according to the inquiry request;
s6, sending a request to the OpenTSDB by the query request object;
s7, acquiring monitoring data or ID values returned by the OpenTSDB in response to the query request object;
and S8, when the OpenTSDB returns the ID value, determining a keyword corresponding to the monitoring data according to the ID value, and inquiring from the HBase database according to the keyword to obtain the monitoring data.
Steps S4 to S8 are monitoring data query steps executed when the storage of the monitoring data is completed through steps S1 and S2 or S1 and S3, and the monitoring data is stored in the OpenTSDB or HBase database.
In this embodiment, the flow of the monitoring data querying steps S4 to S8 is shown in fig. 5.
In steps S4 to S6, after receiving the monitoring data query request, the data processing program first creates an OpenTSDB data query request object, sets fields such as metric, start, end, and tags, and then sends a request to the OpenTSDB.
In step S7, a dataType tag of a result returned by the OpenTSDB is checked, if the value of the dataType tag is 1, it indicates that the monitoring data is numerical monitoring data, and the query result of the OpenTSDB is directly returned until the query of the numerical monitoring data is completed; if dataType has a tag value of 2, indicating that the piece of monitoring data is text type monitoring data, step S8 is performed.
In step S8, a value of the OpenTSDB, that is, an ID of the text data, is taken, then the last 3 characters of the ID are turned, the ID is spliced by the 3 characters to obtain a rowkey, then the rowkey is used to query a pm _ text table of the HBase to obtain the text data, then the value of the OpenTSDB query result is replaced by the text data, and finally the query result of the OpenTSDB is returned, so that the query of the text-type monitoring data is completed.
In the case where the text-type monitoring data occupies a large space, the time for transmitting all the text-type monitoring data at one time may be long. Because the keyword rowkey of the text type monitoring data is only a long integer numerical value, the occupied space is small, and the transmission time is short, all the keyword rowkeys corresponding to the query result can be returned firstly under certain scenes allowing delayed data loading, and then the client side requests the required text data in batches according to the requirement or requests the text type monitoring data corresponding to the keyword rowkey simultaneously, so that a good interaction effect is realized. In this embodiment, a flow of querying a keyword rowkey of text type monitoring data is shown in fig. 6, and a flow of loading text type monitoring data according to the keyword rowkey is shown in fig. 7.
In this embodiment, the principle of executing steps S4 to S8 includes:
1. the OpenTSDB stores a keyword rowkey of the text type monitoring data in the HBase database, firstly queries the keyword rowkey of the text type monitoring data, and then accurately queries the HBase database through the keyword rowkey, namely, a layer of index is made on the text type monitoring data, and the efficiency of querying character string data in a given time range is high;
2. the prefix of the keyword rowkey obtained by turning over a plurality of characters behind the ID can be changed within the same period of time, so that data within the same period of time can be written into different pre-partition regions of HBase, and a write-in hot spot of the keyword HBase is avoided;
3. when the client needs to request the monitoring data of the text type, only the keyword rowkey of the monitoring data of the text type can be requested, and then the text content of the actual monitoring data of the text type is loaded according to the requirement, so that the effect of delaying loading can be realized, and the query speed of the monitoring data is optimized.
In this embodiment, on the basis of executing steps S1 to S3, the following steps may be executed:
s9, setting a delete field of the request object to true, and requesting/api/query of OpenTSDB;
or
S10, inquiring OpenTSDB to obtain keywords;
s11, deleting corresponding monitoring data in the HBase database according to the keywords;
s12, requesting the api/query of the OpenTSDB to delete the key words reserved in the OpenTSDB.
Step S9 and steps S10 to S12 are monitoring data deletion steps executed when the storage of the monitoring data is completed by steps S1 and S2 or S1 and S3 and the monitoring data is stored in the OpenTSDB or HBase database.
In this embodiment, the flow of the monitoring data query steps S9 to S12 is shown in fig. 8.
In this embodiment, if the monitoring data of the value type is to be manually deleted, step S9 may be executed, where the delete field of the request object is set to true, and then the request is made for the/api/query of the OpenTSDB, so as to complete the deletion of the monitoring data of the value type.
In this embodiment, if the text-type monitoring data is to be deleted manually, steps S10 to S12 may be executed, the rowkey information of the text data is obtained by querying the OpenTSDB, then the text data in the HBase is deleted by using the rowkey, and finally the rowkey information retained in the OpenTSDB is requested to be deleted by the/api/query of the OpenTSDB, so that the deletion of the digital-type monitoring data is completed.
In this embodiment, because both OpenTSDB data and text-type monitoring data are stored in the HBase database, the HBase database can specify TTL of the data, and automatic cleaning of stale data is achieved.
The monitoring data storage method based on the OpenTSDB and the HBase in the embodiment may be implemented by writing a computer program for implementing the monitoring data storage method based on the OpenTSDB and the HBase in the embodiment, writing the computer program into a computer device or a storage medium, and executing the monitoring data storage method based on the OpenTSDB and the HBase in the embodiment when the computer program is read out and run, thereby implementing the same technical effect as the monitoring data storage method based on the OpenTSDB and the HBase in the embodiment.
It should be noted that, unless otherwise specified, when a feature is referred to as being "fixed" or "connected" to another feature, it may be directly fixed or connected to the other feature or indirectly fixed or connected to the other feature. Furthermore, the descriptions of upper, lower, left, right, etc. used in the present disclosure are only relative to the mutual positional relationship of the constituent parts of the present disclosure in the drawings. As used in this disclosure, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. In addition, unless defined otherwise, all technical and scientific terms used in this example have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the description of the embodiments herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in this embodiment, the term "and/or" includes any combination of one or more of the associated listed items.
It will be understood that, although the terms first, second, third, etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one type of element from another. For example, a first element could be termed a second element, and, similarly, a second element could be termed a first element, without departing from the scope of the present disclosure. The use of any and all examples, or exemplary language ("e.g.," such as "or the like") provided with this embodiment is intended merely to better illuminate embodiments of the invention and does not pose a limitation on the scope of the invention unless otherwise claimed.
It should be recognized that embodiments of the present invention can be realized and implemented by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer readable memory. The methods may be implemented in a computer program using standard programming techniques, including a non-transitory computer-readable storage medium configured with the computer program, where the storage medium so configured causes a computer to operate in a specific and predefined manner, according to the methods and figures described in the detailed description. Each program may be implemented in a high level procedural or object oriented programming language to communicate with a computer system. However, the program(s) can be implemented in assembly or machine language, if desired. In any case, the language may be a compiled or interpreted language. Furthermore, the program can be run on a programmed application specific integrated circuit for this purpose.
Further, operations of processes described in this embodiment can be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context. The processes described in this embodiment (or variations and/or combinations thereof) may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) collectively executed on one or more processors, by hardware, or combinations thereof. The computer program includes a plurality of instructions executable by one or more processors.
Further, the method may be implemented in any type of computing platform operatively connected to a suitable interface, including but not limited to a personal computer, mini computer, mainframe, workstation, networked or distributed computing environment, separate or integrated computer platform, or in communication with a charged particle tool or other imaging device, and the like. Aspects of the invention may be embodied in machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, optically read and/or write storage medium, RAM, ROM, or the like, such that it may be read by a programmable computer, which when read by the storage medium or device, is operative to configure and operate the computer to perform the procedures described herein. Further, the machine-readable code, or portions thereof, may be transmitted over a wired or wireless network. The invention described in this embodiment includes these and other different types of non-transitory computer-readable storage media when such media include instructions or programs that implement the steps described above in conjunction with a microprocessor or other data processor. The invention also includes the computer itself when programmed according to the methods and techniques described herein.
A computer program can be applied to input data to perform the functions described in the present embodiment to convert the input data to generate output data that is stored to a non-volatile memory. The output information may also be applied to one or more output devices, such as a display. In a preferred embodiment of the invention, the transformed data represents physical and tangible objects, including particular visual depictions of physical and tangible objects produced on a display.
The above description is only a preferred embodiment of the present invention, and the present invention is not limited to the above embodiment, and any modifications, equivalent substitutions, improvements, etc. within the spirit and principle of the present invention should be included in the protection scope of the present invention as long as the technical effects of the present invention are achieved by the same means. The invention is capable of other modifications and variations in its technical solution and/or its implementation, within the scope of protection of the invention.
Claims (10)
1. A monitoring data storage method based on OpenTSDB and HBase is characterized by comprising the following steps:
acquiring monitoring data;
when the data type of the monitoring data is text data, obtaining an ID value corresponding to the monitoring data, determining a keyword corresponding to the monitoring data according to the ID value, writing the monitoring data into an HBase database according to the keyword, and writing the data type information and the ID value of the monitoring data into OpenTSDB.
2. The OpenTSDB and HBase-based monitoring data storage method according to claim 1, wherein the OpenTSDB and HBase-based monitoring data storage method further comprises:
and when the monitoring data belongs to numerical data, writing the data type information of the monitoring data and the numerical value of the monitoring data into the OpenTSDB.
3. The OpenTSDB and HBase-based monitoring data storage method according to claim 1, wherein the obtaining an ID value corresponding to the monitoring data includes:
processing according to the monitoring data by using a snowflake algorithm to generate a globally unique long integer numerical value;
and taking the long integer value as the ID value.
4. The OpenTSDB and HBase-based monitoring data storage method of claim 3, wherein the determining a keyword corresponding to the monitoring data includes:
taking a plurality of characters positioned at the last in the ID value;
turning over the extracted characters;
forming a character string by taking the turned characters as the integral prefix of the ID value;
and taking the character string as the keyword.
5. The OpenTSDB and HBase-based monitoring data storage method according to claim 4, wherein the OpenTSDB and HBase-based monitoring data storage method further comprises:
establishing a pm _ text table in the HBase database;
setting a plurality of pre-partitions of the HBase database through the pm _ text table; each pre-partition is respectively provided with the keywords of the specific prefix corresponding to the pre-partition, and the pre-partition is used for storing the monitoring data corresponding to the keywords.
6. The OpenTSDB and HBase-based monitoring data storage method of claim 5, wherein the taking of the last characters in the ID value comprises:
determining the number of the characters according to the number of the pre-partitions in the HBase database; the number of combinations formed by the characters is greater than the number of the pre-partitions.
7. The OpenTSDB and HBase-based monitoring data storage method according to any one of claims 1-6, wherein the OpenTSDB and HBase-based monitoring data storage method further comprises:
acquiring a query request;
creating a query request object according to the query request;
sending the query request object to an OpenTSDB;
acquiring the monitoring data or the ID value returned by the OpenTSDB in response to the query request object;
and when the OpenTSDB returns the ID value, determining a keyword corresponding to the monitoring data according to the ID value, and inquiring the HBase database according to the keyword to obtain the monitoring data.
8. The OpenTSDB and HBase-based monitoring data storage method according to any one of claims 1-6, wherein the OpenTSDB and HBase-based monitoring data storage method further comprises:
setting a delete field of a request object to true, and requesting/api/query of OpenTSDB;
or
Querying the OpenTSDB to obtain the keywords;
deleting the corresponding monitoring data in the HBase database according to the keywords;
requesting/api/query of the OpenTSDB to delete the keywords reserved in the OpenTSDB.
9. A computer apparatus comprising a memory for storing at least one program and a processor for loading the at least one program to perform the OpenTSDB and HBase-based monitoring data storage method of any one of claims 1-8.
10. A computer readable storage medium in which a processor executable program is stored, wherein the processor executable program, when executed by a processor, is for performing the OpenTSDB and HBase-based monitoring data storage method of any one of claims 1 to 8.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202211565310.9A CN115981951A (en) | 2022-12-07 | 2022-12-07 | Monitoring data storage method based on OpenTSDB and HBase, computer device and storage medium |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202211565310.9A CN115981951A (en) | 2022-12-07 | 2022-12-07 | Monitoring data storage method based on OpenTSDB and HBase, computer device and storage medium |
Publications (1)
Publication Number | Publication Date |
---|---|
CN115981951A true CN115981951A (en) | 2023-04-18 |
Family
ID=85969055
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202211565310.9A Pending CN115981951A (en) | 2022-12-07 | 2022-12-07 | Monitoring data storage method based on OpenTSDB and HBase, computer device and storage medium |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN115981951A (en) |
-
2022
- 2022-12-07 CN CN202211565310.9A patent/CN115981951A/en active Pending
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN110321344B (en) | Information query method and device for associated data, computer equipment and storage medium | |
CN107247808B (en) | Distributed NewSQL database system and picture data query method | |
CN109144994A (en) | Index updating method, system and relevant apparatus | |
CN111339171B (en) | Data query method, device and equipment | |
US20160063107A1 (en) | Data retrieval via a telecommunication network | |
CN107480260B (en) | Big data real-time analysis method and device, computing equipment and computer storage medium | |
CN110704463A (en) | Local caching method and device for common data, computer equipment and storage medium | |
CN109656950B (en) | Recursive query method, device, server and storage medium | |
CN111782452A (en) | Method, system, device and medium for interface contrast test | |
CN112416710A (en) | User operation recording method and device, electronic equipment and storage medium | |
CN110222046B (en) | List data processing method, device, server and storage medium | |
EP2778962A1 (en) | Silo-aware databases | |
CN110928900B (en) | Multi-table data query method, device, terminal and computer storage medium | |
CN111858581B (en) | Paging query method and device, storage medium and electronic equipment | |
CN115981951A (en) | Monitoring data storage method based on OpenTSDB and HBase, computer device and storage medium | |
US11157506B2 (en) | Multiform persistence abstraction | |
EP2990960A1 (en) | Data retrieval via a telecommunication network | |
US8843708B2 (en) | Control block linkage for database converter handling | |
CN112181391A (en) | Method and system capable of dynamically expanding data | |
CN112346949A (en) | AOP-based operation log recording method and system, electronic device and medium | |
CN110866005A (en) | Internet of things data acquisition management method and system, storage medium and terminal | |
CN110781170B (en) | Historical data protection method and device based on AOP | |
WO2024016789A1 (en) | Log data query method and apparatus, and device and medium | |
US9672294B2 (en) | Partial data report generation with data costing notification | |
CN118349593A (en) | Information query method, device, equipment and storage medium |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination |