CN113190623A - Data processing method, device, server and storage medium - Google Patents
Data processing method, device, server and storage medium Download PDFInfo
- Publication number
- CN113190623A CN113190623A CN202110528221.6A CN202110528221A CN113190623A CN 113190623 A CN113190623 A CN 113190623A CN 202110528221 A CN202110528221 A CN 202110528221A CN 113190623 A CN113190623 A CN 113190623A
- Authority
- CN
- China
- Prior art keywords
- data
- target
- node
- pool
- time period
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 20
- 238000000034 method Methods 0.000 claims abstract description 44
- 238000013499 data model Methods 0.000 claims abstract description 38
- 238000012545 processing Methods 0.000 claims abstract description 17
- 238000013500 data storage Methods 0.000 claims description 35
- 238000004590 computer program Methods 0.000 claims description 9
- 230000004044 response Effects 0.000 claims description 5
- 238000012163 sequencing technique Methods 0.000 claims description 5
- 230000008569 process Effects 0.000 description 11
- 238000013461 design Methods 0.000 description 10
- 238000012544 monitoring process Methods 0.000 description 7
- 238000004891 communication Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 5
- 230000002159 abnormal effect Effects 0.000 description 3
- 238000005192 partition Methods 0.000 description 3
- 238000011160 research Methods 0.000 description 2
- 238000012827 research and development Methods 0.000 description 2
- 230000005856 abnormality Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000014509 gene expression Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000002688 persistence Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000007619 statistical method Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2308—Concurrency control
- G06F16/2315—Optimistic concurrency control
- G06F16/2322—Optimistic concurrency control using timestamps
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2471—Distributed queries
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Fuzzy Systems (AREA)
- Mathematical Physics (AREA)
- Probability & Statistics with Applications (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application provides a data processing method, a data processing device, a server and a storage medium, wherein the method comprises the following steps: extracting a target node and a target time period from a data acquisition instruction sent by a client, acquiring a target data pool matched with the target node and the target time period in a preset data model according to the target node and the target time period, acquiring a timestamp of each node data in the target data pool, and acquiring the total amount of the target node data and the target node data matched with the target time period from the target data pool according to the timestamp. According to the technical scheme, the node data are stored to the corresponding data pools by utilizing the timestamps of the node data, when the node data in the preset data model are read, the target node data matched with the target time period can be quickly extracted from the target data pools according to the target nodes and the target time period, the reading speed of the data is increased, and the acquisition efficiency of the node data is improved.
Description
Technical Field
The present application relates to the field of technologies, and in particular, to a data processing method, an apparatus, a server, and a storage medium.
Background
In the data processing process, data often flows through a plurality of nodes, and in order to ensure normal operation of the whole data processing flow, statistical analysis needs to be performed on the node data flowing in and the node data flowing out of each node to determine whether data processing of the current node is abnormal.
In the prior art, when acquiring node data flowing in or node data flowing out from a node, data flowing into the current node is stored by using a time sequence database, and then data query is performed on the time sequence database to acquire the flowing-in and flowing-out data of the current node.
However, in the time sequence database used in the prior art, the data storage structure is poor, the performance of the database is affected, and a great pressure exists in the data reading and writing process, which results in low node data acquisition efficiency.
Disclosure of Invention
The application provides a data processing method, a data processing device, a server and a storage medium, which are used for solving the problem of low acquisition efficiency of the existing node data.
In a first aspect, an embodiment of the present application provides a data processing method, including:
responding to a data acquisition instruction sent by a client, and extracting a target node and a target time period from the data acquisition instruction;
acquiring a target data pool matched with the target node and the target time period in a preset data model according to the target node and the target time period, wherein the target data pool is used for storing node data;
acquiring a timestamp of each node data in the target data pool, wherein the timestamp is used for indicating the time of the node data flow to the target node;
and acquiring the target node data matched with the target time period and the total amount of the target node data from the target data pool according to the timestamp.
In a possible design of the first aspect, the obtaining, according to the target node and the target time period, a target data pool matched with the target node and the target time period in a preset data model includes:
according to the target node, acquiring a target data pool set matched with the target node from the preset data model;
and acquiring a target data pool matched with the target time period from the target data pool set according to the target time period.
In another possible design of the first aspect, the obtaining, according to the target time period, a target data pool that matches the target time period from the target data pool set includes:
obtaining indexes of all data pools in the target data pool set;
and acquiring a target data pool matched with the target time period from each data pool according to the index of each data pool.
In yet another possible design of the first aspect, before obtaining, from each data pool according to the index of each data pool, a target data pool that matches the target time period, the method further includes:
determining a time zone corresponding to each data pool in the target data pool set according to preset time length and current time;
and constructing and obtaining indexes of the data pools according to the time sections corresponding to the data pools.
In another possible design of the first aspect, after the constructing the index of each data pool according to the time segment corresponding to each data pool, the method further includes:
acquiring a timestamp of each node data, and determining a data pool matched with each node data according to the timestamp and an index corresponding to each data pool;
and storing the data of each node into the data storage area of the matched data pool.
In yet another possible design of the first aspect, the storing the respective node data to the data storage area of the matching data pool includes:
acquiring identifiers and timestamps of each node data, wherein the timestamps are used for indicating the time of the node data flow to a target node;
storing the identifier and the timestamp of the node data to a data storage area of a matching data pool.
In yet another possible design of the first aspect, the storing the identifier and the timestamp of the node data to a data storage area of a matching data pool includes:
constructing an identification area of the data storage area and a scoring area associated with the identification area by using a data storage structure of a remote dictionary service;
storing an identifier of the node data into the identification area;
and storing the timestamp of the node data into the scoring area of the data storage area.
In yet another possible design of the first aspect, the obtaining, from the target data pool, the target node data that matches the target time period and the total number of the target node data according to the timestamp includes:
sequencing all node data stored in the target data pool according to the time stamp, and determining the arrangement sequence of all node data;
acquiring target node data matched with the target time period from the target data pool according to the arrangement sequence;
and counting the target node data to obtain the total amount of the target node data.
In yet another possible design of the first aspect, after obtaining, from the target data pool, the target node data that matches the target time period and the total number of the target node data according to the timestamp, the method further includes:
comparing the total amount of the target node data with a preset threshold;
and if the total amount of the target node data exceeds a preset threshold value, outputting an alarm prompt.
In yet another possible design of the first aspect, after obtaining, from the target data pool, the target node data that matches the target time period and the total number of the target node data, the method further includes:
determining a time zone corresponding to each data pool according to the index in each data pool in the preset data model;
acquiring the storage duration of the node data stored in each data pool according to the time section and the current time point;
and clearing the node data with the storage time length exceeding the preset time length.
In another possible design of the first aspect, before the obtaining, according to the target node and the target time period, a target data pool in a preset data model, which is matched with the target node and the target time period, the method further includes:
and caching the preset data model into a remote dictionary service.
In a second aspect, an embodiment of the present application provides a data processing apparatus, including:
the response module is used for responding to a data query instruction sent by the client and acquiring a target node and a target time period;
the matching module is used for acquiring a target data pool matched with the target node and the target time period in a preset data model according to the target node and the target time period, and the target data pool is used for storing node data;
an obtaining module, configured to obtain a timestamp of each node data in the target data pool, where the timestamp is used to indicate a time when the node data stream is transferred to a target node;
and the output module is used for acquiring the target node data matched with the target time period and the total amount of the target node data from the target data pool according to the timestamp.
In a third aspect, an embodiment of the present application provides a server, including a memory and at least one processor;
the memory stores computer-executable instructions;
the at least one processor executing the computer-executable instructions stored by the memory causes the at least one processor to perform the method of any one of the above.
In a fourth aspect, the present application provides a readable storage medium, in which computer instructions are stored, and when executed by a processor, the computer instructions are used to implement the method described above.
In a fifth aspect, the present application provides a computer program product comprising a computer program/instructions, which when executed by a processor, implement the method of any one of the above.
According to the data processing method, the data processing device, the server and the storage medium, the node data are stored to the corresponding data pools by utilizing the timestamps of the node data, when the node data in the preset data model are read, the target node data matched with the target time period can be quickly extracted from the target data pools according to the target nodes and the target time period, the data reading speed is increased, and the node data acquisition efficiency is improved.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and together with the description, serve to explain the principles of the application;
fig. 1 is a schematic view of a scene of a data processing method according to an embodiment of the present application;
fig. 2 is a schematic flowchart of a first embodiment of a data processing method according to an embodiment of the present application;
fig. 3 is a schematic flowchart of a second data processing method according to an embodiment of the present application;
fig. 4 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present application;
fig. 5 is a schematic structural diagram of a server according to an embodiment of the present application.
With the above figures, there are shown specific embodiments of the present application, which will be described in more detail below. These drawings and written description are not intended to limit the scope of the inventive concepts in any manner, but rather to illustrate the inventive concepts to those skilled in the art by reference to specific embodiments.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
The terms referred to in this application are explained first:
Redis：
remote Dictionary service (Remote Dictionary Server, redis) refers to an open source, log-type, Key-Value database written in a programming language, supporting a network, and based on a memory or a persistence.
Fig. 1 is a scene schematic diagram of a data processing method provided in an embodiment of the present application, as shown in fig. 1, a user may browse various articles on a website platform through a terminal device 10, for an article of interest, the user may pay a fee to perform a transaction, generate a transaction order, corresponding order data may be cached in a server 11, the order data may sequentially flow in order flow nodes, for example, the order data may flow in flow nodes such as a next order node, a node to be paid, a payment completion node, a node to be issued, and an order completion node, and in order to ensure that the transaction can be completed normally, a special monitoring platform may be set, the flow nodes may be monitored, inflow and outflow data of the flow nodes may be monitored, for example, in the node to be paid, the order data flowing into the node to be paid and the order data flowing out from the node to be paid are monitored.
In the prior art, the monitoring of the inflow and outflow data mainly uses a time sequence database, for example, the server 20 uses an infiluxdb or other time sequence database to store real-time data, and when monitoring is needed, data query is performed from the time sequence database, and when there is an abnormality in the inflow and outflow data, an alarm is given to an operation and maintenance person of the website platform. However, due to the fact that the number of transaction orders of the website platform is large, the performance of a time sequence database used in the prior art is poor, the read-write pressure of data is very large, when data is inquired from the time sequence database, the efficiency of data inquiry is poor, and the process node data cannot be effectively monitored.
In order to solve the above problems, embodiments of the present application provide a data processing method, an apparatus, a server, and a readable storage medium, where a preset data model is constructed, node data is stored in a matched data pool in the preset data model according to a timestamp of the node data, and when a client needs to query the node data, the matched node data can be quickly selected from the preset data model, so as to enhance read-write performance, thereby effectively monitoring the node data.
The technical solution of the present application will be described in detail below with reference to specific examples. It should be noted that the following specific embodiments may be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.
Fig. 2 is a schematic flowchart of a first embodiment of a data processing method provided in the embodiment of the present application, where the method may be applied to a server, and as shown in fig. 2, the method specifically includes the following steps:
s201, responding to a data acquisition instruction sent by a client, and extracting a target node and a target time period from the data acquisition instruction.
Specifically, the client may be a monitoring platform, and in the process of monitoring the node data, research and development personnel of the monitoring platform may specify the target node and the target time period through the data acquisition instruction to monitor the node data of the target node and the target time period.
In this embodiment, the target node may be a flow node for payment, delivery, receipt, order completion, order closing, and the like, and the target time period may be the first five minutes or the first ten minutes of the current time.
For example, taking the node to be paid as the target node, the developer may set the first 10 minutes of the current time as the target time period, and search the data pool of the server for the amount of the node data flowing into the node to be paid in the first ten minutes.
Optionally, the research and development staff may set a time period, each time a time period passes, the client sends a data acquisition instruction, and the target node and the target time period included in the data acquisition instruction sent in each time period may be different.
S202, acquiring a target data pool matched with the target node and the target time period in the preset data model according to the target node and the target time period.
The target data pool is used for storing node data. Illustratively, the node data may be order data, wherein the order data includes at least an order number.
Specifically, the preset data model includes a plurality of data pools, and each data pool can store a plurality of node data. For example, the data pools in the preset data model may be partitioned according to the nodes and the timestamps, for example, four data pools are included in the node to be paid, a first data pool is used for storing node data flowing into the node to be paid 5 minutes before the current time, a second data pool is used for storing data flowing into the node to be paid 10 minutes before the current time and 5 minutes before the current time, and the third data pool and the fourth data pool are analogized in sequence.
For example, to distinguish the data pools, an index may be constructed for each data pool, and the index may be named by a timestamp, for example, if the current time is 20 points at 09/2021, the index of the first data pool of the nodes to be paid may be named 202103091220, the index of the second data pool may be named 202103091215, the index of the third data pool may be named 202103091210, and the index of the fourth data pool may be named 202103091205.
For example, the data structure of the data pool set of the preset data model is shown in the following table:
in the above table, the index of the first data pool is named 202103091220, the index of the second data pool is named 202103091215, the index of the third data pool is named 202103091210, the index of the fourth data pool is named 202103091205, wherein the members stored in the first data pool include node data having time stamps between 12 hours 15 minutes at 03 month 09 at 2021 and 12 hours at 12 days at 2021 month 03, the members stored in the second data pool include node data having time stamps between 10 minutes at 12 hours at 03 month 09 at 2021 year 12 and 15 minutes at 12 hours at 2021 month 03, the members stored in the third data pool include node data having time stamps between 05 minutes at 12 hours at 09 months at 2021 year 03 and 12 hours at 09 months at 2021 and 10 minutes at 12 hours at 09 months at 2021 year 03, and the members stored in the fourth data pool include node data having time stamps between 00 minutes at 12 hours at 09 months at 2021 year 03 and 12 hours at 09 months at 2021 and 05 minutes at 12 hours at 12 days at 2021 year 03.
In this embodiment, if the target node is a node to be paid, a data pool matched with the target time period is selected from the nodes to be paid as the target data pool according to the target time period.
For example, if the current time is 20 points at 09/12/3/2021, and the target time period is from 10 minutes before the current time to the current time, the target data pools matched with the target time period are the first data pool and the second data pool.
S203, obtaining the time stamp of each node data in the target data pool.
Wherein the timestamp is used for indicating the time when the node data stream is transferred to the target node.
Specifically, taking a node to be paid and a payment completion node as an example, when a user completes payment in a transaction process, node data is transferred from the node to be paid to the payment completion node, and the time when the node data is transferred to the payment completion node is used as a timestamp of the node data.
It can be understood that after the node data is circulated to a node, the node may stay at the node for a certain time due to a delay or a wait, and therefore, when the node data is circulated in the node flow, the timestamp of the node data at different nodes may change.
In this embodiment, the data pools are used for storing node data, each node corresponds to a plurality of data pools, when node data flows into a certain node, the node data is stored in the data pool corresponding to the node, and the target data pool is one or more selected from the data pools corresponding to the node.
Illustratively, the node data may include a timestamp and an order number, and when the node data is stored in the data pool, the timestamp of the node data is also stored in the data pool.
Optionally, when the node data is stored in the data pool, the timestamp of the node data may be converted into score data by using a preset algorithm, and the score data is also stored in the data pool.
And S204, acquiring the target node data matched with the target time period and the total amount of the target node data from the target data pool according to the timestamp.
Specifically, each node data corresponds to a time stamp when flowing into the node, and if the time stamp of the node data is within the target time period, the node data in the target data pool may be used as the target node data.
Illustratively, taking a target time period as [ a, b ] as an example, where a represents the last five minutes, and b represents the current time, then all the node data with the timestamp of [ a, b ] in the target data pool are target node data, and these target node data are counted to obtain the total amount of the target node data.
According to the embodiment of the application, the node data are stored to the corresponding data pools by utilizing the timestamps of the node data, when the node data in the preset data model are read, the target node data matched with the target time period can be quickly extracted from the target data pools according to the target nodes and the target time period, the data reading speed is increased, and the node data acquisition efficiency is improved.
On the basis of the foregoing embodiments, in some embodiments, the step S202 may be specifically implemented by the following steps:
acquiring a target data pool set matched with a target node from a preset data model according to the target node;
and acquiring a target data pool matched with the target time period from the target data pool set according to the target time period.
Specifically, the target node and the target time period may be specified by a developer, for example, the target node may be a node to be paid, each data pool set corresponds to one process node according to a preset corresponding relationship, and the target data pool set corresponding to the target node may be found according to the corresponding relationship.
In this embodiment, each data pool set includes a plurality of data pools, each data pool may be divided into different time zones, and node data is stored in the data pools divided into different time zones according to timestamps of the node data, for example, a target data pool includes a first data pool, a second data pool, a third data pool, and a fourth data pool, the time zone divided by the first data pool is [ five minutes before the current time, the current time ], the time zone divided by the second data pool is [ ten minutes before the current time, five minutes before the current time ], and so on, if the timestamp of the node data is in the [ five minutes before the current time, the current time ] time zone, the node data is stored in the first data pool.
And matching the target time period with the divided time sections so as to determine a target data pool. Illustratively, if the target time period is [ ten minutes before the current time, the current time ], the target data pools matched with the target time period are the first data pool and the second data pool.
According to the method and the device, the target data pool in the target data pool set is determined by using the target node and the target time period, so that the server can quickly inquire the target node data from the target data pool, and the data inquiry efficiency is improved.
Optionally, on the basis of the foregoing embodiments, in some embodiments, the "obtaining a target data pool matched with a target time period from a target data pool set according to the target time period" may specifically be implemented by the following steps:
acquiring indexes of all data pools in a target data pool set;
and acquiring a target data pool matched with the target time period from each data pool according to the index of each data pool.
In this embodiment, after each data pool is divided according to time zones, an index may be configured for each data pool, where the index is used to distinguish the time zone corresponding to each data pool, for example, the time zone corresponding to the first data pool is [ five minutes before the current time, the current time ], the index of the first data pool may be named after the current time, for example, 202103091220, the time zone corresponding to the second data pool is [ ten minutes before the current time, five minutes before the current time ], and the index of the second data pool may be named after five minutes before the current time, for example, 202103091215.
If the target time period is [202103091215, 202103091220], the matched target data pools are the first data pool and the second data pool.
According to the embodiment of the application, each data pool can be distinguished by constructing the index of each data pool, so that the node data can be stored in the matched data pool according to the index of the data pool, and meanwhile, the node data stored in the data pool can be conveniently inquired.
Optionally, in some embodiments, if each data pool set includes a plurality of data pools, the data processing method may further include the following steps:
determining a time zone corresponding to each data pool in the target data pool set according to preset time length and current time;
and constructing and obtaining indexes of the data pools according to the time sections corresponding to the data pools.
Specifically, the preset duration may be input by a developer, and the preset duration is used to indicate the length of the time zone, for example, if the preset duration is five minutes, the current time is 202103091220, and the data pool set includes a first data pool, a second data pool, a third data pool, and a fourth data pool, the index of the first data pool is 202103091220, the index of the second data pool is 202103091215, the index of the third data pool is 202103091210, and the index of the fourth data pool is 202103091205.
According to the embodiment of the application, the corresponding time zones are divided for each data pool through the preset duration and the current time, the index of each data pool is built, the node data can be stored to the matched data pools according to the timestamps of the node data, and when the node data is inquired, the target data pools can be rapidly determined according to the indexes of the data pools, so that the data inquiry efficiency is improved.
On the basis of the foregoing embodiment, in some embodiments, if each data pool has a corresponding index, the foregoing data processing method may further include the following steps:
acquiring a timestamp of each node data, and determining a data pool matched with each node data according to the timestamp and an index corresponding to each data pool;
and storing the data of each node into the data storage area of the matched data pool.
In this embodiment, the data pool includes an index area and a data storage area, where the index area is used to store an index corresponding to the data pool, and the data storage area is used to store node data.
Specifically, each node data has a corresponding timestamp, the index of the data pool may be used to indicate the corresponding time segment, for example, if the index of the data pool is 202103091220, the corresponding time segment may be [202103091215, 202103091220], and if the timestamp of the node data is located in the time segment, the node data is stored in the data storage area of the data pool.
According to the data pool matching method and device, the index of the data pool is utilized to determine the data pool matched with each node data, the node data with different timestamps can be matched with the corresponding data pool and stored in the data storage area of the corresponding data pool, different node data can be conveniently classified and stored, and data query is conveniently performed subsequently.
On the basis of the foregoing embodiments, in some embodiments, the "storing each node data in the data storage area of the matched data pool" may be specifically implemented by the following steps:
acquiring identifiers and timestamps of data of all nodes;
the identifier and the timestamp of the node data are stored to a data storage area of the matching data pool.
In this embodiment, the identifier may be a digital code, each node data corresponds to a unique identifier, and for example, if the node data is order data, the identifier may be an order number, and order information and an order status may be queried through the order number.
According to the embodiment of the application, different node data are identified through the identifier, confusion of each node data stored in the data storage area can be avoided, and the accuracy of data storage is improved.
On the basis of the foregoing embodiments, in some embodiments, the "storing the identifier and the timestamp of the node data in the data storage area of the matched data pool" may be specifically implemented by the following steps:
constructing an identification area of the data storage area and a scoring area associated with the identification area by using a data storage structure of a remote dictionary service;
storing the identifier of the node data into an identification area;
and storing the timestamp of the node data into the scoring area of the data storage area.
In this embodiment, the remote dictionary service, Redis, builds a data storage area by using set scaling of Redis, where an identification area is used to store an identifier and a scoring area associated with the identification area is used to store a timestamp.
Illustratively, taking the node data as the order data as an example, each order data as a member occupies a storage position, and the storage position comprises an identification area and a scoring area, wherein the order number is stored in the identification area, and the timestamp of the order data is stored in the scoring area.
According to the embodiment of the application, the data storage area is partitioned, so that the node data can store corresponding information to the corresponding partition according to the partition, the information of the node data is conveniently extracted, the timestamp stored in the partition can be utilized subsequently, the node data is screened and sorted, and the query efficiency of the data is improved.
On the basis of the foregoing embodiments, in some embodiments, the step S204 may be specifically implemented by the following steps:
according to the time stamps, sequencing all node data stored in a target data pool, and determining the arrangement sequence of all node data;
acquiring target node data matched with the target time period from a target data pool according to the arrangement sequence;
and counting the target node data to obtain the total amount of the target node data.
In this embodiment, the target data pool stores a plurality of node data, and each node data has a corresponding timestamp.
For example, target node data in the target data pool may be counted by using a zcount command of Redis, where the zcount command is used to calculate the number of node data in a specified timestamp interval in the target data pool.
According to the method and the device, the node data are sequenced by utilizing the timestamps, the target node data matched with the target time period can be quickly found from the sequencing sequence, and the data query efficiency is improved.
Fig. 3 is a schematic flow chart of a second embodiment of a data processing method provided in the embodiment of the present application, and as shown in fig. 3, the data processing method includes the following steps:
s301, responding to a data acquisition instruction sent by the client, and extracting a target node and a target time period from the data acquisition instruction.
S302, according to the target node and the target time period, a target data pool matched with the target node and the target time period in the preset data model is obtained.
S303, obtaining the time stamp of each node data in the target data pool.
And S304, acquiring the target node data matched with the target time period and the total amount of the target node data from the target data pool according to the timestamp.
S305, comparing the total amount of the target node data with a preset threshold.
And S306, if the total amount of the target node data exceeds a preset threshold, outputting an alarm prompt.
The target data pool is used for storing node data, and the timestamp is used for indicating the time when the node data stream is transferred to the target node.
In the present embodiment, steps S301 to S304 are the same as steps S201 to S204 in the above embodiment, and the present embodiment mainly describes steps S305 to S306.
The alarm prompt can be a short message prompt and the like, when the total amount of the target node data exceeds a preset threshold value, the condition that the node data is overstocked is shown, the node in the whole process is possibly abnormal, and the server sends a short message to terminal equipment of research personnel to perform alarm prompt.
According to the embodiment of the application, the total amount of the target node data is compared with the preset threshold value, when the total amount of the target node data exceeds the preset threshold value, the alarm prompt is output, the research personnel can conveniently and quickly react to abnormal conditions, and the node data is effectively monitored.
For example, in some embodiments, the data processing method may further include the following steps:
acquiring indexes of all data pools in a preset data model, and determining time sections corresponding to all the data pools according to the indexes;
acquiring the storage duration of the node data in each data pool according to the time section and the current time;
and clearing the node data with the storage time length exceeding the preset time length.
Specifically, the preset time duration may be 30 minutes, a plurality of data pool sets are created in the preset data model, each data pool set creates a plurality of data pools, each data pool has its corresponding time segment, for example, the time segment corresponding to the created first data pool is [202103091205, 202103091210], and if the current time is 202103091240, it indicates that the time of creating the first data pool exceeds 30 minutes, and the node data stored in the first data pool may be cleared.
For example, the developer may set a delay task, and the server may clear the node data according to the delay task.
According to the embodiment of the application, the node data with the storage time length exceeding the preset time length is cleared, the storage space of the preset data model occupied by the node data can be reduced, the reduction of the read-write performance of the data is avoided, and the data query efficiency is improved.
Optionally, in some embodiments, the data processing method further includes the following steps:
and caching the preset data model into a remote dictionary service.
For example, the preset data model may also be stored in other storage media, such as a relational database, a non-relational database, and the like.
According to the embodiment of the application, the preset data model is cached in the remote dictionary service, the distributed mechanism and the high-performance storage characteristic of the remote dictionary service can be utilized, the data reading and writing performance is improved, and the node data is effectively inquired and monitored.
The following are embodiments of the apparatus of the present application that may be used to perform embodiments of the method of the present application. For details which are not disclosed in the embodiments of the apparatus of the present application, reference is made to the embodiments of the method of the present application.
Fig. 4 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present application, where the data processing apparatus may be integrated in a server, or may be independent of the server and cooperate with the server to implement the technical solution of the present application. As shown in fig. 4, the data processing apparatus 40 includes a response module 41, a matching module 42, an obtaining module 43, and an output module 44.
The response module 41 is configured to, in response to a data query instruction issued by a client, acquire a target node and a target time period. The matching module 42 is configured to obtain a target data pool in the preset data model, where the target data pool matches the target node and the target time period, according to the target node and the target time period. The obtaining module 43 is configured to obtain a timestamp of each node data in the target data pool. And the output module 44 is configured to obtain the target node data and the total amount of the target node data, which are matched with the target time period, from the target data pool according to the timestamp.
The target data pool is used for storing node data, and the timestamp is used for indicating the time when the node data stream is transferred to the target node.
In some embodiments, the matching module 42 may be specifically configured to:
acquiring a target data pool set matched with a target node from a preset data model according to the target node;
and acquiring a target data pool matched with the target time period from the target data pool set according to the target time period.
Optionally, in some embodiments, the matching module 42 may be specifically configured to:
acquiring indexes of all data pools in a target data pool set;
and acquiring a target data pool matched with the target time period from each data pool according to the index of each data pool.
On the basis of the foregoing embodiments, in some embodiments, the data processing apparatus 40 further includes a building module, configured to:
determining a time zone corresponding to each data pool in the target data pool set according to preset time length and current time;
and constructing and obtaining indexes of the data pools according to the time sections corresponding to the data pools.
On the basis of the foregoing embodiments, in some embodiments, the data processing apparatus 40 further includes a storage module, configured to:
acquiring a timestamp of each node data, and determining a data pool matched with each node data according to the timestamp and an index corresponding to each data pool;
and storing the data of each node into the data storage area of the matched data pool.
On the basis of the foregoing embodiments, in some embodiments, the storage module is specifically configured to:
acquiring identifiers and timestamps of data of all nodes;
the identifier and the timestamp of the node data are stored to a data storage area of the matching data pool.
Optionally, on the basis of the foregoing embodiments, in some embodiments, the storage module is specifically configured to:
constructing an identification area of the data storage area and a scoring area associated with the identification area by using a data storage structure of a remote dictionary service;
storing the identifier of the node data into an identification area;
and storing the timestamp of the node data into the scoring area of the data storage area.
In some embodiments, the output module 44 may be specifically configured to:
according to the time stamps, sequencing all node data stored in a target data pool, and determining the arrangement sequence of all node data;
acquiring target node data matched with the target time period from a target data pool according to the arrangement sequence;
and counting the target node data to obtain the total amount of the target node data.
On the basis of the foregoing embodiment, in some embodiments, the data processing apparatus 40 further includes an alarm module, where the alarm module is configured to:
comparing the total amount of the target node data with a preset threshold;
and if the total amount of the target node data exceeds a preset threshold value, outputting an alarm prompt.
On the basis of the foregoing embodiment, in some embodiments, the data processing apparatus 40 further includes a clearing module, configured to:
acquiring indexes of all data pools in a preset data model, and determining time sections corresponding to all the data pools according to the indexes;
acquiring the storage duration of the node data in each data pool according to the time section and the current time;
and clearing the node data with the storage time length exceeding the preset time length.
On the basis of the foregoing embodiment, in some embodiments, the data processing apparatus 40 further includes a cache module, and the cache module is configured to:
and caching the preset data model into a remote dictionary service.
Fig. 5 is a schematic structural diagram of a server according to an embodiment of the present application. As shown in fig. 5, the server 50 includes: at least one processor 51, memory 52, bus 53, and communication interface 54.
Wherein: the processor 51, communication interface 54, and memory 52 communicate with each other via a communication bus 54. A communication interface 54 for communicating with other devices. The bus 53 is used for connecting the processor 51, the memory 52 and the communication interface 53
The memory 52 stores computer-executable instructions, and optionally, the memory 52 may also store a preset data model.
The processor 51 executes computer-executable instructions stored by the memory 52 to cause the at least one processor 41 to perform the methods described above.
Optionally, the present embodiment also provides a readable storage medium, in which computer instructions are stored, and when executed by a processor, the computer instructions are used to implement the method as described above.
Optionally, the present embodiment also provides a computer program product, which includes a computer program/instruction, and the computer program/instruction is stored in a readable storage medium. The computer program/instructions may be read by at least one processor from a readable storage medium and when executed by the at least one processor, perform the method as described above.
In the present application, "at least one" means one or more, "a plurality" means two or more. "and/or" describes the association relationship of the associated objects, meaning that there may be three relationships, e.g., a and/or B, which may mean: a exists alone, A and B exist simultaneously, and B exists alone, wherein A and B can be singular or plural. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship; in the formula, the character "/" indicates that the preceding and following related objects are in a relationship of "division". "at least one of the following" or similar expressions refer to any combination of these items, including any combination of the singular or plural items. For example, at least one (one) of a, b, or c, may represent: a, b, c, a-b, a-c, b-c, or a-b-c, wherein a, b, c may be single or multiple.
It is to be understood that the various numerical references referred to in the embodiments of the present application are merely for convenience of description and distinction and are not intended to limit the scope of the embodiments of the present application. In the embodiment of the present application, the sequence numbers of the above-mentioned processes do not mean the execution sequence, and the execution sequence of each process should be determined by its function and inherent logic, and should not constitute any limitation to the implementation process of the embodiment of the present application.
Finally, it should be noted that: the above embodiments are only used for illustrating the technical solutions of the present application, and not for limiting the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present application.
Claims (15)
1. A data processing method, comprising:
responding to a data acquisition instruction sent by a client, and extracting a target node and a target time period from the data acquisition instruction;
acquiring a target data pool matched with the target node and the target time period in a preset data model according to the target node and the target time period, wherein the target data pool is used for storing node data;
acquiring a timestamp of each node data in the target data pool, wherein the timestamp is used for indicating the time of the node data flow to the target node;
and acquiring the target node data matched with the target time period and the total amount of the target node data from the target data pool according to the timestamp.
2. The method according to claim 1, wherein the obtaining a target data pool in a preset data model, which matches the target node and the target time period, according to the target node and the target time period comprises:
acquiring a target data pool set matched with the target node from the preset data model according to the target node;
and acquiring a target data pool matched with the target time period from the target data pool set according to the target time period.
3. The method of claim 2, wherein obtaining a target data pool from the set of target data pools according to the target time period, the target data pool matching the target time period comprises:
obtaining indexes of all data pools in the target data pool set;
and acquiring a target data pool matched with the target time period from each data pool according to the index of each data pool.
4. The method according to claim 3, wherein before obtaining the target data pool matching the target time period from each data pool according to the index of each data pool, further comprising:
determining a time zone corresponding to each data pool in the target data pool set according to preset time length and current time;
and constructing and obtaining indexes of the data pools according to the time sections corresponding to the data pools.
5. The method according to claim 4, wherein after the constructing the index of each data pool according to the time segment corresponding to each data pool, further comprising:
acquiring a timestamp of each node data, and determining a data pool matched with each node data according to the timestamp and an index corresponding to each data pool;
and storing the data of each node into the data storage area of the matched data pool.
6. The method of claim 5, wherein storing the respective node data to the data storage area of the matching data pool comprises:
acquiring identifiers and timestamps of data of all nodes;
storing the identifier and the timestamp of the node data to a data storage area of a matching data pool.
7. The method of claim 6, wherein storing the identifier and the timestamp of the node data to a data storage area of a matching data pool comprises:
constructing an identification area of the data storage area and a scoring area associated with the identification area by using a data storage structure of a remote dictionary service;
storing an identifier of the node data into the identification area;
and storing the timestamp of the node data into the scoring area of the data storage area.
8. The method of claim 1, wherein obtaining the target node data matching the target time period and the total number of the target node data from the target data pool according to the timestamp comprises:
sequencing all node data stored in the target data pool according to the time stamp, and determining the arrangement sequence of all node data;
acquiring target node data matched with the target time period from the target data pool according to the arrangement sequence;
and counting the target node data to obtain the total amount of the target node data.
9. The method according to any one of claims 1-8, wherein after obtaining the target node data matching the target time period and the total number of the target node data from the target data pool according to the timestamp, further comprising:
comparing the total amount of the target node data with a preset threshold;
and if the total amount of the target node data exceeds a preset threshold value, outputting an alarm prompt.
10. The method according to any one of claims 1-8, wherein after obtaining the target node data matching the target time period and the total number of the target node data from the target data pool, further comprising:
acquiring indexes of all data pools in the preset data model, and determining time sections corresponding to all the data pools according to the indexes;
acquiring the storage duration of the node data in each data pool according to the time section and the current time;
and clearing the node data with the storage time length exceeding the preset time length.
11. The method according to any one of claims 1 to 8, wherein before obtaining the target data pool matched with the target node and the target time period in the preset data model according to the target node and the target time period, the method further comprises:
and caching the preset data model into a remote dictionary service.
12. A data processing apparatus, comprising:
the response module is used for responding to a data query instruction sent by the client and acquiring a target node and a target time period;
the matching module is used for acquiring a target data pool matched with the target node and the target time period in a preset data model according to the target node and the target time period, and the target data pool is used for storing node data;
an obtaining module, configured to obtain a timestamp of each node data in the target data pool, where the timestamp is used to indicate a time when the node data stream is transferred to a target node;
and the output module is used for acquiring the target node data matched with the target time period and the total amount of the target node data from the target data pool according to the timestamp.
13. A server, comprising a memory and at least one processor;
the memory stores computer-executable instructions;
the at least one processor executing the computer-executable instructions stored by the memory causes the at least one processor to perform the method of any one of claims 1-11.
14. A readable storage medium having stored therein computer instructions, which when executed by a processor, are adapted to implement the method of any one of claims 1-11.
15. A computer program product comprising computer programs/instructions, characterized in that the computer programs/instructions, when executed by a processor, implement the method of any of claims 1-11.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110528221.6A CN113190623B (en) | 2021-05-14 | 2021-05-14 | Data processing method, device, server and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110528221.6A CN113190623B (en) | 2021-05-14 | 2021-05-14 | Data processing method, device, server and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113190623A true CN113190623A (en) | 2021-07-30 |
| CN113190623B CN113190623B (en) | 2024-05-17 |
Family
ID=76981897
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110528221.6A Active CN113190623B (en) | 2021-05-14 | 2021-05-14 | Data processing method, device, server and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113190623B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113609139A (en) * | 2021-09-30 | 2021-11-05 | 苏州浪潮智能科技有限公司 | Monitoring data management method and device, electronic equipment and storage medium |
| CN114756591A (en) * | 2022-04-15 | 2022-07-15 | 成都卓讯智安科技有限公司 | Data screening method and system based on bidirectional linked list |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20170170921A1 (en) * | 2015-12-09 | 2017-06-15 | Red Hat, Inc. | Timestamp alignment across a plurality of computing devices |
| CN107590157A (en) * | 2016-07-08 | 2018-01-16 | 腾讯科技（深圳）有限公司 | A kind of date storage method, data query method and relevant device |
| US20190034272A1 (en) * | 2012-06-05 | 2019-01-31 | International Business Machines Corporation | Utilizing concentric storage pools in a dispersed storage network |
| CN109447598A (en) * | 2018-09-25 | 2019-03-08 | 平安科技（深圳）有限公司 | Payment data processing method, device, computer equipment and storage medium |
| CN110489417A (en) * | 2019-07-25 | 2019-11-22 | 深圳壹账通智能科技有限公司 | A kind of data processing method and relevant device |
| CN111475105A (en) * | 2020-03-11 | 2020-07-31 | 平安科技（深圳）有限公司 | Monitoring data storage method, device, server and storage medium |
-
2021
- 2021-05-14 CN CN202110528221.6A patent/CN113190623B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20190034272A1 (en) * | 2012-06-05 | 2019-01-31 | International Business Machines Corporation | Utilizing concentric storage pools in a dispersed storage network |
| US20170170921A1 (en) * | 2015-12-09 | 2017-06-15 | Red Hat, Inc. | Timestamp alignment across a plurality of computing devices |
| CN107590157A (en) * | 2016-07-08 | 2018-01-16 | 腾讯科技（深圳）有限公司 | A kind of date storage method, data query method and relevant device |
| CN109447598A (en) * | 2018-09-25 | 2019-03-08 | 平安科技（深圳）有限公司 | Payment data processing method, device, computer equipment and storage medium |
| CN110489417A (en) * | 2019-07-25 | 2019-11-22 | 深圳壹账通智能科技有限公司 | A kind of data processing method and relevant device |
| WO2021012553A1 (en) * | 2019-07-25 | 2021-01-28 | 深圳壹账通智能科技有限公司 | Data processing method and related device |
| CN111475105A (en) * | 2020-03-11 | 2020-07-31 | 平安科技（深圳）有限公司 | Monitoring data storage method, device, server and storage medium |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113609139A (en) * | 2021-09-30 | 2021-11-05 | 苏州浪潮智能科技有限公司 | Monitoring data management method and device, electronic equipment and storage medium |
| CN114756591A (en) * | 2022-04-15 | 2022-07-15 | 成都卓讯智安科技有限公司 | Data screening method and system based on bidirectional linked list |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113190623B (en) | 2024-05-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106339274B (en) | A kind of method and system that data snapshot obtains | |
| US11294960B1 (en) | Entity mapping | |
| CN107608862A (en) | Monitoring alarm method, monitoring alarm device and computer-readable recording medium | |
| CN113190623A (en) | Data processing method, device, server and storage medium | |
| CN104966172A (en) | Large data visualization analysis and processing system for enterprise operation data analysis | |
| CN113283675B (en) | Index data analysis method, device, equipment and storage medium | |
| CN113590824A (en) | Method and device for constructing causal graph and related equipment | |
| CN113592019A (en) | Fault detection method, device, equipment and medium based on multi-model fusion | |
| CN110674014A (en) | Method and device for determining abnormal query request | |
| CN112445875A (en) | Data association and inspection method and device, electronic equipment and storage medium | |
| CN112800061B (en) | Data storage method, device, server and storage medium | |
| CN110060053A (en) | A kind of recognition methods, equipment and computer-readable medium | |
| CN115274122A (en) | Health medical data management method, system, electronic device and storage medium | |
| CN109871368A (en) | Database detection method, apparatus, computer installation and storage medium | |
| CN112925664A (en) | Target user determination method and device, electronic equipment and storage medium | |
| CN115577701A (en) | Risk behavior identification method, device, equipment and medium for big data security | |
| CN111695979A (en) | Method, device and equipment for analyzing relation between raw material and finished product | |
| CN106909492A (en) | The method for tracing and device of business datum | |
| CN113590835A (en) | Method and device for establishing knowledge graph of textile industry data and processor | |
| CN105719072A (en) | System and method for associating multistage assembly transactions | |
| CN110941536B (en) | Monitoring method and system, and first server cluster | |
| CN111523921B (en) | Funnel analysis method, analysis device, electronic device, and readable storage medium | |
| CN115470279A (en) | Data source conversion method, device, equipment and medium based on enterprise data | |
| CN115495587A (en) | Alarm analysis method and device based on knowledge graph | |
| CN113535458B (en) | Abnormal false alarm processing method and device, storage medium and terminal |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information | ||
| CB02 | Change of applicant information |
Address after: 601, 6 / F, building 2, No. 18, Kechuang 11th Street, Daxing District, Beijing, 100176 Applicant after: Jingdong Technology Information Technology Co.,Ltd. Address before: 601, 6 / F, building 2, No. 18, Kechuang 11th Street, Beijing Economic and Technological Development Zone, Beijing 100176 Applicant before: Jingdong Shuke Haiyi Information Technology Co.,Ltd. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |