WO2018121335A1 - 实时数据处理方法和装置 - Google Patents
实时数据处理方法和装置 Download PDFInfo
- Publication number
- WO2018121335A1 WO2018121335A1 PCT/CN2017/117082 CN2017117082W WO2018121335A1 WO 2018121335 A1 WO2018121335 A1 WO 2018121335A1 CN 2017117082 W CN2017117082 W CN 2017117082W WO 2018121335 A1 WO2018121335 A1 WO 2018121335A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- real
- time data
- time
- completeness
- predetermined condition
- Prior art date
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/485—Task life-cycle, e.g. stopping, restarting, resuming execution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/302—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system component is a software system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3466—Performance evaluation by tracing or monitoring
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3466—Performance evaluation by tracing or monitoring
- G06F11/3476—Data logging
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2358—Change logging, detection, and notification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24568—Data stream processing; Continuous queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2462—Approximate or statistical queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5061—Partitioning or combining of resources
- G06F9/5072—Grid computing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2201/00—Indexing scheme relating to error detection, to error correction, and to monitoring
- G06F2201/80—Database-specific techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2201/00—Indexing scheme relating to error detection, to error correction, and to monitoring
- G06F2201/805—Real-time
Definitions
- the present application relates to the field of data processing, and in particular to a real-time data processing method and apparatus.
- stream data is continuous, and I don't know when it is a head, so the data collected in real time changes in real time. Since the data of real-time statistics is constantly changing, the business side using the data has no sense of the progress of the data processing. Therefore, it is impossible to make some decisive behavior based on the data, for example, the sales within the preset minutes of the statistics. The amount is constantly changing. Decision makers cannot determine whether to make changes to the sales strategy based on the data. For example, if the real-time data in the alarm monitoring system is constantly changing, the system cannot make accurate data based on the data. Alarm behavior. That is, no parameters can be used in the prior art to evaluate the progress of real-time data processing.
- each time 25 is calculated once, and the second time is calculated again, and the real-time data is processed by such accumulation.
- the data is stored in the database each time it is processed. That is, every 25 will immediately go through the real-time calculation engine, and then go through the memory and finally reach the database, which is from the data source all the way to the database.
- the first option is currently the most used method in the industry. Because the integrity of the data can be guaranteed. But the downside is that the data in the database will constantly change, so you can't make accurate decisions.
- an expected timeout period for example, a 2 minute timeout. Assume that 4 of the 25 times are divided into 4 time periods, the first 3 times arrive in 2 minutes, and the 4th time arrives in 2 minutes. Then this kind of scheme will only record the first three 75, only one database, that is, the memory in Figure 3 will wait for 2 minutes, and will not be stored in the database after 2 minutes.
- the advantage of this is that for the database, as long as there is data, it will not change any more. You can make decisions directly based on this data. For example, if the transaction falls, you can quickly change the brand of the advertising space, which can be in the fastest time. Recover the transaction amount.
- the disadvantage of this scheme is that data that has passed the timeout period will be lost (25 of the fourth time as shown by the dotted line in the figure), so there will be errors.
- the embodiment of the present application provides a real-time data processing method and apparatus, so as to at least solve the problem that the prior art cannot know the progress caused by the progress of real-time data processing.
- a real-time data processing method includes: collecting real-time data received by one or more nodes in a distributed system; and real-time data satisfying the same predetermined condition on all nodes Performing a summary to obtain a total amount of real-time data corresponding to the same predetermined condition, wherein the predetermined condition is used to represent the time at which the real-time data is collected; and the completeness of the real-time data satisfying the same predetermined condition is calculated, wherein the completeness is used to indicate: The proportion of processed real-time data that meets the same predetermined conditions in real-time data on all nodes aggregated.
- a real-time data processing apparatus comprising: a collection module, configured to collect real-time data received by one or more nodes in a distributed system; Realizing the real-time data satisfying the same predetermined condition on all nodes, and obtaining the total amount of real-time data corresponding to the same predetermined condition, wherein the predetermined condition is used to represent the time when the real-time data is collected; and the calculation module is used to calculate the same The completeness of the real-time data of the predetermined condition, wherein the completeness is used to indicate the proportion of the processed real-time data satisfying the same predetermined condition in the real-time data on all the nodes aggregated.
- real-time data received by one or more nodes in a distributed system is collected, real-time data satisfying the same predetermined condition on all nodes is aggregated, and the total amount of real-time data corresponding to the same predetermined condition is obtained.
- the completeness of the real-time data satisfying the same predetermined condition is calculated, thereby realizing the calculation of the completeness of the real-time data.
- the processing of real-time data introduces a parameter, that is, the degree of completeness, through which the progress of real-time data processing can be known.
- the completeness is used to indicate the progress of processing the real-time data that satisfies the predetermined condition in the processed real-time data.
- the embodiment of the present application solves the technical problem that the prior art cannot know the progress of real-time data processing.
- FIG. 1 is a schematic diagram of a real-time data processing method according to the prior art
- FIG. 2 is a schematic diagram of a first real-time data processing method according to the prior art
- FIG. 3 is a schematic diagram of a second real-time data processing method according to the prior art
- FIG. 4 is a block diagram showing the hardware structure of a computer device of a real-time data processing method according to an embodiment of the present application
- FIG. 5 is a flowchart of a real-time data processing method according to an embodiment of the present application.
- FIG. 6 is a schematic diagram of an optional real-time data processing method according to an embodiment of the present application.
- FIG. 7 is a flowchart of an optional real-time data processing method according to an embodiment of the present application.
- FIG. 8 is a schematic diagram of a real-time data processing apparatus according to an embodiment of the present application.
- FIG. 9 is a structural block diagram of a computer terminal according to an embodiment of the present application.
- an embodiment of a real-time data processing method is provided, and it should be noted that the steps illustrated in the flowchart of the accompanying drawings may be executed in a computer system such as a set of computer executable instructions, and Although a logical order is shown in the flowcharts, in some cases the steps shown or described may be performed in a different order than the ones described herein.
- the method embodiments provided by the embodiments of the present application may be processed on a server.
- a processing result query service may also be provided.
- the operation result on the server may be viewed through a webpage or a client.
- a server can be understood as a computer.
- cloud computing has been widely used, and the method provided in the embodiments of the present application can also be widely used in cloud computing.
- the computing power of the terminal is also enhanced with the development of the technology.
- the terminal When the terminal can obtain corresponding data for calculation, for example, the terminal may include, but is not limited to, a mobile phone, a tablet computer, and other portable devices.
- deploying the following embodiments on the server is a preferred option.
- the hardware architectures that servers, terminals, and cloud computing rely on are similar, and can be regarded as a kind of computer equipment.
- Embodiments of the present application can be implemented in such a computer device.
- the hardware architecture of the computer device has changed, or the computing device of the new architecture has appeared.
- the following embodiments of the present application can also be implemented.
- the architecture of the computing device in FIG. 4 will be described below as an example.
- computer terminal 40 may include one or more (shown in the figures 402a, 402b, ..., 402n) processor 402 (processor 402 may include, but is not limited to, a microprocessor MCU or programmable A processing device such as a logic device FPGA, a memory 404 for storing data, and a transmission device for communication functions. In addition, it can also include: display, input/output interface (I/O interface), universal serial bus (USB) port (which can be included as one of the ports of the I/O interface), network interface, power supply And / or camera.
- processor 402 may include, but is not limited to, a microprocessor MCU or programmable A processing device such as a logic device FPGA, a memory 404 for storing data, and a transmission device for communication functions.
- processor 402 may include, but is not limited to, a microprocessor MCU or programmable A processing device such as a logic device FPGA, a memory 404 for storing data, and a transmission device for
- computer terminal 40 may also include more or fewer components than those shown in FIG. 4, or have a different configuration than that shown in FIG.
- processors 402 and/or other data processing circuits may be referred to herein generally as "data processing circuits.”
- the data processing circuit may be embodied in whole or in part as software, hardware, firmware or any other combination.
- the data processing circuitry can be a single, separate processing module, or incorporated in whole or in part into any of the other components in computer terminal 40.
- the data processing circuit is controlled as a processor (e.g., selection of a variable resistance terminal path connected to the interface).
- the memory 404 can be used to store software programs and modules of the application software, such as program instructions/modules corresponding to the real-time data processing method in the embodiment of the present application, and the processor 402 executes each by running a software program and a module stored in the memory 404.
- Memory 404 can include high speed random access memory and can also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid state memory.
- memory 404 can further include memory remotely located relative to processor 402, which can be connected to mobile terminal 40 over a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.
- the transmission device is for receiving or transmitting data via a network.
- the above-described network specific examples may include a wireless network provided by a communication provider of the computer terminal 40.
- the transmission device includes a Network Interface Controller (NIC) that can be connected to other network devices through the base station to communicate with the Internet.
- the transmission device can be a Radio Frequency (RF) module for communicating with the Internet wirelessly.
- NIC Network Interface Controller
- RF Radio Frequency
- Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.
- the display can be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of computer terminal 40 (or mobile device).
- LCD liquid crystal display
- the computer device 40 shown in FIG. 4 above may include hardware components (including circuits), software components (including computer code stored on a computer readable medium), or A combination of both hardware and software components. It should be noted that FIG. 4 is only one example of a specific embodiment, and is intended to show the types of components that may be present in the above-described computer device (or mobile device).
- FIG. 5 is a flowchart of a real-time data processing method according to an embodiment of the present application. As shown in FIG. 5, the method may include the following steps:
- Step S502 collecting real-time data received by one or more nodes in the distributed system.
- the real-time data mentioned above may be real-time statistical data, and the data is constantly changing with time.
- Step S504 real-time data satisfying the same predetermined condition on all nodes is aggregated to obtain a total amount of real-time data corresponding to the same predetermined condition, wherein the predetermined condition is used to represent the time when the real-time data is collected.
- the predetermined condition may be a time limit.
- the real-time data of the predetermined condition described above may be real-time data collected by the real-time data within a predetermined time.
- the predetermined time is a different time corresponding to the collection of real-time data.
- Step S506 calculating the completeness of the real-time data that satisfies the same predetermined condition, wherein the completeness is used to indicate the proportion of the processed real-time data satisfying the same predetermined condition in the real-time data on all the nodes summarized.
- the percentage of the real-time data processing can be known by using the percentage.
- the progress of the real-time data processing can be made more intuitive, and the business party can make more precise decisions according to the completeness of the real-time data. .
- the above degree of completeness may be calculated by using a quantity parameter, that is, according to the processed quantity of real-time data satisfying the predetermined condition, in the total amount of the processed real-time data after processing. The proportion of the calculation to complete the degree.
- the processing of real-time data may include various operations. For example, a relatively simple process is to collect real-time data. For example, the processing of the real-time data may be considered completed after the real-time data is collected. Of course, there are some more complicated processing methods. For example, after collecting real-time data, some processing of real-time data is required (for example, format adjustment, or data extraction, or data aggregation). It can be considered that the processing is completed after the completion of the processing.
- the following is an example in which the above embodiment is applied to a tlog product.
- the following functions can be implemented. If the current time (in minutes) of the real-time data is (1000, 95%), that is, at the current moment, the number of real-time data statistics is 1000, and the data is The completeness is 95%. Since the real-time data is constantly changing, the data within 2 minutes after the current time is continuously counted, and the number of real-time data in the 2 minutes is 200, and it is assumed that the data of the time (that is, the time corresponding to 2 minutes after the current time) is calculated. The completeness of the data is 100%, that is, the statistical result of the real-time data corresponding to the time is (1200, 100%). Through the above scheme, it is possible to ensure that the data of the tlog product is perceived by the service party without discarding the data.
- the quantity is used as the basis for the completion of the progress processing, that is, when 1200 pieces of data are collected, it is considered that the data completeness is 100%.
- the processing of real-time data introduces a parameter, that is, the degree of completeness, through which the progress of real-time data processing can be known.
- the completeness is used to indicate the progress of processing the real-time data that satisfies the predetermined condition in the processed real-time data.
- the solution of the first embodiment provided by the present application solves the technical problem that the prior art cannot know the progress of real-time data processing.
- the completeness of real-time data processing can be evaluated from the following dimensions, the amount of real-time data collected over a predetermined period of time, which is somewhat similar to evaluating how much real-time data can be collected over a period of time.
- the degree of completeness can also be evaluated from other angles. The dimension of the evaluation selected from any angle, as long as the progress can be displayed, or the progress of using the proportion of the processing is included in the protection scope of the embodiment of the present application.
- the degree of completeness is used to indicate the proportion of the total amount of processed real-time data satisfying the same predetermined condition in the total amount of real-time data on all the nodes aggregated.
- the above completeness can be calculated according to the quantity parameter.
- the degree of completeness is used to indicate the proportion of the total amount of real-time data that satisfies the same predetermined condition in the predetermined time period in the total amount of real-time data on all the nodes aggregated, wherein the same is satisfied.
- the real-time data of the predetermined condition is real-time data collected at a predetermined time
- the predetermined time period is a predetermined time period and a time period before the predetermined time.
- the foregoing preset time may be a specific moment that needs to calculate the completeness of the real-time data.
- the completeness of the real-time data may be real-time data acquired for a specific time, the proportion or progress of the real-time data being processed at a specific time and before a specific time.
- the method may further include the following steps:
- Step S508 obtaining a timestamp of each collected real-time data, wherein the timestamp is used to indicate the time for processing the real-time data.
- the predetermined condition is a time limit, for example, summing up 5 minutes of real-time data
- the data integrity at 2 minutes of real-time data collection.
- each collected real-time data is time stamped, which is used to indicate when the real-time data is collected.
- step S506 calculating the completeness of the real-time data that satisfies the same predetermined condition may include the following steps:
- Step S5062 in the case of the current timestamp, the proportion of the total amount of all real-time data processed in the total amount of real-time data on all the nodes aggregated as the completeness.
- a time stamp of each collected real-time data may be obtained, the number of real-time data of each time stamp is determined, and the processing is completed.
- the proportion of the amount of real-time data in the total amount of real-time data after aggregation is taken as the completeness.
- the timestamp of each collected real-time data is obtained, and the number of real-time data collected within a predetermined time can be intuitively obtained according to the timestamp, so that the completeness of the data can be quickly calculated.
- step S504 real-time data satisfying the same predetermined condition on all nodes is aggregated, and the total amount of real-time data corresponding to the same predetermined condition is obtained, including:
- Step S5042 acquiring real-time data on each distributed node.
- Step S5044 the acquired real-time data with the same time stamp on each distributed node is summarized, and the sum of the real-time data corresponding to each time stamp is obtained.
- each real-time data in the server has its corresponding processing time, and each real-time data is distributed across different nodes.
- each real-time data is distributed across different nodes.
- a unified data structure (for example, time: number of files) is generated on each distributed node, and then the data structure on the distributed node is transmitted to On a single node, the merge operation is performed on a single node, and the real-time data with the same processing time on each distributed node is merged, and the summary result of the real-time data is formed on a single node, and the result includes: Multiple processing times of real-time data, the amount of real-time data corresponding to each processing time.
- each real-time data is distributed on different nodes, and the real-time data on each distributed node is aggregated, that is, the total number of real-time data in a predetermined time period is obtained and collected. The time taken for the real-time data in the predetermined time period. After that, the summary results on each distributed node are combined on a single node to obtain the total amount of the aggregated real-time data within the predetermined time and after collecting the summary. The time used for real-time data.
- the real-time data on each distributed node is first summarized, and then the merge operation is performed on a single node, and finally the summary result of the real-time data in the predetermined time period is obtained, and the result is adopted.
- the solution can achieve the purpose of quickly and accurately summarizing real-time data within a predetermined time period without discarding the data.
- the above embodiment of the present application is described in detail below by taking the real-time statistical completeness of the log file generated by the server as an example.
- FIG. 6 a schematic diagram of an optional real-time data processing method is provided.
- the real-time computing engine obtains the first time from the a file to the 20 real-time data, and the calculated completeness is 25%; the second time from b
- the file gets 25 real-time data, and the calculation completeness is 35%; the third time from the c file to 35 real-time data, the calculation completeness is 75%; the fourth time from the d file to 20 real-time data,
- the calculated completeness is 100%.
- real-time data and completeness are stored from memory to the database.
- this embodiment may include the following steps:
- Step S702 collecting data sources and marking each data source with a time stamp.
- the data source is collected, that is, each log file in a distributed environment is collected.
- each log file is pulled back, it corresponds to the processing time.
- a.log, b.log, and c.log respectively represent three log files.
- the real-time system it has a corresponding processing time (that is, the above-mentioned timestamp), and the processing time is used to indicate the log file.
- the time of processing As shown in Table 1, the processing time corresponding to a.log is 20160113 13:42:50, the processing time corresponding to b.log is 20160113 13:42:52, and the processing time corresponding to c.log is 20160113 13:42:52 .
- Step S704 obtaining a identifiable data structure according to the timestamp.
- the above data structure is, time: the number of files.
- the data structure in Table 1 can be updated to an identifiable data structure according to the timestamp, as shown in Table 2.
- Step S706 the identifiable data structure on the distributed node is transmitted to a single node for aggregation.
- each of the distributed nodes in the distributed environment produces a identifiable data structure (ie, time: number of files), and then performs a merge operation on a single node.
- the contents of the identifiable data structure generated on the A node are as shown in Table 2.
- the a.log file is being processed and not processed. It can be considered that the two files b.log, c.log and other files on the A node have been processed. It has been processed.
- the two files b.log and c.log are being processed and are not processed. It can be considered that other files on the A node have been processed, and the a.log file is still not processed. It has been processed.
- the time point of 20160113 13:42:49 and the previous time point it can be considered that the three files a.log, b.log, c.log and other files on the A node have been processed.
- d.log, e.log, f.log, g.log, h.log, k.log, m.log, n.log can also be considered.
- Eight files and other files on Node B have been processed.
- the data structure in the A node and the B node are combined, and the contents of the data structure generated on the merged node are as shown in Table 4.
- Step S708 when the data is submitted, on the basis of the summarized data structure, calculate the completeness of the real-time data, and store the real-time data and the completeness index together in the database.
- the completeness of each time is a completeness of multiple timestamps, and after calculating the completeness, each timestamp and the completeness calculated for each timestamp are two.
- the indicators (timestamp) are stored in the database together, so that each data in the database has a completeness of data processing. According to the integrity of the data processing, the processing progress of the real-time data can be known, and the real-time data is further processed. More precise decision making.
- the real-time data collected by several nodes is calculated for the completeness of the real-time data:
- the real-time data collected by several nodes is calculated for the completeness of the real-time data:
- the embodiment of the present application adds a very important attribute to the real-time data, that is, proposes the concept of completeness, so that each data has a concept of a percentage (0% to 100%), and the percentage is used to represent the data. Completion. Without the completeness of the data, the results of real-time data have no concept of progress, so in many fast decision-making areas, it is not known whether decisions can be made based on real-time data. If you add completeness to the data, especially to the real-time data, you can make more accurate decisions based on real-time data.
- the completeness of the data can be applied in many fields. For example, due to the real-time calculation, the current statistical data is constantly changing, such as the sales data per minute, and the business party has no sense of the progress of the data processing. Because the data is constantly coming in, for the problem, the embodiment of the present application adds completeness to the real-time data, and according to the calculated completeness of the real-time data satisfying the predetermined condition, the real-time data can have a concept of a percentage, and the percentage can be known.
- the progress of real-time data processing For example, the sales within the preset minutes are always changing.
- the decision maker can determine whether to change the sales strategy based on the calculated completeness of the real-time data that meets the predetermined conditions. For example, if it is in the alarm monitoring system.
- the real-time data can be judged according to the completeness of the calculation, that is, the integrity of the data processing represented by the completeness.
- the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, by hardware, but in many cases, the former is A better implementation.
- the technical solution of the present application which is essential or contributes to the prior art, may be embodied in the form of a software product stored in a storage medium (such as ROM/RAM, disk,
- the optical disc includes a number of instructions for causing a terminal device (which may be a mobile phone, a computer, a server, or a network device, etc.) to perform the methods described in various embodiments of the present application.
- the apparatus 800 may include: a collection module 802, a summary module. 804 and a calculation module 806.
- the collecting module 802 is configured to collect real-time data received by one or more nodes in the distributed system;
- the summary module 804 is configured to summarize real-time data that meets the same predetermined condition on all nodes, and obtain real-time corresponding to the same predetermined condition. a total amount of data, wherein the predetermined condition is used to characterize the time at which the real-time data is collected;
- the calculation module 806 is configured to calculate the completeness of the real-time data that satisfies the same predetermined condition, wherein the completeness is used to indicate that the processed one satisfies the same reservation The proportion of real-time data of the condition in the real-time data on all nodes aggregated.
- the real-time data mentioned above may be data that is calculated from real-time, and the data is constantly changing with time.
- the above predetermined condition may be a time limit.
- the real-time data of the predetermined condition described above may be real-time data collected by the real-time data within a predetermined time.
- the predetermined time is a different time corresponding to the collection of real-time data.
- the above progress can be a percentage, that is, the ratio of the processed data that has been processed to the processed data after processing. The percentage can be used to know the progress of real-time data processing. When the percentage is used, the progress of real-time data processing can be made more intuitive, and it is easier for the business party to make more precise decisions based on the completeness of real-time data.
- the processing of real-time data may include various operations. For example, a relatively simple process is to collect real-time data. For example, the processing of the real-time data may be considered completed after the real-time data is collected. Of course, there are some more complicated processing methods. For example, after collecting real-time data, some processing of real-time data is required (for example, format adjustment, or data extraction, or data aggregation). It can be considered that the processing is completed after the completion of the processing.
- the foregoing collection module 802, the summary module 804, and the calculation module 806 correspond to the steps S502 to S506 in the first embodiment, and the three modules are the same as the examples and application scenarios implemented by the corresponding steps, but It is not limited to the contents disclosed in the above embodiment 1. It should be noted that the above module can be operated as a part of the device in the computer terminal 40 provided in the first embodiment.
- the processing of real-time data introduces a parameter, that is, the degree of completeness, through which the progress of real-time data processing can be known.
- the completeness is used to indicate the progress of processing the real-time data that satisfies the predetermined condition in the processed real-time data.
- For progress it can be a percentage, that is, the ratio of the processed data that has been processed to the processed data.
- the percentage can be used to know the progress of real-time data processing. When the percentage is used, the progress of real-time data processing can be made more intuitive, and it is easier for the business party to make more precise decisions based on the completeness of real-time data.
- the processing of real-time data can include a variety of operations. For example, a more concise process is to collect real-time data. For example, the processing of the real-time data can be considered complete after the real-time data is collected. Of course, there are some more complicated processing methods. For example, after collecting real-time data, some processing of real-time data is required (for example, format adjustment, or data extraction, or data aggregation). It can be considered that the processing is completed after the completion of the processing.
- the use of "collecting" to real-time data is considered to be the processing of real-time data, because the collected real-time data can be used to some extent to evaluate the integrity of real-time data. .
- the completeness of real-time data processing can be evaluated in the following dimensions, the amount of real-time data collected over a predetermined period of time, which is somewhat similar to evaluating how much real-time data can be collected over a period of time.
- the degree of completeness can also be evaluated from other angles. The dimension of the evaluation selected from any angle, as long as the progress can be displayed, or the progress of using the proportion of the processing is included in the protection scope of the embodiment of the present application.
- the degree of completeness is used to indicate the proportion of the total amount of processed real-time data satisfying the same predetermined condition in the total amount of real-time data on all the nodes aggregated.
- the above completeness can be calculated according to the quantity parameter.
- the degree of completeness is used to indicate the proportion of the total amount of real-time data that satisfies the same predetermined condition in the predetermined time period in the total amount of real-time data on all the nodes aggregated, wherein the same is satisfied.
- the real-time data of the predetermined condition is real-time data collected at a predetermined time
- the predetermined time period is a predetermined time period and a time period before the predetermined time.
- the foregoing preset time may be a specific moment that needs to calculate the completeness of the real-time data.
- the completeness of the real-time data may be real-time data acquired for a specific time, the proportion or progress of the real-time data being processed at a specific time and before a specific time.
- the real-time data processing apparatus 800 further includes an acquisition module 808.
- the obtaining module 808 is configured to obtain a timestamp of each collected real-time data after collecting the real-time data, where the timestamp is used to indicate the time for processing the real-time data.
- the above-mentioned acquisition module 808 corresponds to the step S508 in the first embodiment, and the module is the same as the example and the application scenario implemented by the corresponding steps, but is not limited to the content disclosed in the first embodiment. It should be noted that the above module can be operated as a part of the device in the computer terminal 40 provided in the first embodiment.
- each real-time data collected is time-stamped, and the number of real-time data collected within a predetermined time can be intuitively obtained according to the timestamp, so that the completeness of the data can be quickly calculated.
- the calculation module 806 is further configured to use, in the case of the current timestamp, the total amount of all real-time data processed by the total amount of real-time data on all the nodes aggregated. The proportion of the total, as a completeness.
- the foregoing calculation module 806 corresponds to the step S5062 in the first embodiment, and the module is the same as the example and the application scenario implemented by the corresponding steps, but is not limited to the content disclosed in the first embodiment. It should be noted that the above module can be operated as a part of the device in the computer terminal 40 provided in the first embodiment.
- the above summary module 806 includes: an obtaining unit 810 and a summary unit 812.
- the obtaining unit 810 is configured to acquire real-time data on each distributed node
- the summarizing unit 812 is configured to summarize real-time data with the same time stamp on each distributed node that is obtained, and obtain a corresponding timestamp. The total amount of real-time data.
- the foregoing obtaining unit 810 and the summarizing unit 812 correspond to steps S5042 to S5044 in the first embodiment, and the two modules are the same as the examples and application scenarios implemented by the corresponding steps, but are not limited to the above embodiments.
- the real-time data on each distributed node is first summarized, and then the merge operation is performed on a single node, and finally the summary result of the real-time data in the predetermined time period is obtained, and the result is adopted.
- the solution can achieve the purpose of quickly and accurately summarizing real-time data within a predetermined time period without discarding the data.
- the embodiment of the present application adds a very important attribute to the real-time data, that is, proposes the concept of completeness, so that each data has a concept of a percentage (0% to 100%), and the percentage is used to represent the data. Completion. Without the completeness of the data, the results of real-time data have no concept of progress, so in many fast decision-making areas, it is not known whether decisions can be made based on real-time data. If you add completeness to the data, especially to the real-time data, you can make more accurate decisions based on real-time data.
- the completeness of the data can be applied in many fields. For example, due to the real-time calculation, the current statistical data is constantly changing, such as the sales data per minute, and the business party has no sense of the progress of the data processing. Because the data is constantly coming in, for this problem, the embodiment of the present application proposes the concept of completeness, that is, the completeness of the real-time data, according to the calculated completeness of the real-time data satisfying the predetermined condition, the real-time data can have a percentage
- the concept, through which the real-time data processing progress can be known. For example, the sales within the preset minutes are always changing. The decision maker can determine whether to change the sales strategy based on the calculated completeness of the real-time data that meets the predetermined conditions.
- the real-time data can be judged according to the completeness of the calculation, that is, the integrity of the data processing represented by the completeness.
- Embodiments of the present application may provide a computer terminal, which may be any one of computer terminal groups.
- the foregoing computer terminal may also be replaced with a terminal device such as a mobile terminal.
- the computer terminal may be located in at least one network device of the plurality of network devices of the computer network.
- the computer terminal may execute the program code of the following steps in the real-time data processing method: collecting real-time data received by one or more nodes in the distributed system; and summarizing real-time data satisfying predetermined conditions on all nodes Obtaining a total amount of real-time data corresponding to the same predetermined condition, wherein the predetermined condition is used to represent the time at which the real-time data is collected; and the completeness of the real-time data satisfying the same predetermined condition is calculated, wherein the completeness is used to indicate: the processing is completed The proportion of real-time data that satisfies the same predetermined conditions in real-time data on all nodes aggregated.
- FIG. 9 is a structural block diagram of a computer terminal according to an embodiment of the present application.
- the computer terminal 900 may include one or more (only one shown in the figure) processor 902 and memory 904.
- the terminal can be understood as a computing terminal, and the computing terminal has a certain computing power, which can perform operations and feed back the operation result.
- the computing terminal can be understood as a service provider. Multiple terminals can jointly perform computing services.
- the terminal can be a computer, a mobile phone, a tablet, or the like.
- multiple computers can jointly provide services, and can also be understood as a server group; from another perspective, these devices can also provide cloud computing services.
- the memory can be used to store the software program and the module, such as the real-time data processing method and the program instruction/module corresponding to the device in the embodiment of the present application, and the processor executes various functions by running the software program and the module stored in the memory.
- Application and data processing that is, the above method is implemented.
- the memory may include a high speed random access memory, and may also include non-volatile memory such as one or more magnetic storage devices, flash memory, or other non-volatile solid state memory.
- the memory can further include memory remotely located relative to the processor, which can be connected to computer terminal 900 over a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.
- the processor may invoke the memory stored information and the application by the transmission device to perform the steps of: collecting real-time data received by one or more nodes in the distributed system; and summarizing real-time data on all nodes that satisfy the same predetermined condition Obtaining a total amount of real-time data corresponding to the same predetermined condition, wherein the predetermined condition is used to represent the time at which the real-time data is collected; and the completeness of the real-time data satisfying the same predetermined condition is calculated, wherein the completeness is used to indicate: the processing is completed The proportion of real-time data that satisfies the same predetermined conditions in real-time data on all nodes aggregated.
- the foregoing processor may further execute the program code of the following steps: the completeness is used to indicate the proportion of the total amount of real-time data that is processed to satisfy the same predetermined condition in the total amount of real-time data on all the nodes aggregated. .
- the foregoing processor may further execute the program code of the following steps: the completeness is used to indicate the total amount of real-time data on all the nodes aggregated by the total amount of real-time data that satisfies the same predetermined condition in the predetermined time period.
- the ratio of the real-time data that satisfies the same predetermined condition is real-time data collected at a predetermined time, and the predetermined time period is a predetermined time period and a time period before the predetermined time.
- the foregoing processor may further execute the following program code: after collecting real-time data received by one or more nodes in the distributed system, acquiring a timestamp of each collected real-time data, where the timestamp Used to indicate when the real-time data is processed.
- the foregoing processor may further execute program code of the following steps: in the case of the current timestamp, the proportion of the total amount of all real-time data processed in the total amount of real-time data on all the nodes aggregated As a completeness.
- the foregoing processor may further execute the following program code: acquire real-time data on each distributed node; and summarize real-time data with the same time stamp on each distributed node that is obtained, and obtain each The total amount of real-time data corresponding to the timestamp.
- the real-time data received by one or more nodes in the distributed system is collected, and real-time data satisfying the same predetermined condition on all nodes is aggregated, and the total amount of real-time data corresponding to the same predetermined condition is obtained, and the calculation is performed.
- the completeness of the real-time data satisfying the same predetermined conditions thereby realizing the calculation of the completeness of the real-time data.
- the processing of real-time data introduces a parameter, that is, the degree of completeness, through which the progress of real-time data processing can be known.
- the completeness is used to indicate the progress of processing the real-time data that satisfies the predetermined condition in the processed real-time data.
- the embodiment of the present application solves the technical problem that the prior art cannot know the progress of real-time data processing.
- FIG. 9 is only an illustration, and the computer terminal can also be a smart phone (such as an Android mobile phone, an iOS mobile phone, etc.), a tablet computer, an applause computer, and a mobile Internet device (Mobile Internet Devices, MID). ), PAD and other terminal devices.
- FIG. 9 does not limit the structure of the above electronic device.
- computer terminal 900 may also include more or fewer components (such as a network interface, display device, etc.) than shown in FIG. 9, or have a different configuration than that shown in FIG.
- the computer of Fig. 9 can also be used as a server to provide external services.
- Embodiments of the present application also provide a storage medium.
- the foregoing storage medium may be used to save the program code executed by the real-time data processing method provided in Embodiment 1 above.
- the foregoing storage medium may be located in any one of the computer terminal groups in the computer network, or in any one of the mobile terminal groups.
- the storage medium is configured to store program code for performing the following steps: collecting real-time data received by one or more nodes in the distributed system; meeting the same predetermined condition on all nodes Real-time data is aggregated to obtain a total amount of real-time data corresponding to the same predetermined condition, wherein the predetermined condition is used to represent the time at which the real-time data is collected; and the completeness of the real-time data satisfying the same predetermined condition is calculated, wherein the completeness is used for Indicates the proportion of processed real-time data that satisfies the same predetermined condition in real-time data on all nodes aggregated.
- the storage medium is further configured to store program code for performing the following steps: the completeness is used to indicate the total amount of real-time data on all nodes aggregated for the total amount of processed real-time data satisfying the same predetermined condition. The proportion in the middle.
- the storage medium is further configured to store program code for performing the following steps: the completeness is used to indicate the real time of the total amount of real-time data that satisfies the same predetermined condition processed in the predetermined time period on all the nodes in the summary.
- the storage medium is further configured to store program code for performing the following steps: after collecting real-time data received by one or more nodes in the distributed system, obtaining a timestamp of each collected real-time data, The timestamp is used to indicate the time when the real-time data is processed.
- the storage medium is further arranged to store program code for performing the following steps: in the case of the current timestamp, the total amount of real-time data on all nodes aggregated for processing the total amount of all real-time data completed. The proportion in the total, as a completeness.
- the storage medium is further configured to store program code for performing the following steps: acquiring real-time data on each distributed node; summarizing the acquired real-time data having the same time stamp on each distributed node Get the total amount of real-time data corresponding to each timestamp.
- the disclosed technical contents may be implemented in other manners.
- the device embodiments described above are only schematic.
- the division of the unit may be a logical function division.
- there may be another division manner for example, multiple units or components may be combined or may be Integrate into another system, or some features can be ignored or not executed.
- the coupling or direct coupling or communication connection shown or discussed herein may be an indirect coupling or communication connection through some interface, unit or module, and may be electrical or otherwise.
- the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of the embodiment.
- each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
- the above integrated unit can be implemented in the form of hardware or in the form of a software functional unit.
- the integrated unit if implemented in the form of a software functional unit and sold or used as a standalone product, may be stored in a computer readable storage medium.
- a computer readable storage medium A number of instructions are included to cause a computer device (which may be a personal computer, server or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present application.
- the foregoing storage medium includes: a U disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a removable hard disk, a magnetic disk, or an optical disk, and the like. .
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Quality & Reliability (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Mathematical Physics (AREA)
- Computer Hardware Design (AREA)
- Computational Linguistics (AREA)
- Computing Systems (AREA)
- Probability & Statistics with Applications (AREA)
- Fuzzy Systems (AREA)
- Debugging And Monitoring (AREA)
Abstract
一种实时数据处理方法和装置。其中,该处理方法包括:收集分布式***中一个或多个节点接收到的实时数据(S502);将所有节点上满足相同预定条件的实时数据进行汇总,得到相同预定条件下对应的实时数据的总量,其中,预定条件用于表征收集到实时数据的时间(S504);计算满足相同预定条件的实时数据的齐全度,其中,齐全度用于指示:处理完的满足相同预定条件的实时数据在汇总的所有节点上的实时数据中的比例(S506)。解决了现有技术无法获知对实时数据处理的进度所导致的问题,从而增加了对实时数据处理的评估维度。
Description
本申请要求2016年12月30日递交的申请号为201611270978.5、发明名称为“实时数据处理方法和装置”的中国专利申请的优先权,其全部内容通过引用结合在本申请中。
本申请涉及数据处理领域,具体而言,涉及一种实时数据处理方法和装置。
在现有的实时数据计算领域中,流数据是源源不断的,不知道什么时候是个头,这样实时统计出来的数据也是实时变化的。由于实时统计的数据是一直变动的,使用这些数据的业务方对这些数据处理的进度完全无感知,因此,无法根据这些数据做出一些决断性质的行为,比如,统计的预设分钟内的销售额,是一直变动的,决策者无法根据这些数据确定是否进行销售策略的改变;又如,若是报警监控***里的实时数据,由于实时数据的不断变动,该***无法根据这些数据做出准确的报警行为。即在现有技术中没有参数可以用来评估实时数据处理的进度。
目前,常用的实时数据的处理方式为以下两种:
如图1所示,对于统计1分钟的日志的错误个数,如果超过100个就报警。比如有一分钟的100个是分4次来的,即每次都来25个。
对于第一种方案,如图2所示,每来25个就计算一次,第二次来就再计算一次,通过这样累加的方式对实时数据进行处理。每次处理完都会将数据存到数据库里。即,每来25个都会马上经过实时计算引擎,然后经过内存,最终到达数据库,是从数据源一路到达数据库的。目前第一种方案是业内采用最多的方式。因为可以保证数据的完整性。但是缺点是,数据库的数据会不断的变化,这样就不能做精准的决策。
对于第二种方案,如图3所示,可以是一开始指定一个预计的超时时间,例如,2分钟超时。假设4次25个是分4个时间段进来的,前3次在2分钟内到达,第4次在2分钟后才到。那么这种方案只会记录前三次的75个,只存一次数据库,即图3中的内存会等待2分钟,2分钟过了之后才会存入数据库。这样做的好处是可以对于数据库来说,只要有数据,就是不会再变了,可以根据这个数据直接做出决策,比如交易下跌,赶紧马上更换广告位的品牌,可以在最快的时间内挽回交易额。但是这种方案的缺点就是, 过了超时时间的数据会丢掉(如图中虚线所示的第四次的25个),因此,会存在误差。
针对现有技术无法获知对实时数据处理的进度所导致的问题,目前尚未提出有效的解决方案。
发明内容
本申请实施例提供了一种实时数据处理方法和装置,以至少解决现有技术无法获知对实时数据处理的进度所导致的问题。
根据本申请实施例的一个方面,提供了一种实时数据处理方法,该处理方法包括:收集分布式***中一个或多个节点接收到的实时数据;将所有节点上满足相同预定条件的实时数据进行汇总,得到相同预定条件下对应的实时数据的总量,其中,预定条件用于表征收集到实时数据的时间;计算满足相同预定条件的实时数据的齐全度,其中,齐全度用于指示:处理完的满足相同预定条件的实时数据在汇总的所有节点上的实时数据中的比例。
根据本申请实施例的另一方面,还提供了一种实时数据处理装置,该处理装置包括:收集模块,用于收集分布式***中一个或多个节点接收到的实时数据;汇总模块,用于将所有节点上满足相同预定条件的实时数据进行汇总,得到相同预定条件下对应的实时数据的总量,其中,预定条件用于表征收集到实时数据的时间;计算模块,用于计算满足相同预定条件的实时数据的齐全度,其中,齐全度用于指示:处理完的满足相同预定条件的实时数据在汇总的所有节点上的实时数据中的比例。
在本申请实施例中,收集分布式***中一个或多个节点接收到的实时数据,将所有节点上满足相同预定条件的实时数据进行汇总,得到相同预定条件下对应的实时数据的总量,计算满足相同预定条件的实时数据的齐全度,从而实现对实时数据的齐全度进行计算。
容易注意到,对实时数据的处理引入了一个参数,即齐全度,通过该参数可以得知实时数据处理的进度。该齐全度用于指示处理完满足预定条件的实时数据在处理完汇总后的实时数据中所占的进度。在引入齐全度之后,可以收集实时数据并对收集到的实时数据进行汇总;然后从汇总后的实时数据中获取满足预定条件的实时数据;从而可以计算满足预定条件的实时数据的齐全度,达到增加了对实时数据处理的评估维度的效果。
由此,本申请实施例解决了现有技术无法获知对实时数据处理的进度的技术问题。
此处所说明的附图用来提供对本申请的进一步理解,构成本申请的一部分,本申请的示意性实施例及其说明用于解释本申请,并不构成对本申请的不当限定。在附图中:
图1是根据现有技术的一种实时数据处理方法的示意图;
图2是根据现有技术的第一种实时数据处理方法的示意图;
图3是根据现有技术的第二种实时数据处理方法的示意图;
图4是根据本申请实施例的一种实时数据处理方法的计算机设备的硬件结构框图;
图5是根据本申请实施例的一种实时数据处理方法的流程图;
图6是根据本申请实施例的一种可选的实时数据处理方法的示意图;
图7是根据本申请实施例的一种可选的实时数据处理方法的流程图;
图8是根据本申请实施例的一种实时数据处理装置的示意图;
图9是根据本申请实施例的一种计算机终端的结构框图。
为了使本技术领域的人员更好地理解本申请方案,下面将结合本申请实施例中的附图,对本申请实施例中的技术方案进行清楚、完整地描述,显然,所描述的实施例仅仅是本申请一部分的实施例,而不是全部的实施例。基于本申请中的实施例,本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例,都应当属于本申请保护的范围。
需要说明的是,本申请的说明书和权利要求书及上述附图中的术语“第一”、“第二”等是用于区别类似的对象,而不必用于描述特定的顺序或先后次序。应该理解这样使用的数据在适当情况下可以互换,以便这里描述的本申请的实施例能够以除了在这里图示或描述的那些以外的顺序实施。此外,术语“包括”和“具有”以及他们的任何变形,意图在于覆盖不排他的包含,例如,包含了一系列步骤或单元的过程、方法、***、产品或设备不必限于清楚地列出的那些步骤或单元,而是可包括没有清楚地列出的或对于这些过程、方法、产品或设备固有的其它步骤或单元。
实施例1
根据本申请实施例,提供了一种实时数据处理方法的实施例,需要说明的是,在附图的流程图示出的步骤可以在诸如一组计算机可执行指令的计算机***中执行,并且,虽然在流程图中示出了逻辑顺序,但是在某些情况下,可以以不同于此处的顺序执行所 示出或描述的步骤。
本申请实施例所提供的方法实施例可以在服务器上进行处理,为了提供更好的使用体验,还可以提供处理结果查询服务,例如,可以通过网页、或者客户端来查看服务器上的运算结果。服务器可以理解为一种计算机。当然,随着技术的发展,云计算得到了越来越广的应用,本申请实施例中所提供的方法也可以在云计算中推广使用。终端的计算能力也会随着技术的发展而增强,当终端可以获取到相应的数据而进行计算时,例如,终端可以包括但是不限于:手机、平板电脑以及其他的便携式设备。但是就目前而言,在服务器部署本申请以下实施例是一种较优的选择。
在目前的技术条件下,服务器、终端、云计算所依赖的硬件架构都是类似的,均可以看作是一种计算机设备。本申请实施例可以在这种计算机设备中执行。随着技术的发展,计算机设备硬件架构发生了变化,或者出现了新架构的运算设备,本申请以下实施例也可以实施。下面以图4中的计算设备的架构为例进行说明。
图4是根据本申请实施例的一种实时数据处理方法的计算机设备的硬件结构框图。如图4所示,计算机终端40可以包括一个或多个(图中采用402a、402b,……,402n来示出)处理器402(处理器402可以包括但不限于微处理器MCU或可编程逻辑器件FPGA等的处理装置)、用于存储数据的存储器404、以及用于通信功能的传输装置。除此以外,还可以包括:显示器、输入/输出接口(I/O接口)、通用串行总线(USB)端口(可以作为I/O接口的端口中的一个端口被包括)、网络接口、电源和/或相机。本领域普通技术人员可以理解,图4所示的结构仅为示意,其并不对上述电子装置的结构造成限定。例如,计算机终端40还可包括比图4中所示更多或者更少的组件,或者具有与图4所示不同的配置。
应当注意到的是上述一个或多个处理器402和/或其他数据处理电路在本文中通常可以被称为“数据处理电路”。该数据处理电路可以全部或部分的体现为软件、硬件、固件或其他任意组合。此外,数据处理电路可为单个独立的处理模块,或全部或部分的结合到计算机终端40中的其他元件中的任意一个内。如本申请实施例中所涉及到的,该数据处理电路作为一种处理器控制(例如与接口连接的可变电阻终端路径的选择)。
存储器404可用于存储应用软件的软件程序以及模块,如本申请实施例中的实时数据处理方法对应的程序指令/模块,处理器402通过运行存储在存储器404内的软件程序以及模块,从而执行各种功能应用以及数据处理。存储器404可包括高速随机存储器,还可包括非易失性存储器,如一个或者多个磁性存储装置、闪存、或者其他非易失性固 态存储器。在一些实例中,存储器404可进一步包括相对于处理器402远程设置的存储器,这些远程存储器可以通过网络连接至移动终端40。上述网络的实例包括但不限于互联网、企业内部网、局域网、移动通信网及其组合。
传输装置用于经由一个网络接收或者发送数据。上述的网络具体实例可包括计算机终端40的通信供应商提供的无线网络。在一个实例中,传输装置包括一个网络适配器(Network Interface Controller,NIC),其可通过基站与其他网络设备相连从而可与互联网进行通讯。在一个实例中,传输装置可以为射频(Radio Frequency,RF)模块,其用于通过无线方式与互联网进行通讯。
上述网络的实例包括但不限于互联网、企业内部网、局域网、移动通信网及其组合。
显示器可以例如触摸屏式的液晶显示器(LCD),该液晶显示器可使得用户能够与计算机终端40(或移动设备)的用户界面进行交互。
此处需要说明的是,在一些可选实施例中,上述图4所示的计算机设备40可以包括硬件元件(包括电路)、软件元件(包括存储在计算机可读介质上的计算机代码)、或硬件元件和软件元件两者的结合。应当指出的是,图4仅为特定具体实例的一个实例,并且旨在示出可存在于上述计算机设备(或移动设备)中的部件的类型。
在上述运行环境下,本申请实施例提供了一种实时数据处理方法。图5是根据本申请实施例的一种实时数据处理方法的流程图,如图5所示,该方法可以包括如下步骤:
步骤S502,收集分布式***中一个或多个节点接收到的实时数据。
具体的,上述的实时数据可以是实时统计出来的数据,该数据随着时间不断变化。
步骤S504,将所有节点上满足相同预定条件的实时数据进行汇总,得到相同预定条件下对应的实时数据的总量,其中,预定条件用于表征收集到实时数据的时间。
具体的,上述预定条件可以是对时间的限制。上述的预定条件的实时数据可以为实时数据在预定时间内收集到的实时数据。其中,预定时间为收集实时数据时对应的不同时间。
步骤S506,计算满足相同预定条件的实时数据的齐全度,其中,齐全度用于指示:处理完的满足相同预定条件的实时数据在汇总的所有节点上的实时数据中的比例。
具体的,通过该百分比可以获知实时数据处理的进度,使用百分比的情况下,可以使实时数据处理的进度更加直观,更容易使业务方根据实时数据的齐全度,做出一些更为精确的决策。
在一种可选的方案中,上述的齐全度可以采用数量参数来进行计算,也即,可以根 据处理完的满足预定条件的实时数据的数量在处理完的汇总后的实时数据的总量中的占比来计算齐全度。
此处需要说明的是,对实时数据的处理可以包括多种操作,例如,一种比较简明的处理就是收集实时数据,例如,可以在收集到实时数据之后就认为对该实时数据的处理完成。当然,还有一些比较复杂的处理方式,例如,在收集到实时数据之后,还需要对实时数据进行一些加工(例如,格式的调整,或者是数据的抽取,或者是数据的汇总),此时,可以认为加工完成之后才意味着处理的完成。
此处还需要说明的是,在以下实施例的描述中,使用了“收集”到实时数据就认为是实时数据的处理完成,这是因为,收集到的实时数据之后在某种程度上就可以用来评估实时数据的完整度。
下面以将上述的实施方案应用于tlog产品中为例。根据上述实施例,可以实现下面的功能,若当前时刻(以分钟为单位)实时数据的统计结果为(1000,95%),也即在该当前时刻,实时数据统计的数量为1000,这些数据的齐全度为95%。由于实时数据一直在变动,继续统计当前时刻之后的2分钟内的数据,该2分钟内实时数据的数量为200,并假设计算得到该时刻(即当前时刻之后的2分钟对应的时刻)的数据的齐全度为100%,也即,对应该时刻实时数据的统计结果为(1200,100%)。通过上述方案,可以在保证不丢弃数据的同时,又能使使用tlog产品的业务方感知到数据的精准度。
在该例子中,将数量作为了进度处理完成的依据,即当收集到1200个数据时就认为是数据齐全度为100%。
由上可知,本申请上述实施例一公开的方案中,收集分布式***中一个或多个节点接收到的实时数据,将所有节点上满足相同预定条件的实时数据进行汇总,得到相同预定条件下对应的实时数据的总量,计算满足相同预定条件的实时数据的齐全度,从而实现对实时数据的齐全度进行计算。
容易注意到,对实时数据的处理引入了一个参数,即齐全度,通过该参数可以得知实时数据处理的进度。该齐全度用于指示处理完满足预定条件的实时数据在处理完汇总后的实时数据中所占的进度。在引入齐全度之后,可以收集实时数据并对收集到的实时数据进行汇总;然后从汇总后的实时数据中获取满足预定条件的实时数据;从而可以计算满足预定条件的实时数据的齐全度,达到增加了对实时数据处理的评估维度的效果。
由此,本申请提供的上述实施例一的方案解决了现有技术无法获知对实时数据处理的进度的技术问题。
在以下实施例中,实时数据处理的齐全度可以从以下维度进行评估,预定时间内收集到的实时数据的数量,该维度有些类似于评估在一个时间段内究竟能够收集多少实时数据。当然,齐全度也可以从其他的角度来进行评估,无论从什么角度选择的评估的维度,只要是可以展示进度,或者使用处理的比例的方式展示进度均涵盖在本申请实施例的保护范围之内。
根据本申请上述实施例,齐全度用于指示处理完的满足相同预定条件的实时数据的总量在汇总的所有节点上的实时数据的总量中的占比。
在一种可选的方案中,上述的齐全度可以根据数量参数进行计算。
根据本申请上述实施例,齐全度用于指示预定时间段内处理完的满足相同预定条件的实时数据的总量在汇总的所有节点上的实时数据的总量中的占比,其中,满足相同预定条件的实时数据为预定时刻收集到的实时数据,预定时间段为预定时刻和预定时刻之前的时间段。
具体的,上述的预设时刻可以是需要对实时数据的齐全度进行计算的特定时刻。
在一种可选的方案中,实时数据的齐全度可以是针对特定时刻获取到的实时数据,在特定时刻和特定时刻之前的实时数据被处理的比例或者进度。
根据本申请上述实施例,在步骤S502,收集分布式***中一个或多个节点接收到的实时数据之后,上述方法还可以包括如下步骤:
步骤S508,获取收集到的每个实时数据的时间戳,其中,时间戳用于指示对该实时数据进行处理的时间。
在一种可选的方案中,如果预定条件是对时间限制,例如,汇总5分钟的实时数据,此时需要知道在实时数据收集2分钟时的数据齐全度。可以收集实时数据之后,为收集到的每个实时数据标上时间戳,该时间戳用于指示收集到该实时数据的时间。
根据本申请上述实施例,步骤S506,计算满足相同预定条件的实时数据的齐全度可以包括如下步骤:
步骤S5062,在当前时间戳的情况下,将处理完成的所有实时数据的总量在汇总的所有节点上的实时数据的总量中的占比,作为齐全度。
在一种可选的方案中,在一个或多个节点接收到实时数据之后,可以获取收集到的每个实时数据的时间戳,确定每个时间戳的实时数据的数量,并将处理完成的实时数据的数量在汇总后的实时数据总量中占的比例作为齐全度。
通过上述实施例,获取收集到的每个实时数据的时间戳,可以根据时间戳直观的得 到在预定时间内收集到的实时数据的数量,从而可以快速计算出数据的齐全度。
根据本申请上述实施例,步骤S504,将所有节点上满足相同预定条件的实时数据进行汇总,得到相同预定条件下对应的实时数据的总量,包括:
步骤S5042,获取每个分布式节点上的实时数据。
步骤S5044,将获取到的每个分布式节点上具有相同时间戳的实时数据进行汇总,得到每个时间戳对应的实时数据的总和。
在一种可选的方案中,服务器中的每个实时数据都有其对应的处理时间,且每个实时数据被分布在不同的节点上。在对预定时间段内的实时数据进行汇总时,首先获取预定时间段内的每个分布式节点上的实时数据,并对每个分布式节点上的实时数据进行汇总,即将每个分布节点上对应的处理时间相同的实时数据进行合并操作,经过合并操作之后,在各个分布式节点上将生成统一的数据结构(如,时间:文件个数),然后将分布式节点上的数据结构传输到单台节点上,最后在一个单节点上做合并操作,即将各个分布式节点上对应的处理时间相同的实时数据做合并操作,在单节点上形成对实时数据的汇总结果,该结果中包括:实时数据的多个处理时间,与每个处理时间对应的实时数据的数量。
在另一种可选的方案中,每个实时数据被分布在不同的节点上,对每个分布式节点上的实时数据进行汇总,即得到预定时间段内的实时数据的总数量和收集完预定时间段内的实时数据所用的时间,之后,将每个分布式节点上的汇总结果在一个单节点上进行合并操作,得到预定时间内的汇总后的实时数据的总量和收集完汇总后的实时数据所用的时间。
通过上述实施例,由于是分布式环境,首先将每个分布式节点上的实时数据进行汇总,然后在一个单节点上进行合并操作,最终得到预定时间段内的实时数据的汇总结果,采用该方案,可以在不丢弃数据的同时,达到快速且准确对预定时间段内的实时数据的汇总的目的。
下面以对服务器产生的日志文件的实时统计齐全度为例,详述本申请的上述实施例。如图6所示,提供了一种可选的实时数据处理方法的示意图,实时计算引擎获取第一次从a文件获取到20个实时数据,计算得到齐全度是25%;第二次从b文件获取到25个实时数据,计算得到齐全度是35%;第三次从c文件获取到35个实时数据,计算得到齐全度是75%;第四次从d文件获取到20个实时数据,计算得到齐全度是100%。当实时数据的齐全度到达100%,或者到达95%的时候,将实时数据和齐全度从内存存到数据 库。
如图7所示,该实施例可以包括如下步骤:
步骤S702,收集数据源并为每一个数据源标上一个时间戳。
在该步骤中,收集数据源,即为收集分布式环境下的每个日志文件。每个日志文件在拉回来的时候,都对应有处理的时间。例如,a.log,b.log,c.log分别表示三个日志文件,到实时***时,都带有对应的处理时间(即上述的时间戳),该处理时间用于表示对该日志文件进行处理的时间。如表1所示,a.log对应的处理时间为20160113 13:42:50,b.log对应的处理时间为20160113 13:42:52,c.log对应的处理时间为20160113 13:42:52。
表1
日志文件 | 处理时间 |
a.log | 20160113 13:42:50 |
b.log | 20160113 13:42:52 |
c.log | 20160113 13:42:52 |
步骤S704,根据时间戳得到可以识别的数据结构。
上述的数据结构为,时间:文件个数。根据时间戳可以将表1中的数据结构更新为可以识别的数据结构,如表2所示。
表2
时间 | 日志文件个数 |
20160113 13:42:50 | 1(a.log) |
20160113 13:42:52 | 2(包括b.log和c.log) |
步骤S706,将分布式节点上的可以识别的数据结构传输到单台节点上做汇总。
由于是分布式环境,在分布式环境中各个分布式节点上都生产一个上述的可以识别的数据结构(即时间:文件个数),然后在一个单节点上做合并操作。
如在A节点上生成的可以识别的数据结构的内容,如表2所示。
如表2所示,对于20160113 13:42:50这个时间点,a.log文件正在处理中,未被处理完毕,可以认为b.log,c.log两个文件以及A节点上的其他文件已经被处理完毕。对于20160113 13:42:52这个时间点,b.log和c.log两个文件正在处理中,未被处理完毕,可以认为A节点上的其他文件已经被处理完毕,且a.log文件仍然未被处理完毕。进一步对于20160113 13:42:49这个时间点以及之前的时间点,还可以认为a.log,b.log,c.log三个文件以及A节点上的其他文件均已经被处理完毕。
在B节点上生成的可以识别的数据结构的内容,如表3所示。
表3
如表3所示,对于20160113 13:42:51这个时间点,k.log,m.log,n.log三个文件正在处理中,未被处理完毕,可以认为d.log,e.log,f.log,g.log,h.log五个文件以及B节点上的其他文件已经被处理完毕。对于20160113 13:42:52这个时间点,d.log,e.log,f.log,g.log,h.log五个文件正在处理中,未被处理完毕,可以认为B节点上的其他文件已经被处理完毕,且k.log,m.log,n.log三个文件仍然未被处理完毕。进一步对于2016011313:42:50这个时间点以及之前的时间点,还可以认为d.log,e.log,f.log,g.log,h.log,k.log,m.log,n.log八个文件以及B节点上的其他文件均已经被处理完毕。
将A节点和B节点中的数据结构做合并操作,在合并后的节点上生成的数据结构的内容如表4所示。
表4
步骤S708,在数据提交的时候,在汇总的数据结构的基础上,算出实时数据的齐全度,并将实时数据和齐全度两个指标一起存入数据库中。
可选的,每个时刻对应的齐全度是多个时间戳对应的齐全度,可以在计算出齐全度之后,将每个时间戳(timestamp)和对于每个时间戳计算得到的齐全度两个指标(timestamp,齐全度)一起存入数据库中,从而使得数据库中的每份数据都具有一个数据处理的完整度,根据数据处理的完整度可以获知实时数据的处理进度,并进一步针对实时数据做更为精准的决策。
此处需要说明的是,上述实施例中仅收集日志文件,即上述的数据源,并根据收集到的日志文件计算齐全度,并未说明数据源中实时数据的数量,因此,本申请实施例中 不举例说明如何将实时数据和齐全度两个指标一起存入数据库中。
根据合并后节点上的数据结构,计算上述表1至表4所示的实施例中的实时数据的齐全度,具体如下所示:
1:在20160113 13:42:53秒这个时刻,对于时间戳为20160113 13:42:49的11个日志文件,因为处理上述日志文件中的最小时间是20160113 13:42:50,比49秒大,所以认为时间戳为20160113 13:42:49的11个文件均已经被处理完毕,所以49秒的齐全度是100%。
2:在20160113 13:42:53秒这个时刻,对于时间戳为20160113 13:42:50的11个文件。因为50秒低于52秒和51秒的时间,所以在时间戳为20160113 13:42:50的11个文件中,A节点中的a.log文件正在处理中,还未被处理完毕,而A节点中的b.log,c.log两个文件和B节点中的d.log,e.log,f.log,g.log,h.log五个文件已经被处理完毕,且B节点中的k.log,m.log,n.log三个文件也已经被处理完毕,因此,时间戳为20160113 13:42:50的11个日志文件中已经处理的日志文件个数是7+3=10,此时的齐全度为10/11=91%。
3:在20160113 13:42:53秒这个时刻,对于时间戳为20160113 13:42:51的是11个文件。因为51秒介于52秒和50秒之间,所以对于时间戳为20160113 13:42:51的11个文件,A节点中的a.log文件仍未被处理完毕,B节点中的k.log,m.log,n.log三个文件正在处理中,还未被处理完毕,而A节点中的b.log,c.log两个文件和B节点中的d.log,e.log,f.log,g.log,h.log五个文件已经处理完毕。因此,时间戳为20160113 13:42:51的11个日志文件中已经处理的日志文件个数是7,可以得到51秒的所有实时数据的齐全度为7/11=63.6%。
4:在20160113 13:42:53秒这个时刻,对于时间戳为20160113 13:42:52的是11个文件。因为52秒高于51秒和50秒的时间,所以在时间戳为20160113 13:42:52的11个文件中,A节点中的b.log,c.log两个文件和B节点中的d.log,e.log,f.log,g.log,h.log五个文件正在处理中,还未被处理完毕,而且,A节点中的a.log文件和B节点中的k.log,m.log,n.log三个文件仍未被处理完毕。因此,认为时间戳为20160113 13:42:52的11个日志文件均未被处理完毕,得到52秒的所有实时数据的齐全度为0%。
5:在20160113 13:42:53秒这个时刻,对于时间戳为20160113 13:42:53的11个文件,因为所有日志文件最大处理到20160113 13:42:52,即,还未对53秒的日志文件进行收集,所以53秒的数据的统计结果是0%。
根据上述方法步骤,在20160113 13:42:54秒这个时刻,再对几个节点所收集到的实时数据计算实时数据的齐全度:
对于时间戳为20160113 13:42:49的11个日志文件,已经处理的日志文件个数是11,可以得到49秒的所有实时数据的齐全度为11/11=100%;
对于时间戳为20160113 13:42:50的11个日志文件,已经处理的日志文件个数是11,可以得到50秒的所有实时数据的齐全度为11/11=100%;
对于时间戳为20160113 13:42:51的11个日志文件,已经处理的日志文件个数是8,可以得到51秒的所有实时数据的齐全度为8/11=72.3%;
对于时间戳为20160113 13:42:52的11个日志文件,已经处理的日志文件个数是5,可以得到52秒的所有实时数据的齐全度为5/11=45.5%;
对于时间戳为20160113 13:42:53的11个日志文件,已经处理的日志文件个数是2,可以得到53秒的所有实时数据的齐全度为2/11=18.2%;
对于时间戳为20160113 13:42:54的11个日志文件,已经处理的日志文件个数是0,可以得到54秒的所有实时数据的齐全度为0。
根据上述方法步骤,在20160113 13:42:55秒这个时刻,再对几个节点所收集到的实时数据计算实时数据的齐全度:
对于时间戳为20160113 13:42:49的11个日志文件,已经处理的日志文件个数是11,可以得到49秒的所有实时数据的齐全度为11/11=100%;
对于时间戳为20160113 13:42:50的11个日志文件,已经处理的日志文件个数是11,可以得到50秒的所有实时数据的齐全度为11/11=100%;
对于时间戳为20160113 13:42:51的11个日志文件,已经处理的日志文件个数是9,可以得到51秒的所有实时数据的齐全度为9/11=81.8%;
对于时间戳为20160113 13:42:52的11个日志文件,已经处理的日志文件个数是6,可以得到52秒的所有实时数据的齐全度为6/11=54.5%;
对于时间戳为20160113 13:42:53的11个日志文件,已经处理的日志文件个数是4,可以得到53秒的所有实时数据的齐全度为4/11=36.4%;
对于时间戳为20160113 13:42:54的11个日志文件,已经处理的日志文件个数是2,可以得到54秒的所有实时数据的齐全度为2/11=18.2%;
对于时间戳为20160113 13:42:55的11个日志文件,已经处理的日志文件个数是0,可以得到55秒的所有实时数据的齐全度为0。
本申请实施例为实时数据添加了一个非常重要的属性,即提出了齐全度的概念,让每份数据都具有一个百分比(0%~100%)的概念,用该百分比来表示该数据代表的完 整度。如果没有数据的齐全度,那么实时数据的结果就没有进度的概念,这样在很多快速决策性的领域里面,就不知道是否可以根据实时数据做出决策。如果给数据加上齐全度,尤其是给实时数据加上齐全度,这样就可以根据实时数据做出更为精准的决策。
为实时数据加上齐全度,让每个时间戳(timestamp)的实时数据变成(timestamp,齐全度),也即将实时数据一个指标变成包含每个时间戳的实时数据和齐全度的两个指标,并将两个指标一起存储数据库中。齐全度的功能可以应用于如下计算中:报警监控***里的实时计算中,数据大屏里的实时计算中,tlog产品的数据统计中以及一种单值聚合的实时计算中。其中,如果是报警监控***里的实时计算,可以根据计算出的每份数据的齐全度,来获知每份数据处理的完整度,来做是否报警的判断,从而保证每一次报警的准确性。如果是数据大屏里的实时计算,可以为当前的数据加上一个百分比的处理进度。让业务人员和决策层可以针对这个数据报表,做出最准确快速正确的决策。比如,如果当前广告这分钟带来的收益是100万,如果能确定这分钟的齐全度是100%了,那么可以马上更换广告。在争分夺秒的大促,如在11.11活动中,让广告带来更多的收益。
数据的齐全度可以在多个领域应用,例如,由于实时计算的时候,当前统计的数据是一直变动的,比如每分钟的销售额数据,业务方对这份数据处理的进度完全无感知。因为数据在不断的进来,针对这个问题,本申请实施例对实时数据加入齐全度,根据计算的满足预定条件的实时数据的齐全度,可以使实时数据具有一个百分比的概念,通过该百分比可以获知实时数据处理的进度。比如,统计的预设分钟内的销售额,是一直变动的,决策者可以根据计算的满足预定条件的实时数据的齐全度,确定出是否进行销售策略的改变;又如,若是报警监控***里的实时数据,可以根据计算得到的齐全度,即通过齐全度表示的数据处理的完整度,来做是否报警的判断。通过上述方案,解决了现有技术无法获知对实时数据处理的进度的问题,从而可以使业务方根据实时数据的齐全度,做出一些更为精确的决策。
需要说明的是,对于前述的各方法实施例,为了简单描述,故将其都表述为一系列的动作组合,但是本领域技术人员应该知悉,本申请并不受所描述的动作顺序的限制,因为依据本申请,某些步骤可以采用其他顺序或者同时进行。其次,本领域技术人员也应该知悉,说明书中所描述的实施例均属于优选实施例,所涉及的动作和模块并不一定是本申请所必须的。
通过以上的实施方式的描述,本领域的技术人员可以清楚地了解到根据上述实施例 的方法可借助软件加必需的通用硬件平台的方式来实现,当然也可以通过硬件,但很多情况下前者是更佳的实施方式。基于这样的理解,本申请的技术方案本质上或者说对现有技术做出贡献的部分可以以软件产品的形式体现出来,该计算机软件产品存储在一个存储介质(如ROM/RAM、磁碟、光盘)中,包括若干指令用以使得一台终端设备(可以是手机,计算机,服务器,或者网络设备等)执行本申请各个实施例所述的方法。
实施例2
根据本申请实施例,还提供了一种用于实施上述实施例1的实时数据处理方法的实时数据处理装置的实施例,如图8所示,该装置800可以包括:收集模块802,汇总模块804以及计算模块806。
其中,收集模块802用于收集分布式***中一个或多个节点接收到的实时数据;汇总模块804用于将所有节点上满足相同预定条件的实时数据进行汇总,得到相同预定条件下对应的实时数据的总量,其中,预定条件用于表征收集到实时数据的时间;计算模块806用于计算满足相同预定条件的实时数据的齐全度,其中,齐全度用于指示:处理完的满足相同预定条件的实时数据在汇总的所有节点上的实时数据中的比例。
具体的,上述的实时数据可以是从实时统计出来的数据,该数据随着时间不断变化。上述预定条件可以是对时间的限制。上述的预定条件的实时数据可以为实时数据在预定时间内收集到的实时数据。其中,预定时间为收集实时数据时对应的不同时间。上述进度可以是一个百分比,即已经处理完的实施数据占处理完汇总后的数据的比例。通过该百分比可以获知实时数据处理的进度,使用百分比的情况下,可以使实时数据处理的进度更加直观,更容易使业务方根据实时数据的齐全度,做出一些更为精确的决策。
此处需要说明的是,对实时数据的处理可以包括多种操作,例如,一种比较简明的处理就是收集实时数据,例如,可以在收集到实时数据之后就认为对该实时数据的处理完成。当然,还有一些比较复杂的处理方式,例如,在收集到实时数据之后,还需要对实时数据进行一些加工(例如,格式的调整,或者是数据的抽取,或者是数据的汇总),此时,可以认为加工完成之后才意味着处理的完成。
此处还需要说明的是,上述收集模块802,汇总模块804以及计算模块806对应于实施例一中的步骤S502至步骤S506,三个模块与对应的步骤所实现的实例和应用场景相同,但不限于上述实施例一所公开的内容。需要说明的是,上述模块作为装置的一部分可以运行在实施例一提供的计算机终端40中。
由上可知,本申请上述实施例二公开的方案中,收集分布式***中一个或多个节点 接收到的实时数据,将所有节点上满足相同预定条件的实时数据进行汇总,得到相同预定条件下对应的实时数据的总量,计算满足相同预定条件的实时数据的齐全度,从而实现对实时数据的齐全度进行计算。
容易注意到,对实时数据的处理引入了一个参数,即齐全度,通过该参数可以得知实时数据处理的进度。该齐全度用于指示处理完满足预定条件的实时数据在处理完汇总后的实时数据中所占的进度。在引入齐全度之后,可以收集实时数据并对收集到的实时数据进行汇总;然后从汇总后的实时数据中获取满足预定条件的实时数据;从而可以计算满足预定条件的实时数据的齐全度,达到增加了对实时数据处理的评估维度的效果。
由此,本申请提供的上述实施例二的方案解决了现有技术无法获知对实时数据处理的进度的技术问题。
对于进度,可以是一个百分比,即已经处理完的实施数据占处理完汇总后的数据的比例。通过该百分比可以获知实时数据处理的进度,使用百分比的情况下,可以使实时数据处理的进度更加直观,更容易使业务方根据实时数据的齐全度,做出一些更为精确的决策。
对实时数据的处理可以包括多种操作,例如,一种比较简明的处理就是收集实时数据,例如,可以在收集到实时数据之后就认为对该实时数据的处理完成。当然,还有一些比较复杂的处理方式,例如,在收集到实时数据之后,还需要对实时数据进行一些加工(例如,格式的调整,或者是数据的抽取,或者是数据的汇总),此时,可以认为加工完成之后才意味着处理的完成。
在以下实施例的描述中,使用了“收集”到实时数据就认为是实时数据的处理完成,这是因为,收集到的实时数据之后在某种程度上就可以用来评估实时数据的完整度。
在以下实施例中,实时数据处理的齐全度可以以下维度进行评估,预定时间内收集到的实时数据的数量,该维度有些类似于评估在一个时间段内究竟能够收集多少实时数据。当然,齐全度也可以从其他的角度来进行评估,无论从什么角度选择的评估的维度,只要是可以展示进度,或者使用处理的比例的方式展示进度均涵盖在本申请实施例的保护范围之内。
根据本申请上述实施例,齐全度用于指示处理完的满足相同预定条件的实时数据的总量在汇总的所有节点上的实时数据的总量中的占比。
在一种可选的方案中,上述的齐全度可以根据数量参数进行计算。
根据本申请上述实施例,齐全度用于指示预定时间段内处理完的满足相同预定条件 的实时数据的总量在汇总的所有节点上的实时数据的总量中的占比,其中,满足相同预定条件的实时数据为预定时刻收集到的实时数据,预定时间段为预定时刻和预定时刻之前的时间段。
具体的,上述的预设时刻可以是需要对实时数据的齐全度进行计算的特定时刻。
在一种可选的方案中,实时数据的齐全度可以是针对特定时刻获取到的实时数据,在特定时刻和特定时刻之前的实时数据被处理的比例或者进度。
根据本申请上述实施例,如图8所示,在预定条件包括实时数据在预定时间内收集到的情况下,其中,上述实时数据处理装置800还包括:获取模块808。
其中,获取模块808用于收集实时数据之后,获取收集到的每个实时数据的时间戳,其中,时间戳用于指示对该实时数据进行处理的时间。
此处需要说明的是,上述获取记模块808对应于实施例一中的步骤S508,该模块与对应的步骤所实现的实例和应用场景相同,但不限于上述实施例一所公开的内容。需要说明的是,上述模块作为装置的一部分可以运行在实施例一提供的计算机终端40中。
通过上述实施例,为收集到的每个实时数据标上时间戳,可以根据时间戳直观的得到在预定时间内收集到的实时数据的数量,从而可以快速计算出数据的齐全度。
根据本申请上述实施例,如图8所示,计算模块806还用于在当前时间戳的情况下,将处理完成的所有实时数据的总量在汇总的所有节点上的实时数据的总量中的占比,作为齐全度。
此处需要说明的是,上述计算模块806对应于实施例一中的步骤S5062,该模块与对应的步骤所实现的实例和应用场景相同,但不限于上述实施例一所公开的内容。需要说明的是,上述模块作为装置的一部分可以运行在实施例一提供的计算机终端40中。
根据本申请上述实施例,如图8所示,上述的汇总模块806包括:获取单元810和汇总单元812。
其中,获取单元810用于获取每个分布式节点上的实时数据;汇总单元812,用于将获取到的每个分布式节点上具有相同时间戳的实时数据进行汇总,得到每个时间戳对应的实时数据的总量。
此处需要说明的是,上述获取单元810和汇总单元812对应于实施例一中的步骤S5042至步骤S5044,两个模块与对应的步骤所实现的实例和应用场景相同,但不限于上述实施例一所公开的内容。需要说明的是,上述模块作为装置的一部分可以运行在实施例一提供的计算机终端40中。
通过上述实施例,由于是分布式环境,首先将每个分布式节点上的实时数据进行汇总,然后在一个单节点上进行合并操作,最终得到预定时间段内的实时数据的汇总结果,采用该方案,可以在不丢弃数据的同时,达到快速且准确对预定时间段内的实时数据的汇总的目的。
本申请实施例为实时数据添加了一个非常重要的属性,即提出了齐全度的概念,让每份数据都具有一个百分比(0%~100%)的概念,用该百分比来表示该数据代表的完整度。如果没有数据的齐全度,那么实时数据的结果就没有进度的概念,这样在很多快速决策性的领域里面,就不知道是否可以根据实时数据做出决策。如果给数据加上齐全度,尤其是给实时数据加上齐全度,这样就可以根据实时数据做出更为精准的决策。
数据的齐全度可以在多个领域应用,例如,由于实时计算的时候,当前统计的数据是一直变动的,比如每分钟的销售额数据,业务方对这份数据处理的进度完全无感知。因为数据在不断的进来,针对这个问题,本申请实施例提出了齐全度的概念,即对实时数据加入齐全度,根据计算的满足预定条件的实时数据的齐全度,可以使实时数据具有一个百分比的概念,通过该百分比可以获知实时数据处理的进度。比如,统计的预设分钟内的销售额,是一直变动的,决策者可以根据计算的满足预定条件的实时数据的齐全度,确定出是否进行销售策略的改变;又如,若是报警监控***里的实时数据,可以根据计算得到的齐全度,即通过齐全度表示的数据处理的完整度,来做是否报警的判断。通过上述方案,解决了现有技术无法获知对实时数据处理的进度的问题,从而可以使业务方根据实时数据的齐全度,做出一些更为精确的决策。
实施例3
本申请的实施例可以提供一种计算机终端,该计算机终端可以是计算机终端群中的任意一个计算机终端设备。可选地,在本实施例中,上述计算机终端也可以替换为移动终端等终端设备。
可选地,在本实施例中,上述计算机终端可以位于计算机网络的多个网络设备中的至少一个网络设备。
在本实施例中,上述计算机终端可以执行实时数据处理方法中以下步骤的程序代码:收集分布式***中一个或多个节点接收到的实时数据;将所有节点上满足预定条件的实时数据进行汇总,得到相同预定条件下对应的实时数据的总量,其中,预定条件用于表征收集到实时数据的时间;计算满足相同预定条件的实时数据的齐全度,其中,齐全度用于指示:处理完的满足相同预定条件的实时数据在汇总的所有节点上的实时数据中的 比例。
图9是根据本申请实施例的一种计算机终端的结构框图,如图9所示,该计算机终端900可以包括:一个或多个(图中仅示出一个)处理器902、存储器904。该终端可以理解为一个计算端,该计算端具有一定的计算能力,其可以进行运算并反馈运算结果。在进行运算时,该计算端可以理解为一个服务提供端。多个终端可以联合进行运算服务。终端可以是计算机、手机、平板电脑等。当然,终端是计算机时,多个计算机可以联合提供服务,也可以理解为是服务器群组;从另一个角度上讲,这些设备也可提供云计算服务。
其中,存储器可用于存储软件程序以及模块,如本申请实施例中的实时数据处理方法和装置对应的程序指令/模块,处理器通过运行存储在存储器内的软件程序以及模块,从而执行各种功能应用以及数据处理,即实现上述的方法。存储器可包括高速随机存储器,还可以包括非易失性存储器,如一个或者多个磁性存储装置、闪存、或者其他非易失性固态存储器。在一些实例中,存储器可进一步包括相对于处理器远程设置的存储器,这些远程存储器可以通过网络连接至计算机终端900。上述网络的实例包括但不限于互联网、企业内部网、局域网、移动通信网及其组合。
处理器可以通过传输装置调用存储器存储的信息及应用程序,以执行下述步骤:收集分布式***中一个或多个节点接收到的实时数据;将所有节点上满足相同预定条件的实时数据进行汇总,得到相同预定条件下对应的实时数据的总量,其中,预定条件用于表征收集到实时数据的时间;计算满足相同预定条件的实时数据的齐全度,其中,齐全度用于指示:处理完的满足相同预定条件的实时数据在汇总的所有节点上的实时数据中的比例。
可选的,上述处理器还可以执行如下步骤的程序代码:齐全度用于指示处理完的满足相同预定条件的实时数据的总量在汇总的所有节点上的实时数据的总量中的占比。
可选的,上述处理器还可以执行如下步骤的程序代码:齐全度用于指示预定时间段内处理完的满足相同预定条件的实时数据的总量在汇总的所有节点上的实时数据的总量中的占比,其中,满足相同预定条件的实时数据为预定时刻收集到的实时数据,预定时间段为预定时刻和预定时刻之前的时间段。
可选的,上述处理器还可以执行如下步骤的程序代码:在收集分布式***中一个或多个节点接收到的实时数据之后,获取收集到的每个实时数据的时间戳,其中,时间戳用于指示对该实时数据进行处理的时间。
可选的,上述处理器还可以执行如下步骤的程序代码:在当前时间戳的情况下,将处理完成的所有实时数据的总量在汇总的所有节点上的实时数据的总量中的占比,作为齐全度。
可选的,上述处理器还可以执行如下步骤的程序代码:获取每个分布式节点上的实时数据;将获取到的每个分布式节点上具有相同时间戳的实时数据进行汇总,得到每个时间戳对应的实时数据的总量。
采用本申请实施例,收集分布式***中一个或多个节点接收到的实时数据,将所有节点上满足相同预定条件的实时数据进行汇总,得到相同预定条件下对应的实时数据的总量,计算满足相同预定条件的实时数据的齐全度,从而实现对实时数据的齐全度进行计算。
容易注意到,对实时数据的处理引入了一个参数,即齐全度,通过该参数可以得知实时数据处理的进度。该齐全度用于指示处理完满足预定条件的实时数据在处理完汇总后的实时数据中所占的进度。在引入齐全度之后,可以收集实时数据并对收集到的实时数据进行汇总;然后从汇总后的实时数据中获取满足预定条件的实时数据;从而可以计算满足预定条件的实时数据的齐全度,达到增加了对实时数据处理的评估维度的效果。
由此,本申请实施例解决了现有技术无法获知对实时数据处理的进度的技术问题。
本领域普通技术人员可以理解,图9所示的结构仅为示意,计算机终端也可以是智能手机(如Android手机、iOS手机等)、平板电脑、掌声电脑以及移动互联网设备(Mobile Internet Devices,MID)、PAD等终端设备。图9其并不对上述电子装置的结构造成限定。例如,计算机终端900还可包括比图9中所示更多或者更少的组件(如网络接口、显示装置等),或者具有与图9所示不同的配置。图9的计算机也可以作为服务器来对外提供服务。
本领域普通技术人员可以理解上述实施例的各种方法中的全部或部分步骤是可以通过程序来指令终端设备相关的硬件来完成,该程序可以存储于一计算机可读存储介质中,存储介质可以包括:闪存盘、只读存储器(Read-Only Memory,ROM)、随机存取器(Random Access Memory,RAM)、磁盘或光盘等。
实施例4
本申请的实施例还提供了一种存储介质。可选地,在本实施例中,上述存储介质可以用于保存上述实施例一所提供的实时数据处理方法所执行的程序代码。
可选地,在本实施例中,上述存储介质可以位于计算机网络中计算机终端群中的任 意一个计算机终端中,或者位于移动终端群中的任意一个移动终端中。
可选地,在本实施例中,存储介质被设置为存储用于执行以下步骤的程序代码:收集分布式***中一个或多个节点接收到的实时数据;将所有节点上满足相同预定条件的实时数据进行汇总,得到相同预定条件下对应的实时数据的总量,其中,预定条件用于表征收集到实时数据的时间;计算满足相同预定条件的实时数据的齐全度,其中,齐全度用于指示:处理完的满足相同预定条件的实时数据在汇总的所有节点上的实时数据中的比例。
可选地,存储介质还被设置为存储用于执行以下步骤的程序代码:齐全度用于指示处理完的满足相同预定条件的实时数据的总量在汇总的所有节点上的实时数据的总量中的占比。
可选的,存储介质还被设置为存储用于执行以下步骤的程序代码:齐全度用于指示预定时间段内处理完的满足相同预定条件的实时数据的总量在汇总的所有节点上的实时数据的总量中的占比,其中,满足相同预定条件的实时数据为预定时刻收集到的实时数据,预定时间段为预定时刻和预定时刻之前的时间段。
可选地,存储介质还被设置为存储用于执行以下步骤的程序代码:在收集分布式***中一个或多个节点接收到的实时数据之后,获取收集到的每个实时数据的时间戳,其中,时间戳用于指示对该实时数据进行处理的时间。
可选地,存储介质还被设置为存储用于执行以下步骤的程序代码:在当前时间戳的情况下,将处理完成的所有实时数据的总量在汇总的所有节点上的实时数据的总量中的占比,作为齐全度。
可选地,存储介质还被设置为存储用于执行以下步骤的程序代码:获取每个分布式节点上的实时数据;将获取到的每个分布式节点上具有相同时间戳的实时数据进行汇总,得到每个时间戳对应的实时数据的总量。
上述本申请实施例序号仅仅为了描述,不代表实施例的优劣。
在本申请的上述实施例中,对各个实施例的描述都各有侧重,某个实施例中没有详述的部分,可以参见其他实施例的相关描述。
在本申请所提供的几个实施例中,应该理解到,所揭露的技术内容,可通过其它的方式实现。其中,以上所描述的装置实施例仅仅是示意性的,例如所述单元的划分,可以为一种逻辑功能划分,实际实现时可以有另外的划分方式,例如多个单元或组件可以结合或者可以集成到另一个***,或一些特征可以忽略,或不执行。另一点,所显示或 讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口,单元或模块的间接耦合或通信连接,可以是电性或其它的形式。
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的,作为单元显示的部件可以是或者也可以不是物理单元,即可以位于一个地方,或者也可以分布到多个单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
另外,在本申请各个实施例中的各功能单元可以集成在一个处理单元中,也可以是各个单元单独物理存在,也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现,也可以采用软件功能单元的形式实现。
所述集成的单元如果以软件功能单元的形式实现并作为独立的产品销售或使用时,可以存储在一个计算机可读取存储介质中。基于这样的理解,本申请的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的全部或部分可以以软件产品的形式体现出来,该计算机软件产品存储在一个存储介质中,包括若干指令用以使得一台计算机设备(可为个人计算机、服务器或者网络设备等)执行本申请各个实施例所述方法的全部或部分步骤。而前述的存储介质包括:U盘、只读存储器(ROM,Read-Only Memory)、随机存取存储器(RAM,Random Access Memory)、移动硬盘、磁碟或者光盘等各种可以存储程序代码的介质。
以上所述仅是本申请的优选实施方式,应当指出,对于本技术领域的普通技术人员来说,在不脱离本申请原理的前提下,还可以做出若干改进和润饰,这些改进和润饰也应视为本申请的保护范围。
Claims (12)
- 一种实时数据处理方法,其特征在于,包括:收集分布式***中一个或多个节点接收到的实时数据;将所有节点上满足相同预定条件的实时数据进行汇总,得到所述相同预定条件下对应的实时数据的总量,其中,所述预定条件用于表征收集到所述实时数据的时间;计算满足所述相同预定条件的实时数据的齐全度,其中,所述齐全度用于指示:处理完的满足所述相同预定条件的实时数据在汇总的所有节点上的实时数据中的比例。
- 根据权利要求1所述的方法,其特征在于,所述齐全度用于指示处理完的满足所述相同预定条件的实时数据的总量在汇总的所有节点上的实时数据的总量中的占比。
- 根据权利要求2所述的方法,其特征在于,所述齐全度用于指示预定时间段内处理完的满足所述相同预定条件的实时数据的总量在汇总的所有节点上的实时数据的总量中的占比,其中,满足所述相同预定条件的实时数据为预定时刻收集到的实时数据,所述预定时间段为所述预定时刻和所述预定时刻之前的时间段。
- 根据权利要求3所述的方法,其特征在于,在收集分布式***中一个或多个节点接收到的实时数据之后,所述方法还包括:获取收集到的每个所述实时数据的时间戳,其中,所述时间戳用于指示对该实时数据进行处理的时间。
- 根据权利要求4所述的方法,其特征在于,计算满足所述相同预定条件的实时数据的齐全度包括:在当前时间戳的情况下,将处理完成的所有实时数据的总量在汇总的所有节点上的实时数据的总量中的占比,作为所述齐全度。
- 根据权利要求1至5中任一项所述的方法,其特征在于,将所有节点上满足相同预定条件的实时数据进行汇总,得到所述相同预定条件下对应的实时数据的总量包括:获取每个分布式节点上的实时数据;将获取到的所述每个分布式节点上具有相同时间戳的实时数据进行汇总,得到每个时间戳对应的实时数据的总量。
- 一种实时数据处理装置,其特征在于,包括:收集模块,用于收集分布式***中一个或多个节点接收到的实时数据;汇总模块,用于将所有节点上满足相同预定条件的实时数据进行汇总,得到所述相同预定条件下对应的实时数据的总量,其中,所述预定条件用于表征收集到所述实时数据的时间;计算模块,用于计算满足所述相同预定条件的实时数据的齐全度,其中,所述齐全度用于指示:处理完的满足所述相同预定条件的实时数据在汇总的所有节点上的实时数据中的比例。
- 根据权利要求7所述的装置,其特征在于,所述齐全度用于指示处理完的满足所述相同预定条件的实时数据的总量在汇总的所有节点上的实时数据的总量中的占比。
- 根据权利要求8所述的装置,其特征在于,所述齐全度用于指示预定时间段内处理完的满足所述相同预定条件的实时数据的总量在汇总的所有节点上的实时数据的总量中的占比,其中,满足所述相同预定条件的实时数据为预定时刻收集到的实时数据,所述预定时间段为所述预定时刻和所述预定时刻之前的时间段。
- 根据权利要求9所述的装置,其特征在于,所述装置还包括:获取模块,用于收集所述实时数据之后,获取收集到的每个所述实时数据的时间戳,其中,所述时间戳用于指示对该实时数据进行处理的时间。
- 根据权利要求10所述的装置,其特征在于,所述计算模块还用于在当前时间戳的情况下,将处理完成的所有实时数据的总量在汇总的所有节点上的实时数据的总量中的占比,作为所述齐全度。
- 根据权利要求7至11中任一项所述的装置,其特征在于,所述汇总模块包括:获取单元,用于获取每个分布式节点上的实时数据;汇总单元,用于将获取到的所述每个分布式节点上具有相同时间戳的实时数据进行汇总,得到每个时间戳对应的实时数据的总量。
Priority Applications (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2019535921A JP7046073B2 (ja) | 2016-12-30 | 2017-12-19 | リアルタイムデータ処理方法及び装置 |
US16/457,540 US11635985B2 (en) | 2016-12-30 | 2019-06-28 | Using degree of completeness of real-time data to maximize product revenue |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201611270978.5 | 2016-12-30 | ||
CN201611270978.5A CN108268357B (zh) | 2016-12-30 | 2016-12-30 | 实时数据处理方法和装置 |
Related Child Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/457,540 Continuation US11635985B2 (en) | 2016-12-30 | 2019-06-28 | Using degree of completeness of real-time data to maximize product revenue |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2018121335A1 true WO2018121335A1 (zh) | 2018-07-05 |
Family
ID=62706875
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2017/117082 WO2018121335A1 (zh) | 2016-12-30 | 2017-12-19 | 实时数据处理方法和装置 |
Country Status (5)
Country | Link |
---|---|
US (1) | US11635985B2 (zh) |
JP (1) | JP7046073B2 (zh) |
CN (1) | CN108268357B (zh) |
TW (1) | TW201824018A (zh) |
WO (1) | WO2018121335A1 (zh) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN113468234A (zh) * | 2021-05-31 | 2021-10-01 | 济南浪潮数据技术有限公司 | 一种监控数据处理方法、装置、***及存储介质 |
Families Citing this family (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN109240848A (zh) * | 2018-07-27 | 2019-01-18 | 阿里巴巴集团控股有限公司 | 一种数据对象标识生成方法及装置 |
CN112307260A (zh) * | 2020-10-30 | 2021-02-02 | 北京字节跳动网络技术有限公司 | 视频鉴别方法、装置、电子设备及计算机可读存储介质 |
CN112364063B (zh) * | 2021-01-12 | 2021-06-04 | 北京智慧星光信息技术有限公司 | 流计算***及其数据处理方法、电子设备及介质 |
CN116980890B (zh) * | 2023-09-20 | 2023-12-22 | 北京集度科技有限公司 | 信息安全通信装置、方法、车辆、计算机程序产品 |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7076543B1 (en) * | 2002-02-13 | 2006-07-11 | Cisco Technology, Inc. | Method and apparatus for collecting, aggregating and monitoring network management information |
CN101170454A (zh) * | 2006-10-27 | 2008-04-30 | 亿阳信通股份有限公司 | 一种监控数据采集汇总状态的方法和*** |
CN102882745A (zh) * | 2012-09-29 | 2013-01-16 | 摩卡软件(天津)有限公司 | 一种用于监控业务服务器的方法和装置 |
CN104899948A (zh) * | 2015-06-05 | 2015-09-09 | 北京京东尚科信息技术有限公司 | 一种封闭空间拥堵监测方法和*** |
CN105812202A (zh) * | 2014-12-31 | 2016-07-27 | 阿里巴巴集团控股有限公司 | 日志实时监控预警方法及其装置 |
Family Cites Families (20)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6035297A (en) | 1996-12-06 | 2000-03-07 | International Business Machines Machine | Data management system for concurrent engineering |
US6215898B1 (en) | 1997-04-15 | 2001-04-10 | Interval Research Corporation | Data processing system and method |
US6542905B1 (en) | 1999-03-10 | 2003-04-01 | Ltcq, Inc. | Automated data integrity auditing system |
US6526044B1 (en) | 1999-06-29 | 2003-02-25 | Wandel & Goltermann Technologies, Inc. | Real-time analysis through capture buffer with real-time historical data correlation |
US20020091991A1 (en) * | 2000-05-11 | 2002-07-11 | Castro Juan Carlos | Unified real-time microprocessor computer |
JP2004021744A (ja) | 2002-06-18 | 2004-01-22 | Hitachi Ltd | 業務工程の状況把握システム、業務工程の状況把握システムの記録媒体、及び業務工程の状況把握表示装置 |
US8606594B2 (en) | 2002-10-29 | 2013-12-10 | Practice Velocity, LLC | Method and system for automated medical records processing |
US20090138415A1 (en) | 2007-11-02 | 2009-05-28 | James Justin Lancaster | Automated research systems and methods for researching systems |
US8429220B2 (en) | 2007-03-29 | 2013-04-23 | International Business Machines Corporation | Data exchange among data sources |
CN101729296B (zh) * | 2009-12-29 | 2012-12-19 | 中兴通讯股份有限公司 | 一种以太网流量统计分析的方法和*** |
US20120284118A1 (en) * | 2011-05-04 | 2012-11-08 | Microsoft Corporation | Using collective data for targeting of advertisements |
WO2013009710A1 (en) | 2011-07-08 | 2013-01-17 | Steamfunk Labs, Inc. | Automated presentation of information using infographics |
CN103262042B (zh) * | 2011-12-19 | 2016-03-30 | 华为技术有限公司 | 一种分布式存储数据恢复方法、装置及*** |
JP2014179000A (ja) | 2013-03-15 | 2014-09-25 | Ricoh Co Ltd | 情報管理システム及びプログラム |
CN103825930B (zh) * | 2013-11-12 | 2017-03-29 | 浙江省水文局 | 一种分布式环境下的实时数据同步方法 |
CN109945844B (zh) | 2014-05-05 | 2021-03-12 | 赫克斯冈技术中心 | 测量子***和测量*** |
US9804951B2 (en) * | 2014-10-08 | 2017-10-31 | Signalfx, Inc. | Quantization of data streams of instrumented software |
CN104331270B (zh) * | 2014-10-24 | 2017-07-21 | 华为技术有限公司 | 一种流数据处理的方法、装置及*** |
CN104468569B (zh) * | 2014-12-04 | 2017-12-22 | 北京国双科技有限公司 | 分布式数据的完整性检测方法和装置 |
US11442804B2 (en) * | 2019-12-27 | 2022-09-13 | Paypal, Inc. | Anomaly detection in data object text using natural language processing (NLP) |
-
2016
- 2016-12-30 CN CN201611270978.5A patent/CN108268357B/zh active Active
-
2017
- 2017-08-10 TW TW106127141A patent/TW201824018A/zh unknown
- 2017-12-19 JP JP2019535921A patent/JP7046073B2/ja active Active
- 2017-12-19 WO PCT/CN2017/117082 patent/WO2018121335A1/zh active Application Filing
-
2019
- 2019-06-28 US US16/457,540 patent/US11635985B2/en active Active
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7076543B1 (en) * | 2002-02-13 | 2006-07-11 | Cisco Technology, Inc. | Method and apparatus for collecting, aggregating and monitoring network management information |
CN101170454A (zh) * | 2006-10-27 | 2008-04-30 | 亿阳信通股份有限公司 | 一种监控数据采集汇总状态的方法和*** |
CN102882745A (zh) * | 2012-09-29 | 2013-01-16 | 摩卡软件(天津)有限公司 | 一种用于监控业务服务器的方法和装置 |
CN105812202A (zh) * | 2014-12-31 | 2016-07-27 | 阿里巴巴集团控股有限公司 | 日志实时监控预警方法及其装置 |
CN104899948A (zh) * | 2015-06-05 | 2015-09-09 | 北京京东尚科信息技术有限公司 | 一种封闭空间拥堵监测方法和*** |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN113468234A (zh) * | 2021-05-31 | 2021-10-01 | 济南浪潮数据技术有限公司 | 一种监控数据处理方法、装置、***及存储介质 |
CN113468234B (zh) * | 2021-05-31 | 2023-12-22 | 济南浪潮数据技术有限公司 | 一种监控数据处理方法、装置、***及存储介质 |
Also Published As
Publication number | Publication date |
---|---|
US20190324794A1 (en) | 2019-10-24 |
JP7046073B2 (ja) | 2022-04-01 |
JP2020507147A (ja) | 2020-03-05 |
US11635985B2 (en) | 2023-04-25 |
TW201824018A (zh) | 2018-07-01 |
CN108268357A (zh) | 2018-07-10 |
CN108268357B (zh) | 2021-10-26 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
WO2018121335A1 (zh) | 实时数据处理方法和装置 | |
US9043317B2 (en) | System and method for event-driven prioritization | |
CN105657003B (zh) | 一种信息处理方法和服务器 | |
AU2013344416B2 (en) | Unsupervised prioritization and visualization of clusters | |
US20150066950A1 (en) | Sentiment scoring for sports entities and filtering techniques | |
CN105608517B (zh) | 基于流的业务交易性能管理及可视化方法和装置 | |
WO2015090241A1 (zh) | 监控业务运行数据入库的方法和相关设备及*** | |
US9479414B1 (en) | System and method for analyzing computing performance | |
CN105791902B (zh) | 用户推荐方法及装置 | |
CN109949172B (zh) | 社交账号影响力评价方法、装置及存储介质 | |
CN107273521B (zh) | Feed内容质量评价方法及装置 | |
US20190050317A1 (en) | Systems and methods for determining event processing delays | |
TW201510720A (zh) | 數據統計方法和裝置 | |
Finley et al. | Does network quality matter? A field study of mobile user satisfaction | |
CN110602215B (zh) | 基于联盟区块链的资源处理方法及联盟区块链*** | |
US11194834B2 (en) | Systems and methods for enabling situational awareness for events via data visualization | |
CN107809363A (zh) | 一种网络信息传播控制方法和装置 | |
CN115185794A (zh) | 服务调用链的分析方法、装置及电子设备 | |
CN104937613A (zh) | 量化数据质量的探试 | |
CN113391741B (zh) | 操作校验方法和装置、存储介质及电子设备 | |
CN109377391B (zh) | 一种信息追踪方法、存储介质和服务器 | |
WO2023045434A1 (zh) | 访问检测方法、***及装置 | |
CN107786528B (zh) | 应用的登录方法及装置、通信*** | |
CN104410874A (zh) | 视频粘度信息的检测方法、装置和*** | |
CN112131276A (zh) | 一种数据统计的方法、电子设备及可读存储介质 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 17886316 Country of ref document: EP Kind code of ref document: A1 |
|
ENP | Entry into the national phase |
Ref document number: 2019535921 Country of ref document: JP Kind code of ref document: A |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 17886316 Country of ref document: EP Kind code of ref document: A1 |