CN112269903B

CN112269903B - Data processing method, data processing apparatus, and readable storage medium

Info

Publication number: CN112269903B
Application number: CN202011301375.3A
Authority: CN
Inventors: 周振江; 吴庆双; 周效军; 王浩然; 邵传贤
Original assignee: Migu Cultural Technology Co Ltd; China Mobile Communications Group Co Ltd
Current assignee: Migu Cultural Technology Co Ltd; China Mobile Communications Group Co Ltd
Priority date: 2020-11-19
Filing date: 2020-11-19
Publication date: 2024-06-11
Anticipated expiration: 2040-11-19
Also published as: CN112269903A

Abstract

The application provides a data processing method, a data processing device and a readable storage medium. The data processing method comprises the following steps: obtaining L first time lengths corresponding to the ith period of the first index, wherein i and L are positive integers; determining a target duration according to the L first durations and the reference duration acquired in advance; and after controlling the first index to store the data acquired by the data processing equipment in the target duration, controlling the first index to stop storing the data. In this way, the data volume stored in the first index can be prevented from being too large, and the data query efficiency of the data processing equipment can be improved.

Description

Data processing method, data processing apparatus, and readable storage medium

Technical Field

The embodiment of the application relates to the technical field of computers, in particular to a data processing method, data processing equipment and a readable storage medium.

Background

Currently, an index of the system is used to store the amount of data over a period of time. However, if the user volume increases sharply (the system operation capability increases and the product popularization effect appears) in the period corresponding to a certain index, the data of the data processing device increases greatly, which results in that the data volume stored in the index is too large, and further, the data query efficiency of the data processing device is lower.

Disclosure of Invention

The embodiment of the application provides a data processing method, data processing equipment and a readable storage medium, which are used for solving the problem that the data query efficiency of the data processing equipment is lower due to the fact that the data volume stored by an index is overlarge.

To solve the above problems, the present application is achieved as follows:

In a first aspect, an embodiment of the present application provides a data processing method, where the method includes:

obtaining L first time lengths corresponding to the ith period of the first index, wherein i and L are positive integers;

Determining a target duration according to the L first durations and the reference duration acquired in advance;

And after controlling the first index to store the data acquired by the data processing equipment in the target duration, controlling the first index to stop storing the data.

Optionally, the determining the target duration according to the L first durations and the reference duration acquired in advance includes:

acquiring a first posterior probability value and a second posterior probability value according to second data, wherein the second data is stored before the end time of the ith period of the first index; the first posterior probability values are probability values of which the first time length is longer than the reference time length when the performance parameter value of the data processing equipment meets a preset condition, and the second posterior probability values are probability values of which the performance parameter value of the data processing equipment meets the preset condition and are smaller than the reference time length;

comparing the first posterior probability value with the second posterior probability value to obtain a comparison result;

And determining a target duration according to at least one of the L first durations and the reference duration based on the comparison result.

Optionally, the determining, based on the comparison result, a target duration according to at least one of the L first durations and the reference duration includes at least one of:

If the comparison result is that the first posterior probability value is greater than the second posterior probability value, P-i periods are determined as target time periods, P is the number of periods included in any one of the L first time periods, and P is an integer greater than i;

And when the comparison result is that the first posterior probability value is smaller than or equal to the first posterior probability value, N-i periods are determined to be target duration, N is the period number included in the reference duration, and N is an integer larger than i.

Optionally, the second data corresponds to H periods, and H is a positive integer;

the obtaining the first posterior probability value and the second posterior probability value according to the second data includes:

Obtaining a first posterior probability value and a second posterior probability value according to L first judgment results respectively corresponding to the H periods and a second judgment result respectively corresponding to the H periods;

Wherein, the first judging result is: judging whether the first time length is longer than the reference time length or not; the second judgment result is as follows: and judging whether the performance parameter value of the data processing equipment meets the judgment result of a preset condition.

Detecting whether a first time length is longer than the reference time length in the L first time lengths, and obtaining a detection result;

And determining a target duration according to at least one of the L first durations and the reference duration based on the detection result.

Optionally, the determining, based on the detection result, a target duration according to at least one of the L first durations and the reference duration includes:

When the detection result is that the first time length of the L first time lengths is larger than the reference time length, M-i periods are determined to be target time lengths, M is the number of periods included in any one of the L first time lengths, and M is an integer larger than i;

And under the condition that the detection result is that L first time periods corresponding to the ith period are smaller than the reference time period, N-i periods are determined to be target time periods, N is the period number included in the reference time period, and N is an integer larger than i.

Optionally, the acquiring L first durations corresponding to the ith period of the first index of the data processing device includes at least one of the following:

obtaining L parameter values corresponding to second data and L parameter values corresponding to third data, wherein the second data is stored before the end time of the ith period of the first index, and the third data is acquired by the data processing equipment in the ith period of the first index;

Determining the cycle number included in a first time length corresponding to the first parameter value as D+i;

wherein D is a ratio of a first value to a first parameter value corresponding to the third data, and the first value is a difference between a first parameter maximum value and the first parameter value corresponding to the second data; the first parameter value is any one of the L parameter values.

Optionally, before determining the target duration according to the L first durations corresponding to the ith period and the reference duration acquired in advance, the method further includes:

Acquiring fourth data, wherein the fourth data is data acquired by the data processing equipment in a period when the performance parameter value of the data processing equipment meets a preset condition;

and determining the reference time length according to the fourth data.

In a second aspect, an embodiment of the present application further provides a data processing apparatus, including: a transceiver, a memory, a processor, and a program or instructions stored on the memory and executable on the processor; wherein the processor is configured to read a program or instructions in the memory to implement the steps of the method as claimed in any one of the preceding claims.

In a third aspect, embodiments of the present application also provide a readable storage medium storing a program or instructions which, when executed by a processor, implement the steps in a method as claimed in any preceding claim.

In the embodiment of the application, L first time lengths corresponding to the ith period of the first index are obtained, wherein i and L are positive integers; determining a target duration according to the L first durations and the reference duration acquired in advance; and after controlling the first index to store the data acquired by the data processing equipment in the target duration, controlling the first index to stop storing the data. In this way, the data volume stored in the first index can be prevented from being too large, and the data query efficiency of the data processing equipment can be improved.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the description of the embodiments of the present application will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and other drawings may be obtained according to these drawings without inventive effort to a person of ordinary skill in the art.

FIG. 1 is a schematic flow chart of a data processing method according to an embodiment of the present application;

FIG. 2 is a second flowchart of a data processing method according to an embodiment of the present application;

FIG. 3 is a schematic diagram of a data processing apparatus according to an embodiment of the present application;

FIG. 4 is a second schematic diagram of a data processing apparatus according to an embodiment of the present application.

Detailed Description

The following description of the embodiments of the present application will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are some, but not all embodiments of the application. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.

The terms "first," "second," and the like in this disclosure are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus. Furthermore, the use of "or" in the present application means at least one of the connected objects, such as a or B or C, means 7 cases including a alone a, B alone, C alone, and both a and B, both B and C, both a and C, and both A, B and C.

The data processing method provided by the embodiment of the application can be applied to data processing equipment. In practical applications, the data processing device (or referred to as a data processing system) may be specifically represented as a search platform, a search engine, a search server, etc., and embodiments of the present application are not limited to the specific form of the data processing device. Such as: the data processing device may be, but is not limited to, ELASTICSEARCH (ES) search engines.

In an embodiment of the present application, the data processing apparatus stores data by Index (Index). The index may also be referred to as a database, a folder, etc., but is not limited thereto.

Referring to fig. 1, fig. 1 is a flow chart of a data processing method according to an embodiment of the present application.

As shown in fig. 1, the data processing method may include the steps of:

step 101, obtaining L first durations corresponding to the ith period of the first index, wherein i and L are positive integers.

In particular implementations, the first index may be any index created for the data processing device. For any index, it may store data collected by the data processing device during at least one period, which may be considered as: the period of the index, or the period corresponding to the data stored in the index. The period length can be in units of days and hours, and the specific length can be determined according to actual requirements, which is not limited in the embodiment of the application.

In the embodiment of the application, the number of periods corresponding to the stored data of different indexes created by the data processing system can be different or the same. Such as: index 1 may store data collected by the data processing device on the first 10 days of month 9 of 2020, index 2 may store data collected by the data processing device on the last 20 days of month 9 of 2020, and index 3 may store data collected by the data processing device on the first half month of month 10 of 2020.

In the embodiment of the present application, each period of the first index may correspond to L first durations. The L first durations correspond to L parameters corresponding to the data, and the parameters corresponding to the data may include, but are not limited to, the number of data (hereinafter referred to as data amount), the size of the data (hereinafter referred to as data size), and the like.

The first duration corresponding to the target parameter in the L first durations corresponding to the ith period may be understood as: the data processing apparatus predicts a number of periods required for the target parameter value of the data stored in the first index to reach the target parameter maximum value at the i-th period. It is understood that the number of cycles included in the first time period corresponding to the different parameters may be different or the same.

Step 102, determining a target duration according to the L first durations and the reference duration acquired in advance.

In the embodiment of the present application, the reference time length may be understood as: the data processing apparatus predicts the number of cycles required for each parameter value of the data stored in the first index to be smaller than the maximum value of the parameter, based on the parameter value of the data stored in one cycle when the performance parameter value thereof satisfies a preset condition.

The performance parameter value of the data processing device meeting the preset condition can be characterized as follows: the data processing device has better performance. Optionally, the performance parameter may include at least one of: slow number of queries, query response time, etc. In the case that the performance parameter value is the slow query number, the performance parameter value satisfies the preset condition, which may be expressed as: the slow query times are less than the preset times. In the case that the performance parameter value is the query response duration, the performance parameter value satisfies the preset condition, which may be expressed as: the query response time is less than a preset time.

The target duration may be understood as: the first index may also store the number of cycles corresponding to the data from the i+1th cycle. Such as: if the target duration includes a cycle number of 2, the first index may further store data acquired by the data processing apparatus within 2 cycles from the i+1th cycle of the first index (i.e., the i+1th cycle and the i+2th cycle of the first index).

In the embodiment of the present application, the manner in which the data processing apparatus determines the target duration may be different in the first case and the second case.

In one implementation, the first case may be: the data processing equipment predicts the situation that the probability value of the first time length which is larger than the reference time length exists in L first time lengths corresponding to the ith period based on the historical data stored in the first index; the second case may be: and the data processing equipment predicts the situation that the probability value of the first time length which is larger than the reference time length is smaller in the L first time lengths corresponding to the ith period based on the historical data stored in the first index.

In another implementation, the first case may be: the condition that the first time length is longer than the reference time length exists in L first time lengths corresponding to the ith period; the second case may be: and the condition that the first time length is longer than the reference time length does not exist in L first time lengths corresponding to the ith period.

Of course, in other embodiments, the data processing apparatus may directly determine any one of the L first durations and the reference duration as the target duration. It should be noted that, the embodiment of the present application is not limited to a specific implementation manner in which the data processing device determines the target duration according to the L first durations and the reference duration.

Step 103, after controlling the first index to store the data acquired by the data processing device in the target duration, controlling the first index to stop storing the data.

In particular, when the data processing device acquires the target duration, data acquired by the data processing device in Q periods from the i+1st period may be stored in the first index, and then, storage of new data in the first index may be stopped, where Q is a period number included in the target duration, and Q is a positive integer.

Alternatively, the data processing device may create a new index again, denoted as a second index, store data acquired by the data processing device during the i+q+1 th period into the second index, and so on.

According to the data processing method of the embodiment, L first time lengths corresponding to the ith period of the first index are obtained, wherein i and L are positive integers; determining a target duration according to the L first durations and the reference duration acquired in advance; and after controlling the first index to store the data acquired by the data processing equipment in the target duration, controlling the first index to stop storing the data. In this way, the data volume stored in the first index can be prevented from being too large, and the data query efficiency of the data processing equipment can be improved.

1. The following specifically describes a method for determining the target duration in the embodiment of the present application:

First embodiment

In a specific implementation, the second data may be all or part of the data stored by the first index before the end of the ith period. In the case where the second data is part of the data of the first index stored before the end of the i-th period, the second data may alternatively be all of the data of the first index stored before the start of the i-th period, but is not limited thereto.

In this embodiment, the data processing apparatus acquires a first posterior probability value and a second posterior probability value according to the second data stored in the first index.

The acquisition of the first posterior probability value and the second posterior probability value is described as follows:

And assuming that the second data corresponds to H periods, and H is a positive integer. That is, the second data includes data acquired by the data processing apparatus in H cycles.

Optionally, the acquiring the first posterior probability value and the second posterior probability value according to the second data may include:

The expression forms of the first judgment result and the second judgment result may be: "yes"; or "no". For the first judgment result, if the expression form of the first judgment result is 'yes', the first time length is larger than the reference time length; if the expression is 'no', the first duration is less than or equal to the reference duration. For the second judgment result, if the expression form of the second judgment result is yes, the performance parameter value of the data processing equipment meets the preset condition; if the expression form is 'no', the performance parameter value of the data processing device does not meet the preset condition.

It can be understood that, for each period of the H periods, based on the L parameters corresponding to the data, L first time lengths may be obtained, and thus L first determination results may be obtained. The L first judging results have corresponding relations with the L parameters.

In a specific implementation, the data processing device may acquire the first posterior probability value and the second posterior probability value according to the L first judgment results corresponding to the H periods respectively and the second judgment results corresponding to the H periods respectively by using a bayesian formula.

The Bayes formula is as follows:

Wherein P (w) is a priori probability and represents the probability of each type of distribution; p (w) is the probability of something happening; p (x|w) is a class conditional probability, which indicates the probability of something happening under the premise of a certain class; p (w|x) is a posterior probability indicating the probability that something happens and that it belongs to a certain class. The greater the posterior probability, the greater the likelihood that something belongs to this category, the more reasonable we can put it under this category.

In the embodiment of the present application, the category may include a category 1 (hereinafter referred to as w 1) and a category 2 (hereinafter referred to as w 2), where w1 is that at least one of the L first determination results has a yes expression, that is, at least one of the L first durations has a first duration longer than the reference duration, that is, at least one of the L first durations has a cycle number greater than the cycle number included in the reference duration; and w2 is represented as yes in the L first judging results, namely that one first time length which is not existed in the L first time lengths is longer than the reference time length. Something happens can be expressed as: the performance parameter value of the data processing device satisfies a preset condition.

For ease of understanding, the following is exemplified in connection with table 1:

Table 1: first judging result and second judging result corresponding to H periods

In table 1, the cycle length is 1 day; the L parameters comprise a parameter 1 and a parameter 2; the value of H is n. It will be appreciated that the values in table 1 are examples, and are not limiting.

Recording the number of the 'yes' expressions in the first judgment result corresponding to the parameter 1 as N1, wherein the probability is P1=N1/N; the number of the "no" expressions in the first judgment result corresponding to the parameter 1 is N2, and the probability is p2=n2/N. Then: p (w ₁)＝P1+P2;P(w₂)＝1-P(w₁).

As can be seen from table 1, P (x|w ₁)＝0;P(x|w₂) =1.

In addition, P (x) =p (x-w ₁)P(w₁)+P(x│w₂)P(w₂).

Bringing P (w ₁)、P(x|w₁) and P (x) into a Bayesian formula to obtain P (w ₁ -x), namely the first posterior value; substituting P (w ₂)、P(x|w₂) and P (x) into a bayesian formula, P (w ₂ -x) is obtained, namely the second posterior value.

In this optional embodiment, after the data processing apparatus acquires the first posterior probability value and the second posterior probability value, the target duration may be determined according to at least one of the L first durations and the reference duration based on a comparison result of the first posterior probability value and the second posterior probability value.

In the present embodiment, it can be assumed that: and if the first posterior probability value is larger than the second posterior probability value, the data processing equipment predicts a period required by the parameter value of the data stored in the first index to reach the maximum parameter value in the ith period, and meets the standard that the performance parameter value of the data processing equipment meets the preset condition. That is, if the first index stores the first data according to the target duration determined according to the L first durations corresponding to the ith period, and the data processing device queries the data through the index, the performance parameter value of the data processing device satisfies the preset condition.

Otherwise, if the first posterior probability value is smaller than or equal to the second posterior probability value, the period required by the data processing device to predict that the parameter value of the data stored in the first index reaches the maximum parameter value in the ith period is not in accordance with the standard. That is, if the first index stores the first data according to the target time length determined according to the L first time lengths corresponding to the ith period, the performance parameter value of the data processing device does not satisfy the preset condition when the data processing device queries the data through the index.

In the case where the first posterior probability value is greater than the second posterior probability value, based on the above assumption, a target duration may be determined according to L first durations corresponding to the i-th period. In the case where the first posterior probability value is less than or equal to the second posterior probability value, based on the above assumption, a target time period may be determined from the reference time period.

In a specific implementation, optionally, the determining, based on the comparison result, a target duration according to at least one of the L first durations and the reference duration includes at least one of:

and when the comparison result is that the first posterior probability value is smaller than or equal to the first posterior probability value, N-i periods from the (i+1) th period are determined as target time periods, N is the period number included in the reference time period, and N is an integer larger than i.

Further, P may be the number of cycles included in the first time period having the largest number of cycles among the L first time periods. In specific implementation, a target first time length of the L first time lengths may be determined first, where the number of cycles included in the target first time length is greater than or equal to the number of cycles included in other first time lengths; then, a difference between the number of cycles included in the target first time period and i may be determined as the number of cycles included in the target time period.

In this optional embodiment, the data processing apparatus may determine the target duration according to a first duration with the largest number of cycles among the L first durations. It will be appreciated that in other embodiments, the data processing apparatus may determine the target duration according to any one of the L first durations.

It should be noted that the above assumption of the present embodiment is merely an example, and the data processing apparatus may determine the target duration from at least one of the L first durations and the reference duration based on the comparison result based on other assumptions. It will be appreciated that the manner in which the target duration is determined may be different based on different assumptions.

Second embodiment

In this embodiment, the data processing device may directly detect whether a first time length is greater than a reference time length in L first time lengths corresponding to the ith period, and determine, based on the detection result, a target time length according to at least one of the L first time lengths and the reference time length.

Alternatively, it may be assumed that: if the first time length of the L first time lengths is greater than the reference time length, the data processing equipment predicts a period required by the parameter value of the data stored by the first index to reach the maximum value of the parameter in the ith period, and meets the standard that the performance parameter value of the data processing equipment meets the preset condition. That is, if the first index stores the first data according to the target time length determined according to the L first time lengths, the performance parameter value of the data processing device satisfies the preset condition when the data processing device queries the data through the index.

Otherwise, if the first time length is not longer than the reference time length in the L first time lengths, that is, the L first time lengths are all shorter than the reference time length, the period required by the data processing device to predict that the parameter value of the data stored in the first index reaches the maximum parameter value in the ith period is described, and the standard is not met. That is, if the first index stores the first data according to the target time periods determined according to the L first time periods, the performance parameter value of the data processing apparatus does not satisfy the preset condition when the data processing apparatus queries the data through the index.

And under the condition that the first time length exists in the L first time lengths and is larger than the reference time length, based on the assumption, determining the target time length according to the L first time lengths corresponding to the ith period. And under the condition that the first time length is not longer than the reference time length in the L first time lengths corresponding to the ith period, based on the assumption, determining the target time length according to the reference time length.

In a specific implementation, optionally, the determining, based on the detection result, a target duration according to at least one of the L first durations and the reference duration includes:

Further, M may be a number of periods included in a first time period having a largest number of periods among the L first time periods. In specific implementation, a target first time length of the L first time lengths may be determined first, where the number of cycles included in the target first time length is greater than or equal to the number of cycles included in other first time lengths; then, a difference between the number of cycles included in the target first time period and i may be determined as the number of cycles included in the target time period.

It should be noted that the above assumption of the present embodiment is merely an example, and the data processing apparatus may determine the target duration based on the comparison result and at least one of the L first durations and the reference durations corresponding to the ith period based on other assumptions. It will be appreciated that the manner in which the target duration is determined may be different based on different assumptions.

2. The following describes the acquisition of the L first durations in the embodiment of the present application:

In the embodiment of the present application, the L first durations are related to L parameter values corresponding to second data, and L parameter values corresponding to third data, where the second data is data stored by the first index before the end time of the ith period, and the third data is a number acquired by the data processing device in the ith period of the first index.

The description of the second data may refer to the foregoing, and will not be repeated herein.

In the embodiment of the present application, for the maximum value of L parameters, that is, the maximum value of L parameters, the data processing apparatus may acquire in various manners, such as reception acquisition or processing acquisition.

In the processing and acquiring mode, when the data processing device can statistically index and store the data with different first parameter values, the performance parameter value of the data processing device can be acquired according to the statistical data, and the value range of the first parameter corresponding to the preset condition is met by the performance parameter value of the data processing device. And extracting any value from the value range, and increasing the preset step length gradually until the performance parameter value of the data processing equipment does not meet the preset condition, and determining the current value of the first parameter as the maximum value of the first parameter. The first parameter may be any one of the L parameters. Such as: assuming that the value of the first parameter extracted from the value range is C _min, and after j preset steps a are added to C _min, the performance parameter value of the data processing apparatus does not satisfy the preset condition, and the first parameter is maximum C _max＝C_min +jxa.

It may be understood that, in a scenario where the data processing apparatus obtains the L parameter maximum values by receiving the L parameter maximum values sent by other apparatuses, the other apparatuses may obtain the L parameter maximum values in the above processing manner, but is not limited thereto.

After the data processing device obtains the L parameter values and the L parameter maximum values corresponding to the second data and the third data, the data processing device may determine L first durations corresponding to the ith period according to the L parameter values and the L parameter maximum values corresponding to the second data and the third data, respectively. Optionally, the period number included in the first time length corresponding to the first parameter value is determined to be d+i.

In specific implementation, d= (C _max-C_x)/C, where C _max is the first parameter maximum value; c _x is a first parameter value corresponding to the second data; and C, the first parameter value corresponding to the third data.

It should be noted that, in the case where (C _max-C_x) cannot be divided by C, D may be equal to the lower rounding or the upper rounding of (C _max-C_x)/C, which may be specifically determined according to the actual situation, and the embodiment of the present application is not limited thereto.

3. The following describes the acquisition of the reference time length in the embodiment of the present application.

In the embodiment of the present application, the data processing device may acquire the reference duration in a manner of receiving acquisition or processing acquisition.

In the processing the acquired scene, optionally, before determining the target duration according to the L first durations corresponding to the ith period and the reference duration acquired in advance, the method further includes:

Determining the reference time length according to the fourth data;

in a specific implementation, the reference duration may be determined according to L parameter values and L parameter maximum values corresponding to the fourth data.

Optionally, for each parameter, a ratio of a maximum value of the parameter to a parameter value corresponding to the fourth data may be calculated to obtain L ratios, and in one implementation, the data processing apparatus may select an up-rounding value or a down-rounding value of a minimum or maximum ratio of the L ratios as N; in another implementation, the data processing apparatus may take an upward rounded value or a downward rounded value of a mean value or a weighted average of the L ratios as N, where N is a number of periods included in the reference duration.

It should be noted that, the various optional implementations described in the embodiments of the present application may be implemented in combination with each other without collision, or may be implemented separately, which is not limited to the embodiments of the present application.

For ease of understanding, examples are illustrated below:

In the following embodiments, the L parameters corresponding to the data include a data amount and a data size; the reference time length includes a period number denoted as D1, a first time length corresponding to the data amount among the L first time lengths denoted as Dc1, and a first time length corresponding to the data size denoted as Dc2. The performance parameter value is the number of slow queries. One cycle was 1 day.

1. Data collection is carried out on the system in the early stage, and the slow query condition of each day is indexed under the condition of testing different data volumes:

Table 2: indexing daily slow query cases for different data volumes

Index data volume	Index data size	Slow number of queries
			C1	G1	N1
C2	G2	N2
			...	...	...
Cn	Gn	Nn

And taking the index quantity and the index size when the average slow query frequency is the minimum (Nmin), and taking the two data as references, and further testing the corresponding index data quantity range and index size range when the index slow query frequency is the minimum.

2. When the slow query times are at least and are Nmin, the data volume of the index is properly increased, and the index size is increased, so that an index data volume maximum Cmax and an index size maximum Gmax under the condition of the least slow query times are obtained.

3. When the elastic search cluster is running, the data amount of one day of the current system is counted, and accordingly, it is derived how many days of data are required presumably when the index data amount is within Cmax and the index space is within Gmax, assuming that this day is D1.

4. The first index is created in the elastic search cluster system and the schedule is initialized to store data for consecutive D days in the index. And calculating how much day the current index can reach the threshold range according to the data volume of the previous day and the existing data volume of the current index in the morning.

5. When the calculation is carried out on any day (Dn days) in the early morning, the current index which also needs Dc days can reach the maximum data amount according to D= (Cmax-Cx)/C.

Wherein, D is the current index and the data volume can reach the maximum value of the current system in a few days, cx is the maximum value of the index data volume when the current index is queried slowly, cx is the existing data volume of the current index, and C is the data volume of the previous day. And obtaining the current index which still needs Dc days to reach the maximum value of the data quantity.

6. And (3) changing Cmax in the formula in the step (5) into the maximum value Gmax of the index size when the current index is the least slowly inquired, changing Cx into the current index size, and changing C into the data size of the previous day, so that the maximum value which can reach the index space in Dg days can be obtained.

7. The result Dc of step 5 and the result Dg of step 6 are added to the result Dn of step 5, respectively, to yield the results Dc1 and Dg1, respectively. If one of Dc1 and Dg1 is greater than D1 of step 3, then the number of days required to judge that the day is predicted to reach maximum meets the criteria. If both values are less than D1, it is determined that the number of days required for the day to be predicted to reach the maximum value does not meet the criterion.

8. Enough system data was collected, combined with step 7, to obtain the following results:

table 3: judgment result corresponding to the first n days of data

9. According to step 8, it can be obtained whether Dc1 is greater than D1, and the number of yes is N1, and the probability is p1=n1/N; whether "Dg1 is greater than D1" results in a number of yes, N2, with a probability of p2=n2/N.

10. According to the Bayesian formula:

Wherein p (w) is a priori probability, representing the probability of each category distribution; p (x|w) is a class conditional probability, which indicates the probability of something happening under the premise of a certain class; and p (w|x) is a posterior probability, which indicates the probability that something happens and that it belongs to a certain class, with which we can classify the samples of step 8. The greater the posterior probability, the greater the likelihood that something belongs to this category, the more reasonable we can put it under this category.

11. Let w1 be the probability that either of Dc1 and Dg1 is greater than D1, i.e., P (w 1) =p1+p2; w2 is the probability that both are smaller than D1, p (w 2) =1-p (w 1); x is the conformity standard. The class conditional probability p (x|w1), i.e., the probability that either Dc1 or Dg1 is larger than D1 and the result accords with the standard, is 0 in the present environment; when the class conditional probabilities p (x|w2), that is, dc1 and Dg1 are smaller than D1, the probability that the result meets the criterion is 1 in the present environment. The probability that p (x) is the least number of slow queries is:

p(x)＝p(x|w₁)p(w₁)+p(x|w₂)p(w₂)

Bringing P (w 1), P (x|w1) and P (x) into the formula of the step 10 to obtain the probability that any one of Dc1 and Dg1 is larger than D1 under the condition of least slow query; taking P (w 2), P (x|w2), P (x) into the formula of step 10 yields the probability that P (w2|x), with minimum slow queries, both Dc1 and Dg1 are less than D1.

12. Finally, comparing P (w1|x) with P (w2|x), and obtaining that P (w2|x) is larger than or equal to P (w1|x), namely when Dc1 and Dg1 are smaller than D1, the probability of minimum slow query times is high, the current index is kept to be initially set, and data of D1 days are stored in total, so that the maximum number of stored data can be reached, the minimum slow query is generated, and the highest query efficiency is maintained.

13. If P (w 1|x) is greater than or equal to P (w 2|x), then the probability that one of Dc1 and Dg1 is greater than D1 is large, then the current index initial setting is modified, and the data for D1 days is no longer saved. Continuing to judge by the magnitudes of Dc1 and Dg 1.

14. If Dc1> =dg 1, the index stores Dg day data again, the index will reach the critical point with the minimum slow query times and the maximum index data volume, i.e. the current index stores Dg day data again, and then the next index is newly built and the data is stored in the next index. The current index is kept at the highest query efficiency.

15. Otherwise, if Dc1 is smaller than Dg1, the index stores the data in Dc again, the index reaches the critical point with the minimum slow query times and the maximum index data quantity, namely, the current index stores the data in Dc again, the next index is newly built, and the data is stored in the next index. The current index is kept to the highest query efficiency and the largest data volume.

16. So, the task of calculating the results of steps 5 and 6 is started every early morning, if either Dc1 and Dg1 is larger than D1, the standard days D1 set by the system are modified, and the smaller days of Dc1 and Dg1 are taken as the reference days, and Dc1 is assumed. The current index stores data in Dc again, namely the standard Dc can be reached for 1 day, then the next index is newly established, and the data in Dc1 day is stored again, so that the purposes of dynamically modifying the data quantity and the space size of the elastic search index are achieved, the related data of the acquisition system are continuously calculated and acquired for calculation, flexible dynamic adjustment can be realized, and the query efficiency of the elastic search index is improved.

The above example is described in connection with fig. 2:

As shown in fig. 2, the data processing method may include the steps of:

And step 201, collecting system data to obtain Cmax and Gmax corresponding to Nmin.

Step 202, calculating to obtain indexes reaching Cmax and Gmax according to the average data amount of the current day of the system, wherein D1 day is needed.

And 203, calculating in the morning on Dn days to obtain the maximum data volume which can be reached by the current index on Dc days and the maximum index space which can be reached by the current index on Dg days.

Step 204, dc1 is calculated from Dc1= (Dn-1) +Dc, and Dc1 is calculated from Dg1= (Dn-1) +Dg.

Step 205, calculating the probability P (w1|x) that either Dc1 or Dg1 is greater than D1, and the probability P (w2|x) that both Dc1 and Dg1 are less than D1.

Step 206, determine whether P (w2|x) is greater than P (w1|x).

If P (w2|x) is greater than or equal to P (w1|x), then the probability that Dc1 and Dg1 are both less than D1 is high, step 207 is performed, so that the maximum number of stores can be reached, resulting in the least slow queries, and thus maintaining the highest query efficiency. If P (w1|x) is greater than P (w2|x), the probability that one of Dc1 and Dg1 is greater than D1 is high, step 208 is performed.

Step 207, the current index is kept in the initial setting, and data of D1 day is saved in total.

Step 208, modify the current index initial setting, and not save the data of D1 day.

Step 209, determining whether Dc1 is greater than or equal to Dg1.

If yes, go to step 210; if not, go to step 211.

Step 210, the index stores the data of Dg days again, and the next index is newly built, and the data is stored in the next index.

And (3) the index stores the data of Dg days again, the index can reach the critical point with the minimum slow query times and the maximum index data quantity, namely, the current index stores the data of Dg days again, the next index is newly built, and the data is stored in the next index. The current index is kept at the highest query efficiency.

Step 211, the index stores the data in Dc again, and the next index is newly built, and the data is stored in the next index.

And (3) the index stores data in Dc again, the index can reach a critical point with the minimum slow query times and the maximum index data quantity, namely, the current index stores data in Dc again, a next index is newly built, and the data is stored in the next index. The current index is kept to the highest query efficiency and the largest data volume.

The embodiment of the application obtains the most reliable influence factor of the elastic search according to the Bayesian formula, judges the proper data volume and size of the elastic search index according to the most reliable influence factor, can be dynamically and flexibly adjusted, and improves the user experience of service inquiry to the maximum extent. By adopting the embodiment of the application, the data quantity and the size of the elastic search index can be dynamically and flexibly adjusted, the normal operation of the system can not be influenced, the excessive time of operators can not be occupied, and the labor is saved.

The embodiment of the application also provides data processing equipment capable of executing the method embodiment. Since the principle of the data processing apparatus for solving the problem is similar to that of the data processing method in the embodiment of the present application, the implementation of the data processing apparatus may refer to the implementation of the method, and the repetition is not repeated.

Referring to fig. 3, fig. 3 is one of the block diagrams of the data processing apparatus provided in the embodiment of the present application. As shown in fig. 3, the data processing apparatus 300 may include:

A first obtaining module 301, configured to obtain L first durations corresponding to an ith period of the first index, where i and L are positive integers;

a first determining module 302, configured to determine a target duration according to the L first durations and a reference duration acquired in advance;

And the control module 303 is used for controlling the first index to stop storing data after controlling the first index to store the data acquired by the data processing device in the target duration.

Optionally, the first determining module 302 includes:

A first obtaining unit, configured to obtain a first posterior probability value and a second posterior probability value according to second data, where the second data is data that has been stored by the first index before an end time of the i-th period; the first posterior probability values are probability values of which the first time length is longer than the reference time length when the performance parameter value of the data processing equipment meets a preset condition, and the second posterior probability values are probability values of which the performance parameter value of the data processing equipment meets the preset condition and are smaller than the reference time length;

The comparison unit is used for comparing the first posterior probability value and the second posterior probability value to obtain a comparison result;

And the first determining unit is used for determining a target duration according to at least one of the L first durations and the reference duration based on the comparison result.

Optionally, the first determining unit is specifically configured to at least one of:

The first obtaining unit is specifically configured to:

obtaining a first posterior probability value and a second posterior probability value according to L first judgment results corresponding to the H periods respectively and the judgment results corresponding to the H periods respectively;

Optionally, the first determining module 302 includes:

the detection unit is used for detecting whether the first time length is longer than the reference time length in the L first time lengths or not, and obtaining a detection result;

and the second determining unit is used for determining a target duration according to at least one of the L first durations and the reference duration based on the detection result.

Optionally, the second determining unit is configured to at least one of:

Optionally, the first obtaining module 301 includes:

The second obtaining unit is configured to obtain L parameter values corresponding to second data and L parameter values corresponding to third data, where the second data is data stored by the first index before the end time of the i-th period, and the third data is data acquired by the data processing device in the i-th period of the first index;

A third determining unit, configured to determine that the first period corresponding to the first parameter value includes a number of cycles d+i;

Optionally, the data processing system further comprises:

The second acquisition module is used for acquiring fourth data, wherein the fourth data is data acquired by the data processing equipment in a period when the performance parameter value of the data processing equipment meets a preset condition;

and the second determining module is used for determining the reference duration according to the fourth data.

The data processing device 300 provided in the embodiment of the present application may execute the above method embodiment, and its implementation principle and technical effects are similar, and this embodiment will not be described herein.

Referring to fig. 4, fig. 4 is a second block diagram of a data processing apparatus according to an embodiment of the present application. As shown in fig. 4, the data processing apparatus 400 may include:

a processor 401 for reading a program or instructions in a memory 402, performing the following process:

Optionally, the processor 401 is further configured to read the program or the instructions, and perform the following steps:

Optionally, the processor 401 is further configured to read the program or the instructions, and perform at least one of the following:

Optionally, the second data corresponds to H periods, and H is a positive integer; the processor 401 is further configured to read the program or instructions, and execute the following steps:

and determining the reference time length according to the fourth data.

A transceiver 403 for receiving and transmitting data under control of the process 401.

Where in FIG. 4, a bus architecture may comprise any number of interconnected buses and bridges, with one or more processors, represented in particular by processor 401, and various circuits of memory, represented by memory 402, linked together. The bus architecture may also link together various other circuits such as peripheral devices, voltage regulators, power management circuits, etc., which are well known in the art and, therefore, will not be described further herein. The bus interface provides an interface. The transceiver 403 may be a number of elements, i.e. comprising a transmitter and a transceiver, providing a unit for communicating with various other apparatus over a transmission medium. The processor 401 is responsible for managing the bus architecture and general processing, and the memory 402 may store data used by the processor 401 in performing operations.

The processor 401 is responsible for managing the bus architecture and general processing, and the memory 402 may store data used by the processor 401 in performing operations.

The data processing device provided in the embodiment of the present application may execute the above method embodiment, and its implementation principle and technical effects are similar, and this embodiment will not be described herein.

Furthermore, a readable storage medium of an embodiment of the present application stores a program or instructions executable by a processor to implement the steps of:

Alternatively, the program or instructions may be executable by a processor to perform the steps of:

Optionally, the program or instructions may be executable by a processor to perform at least one of:

Optionally, the second data corresponds to H periods, and H is a positive integer; the program or instructions may be executable by a processor to perform the steps of:

and determining the reference time length according to the fourth data.

In the several embodiments provided in the present application, it should be understood that the disclosed methods and apparatus may be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative, e.g., the division of the units is merely a logical function division, and there may be additional divisions when actually implemented, e.g., multiple units or components may be combined or integrated into another system, or some features may be omitted or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or units, which may be in electrical, mechanical or other form.

In addition, each functional unit in the embodiments of the present application may be integrated in one processing unit, or each unit may be physically included separately, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in hardware plus software functional units.

The integrated unit implemented in the form of a software functional unit described above may be stored in a readable storage medium. The software functional unit is stored in a storage medium and includes instructions for causing a device (which may be a person, a server, or a network device, etc.) to perform part of the steps of the data processing method according to the embodiments of the present application. And the aforementioned storage medium includes: a usb disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a magnetic disk, or an optical disk, or other various media capable of storing programs or instruction codes.

While the foregoing is directed to the preferred embodiments of the present application, it will be appreciated by those skilled in the art that various modifications and adaptations can be made without departing from the principles of the present application, and such modifications and adaptations are intended to be comprehended within the scope of the present application.

Claims

1. A method of data processing, the method comprising:

Determining a target duration according to the L first durations and the reference duration acquired in advance; the first duration is a period number required by the data processing device to predict that a target parameter value of the data stored in the first index reaches a target parameter maximum value in the ith period; the reference duration is a number of periods required by the data processing device when predicting that each parameter value of the data stored in the first index is smaller than the maximum value of the parameter based on the parameter value of the data stored in one period when the performance parameter value of the data processing device meets a preset condition; the target duration is the cycle number corresponding to the data which can be stored by the first index from the (i+1) th cycle;

after controlling the first index to store the data acquired by the data processing equipment in the target duration, controlling the first index to stop storing the data;

the determining the target duration according to the L first durations and the reference duration acquired in advance includes:

Determining a target duration according to at least one of the L first durations and the reference duration based on the comparison result;

or determining a target duration according to the L first durations and the reference durations acquired in advance, including:

determining a target duration according to at least one of the L first durations and the reference duration based on the detection result;

wherein the determining, based on the detection result, a target duration according to at least one of the L first durations and the reference duration includes:

2. The method of claim 1, wherein the determining a target duration from at least one of the L first durations and the reference duration based on the comparison result comprises at least one of:

3. The method of claim 1, wherein the second data corresponds to H cycles, H being a positive integer;

4. The method of claim 1, wherein the obtaining L first durations corresponding to the ith period of the first index includes at least one of:

5. The method of claim 1, wherein before determining the target time period based on the L first time periods and the reference time period acquired in advance, the method further comprises:

and determining the reference time length according to the fourth data.

6. A data processing apparatus comprising: a transceiver, a memory, a processor, and a program or instructions stored on the memory and executable on the processor; -c h a r a c t e r i z e d in that the processor is arranged to read a program or instructions in a memory for implementing the steps in the method according to any one of claims 1 to 5.

7. A readable storage medium storing a program or instructions which, when executed by a processor, implement the steps in the method of any one of claims 1 to 5.