CN113360527A

CN113360527A - Buffer area size processing method and device and computer equipment

Info

Publication number: CN113360527A
Application number: CN202010149585.9A
Authority: CN
Inventors: 苏桐
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2020-03-06
Filing date: 2020-03-06
Publication date: 2021-09-07

Abstract

The application relates to a method and a device for processing the size of a buffer area and computer equipment. The method comprises the following steps: detecting attribute values of read-write data when reading or writing data; comparing the read-write data attribute value with a preset read-write data attribute threshold value, and determining the data volume state when reading or writing data; acquiring a first size value and a size adjustment value of a buffer area; determining a second size value of the buffer according to the data volume state, the first size value and the resizing value; adjusting the size value of the buffer to the second size value. By adopting the method, the system overhead can be reduced, and the data delay can be reduced.

Description

Buffer area size processing method and device and computer equipment

Technical Field

The present application relates to the field of computer technologies, and in particular, to a method and an apparatus for processing a buffer size, a computer device, and a computer-readable storage medium.

Background

With the development of internet technology, more and more internet products and systems have requirements on the real-time performance of data, namely: operators and research and development staff of the product need to know the feedback, behavior data and system state of the user as soon as possible, and then adjust the strategy in time, so as to provide service for the user better. Many conventional big data system facilities can freely cope with huge amount of internet user data, but they all have a defect: the delay of data processing is very high, and after a lot of data are generated at the user side, dozens of minutes or even one day is needed to be fed back to product personnel, so that the capability of providing high-quality service for the user is greatly reduced, and a buffer area appears. Therefore, the conventional buffer still has a problem of excessive data delay.

Disclosure of Invention

In view of the above, it is necessary to provide a buffer size processing method, apparatus, computer device and computer readable storage medium capable of improving accuracy of a buffer size.

A buffer size processing method, the method comprising:

detecting attribute values of read-write data when reading or writing data;

comparing the read-write data attribute value with a preset read-write data attribute threshold value, and determining the data volume state when reading or writing data;

acquiring a first size value and a size adjustment value of a buffer area;

determining a second size value of the buffer according to the data volume state, the first size value and the resizing value;

adjusting the size value of the buffer to the second size value.

A buffer size processing apparatus, the apparatus comprising:

the detection module is used for detecting the attribute value of read-write data when the data is read or written;

the state determining module is used for comparing the read-write data attribute value with a preset read-write data attribute threshold value and determining the data volume state when the data is read or written;

the device comprises an acquisition module, a storage module and a processing module, wherein the acquisition module is used for acquiring a first size value and a size adjustment value of a buffer area;

the size determining module is used for determining a second size value of the buffer area according to the data volume state, the first size value and the size adjusting value;

and the adjusting module is used for adjusting the size value of the buffer area to the second size value.

A computer device comprising a memory and a processor, the memory storing a computer program, the processor implementing the following steps when executing the computer program:

detecting attribute values of read-write data when reading or writing data;

acquiring a first size value and a size adjustment value of a buffer area;

adjusting the size value of the buffer to the second size value.

A computer-readable storage medium, on which a computer program is stored which, when executed by a processor, carries out the steps of:

detecting attribute values of read-write data when reading or writing data;

acquiring a first size value and a size adjustment value of a buffer area;

adjusting the size value of the buffer to the second size value.

According to the buffer size processing method, the device, the computer equipment and the storage medium, the read-write data attribute value during reading or writing data is detected, the read-write data attribute value is compared with the preset read-write data attribute threshold value to determine the data volume state, the size of the buffer is adjustable according to the data volume state, the first size value and the size adjusting value, and the size correctness of the buffer is improved, so that the size of the buffer does not need to be evaluated before the buffer is created, the test time is reduced, the phenomenon that the consumed performance cost is too much due to too frequent exchange of the buffer is avoided, and meanwhile, the data delay of writing the buffer into a downstream system is avoided.

Drawings

FIG. 1 is a diagram of an exemplary implementation of a buffer size handling method;

FIG. 2 is a flow diagram of a double buffer in one embodiment;

FIG. 3 is a flow diagram illustrating buffer size processing in one embodiment;

FIG. 4 is a flow diagram illustrating the determination of a second size value for a buffer in one embodiment;

FIG. 5 is a flow diagram illustrating a process for determining a second size value of a buffer in an alternative embodiment;

FIG. 6 is a block diagram of a buffer size processing apparatus in one embodiment;

FIG. 7 is a diagram illustrating an internal structure of a computer device according to an embodiment.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.

The buffer size processing method provided by the application can be applied to the application environment shown in fig. 1. Wherein the first computer device 110 is connected to the second computer device 120 via a network and the second computer device 120 is connected to the third computer device 130 via a network. First computer device 110, second computer device 120, and third computer device 130 may each be a terminal or a server. The terminal can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers and portable wearable devices, and the server can be implemented by an independent server or a server cluster formed by a plurality of servers. Then the first computer device 110 may be the data producer, the buffer included in the second computer device 120, and the third computer device 130 may be the data consumer. Alternatively, the buffer size processing method provided by the present application can be applied to an application environment containing only two computer devices, for example, one of the computer devices is a data producer and includes a buffer, and the other computer device is a data consumer; alternatively, one of the computer devices is a data producer and the other computer device is a data consumer and includes a buffer. Alternatively, the buffer size processing method provided by the present application can be applied to an application environment including only one computer device, for example, a data producer and a data consumer are both threads of the computer device. The buffer size processing method provided by the application can be completed through a buffer size processing program, wherein the buffer size processing program is applied to computer equipment comprising a buffer.

In one embodiment, in order to deal with the situation that a lot of data needs tens of minutes or even one day to be fed back to product personnel after being generated at the user side, and the capability of providing high-quality service for users is greatly reduced, a lot of new systems and facilities aiming at real-time data streams are generated, such as Apache Flink, Apache pulse and clickwouse. Users of these systems often need to develop and deploy data middleware between the systems for the following purposes:

1. the data transmission efficiency between the systems is improved: the data middleware provides certain data distribution capacity, and avoids the situation that a plurality of downstream systems (data consumers) repeatedly read data of the upstream system, thereby wasting network bandwidth.

2. Speed difference between the balance systems: different system designs tend to bring different performance performances, for example, Apache Pulsar (here, producer) can output millions of small data in a short time, while clickwouse (here, consumer) can only accept tens of thousands of write requests in the same time, but the data volume of a single write can be larger. The data middleware can then act as a buffer between the two, namely: a part of the data read from the Apache pulse is accumulated, and then a certain amount of the accumulated data is uniformly written into the ClickHouse.

3. Provide additional data screening capabilities: the message middleware can provide functions of data verification, screening and the like, move out unqualified data, avoid the transmission of the unqualified data among systems, waste network bandwidth and simplify the processing logic of a downstream system.

The function of "balancing the speed difference between systems" can be realized by double buffer queue (double buffer queue). Double-buffered queues are a common high-performance data structure that is commonly used in computer games, high-performance data streaming systems, and the like. The method is characterized in that the method is provided with two buffer areas (buffers), a data producer writes data into one buffer area, a data consumer reads data from the other buffer area, when the producer finishes writing and the consumer finishes reading, the producer and the consumer exchange the originally occupied buffer areas, and then data and consumption data are continuously produced. Double buffer data queue implementation:

1. a thread merges the read-in large amounts of small data from the upstream system before filling into a buffer. (this thread is called the data producer of this buffer)

2. At the same time, the other thread reads and writes the merged data from the other buffer to the downstream system. (this thread is called the data consumer for this buffer)

3. When the first buffer is filled and the second buffer is read empty, the two buffers are swapped, the data producer writes the next upstream data into the buffer that was previously read empty, and the data consumer reads the data from the previously filled buffer and writes it to the downstream system.

The traditional double-buffer design can enable the system to achieve extremely high throughput when the data volume is large, namely, the data volume transferred in unit time can be large. The main reason is that the data producer and the data consumer can work simultaneously, unlike a single buffer data structure: the data consumer needs to wait for the data producer to fill the buffer area with the data before consuming the data, and during consumption, the data producer needs to wait for the data consumer to consume the data in the buffer area before producing the data.

FIG. 2 is a flow diagram illustrating a double buffer in one embodiment. The work flow of the double buffer areas is as follows:

1. the data producer writes small data read from the upstream system into the first buffer:

a) the data producer aggregates a large amount of small data into buffer 0 (graph a).

b) Until the buffer is full of 10 ten thousand pieces of data, and then the buffer is yielded (diagram A).

c) The data producer begins to attempt to write to buffer 1 (fig. B).

d) If there is data in buffer 1 or is being occupied by a data consumer, wait until buffer 1 becomes writable.

2. Meanwhile, the data consumer writes the data merged from the small data in the buffer 1 into the downstream system:

a) the data in buffer 1 is written into clickwouse (diagram a).

b) After consumption is complete, buffer 1 is cleared and freed (fig. a).

c) An attempt to consume read buffer 0 is initiated (figure B).

d) If buffer 0 is occupied by the data producer, wait.

The main drawback of the conventional solution is that the size of the buffer is fixed, which may cause the following problems in use:

1. a program or person needs to evaluate and size the buffer in advance before creating the buffer, and this value often requires trial and error validation for new projects or new data flows.

2. If this value is set too small, the buffer exchange becomes very frequent. On one hand, the excessively frequent switching brings unnecessary performance overhead, and on the other hand, the write frequency of the downstream system is increased, so that the double buffering loses the existing significance.

3. If the value is set too large, on one hand, the overhead occupied by the memory is increased due to an excessively large buffer area; on the other hand, the producer needs more time to accumulate data, which in turn results in higher latency for writing data to the downstream system.

4. The software system of the internet product often has data peaks and data valleys in one day, for example, the data volume will be reduced when the number of active users is small in the morning, and the data volume will be increased when the number of active users is high after dinner. Then setting the buffer size too small, writing data out too frequently to the downstream system during data peaks can cause excessive downstream system stress, while data dips can cause increased data transfer delays.

The invention adds an automatic buffer area size adjusting mechanism on the basis of double buffer queues, further overcomes the defect of fixed buffer area size, solves the problem, and not only ensures high throughput of data peak time, but also ensures low delay of data valley time.

Therefore, as shown in fig. 3, a buffer size processing method is provided, which is exemplified by a computer device applied to a single buffer or a double buffer, and includes the following steps:

step 302, detecting the attribute value of read-write data during reading or writing.

The reading or writing refers to when a data producer reads data from the second database, when a data producer writes data into the buffer, when a data consumer reads data from the buffer, or when a data consumer writes data into the first database. And in the operation process, the attribute values of the read-write data for reading the data from the second database by the data producer and writing the data into the buffer area are the same, and the attribute values of the read-write data for reading the data from the buffer area by the data consumer and writing the data into the first data are the same. The read-write data attribute value refers to a data stream attribute value when reading or writing. For example, the read-write data attribute value may be a frequency value, a time consumption value, or a speed value, etc., but is not limited thereto. The read-write data attribute value can reflect the data volume to a certain extent.

Specifically, the buffer size handler detects read-write data attribute values when the data producer reads data from the second database, when the data producer writes data to the buffer, when the data consumer reads data from the buffer, or when the data consumer writes data to the first database.

And 304, comparing the read-write data attribute value with a preset read-write data attribute threshold value, and determining the data volume state during reading or writing data.

The read-write data attribute threshold is preset in the computer equipment comprising the buffer area. The read-write data attribute threshold comprises a read-write data attribute upper limit value and a read-write data attribute lower limit value. The data volume status is used to characterize how much data is written to the buffer. The data volume state may include a first data volume state, a second data volume state, and a third data volume state. The data volume represented by the first data volume state is larger than the data volume represented by the third data volume state, and the data volume represented by the third data volume state is larger than the data volume represented by the second data volume state.

Specifically, the buffer size processing program compares the read-write data attribute value with a preset read-write data attribute threshold value, and determines the data volume state during reading or writing. For example, if the read/write data attribute value is a time consumption value, the time consumption value is compared with a preset time consumption threshold value, and the data volume state during reading or writing is determined.

In this embodiment, when the data size state is detected to be the third data size state, that is, the data size is within a normal range, the buffer size processing program does not process the buffer size.

Step 306, obtain a first size value and a resizing value of the buffer.

Specifically, the first size value refers to a size value of the current buffer. The first size value may be a variable and the resizing value a fixed value. The resizing value refers to a preset value for resizing the buffer each time. The resizing values for increasing the buffer size and decreasing the buffer size may not be the same. The buffer size handler obtains a first size value and a resizing value for the buffer.

And 308, determining a second size value of the buffer according to the data volume state, the first size value and the size adjustment value.

Specifically, the buffer size handler calculates a second size value of the buffer based on the data size status, the first size value, and the resizing value. Wherein the data volume status corresponds to a size change of the buffer. That is, when the data size state is the first data size state, the size of the buffer is increased to obtain the second size value. And when the data volume state is the second data volume state, reducing the size of the buffer area to obtain a second size value. The first data volume represents a data volume that is greater than the data volume represented by the second data volume state.

In this embodiment, when the data size status is the first data size status, the buffer size processing program may use the sum of the first size value and the resizing value as the second size value of the buffer. Alternatively, the buffer size processing procedure may use the difference between the first size value and the resizing value as the second size value of the buffer. Alternatively, the buffer size processing procedure may use the sum of the resizing value and the first size value, and the product of the resizing value and the first size value as the second size value of the buffer, and the like, but is not limited thereto.

In the embodiment, a first size and a size adjustment value of a buffer area are obtained every preset time length; the second size of the buffer is determined based on the data volume status, the first size value, and the resizing value, which can strip the algorithmic logic from the producer and consumer to facilitate system maintenance and debugging.

In this embodiment, when applied to a computer device including a double buffer, the second size value of the double buffer is determined according to the data volume status, the first size value, and the resizing value.

In step 310, the size of the buffer is adjusted to a second size value.

According to the buffer area size processing method, the read-write data attribute value during reading or writing data is detected, the read-write data attribute value is compared with the preset read-write data attribute threshold value to determine the data volume state, the size of the buffer area is adjustable according to the data volume state, the first size value and the size adjustment value, the size accuracy of the buffer area is improved, the size of the buffer area does not need to be evaluated before the buffer area is created, the test time is reduced, too much consumed performance expense caused by too frequent exchange of the buffer area is avoided, meanwhile, data delay caused by writing of the buffer area into a downstream system is avoided, product personnel or users can be helped to obtain the latest information, and decision making or user experience improvement is made.

In one embodiment, the present application further provides an application scenario, such as data analysis, that applies the above buffer size processing method. Specifically, the application of the buffer size processing method in the application scenario is as follows: the data producer is a user terminal, and the data consumer is a first database. A first database, such as Clickhouse, MySQL, drive, elastic search, etc., is used for persistent storage for use by the administrator terminal. The method is applied to a server comprising a buffer area, and comprises the following steps: detecting a read-write data attribute value when a user terminal triggers product information; comparing the read-write data attribute value with a preset read-write data attribute threshold value, and determining the data volume state when the data is read or written; acquiring a first size value and a size adjustment value of a buffer area; determining a second size value of the buffer area according to the data volume state, the first size value and the size adjustment value; the size value of the buffer is adjusted to a second size value. Data associated with the trigger operation may be written to the first database via the buffer. Product data may be pushed to the administrator terminal via the buffer. The user terminal triggers product information, for example, the user a sees 10 pieces of product information, which are a, b, c, d, etc., and only two pieces of product information, a and c, are triggered through the user terminal.

In one embodiment, the present application further provides an application scenario, such as information search of a user, which applies the above buffer size processing method. The method is applied to computer equipment comprising a buffer area, and comprises the following steps: detecting a read-write data attribute value corresponding to an information search instruction sent by a user terminal to a buffer area; comparing the read-write data attribute value with a preset read-write data attribute threshold value, and determining the data volume state when the data is read or written; acquiring a first size value and a size adjustment value of a buffer area; determining a second size value of the buffer area according to the data volume state, the first size value and the size adjustment value; adjusting the size value of the buffer area to a second size value; and pushing the search information corresponding to the information search instruction to the administrator terminal through the buffer area.

In one embodiment, detecting read and write data attribute values while reading or writing includes: a frequency value is detected when a consumer of the data reads data from the buffer or writes data to the first database.

Comparing the read-write data attribute value with a preset read-write data attribute threshold value to determine the data volume state during reading or writing data, comprising: the read-write data attribute threshold comprises a frequency upper limit value and a frequency lower limit value;

when the frequency value is higher than the upper limit value of the frequency, determining that the data volume state is a first data volume state;

and when the frequency value is lower than the lower frequency limit value, determining that the data volume state is a second data volume state, wherein the data volume represented by the first data volume state is larger than the data volume represented by the second data volume state.

Where frequency may be represented by the number of times divided by time. The frequency value is used to indicate the number of times data is read and written within a certain period of time. The higher the frequency is, the higher the number of times of reading and writing data is; the lower the frequency, the lower the number of times data is read and written. The first data volume state may also be referred to as a high data volume state and the second data volume state may be referred to as a low data volume state. The code logic in this embodiment may be implemented in a data consumer and may be configured to determine the upper frequency limit value and the lower frequency limit value.

In particular, the buffer size handler detects the frequency value of the frequency value when the data consumer reads data from the buffer or writes data to the second database. When the frequency value is higher than the upper limit value of the frequency, the data consumer frequently reads data from the buffer or frequently writes data into the first database, namely the size of the buffer is too small and the system load is large, so that the data volume produced by the data producer is too large and the data volume is in a high data volume state.

If the frequency value is lower than the lower frequency limit, it means that the data consumer is less reading data from the buffer or writing data to the first database, and the size of the buffer is larger, i.e. the data producer is producing too small amount of data, resulting in large data delay and low data amount status.

In this embodiment, since the frequency value is equal to 1/time-consuming value, the frequency value may also be represented by the time-consuming value. Then, when the elapsed time value is lower than the upper frequency limit value, the data volume status may be determined to be the first data volume status. For example, if the upper limit value is written every 2 seconds and this time takes 1.5 seconds, the data amount state is a high data amount state.

According to the method for processing the size of the buffer area, the frequency value when a data consumer reads data from the buffer area or writes data into the first database is detected, when the frequency value is higher than the upper limit value of the frequency, the data volume state is determined to be the first data volume state, when the frequency value is lower than the lower limit value of the frequency, the data volume state is determined to be the second data volume state, wherein the data volume represented by the first data volume state is larger than the data volume represented by the second data volume state, the size of the buffer area can be adjusted according to the data volume state, the correctness of the size of the buffer area is improved, the data consumer can be prevented from writing data into the first database at an overhigh frequency, the power consumption of a system is reduced, and the data delay is reduced.

In one embodiment, detecting read and write data attribute values while reading or writing includes: a time consuming value is detected when the data producer reads data from the second database or writes data to the buffer.

Comparing the read-write data attribute value with a preset read-write data attribute threshold value to determine the data volume state during reading or writing data, comprising: the read-write data attribute threshold comprises a time consumption upper limit value and a time consumption lower limit value;

when the time consumption value is smaller than the time consumption lower limit value, determining that the data volume state is a first data volume state;

and when the consumed time value is larger than the consumed time upper limit value, determining that the data volume state is a second data volume state, wherein the data volume represented by the first data volume state is larger than the data volume represented by the second data volume state.

The time consumption value is the time value consumed by writing the buffer full in the double-buffer queue structure. For example, the elapsed time value may be 2 seconds, 1 second, or the like, but is not limited thereto. The code logic in this embodiment may be implemented in a data producer, and may be configured to determine the upper time-consuming value and the lower time-consuming value.

Specifically, the first data amount state is a high data amount state, and the second data amount state is a low data amount state. When the elapsed time value is less than the elapsed time lower limit, it means that the data producer can easily fill the buffer area with the data, i.e. the size of the buffer area is too small, and therefore the data volume produced by the producer is too large, which is a high data volume state.

Then, when the elapsed time value is higher than the elapsed time upper limit value, it indicates that the data producer produces data that is difficult to fill the buffer, i.e., the size of the buffer is too large, and therefore the data volume produced by the producer is too small, which is a low data volume state.

According to the buffer area size processing method, the time consumption value when a data producer reads data from the second database or writes data into the buffer area is detected, and when the time consumption value is smaller than the time consumption lower limit value, the data volume state is determined to be the first data volume state; when the consumed time value is larger than the consumed time upper limit value, the data volume state is determined to be the second data volume state, wherein the data volume represented by the first data volume state is larger than the data volume represented by the second data volume state, the data volume state can be determined according to the consumed time value, so that the size of the buffer area is adjusted, the correctness of the size of the buffer area is improved, a data consumer can be prevented from writing data into the first database at an excessively high frequency, the power consumption of a system is reduced, and the data delay is reduced.

In one embodiment, comparing the read-write data attribute value with a preset read-write data attribute threshold value to determine the data volume status during reading or writing data includes:

the read-write data attribute threshold comprises a speed upper limit value and a speed lower limit value;

when the read-write speed value is greater than the speed upper limit value, determining that the data volume state is a first data volume state;

and when the read-write speed value is smaller than the speed lower limit value, determining that the data volume state is a second data volume state, wherein the data volume represented by the first data volume state is larger than the data volume represented by the second data volume state.

The read-write speed value refers to the amount of data read or written in a unit time. The upper speed limit may be, for example, 500MB/S per second (Mbyte per second megabyte per second), and the lower speed limit may be, for example, 1MB/S, etc., without being limited thereto.

Specifically, when the read-write speed value is greater than the speed upper limit value, it can be said that the data amount is large, and therefore, the first data amount state can be determined. When the read-write speed value is smaller than the speed lower limit value, the data volume can be indicated to be small, and therefore the state of the second data volume can be determined.

According to the buffer area size processing method, when the read-write speed value is larger than the upper speed limit value, the data volume state is determined to be the first data volume state, when the read-write speed value is smaller than the lower speed limit value, the data volume state is determined to be the second data volume state, the size of the buffer area can be adjusted according to the data volume state, the size correctness of the buffer area is improved, data consumers can be prevented from writing data into the first database at an overhigh frequency, the power consumption of a system is reduced, and the data delay is reduced.

In one embodiment, as shown in FIG. 4, a flowchart of determining a second size value of the buffer in one embodiment is shown. Determining a second size value of the buffer according to the data volume status, the first size and the resizing value, comprising:

step 402, obtaining a size upper limit value.

The upper limit of the size is the maximum size of the buffer.

In step 404, when the data size status is the first data size status, the first size increment is obtained according to the product of the difference between the size upper limit value and the first size adjustment value.

Wherein the size adjustment value is a first size adjustment value.

Specifically, when the data size state is the first data size state, the buffer size handler needs to increase the size of the buffer. That is, when the data amount is in the first data amount state, the first size increase amount is (size upper limit value — first size value) × the first resize value.

At step 406, a first maximum of the first size increase and the second size increase is determined.

Wherein the second size value is increased by a preset size increase. The second size value increase amount may be configured as desired. Wherein the second size value increase may be a second size value minimum increase. For example, the second size value minimum increase is 1. Where 1 indicates that the increased buffer can accommodate a data amount of 1.

Specifically, max (first size increase amount, second size increase amount) which is the first maximum value of the first size increase amount and the second size increase amount is determined.

Step 408, determine the sum of the first maximum value and the first size value to obtain a first candidate size value.

Specifically, the sum of the first maximum value and the first size value is determined, resulting in a first candidate size value, i.e., the first candidate size value is the first maximum value + the first size value.

In step 410, the maximum value of the first candidate size value and the size upper limit value is used as the second size value of the buffer.

Specifically, the second size value is max (first candidate size value, size upper limit value). In summary, the second size value is min (upper size limit, max ((upper size limit — first size value) × 0.05,1) + first size value). Wherein 0.05 is the first size adjustment value and 1 is the second size value increment. Wherein, the first adjustment value can be adjusted according to the actual situation. For example, to increase the adjustment speed of the buffer, the first adjustment value may be set to a larger value, i.e. larger than the preset adjustment value threshold value 0.05, such as 0.1, 0.2, etc., without being limited thereto. In order to make the accuracy of the size adjustment higher, the first adjustment value may be set to a smaller value, i.e., smaller than a preset adjustment value threshold value of 0.05, such as 0.01, 0.02, etc., without being limited thereto.

In the method for processing the size of the buffer area, because the memory resource is limited, the size upper limit value is adopted to limit the size of the buffer area; when the data volume state is the first data volume state, the size of the buffer area can be increased, the throughput of the buffer area is improved, when the first size value is smaller, the first size increment is larger, and along with the increase of the first size value, the size increment approaches to the second size value increment, the size of the buffer area can be adjusted more accurately, so that frequent data reading and writing are avoided, and the performance overhead of computer equipment is reduced.

In one embodiment, as shown in FIG. 5, a flowchart of determining a second size value of a buffer in another embodiment is shown. Determining a second size value of the buffer according to the data volume status, the first size and the resizing value, comprising:

step 502, obtain a size lower limit.

The lower limit of the size is the minimum size of the buffer.

Step 504, when the data size status is the second data size status, the first size reduction is obtained according to the product of the difference between the first size value and the size lower limit value and the second size adjustment value.

Wherein the second resizing value is used for resizing in the second data volume state. The first resizing value may be the same as the second resizing value. For example, the first resizing value is 0.05, and the second resizing value is also 0.05.

Specifically, when the data size status is the second data size status, the buffer size handler needs to reduce the buffer size. That is, the first size reduction amount is (first size value — size lower limit value) × the second size adjustment value.

At step 506, a second maximum of the first size reduction and the second size reduction is determined.

Wherein the second size reduction may be a second size minimum reduction. For example, the second size minimum reduction is 1. Where 1 indicates that the reduced buffer can accommodate a data amount of 1.

Specifically, the second maximum value, i.e., the second maximum value, of the first size reduction and the second size reduction is determined to be max (first size reduction, second size reduction).

Step 508, determining a difference between the first size value and the second maximum value to obtain a second candidate size value.

Specifically, the second candidate size value is the first size value — the second maximum value.

Step 510, the maximum value of the second candidate size value and the size lower limit value is used as the second size value of the buffer.

Specifically, the second size value is max (second candidate size value, size lower limit value). In summary, the second size value is max (the size lower limit value, the first size value is max ((the first size value — the size lower limit value) × 0.05, 1)). Where 0.05 is the second resizing value and 1 is the second size reduction. Wherein, the second adjustment value can be adjusted according to the actual situation. For example, to increase the adjustment speed of the buffer, the second adjustment value may be set to a larger value, i.e. larger than the preset adjustment value threshold value 0.05, such as 0.1, 0.2, etc., without being limited thereto. In order to make the accuracy of the size adjustment higher, the second adjustment value may be set to a smaller value, i.e., smaller than the preset adjustment value threshold value 0.05, e.g., 0.01, 0.02, etc., without being limited thereto.

According to the buffer size processing method, when the data volume state is the second data volume state, the size of the buffer can be reduced, and the data delay is reduced; and when the first size value is larger, the first size reduction is larger, and along with the reduction of the first size value, the size increase approaches to the second size reduction, so that the size of the buffer area can be adjusted more accurately, the frequent data reading and writing is avoided, and the system performance overhead is reduced.

In one embodiment, the buffer size processing method further includes: and when the data volume state is the first data volume state, controlling the data consumer to stop reading data from the buffer area or writing data into the first database until the time corresponding to the upper limit value of the frequency is reached, and controlling the data consumer to resume reading data from the buffer area or writing data into the first database.

Specifically, when the data volume state is the first data volume state, that is, the buffer area is too small, the read-write data is relatively frequent, and the buffer area size processing program controls the data consumer to stop reading data from the buffer area or writing data into the first database until the time corresponding to the upper limit of the frequency is reached, and controls the data consumer to resume reading data from the buffer area or writing data into the first database. For example, the time corresponding to the upper limit of the frequency may be, for example, that the upper limit is written every 2 seconds, and is currently written every 1.5 seconds, so that the data consumer suspends the execution of reading data from the buffer or writing data into the first database, and resumes the reading of data from the buffer or writing data into the first database after 0.5 seconds, and the read-write frequency can be limited, so as to accumulate a lot of data and improve the throughput of the system.

In one embodiment, the buffer size processing method further includes: when the data volume state is the first data volume state, controlling a data producer to stop writing data into the buffer area or reading data from the second database until the time corresponding to the time-consuming upper limit value is reached, and controlling the data producer to resume writing data into the buffer area or reading data from the second database; and when the data volume state is a second data volume state, controlling the data producer to stop writing data into the buffer area or reading data from the second database, wherein the data volume represented by the first data volume state is larger than the data volume represented by the second data volume state.

Specifically, when the data volume state is the first data volume state, the buffer size processing program controls the data producer to stop writing data into the buffer or reading data from the second database until the time corresponding to the time-consuming upper limit value is reached, and controls the data producer to resume writing data into the buffer or reading data from the second database. For example, if the time corresponding to the upper time limit is 2 seconds, and is currently 1.5 seconds, then the buffer size handler controls the data producer to stop writing data to the buffer or reading data from the second database. Until after 0.5 seconds, the control data producer resumes writing data to the buffer or reading data from the second database. I.e. equivalent to blocking data and not writing it into the buffer, it is possible to allow the computer device containing the buffer sufficient time to process the data in the buffer, thereby reducing the load on the buffer.

When the data volume status is the second data volume status, the buffer size handler controls the data producer to stop writing data to the buffer or reading data from the second database, and controls the producer to write the data that has been read into the first database through the buffer. That is, in the low data amount state, if the buffer is too large to be full, the data can be accumulated by stopping writing data into the buffer or reading data from the second database, and the resources occupied by the buffer and the resources for processing data by the consumer are reduced.

In one embodiment, a buffer size processing method includes:

step (a1), detecting a time consuming value when the data producer reads data from the second database or writes data to the buffer.

And (a2) determining the data volume state as a first data volume state when the elapsed time value is less than the elapsed time lower limit value.

And (a3) when the elapsed time value is greater than the elapsed time upper limit value, determining that the data volume state is a second data volume state, wherein the data volume represented by the first data volume state is greater than the data volume represented by the second data volume state.

Step (a4), obtain the first size value and the size adjustment value of the buffer.

And a step (a5) of acquiring a size upper limit value.

And (a6) obtaining a first size increment by taking the product of the difference between the size upper limit and the first size value and the first size adjustment value when the data volume state is the first data volume state.

Step (a7) determines a first maximum of the first size increase and the second size increase.

And (a8) determining the sum of the first maximum value and the first size value to obtain a first candidate size value.

And a step (a9) of setting the minimum value of the first candidate size value and the size upper limit value as the second size value of the buffer.

And (a10) acquiring a size lower limit value.

And (a11) obtaining a first size reduction by multiplying the difference between the first size value and the size lower limit value by the second size adjustment value when the data size status is the second data size status.

And (a12) determining a second maximum value of the first size reduction and the second size reduction.

And (a13) determining the difference between the first size value and the second maximum value to obtain a second candidate size value.

And a step (a14) of setting the maximum value of the second candidate size value and the size lower limit value as the second size value of the buffer area.

And (a15) adjusting the size value of the buffer area to a second size value.

And (a16) when the data volume state is the first data volume state, controlling the data producer to stop writing data into the buffer area or reading data from the second database until reaching the time corresponding to the time-consuming upper limit value, and controlling the data producer to resume writing data into the buffer area or reading data from the second database.

And (a17) controlling the data producer to stop writing data into the buffer or reading data from the second database when the data volume state is a second data volume state, wherein the data volume represented by the first data volume state is larger than the data volume represented by the second data volume state.

The size processing method of the buffer area in the embodiment of the application can increase the size of the buffer area when the data volume is large, thereby reducing the exchange frequency of the buffer area, reducing the performance expense of computer equipment and ensuring high throughput when the data volume is large; when the data volume is small, the size of the buffer area can be reduced, so that the memory occupation is reduced, and the data transmission delay is reduced.

It should be understood that, although the steps in the flowcharts of fig. 3 to 5 are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least some of the steps in fig. 3 to 5 may include multiple steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of performing the steps or stages is not necessarily sequential, but may be performed alternately or alternately with other steps or at least some of the other steps or stages.

In one embodiment, as shown in fig. 6, there is provided a buffer size processing apparatus, which may be a part of a computer device using a software module or a hardware module, or a combination of the two, the apparatus specifically includes: a detection module 602, a state determination module 604, an acquisition module 606, a size determination module 608, and an adjustment module 610, wherein:

a detection module 602, configured to detect a read-write data attribute value when reading or writing data;

a state determining module 604, configured to compare the read-write data attribute value with a preset read-write data attribute threshold value, and determine a data volume state when reading or writing data;

an obtaining module 606, configured to obtain a first size value and a size adjustment value of the buffer;

a size determination module 608 for determining a second size value of the buffer according to the data size status, the first size value and the size adjustment value;

and an adjusting module 610, configured to adjust the size value of the buffer to a second size value.

According to the buffer area size processing device, the read-write data attribute value during reading or writing data is detected, the read-write data attribute value is compared with the preset read-write data attribute threshold value, so that the data volume state is determined, the size of the buffer area is adjustable according to the data volume state, the first size value and the size adjustment value, the size correctness of the buffer area is improved, the size of the buffer area does not need to be evaluated before the buffer area is created, the test time is reduced, too much consumed performance expense caused by too frequent exchange of the buffer area is avoided, meanwhile, data delay caused by writing of the buffer area into a downstream system is avoided, product personnel or users can be helped to obtain the latest information, and decision making or user experience improvement is made.

In one embodiment, the detection module 602 is configured to detect a frequency value when a data consumer reads data from a buffer or writes data to a first database. The state determining module 604 is configured to determine that the data volume state is a first data volume state when the frequency value is higher than the upper frequency limit; and when the frequency value is lower than the lower frequency limit value, determining that the data volume state is a second data volume state, wherein the data volume represented by the first data volume state is larger than the data volume represented by the second data volume state.

The buffer area size processing device detects the frequency value when a data consumer reads data from the buffer area or writes data into the first database, determines that the data volume state is the first data volume state when the frequency value is higher than the upper frequency limit value, and determines that the data volume state is the second data volume state when the frequency value is lower than the lower frequency limit value, wherein the data volume represented by the first data volume state is larger than the data volume represented by the second data volume state, so that the size of the buffer area can be adjusted according to the data volume state, the correctness of the size of the buffer area is improved, the data consumer can be prevented from writing data into the first database at an excessively high frequency, the power consumption of a system is reduced, and the data delay is reduced.

In one embodiment, the detection module 602 is used to detect a time consuming value when a data producer reads data from a second database or writes data to a buffer. The state determining module 604 is configured to determine that the data volume state is the first data volume state when the time consumption value is smaller than the time consumption lower limit value; and when the consumed time value is larger than the consumed time upper limit value, determining that the data volume state is a second data volume state, wherein the data volume represented by the first data volume state is larger than the data volume represented by the second data volume state.

The buffer area size processing device detects a time consumption value when a data producer reads data from the second database or writes data into the buffer area, and determines that the data volume state is a first data volume state when the time consumption value is smaller than a time consumption lower limit value; when the consumed time value is larger than the consumed time upper limit value, the data volume state is determined to be the second data volume state, wherein the data volume represented by the first data volume state is larger than the data volume represented by the second data volume state, the data volume state can be determined according to the consumed time value, so that the size of the buffer area is adjusted, the correctness of the size of the buffer area is improved, a data consumer can be prevented from writing data into the first database at an excessively high frequency, the power consumption of a system is reduced, and the data delay is reduced.

In one embodiment, the state determining module 604 is configured to determine that the data size state is a first data size state when the read/write speed value is greater than the speed upper limit value; and when the read-write speed value is smaller than the speed lower limit value, determining that the data volume state is a second data volume state, wherein the data volume represented by the first data volume state is larger than the data volume represented by the second data volume state.

According to the buffer area size processing device, when the read-write speed value is larger than the upper speed limit value, the data volume state is determined to be the first data volume state, when the read-write speed value is smaller than the lower speed limit value, the data volume state is determined to be the second data volume state, the size of the buffer area can be adjusted according to the data volume state, the size correctness of the buffer area is improved, data consumers can be prevented from writing data into the first database at an overhigh frequency, the power consumption of a system is reduced, and the data delay is reduced.

In one embodiment, the size determination module 608 is configured to obtain a size upper limit value; when the data volume state is a first data volume state, obtaining a first size increment according to the product of the difference value of the size upper limit value and the first size adjustment value; determining a first maximum value of the first size increase amount and the second size value increase amount; determining the sum of the first maximum value and the first size value to obtain a first candidate size value; the maximum value of the first candidate size value and the size upper limit value is used as the second size value of the buffer area.

The buffer area size processing device adopts the size upper limit value to limit the size of the buffer area because the memory resource is limited; when the data volume state is the first data volume state, the size of the buffer area can be increased, the throughput of the buffer area is improved, when the first size value is smaller, the first size increment is larger, and along with the increase of the first size value, the size increment approaches to the second size value increment, the size of the buffer area can be adjusted more accurately, so that frequent data reading and writing are avoided, and the performance overhead of computer equipment is reduced.

In one embodiment, the size determination module 608 is configured to obtain a size lower limit value; when the data volume state is a second data volume state, obtaining a first size reduction value according to the product of the difference value between the first size value and the size lower limit value and the second size adjustment value; determining a second maximum of the first size reduction and the second size reduction; determining the difference value between the first size value and the second maximum value to obtain a second candidate size value; and taking the maximum value of the second candidate size value and the size lower limit value as the second size value of the buffer area.

When the data volume state is the second data volume state, the buffer size processing device can reduce the size of the buffer and reduce the delay of data; and when the first size value is larger, the first size reduction is larger, and along with the reduction of the first size value, the size increase approaches to the second size reduction, so that the size of the buffer area can be adjusted more accurately, the frequent data reading and writing is avoided, and the system performance overhead is reduced.

In one embodiment, the buffer size processing apparatus further includes a control module. The control module is used for controlling a data consumer to stop reading data from the buffer area or writing data into the first database when the data volume state is the first data volume state, and controlling the data consumer to recover to read data from the buffer area or write data into the first database when the time corresponding to the upper limit value of the frequency is reached. The buffer size processing device can limit the read-write frequency so as to accumulate a plurality of data and improve the throughput of the system.

In one embodiment, the buffer size processing apparatus further includes a control module. The control module is used for: when the data volume state is the first data volume state, controlling a data producer to stop writing data into the buffer area or reading data from the second database until the time corresponding to the time-consuming upper limit value is reached, and controlling the data producer to resume writing data into the buffer area or reading data from the second database; and when the data volume state is a second data volume state, controlling the data producer to stop writing data into the buffer area or reading data from the second database, wherein the data volume represented by the first data volume state is larger than the data volume represented by the second data volume state. The buffer area size processing device can enable the computer equipment containing the buffer area to have enough time to process the data in the buffer area, thereby reducing the load of the buffer area; in a low data volume state, the buffer area is too large to be full, and then data can be accumulated by stopping writing data into the buffer area or reading data from the second database, so that resources occupied by the buffer area and resources for processing data by consumers are reduced.

For the specific definition of the buffer size processing device, reference may be made to the above definition of the buffer size processing method, which is not described herein again. The respective modules in the buffer size processing apparatus described above may be implemented in whole or in part by software, hardware, and a combination thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.

In one embodiment, a computer device is provided, which may be a server, the internal structure of which may be as shown in fig. 7. The computer device includes a processor, a memory, and a network interface connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The database of the computer device is used for storing data. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a buffer size handling method.

Those skilled in the art will appreciate that the architecture shown in fig. 7 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.

In one embodiment, a computer device is further provided, which includes a memory and a processor, the memory stores a computer program, and the processor implements the steps of the above method embodiments when executing the computer program.

In an embodiment, a computer-readable storage medium is provided, in which a computer program is stored which, when being executed by a processor, carries out the steps of the above-mentioned method embodiments.

It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, database or other medium used in the embodiments provided herein can include at least one of non-volatile and volatile memory. Non-volatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical storage, or the like. Volatile Memory can include Random Access Memory (RAM) or external cache Memory. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM), among others.

The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.

The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.

Claims

1. A buffer size processing method, the method comprising:

detecting attribute values of read-write data when reading or writing data;

acquiring a first size value and a size adjustment value of a buffer area;

adjusting the size value of the buffer to the second size value.

2. The method of claim 1, wherein detecting the read or write data attribute value when reading or writing comprises:

detecting a frequency value when a data consumer reads data from the buffer or writes data to a first database;

the comparing the read-write data attribute value with a preset read-write data attribute threshold value to determine the data volume state during the data reading or writing includes: the read-write data attribute threshold comprises a frequency upper limit value and a frequency lower limit value;

3. The method of claim 1, wherein detecting the read or write data attribute value when reading or writing comprises:

detecting a time consuming value when a data producer reads data from a second database or writes data to the buffer;

the comparing the read-write data attribute value with a preset read-write data attribute threshold value to determine the data volume state during the data reading or writing includes: the read-write data attribute threshold comprises a time consumption upper limit value and a time consumption lower limit value;

4. The method of claim 1, wherein comparing the read-write data attribute value with a preset read-write data attribute threshold value to determine the data volume status during reading or writing data comprises:

5. The method of claim 1, wherein the resizing value is a first resizing value;

the determining a second size value of the buffer according to the data size status, the first size, and the resizing value comprises:

acquiring an upper limit value of the size;

when the data volume state is a first data volume state, obtaining a first size increment according to the product of the difference value between the size upper limit value and the first size adjustment value;

determining a first maximum value of the first size increase amount and the second size value increase amount;

determining the sum of the first maximum value and a first size value to obtain a first candidate size value;

and taking the minimum value in the first candidate size value and the size upper limit value as the second size value of the buffer area.

6. The method of claim 1, wherein the resizing value is a second resizing value;

acquiring a lower limit value of the size;

when the data volume state is a second data volume state, obtaining a first size reduction value according to the product of the difference value between the first size value and the size lower limit value and the second size adjustment value;

determining a second maximum of the first and second size reductions;

determining the difference value between the first size value and the second maximum value to obtain a second candidate size value;

and taking the maximum value of the second candidate size value and the size lower limit value as the second size value of the buffer area.

7. The method of any one of claims 1 to 6, further comprising:

and when the data volume state is a first data volume state, controlling a data consumer to stop reading data from the buffer area or writing data into the first database until reaching the time corresponding to the upper limit value of the frequency, and controlling the data consumer to resume reading data from the buffer area or writing data into the first database.

8. The method of any one of claims 1 to 6, further comprising:

when the data volume state is a first data volume state, controlling a data producer to stop writing data into the buffer area or reading data from a second database until reaching the time corresponding to the time-consuming upper limit value, and controlling the data producer to resume writing data into the buffer area or reading data from the second database;

and when the data volume state is a second data volume state, controlling the data producer to stop writing data into the buffer area or reading data from a second database, wherein the data volume represented by the first data volume state is larger than the data volume represented by the second data volume state.

9. A buffer size processing apparatus, characterized in that the apparatus comprises:

10. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor, when executing the computer program, implements the steps of the method of any of claims 1 to 8.