CN111309257A - Pre-reading method and device for reading file at constant speed and computer readable storage medium - Google Patents
Pre-reading method and device for reading file at constant speed and computer readable storage medium Download PDFInfo
- Publication number
- CN111309257A CN111309257A CN202010084184.XA CN202010084184A CN111309257A CN 111309257 A CN111309257 A CN 111309257A CN 202010084184 A CN202010084184 A CN 202010084184A CN 111309257 A CN111309257 A CN 111309257A
- Authority
- CN
- China
- Prior art keywords
- reading
- read
- data
- constant
- current
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 49
- 238000004364 calculation method Methods 0.000 claims description 10
- 238000004590 computer program Methods 0.000 claims description 5
- 239000002699 waste material Substances 0.000 abstract description 2
- 230000008569 process Effects 0.000 description 12
- 230000007423 decrease Effects 0.000 description 11
- 230000008859 change Effects 0.000 description 8
- 238000004422 calculation algorithm Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 4
- 238000012360 testing method Methods 0.000 description 4
- 230000008901 benefit Effects 0.000 description 3
- 230000007246 mechanism Effects 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000006872 improvement Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000005192 partition Methods 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0604—Improving or facilitating administration, e.g. storage management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/172—Caching, prefetching or hoarding of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
- G06F3/0656—Data buffering arrangements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0673—Single storage device
- G06F3/0674—Disk device
- G06F3/0676—Magnetic disk device
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Memory System Of A Hierarchy Structure (AREA)
Abstract
The invention discloses a pre-reading method, a device and a computer readable storage medium for reading files at a constant speed, wherein the method comprises the following steps: when reading a file at a constant speed for pre-reading, judging whether the current pre-reading is the first pre-reading; if the current pre-reading is the first pre-reading, performing the current pre-reading according to the preset pre-reading data size set in the configuration file; and if the current pre-reading is not the first pre-reading, calculating the size of the current pre-reading data, and performing the current pre-reading according to the calculated size of the current pre-reading data. The invention can ensure that the pre-reading of each file can obtain more reasonable pre-reading data size, and avoids the waste of cache resources.
Description
Technical Field
The present invention relates to the field of computer storage, and in particular, to a method and an apparatus for pre-reading a constant-rate read file, and a computer-readable storage medium.
Background
In the prior art, a File System (FS) is a method and data structure used by an operating System to specify files on a storage device (usually a disk and a solid state disk) or a partition; i.e. a method of organizing files on a storage device. From a system perspective, a file system is a system that organizes and allocates space of a file storage device, is responsible for file storage, and protects and retrieves stored files. In particular, it is responsible for creating files for the user, storing, reading, modifying, dumping files, controlling access to files, revoking files when the user is no longer in use, etc.
At present, almost all FS implementations use a cache (cache) mechanism to speed up the read-write performance of a file. Because the read/write speed of the memory is much higher than that of the disk, in the existing FS implementation, a part of the operating system memory is divided to serve as the cache of the FS. When an application program reads a large file in sequence, the FS takes out the data of the current reading position and a subsequent continuous large block of data from the disk and puts the data into the cache, and the subsequent file reading operation of the application program can directly read the required data in the cache without accessing the slow disk, so that the reading performance of the FS is improved, and the mechanism is called as a pre-reading mechanism of a file system.
The reading mode of the application program for the file is divided into sequential reading and random reading, and because the random reading cannot predict the subsequent reading position of the file, the pre-reading only aims at the sequential reading of the file.
In the prior art, after all the data that was previously read into the cache is completely read by the application, the next reading operation of the file by the application will not find the needed data in the cache, which is called miss. Then, this operation needs to access the disk, and a large block of data is read out from the disk again into the cache as the pre-read data, and the subsequent read operation can continue to read the required data from the cache, and so on. As can be seen from the above description, as shown in FIG. 1, a cache hit reads the cache, and a miss reads the disk and reads ahead.
In the field of file system application, there is an application mode in which a user can read a plurality of different files at a constant rate, and the application mode is widely used in the industries of video monitoring and broadcasting and television media. When reading a file at such a constant speed, the application itself controls the speed of reading the file, for example, a read request is sent every 100ms (milliseconds), and each time data with a size of 1MB is read, the rate of reading the file is 10 MB/s. Typically, the read file rate for this application mode in the industry is in the tens of MB/s.
The existing implementation of pre-reading of various file systems is to read out a large block of data from a back-end disk at a time and put the data into a cache, the size of the pre-read data is usually set in a configuration file of the file system, and after the file system is started, the pre-reading is performed according to the preset size. For reading files in a non-constant-rate sequence, an application program does not control the reading speed, after a reading request returns, a next request is initiated immediately, pre-reading data in a cache can be read quickly, and the file system can release the cache for future pre-reading. However, for a constant-rate read file, the read speed is significantly lower than that of a non-constant read, and if the data in the cache is much read in advance each time, part of the read-ahead data in the cache stays for a long time to wait for the reading of the application program. The total size of the cache is fixed, if the pre-read data of some files occupies too much cache, some files cannot obtain enough cache to store the pre-read data, and the number of files which can be read by an application program at a constant speed is influenced finally.
Disclosure of Invention
Embodiments of the present invention provide a method and an apparatus for pre-reading a constant-rate read file, and a computer-readable storage medium, so as to solve the above problems in the prior art in which a constant-rate read file exists.
The embodiment of the invention provides a pre-reading method for reading a file at a constant speed, which comprises the following steps:
when reading a file at a constant speed for pre-reading, judging whether the current pre-reading is the first pre-reading;
if the current pre-reading is the first pre-reading, performing the current pre-reading according to the preset pre-reading data size set in the configuration file; and if the current pre-reading is not the first pre-reading, calculating the size of the current pre-reading data, and performing the current pre-reading according to the calculated size of the current pre-reading data.
An embodiment of the present invention further provides a pre-reading apparatus for reading a file at a constant rate, including:
the judging module is used for judging whether the current pre-reading is the first pre-reading when the file is read at a constant speed for pre-reading;
the calculation module is used for performing current pre-reading according to the size of the preset pre-reading data set in the configuration file when the current pre-reading is the first pre-reading; and when the current pre-reading is not the first pre-reading, calculating the size of the current pre-reading data, and performing the current pre-reading according to the calculated size of the current pre-reading data.
An embodiment of the present invention further provides a pre-reading apparatus for reading a file at a constant rate, including: the computer program is executed by the processor to realize the steps of the pre-reading method for reading the file at the constant speed.
The embodiment of the invention also provides a computer readable storage medium, wherein an implementation program for information transmission is stored on the computer readable storage medium, and the program is executed by a processor to realize the steps of the pre-reading method for reading the file at the constant speed.
By adopting the embodiment of the invention, the cache pre-reading strategy is optimized aiming at the application mode of reading the files at the constant speed, and the size of the pre-reading data volume is dynamically adjusted according to the requirement, so that a reasonable cache space is allocated to each file, and the total number of different files which can be simultaneously read at the constant speed and can be supported by a file system is increased. The pre-reading of each file can obtain more reasonable pre-reading data size, and the waste of cache resources is avoided.
The foregoing description is only an overview of the technical solutions of the present invention, and the embodiments of the present invention are described below in order to make the technical means of the present invention more clearly understood and to make the above and other objects, features, and advantages of the present invention more clearly understandable.
Drawings
Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the invention. Also, like reference numerals are used to refer to like parts throughout the drawings. In the drawings:
FIG. 1 is a flow chart of sequential reading in the prior art;
FIG. 2 is a flow chart of a method for pre-reading a constant rate read file according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of a pre-reading method for reading a file at a constant rate according to an embodiment of the present invention;
FIG. 4 is a detailed flowchart of a pre-read method for reading a file at a constant rate according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of a pre-reading apparatus for reading a file at a constant rate according to a first embodiment of the present invention;
fig. 6 is a schematic diagram of a pre-reading apparatus for reading a file at a constant rate according to a second embodiment of the present invention.
Detailed Description
Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
Method embodiment
According to an embodiment of the present invention, a method for pre-reading a constant-rate read file is provided, fig. 2 is a flowchart of the method for pre-reading a constant-rate read file according to the embodiment of the present invention, and as shown in fig. 2, the method for pre-reading a constant-rate read file according to the embodiment of the present invention specifically includes:
step 201, when reading a file at a constant rate for pre-reading, judging whether the current pre-reading is the first pre-reading;
step 202, if the current pre-reading is the first pre-reading, the current pre-reading is carried out according to the preset pre-reading data size set in the configuration file; and if the current pre-reading is not the first pre-reading, calculating the size of the current pre-reading data, and performing the current pre-reading according to the calculated size of the current pre-reading data.
In step 201, calculating the size of the current pre-read data specifically includes:
step 2021, calculating a constant reading rate of the current file read at a constant rate; specifically, a time difference is calculated according to the end time of the previous pre-reading and the current time, wherein the time difference represents the time taken by the application program to finish reading all the data in the cache in the previous pre-reading; and then dividing the total size of the data read in advance last time by the time difference to calculate the constant reading rate of the current file read at the constant rate.
Step 2022, determine the size of the current pre-read data according to the constant read rate. Specifically, the calculated constant reading rate is multiplied by a preset time parameter to obtain the size of the current pre-read data, wherein the time parameter represents the time required by the application program to completely read the data pre-read into the cache each time.
That is to say, in the embodiment of the present invention, as shown in fig. 3, the technical solution of the embodiment of the present invention is only for constant rate file reading. Firstly, a parameter T (i.e. the time parameter) needs to be defined, where T represents that data pre-read into the cache at each time should be completely read by the application program within T seconds, and the parameter value may be defined in the configuration file before the file system is started, for example, 10 seconds; t is an important parameter, needs to be set before the file system is started, cannot be changed in the operation process of the file system, and can be adjusted to an optimal value according to a test result. Reducing the value of T can correspondingly reduce the cache occupied by the pre-read data of each file, thereby increasing the number of files which can be simultaneously read and supported by a system, but the disadvantages are that the pre-read frequency is increased, and the file read request often has high delay; conversely, increasing the value of T reduces the number of simultaneous reading of files that the system can support, but reduces the frequency of high latency of file read requests; how to take the value of T is adjusted according to a specific application scenario. After T is set, during the first pre-reading, using the pre-reading amount (i.e. the size of the pre-read data) preset in the configuration file, for example, 100 MB; when the pre-reading is not performed for the first time, the time t of the end of the previous pre-reading can be used1And the current time t2Calculating a time difference t, wherein t represents the time taken by the application program to finish reading all the data in the cache in the previous pre-reading, and dividing the total size d of the data in the previous pre-reading by t to obtain an approximate constant reading speed v of the application program: v ═ d/(t)2–t1) (ii) a The reading rate v of the application program is multiplied by T to obtain the current timePre-reading the size of the data volume that should be read: d ═ v × T. It should be noted that the technical solution of the embodiment of the present invention supports reading of all files at the same rate, and also supports reading of each file at a different rate.
In summary, in the technical solution of the embodiment of the present invention, if the application program increases or decreases the constant reading rate during the reading process of the file, the change can be sensed and the amount of the pre-read data can be adjusted in time. The faster the constant read rate of the file is, the more cache is occupied, and the number of files that can be read simultaneously by the system is reduced, and vice versa.
The above technical solutions of the embodiments of the present invention are explained below with reference to the drawings.
Fig. 4 is a detailed flowchart of a pre-reading method for reading a file at a constant rate according to an embodiment of the present invention, and as shown in fig. 4, the method specifically includes the following steps:
step 401, an application program reads files in a constant rate sequence, wherein all files can be read at the same rate, and each file can be read at different rates;
step 402, judging whether the data required to be read is stored in a cache, if so, executing step 401, otherwise, executing step 403;
step 403, preparing to perform pre-reading from a disk;
step 404, judging whether the current pre-reading is the first pre-reading, if so, executing step 406, otherwise, executing step 405;
step 405, calculating the size of the data to be read in advance by using a read-ahead algorithm, wherein the read-ahead algorithm specifically comprises: according to the ending time t of the last preread1And the current time t2Calculating a time difference t, wherein t represents the time taken by the application program to finish reading all the data in the cache in the previous pre-reading, and dividing the total size d of the data in the previous pre-reading by t to obtain an approximate constant reading speed v of the application program: v ═ d/(t)2–t1) (ii) a Multiplying the reading speed v of the application program by T to obtain the size of the data volume which should be read by the pre-reading: d ═ v × T, size D of the resulting pre-read data, execution of the stepsStep 407;
step 406, obtaining the size of the first pre-read data according to the configuration file;
step 407; a pre-read is made from disk and step 402 is performed.
In summary, the technical solution of the embodiments of the present invention optimizes the cache read-ahead strategy for the application mode of reading files at a constant rate, and can dynamically adjust the size of the read-ahead data amount as required. The technical scheme of the embodiment of the invention can be adopted in all file systems, and the modifiable configuration parameter added in the embodiment of the invention can be adjusted and optimized according to different application scenes, so that the method is more elastic. In the process of reading the file, if the application program increases or decreases the constant reading rate, the strategy provided by the technical scheme of the embodiment of the invention can sense the change and adjust the pre-reading data volume.
Apparatus embodiment one
According to an embodiment of the present invention, there is provided a pre-reading apparatus for reading a file at a constant rate, fig. 5 is a schematic diagram of a pre-reading apparatus for reading a file at a constant rate according to a first embodiment of the present invention, and as shown in fig. 5, the pre-reading apparatus for reading a file at a constant rate according to an embodiment of the present invention specifically includes:
the judging module 50 is configured to judge whether the current pre-reading is the first pre-reading when the file is read at a constant rate for pre-reading;
the calculation module 52 is configured to perform current pre-reading according to a size of pre-reading data set in the configuration file when the current pre-reading is the first pre-reading; and when the current pre-reading is not the first pre-reading, calculating the size of the current pre-reading data, and performing the current pre-reading according to the calculated size of the current pre-reading data.
The calculation module 52 specifically includes:
the first calculation module is used for calculating the constant reading rate of the current file read at the constant rate; the first calculation module is specifically configured to: calculating a time difference according to the ending time of the last pre-reading and the current time, wherein the time difference represents the time spent by the application program for reading all the data in the cache in the last pre-reading; and dividing the total size of the data read in advance last time by the time difference to calculate the constant reading rate of the current file read at the constant rate.
And the second calculation module is used for determining the size of the current pre-reading data according to the constant reading rate. The second calculation module is specifically configured to: and multiplying the calculated constant reading rate by a preset time parameter to obtain the size of the current pre-read data, wherein the time parameter represents the time required by the application program to completely read the data pre-read into the cache each time.
That is to say, in the embodiment of the present invention, as shown in fig. 3, the technical solution of the embodiment of the present invention is only for constant rate file reading. Firstly, a parameter T (i.e. the time parameter) needs to be defined, where T represents that data pre-read into the cache at each time should be completely read by the application program within T seconds, and the parameter value may be defined in the configuration file before the file system is started, for example, 10 seconds; t is an important parameter, needs to be set before the file system is started, cannot be changed in the operation process of the file system, and can be adjusted to an optimal value according to a test result. Reducing the value of T can correspondingly reduce the cache occupied by the pre-read data of each file, thereby increasing the number of files which can be simultaneously read and supported by a system, but the disadvantages are that the pre-read frequency is increased, and the file read request often has high delay; conversely, increasing the value of T reduces the number of simultaneous reading of files that the system can support, but reduces the frequency of high latency of file read requests; how to take the value of T is adjusted according to a specific application scenario. After T is set, during the first pre-reading, using the pre-reading amount (i.e. the size of the pre-read data) preset in the configuration file, for example, 100 MB; when the pre-reading is not performed for the first time, the time t of the end of the previous pre-reading can be used1And the current time t2Calculating the time difference t, wherein t represents the time spent by the application program for reading all the data in the cache in the last pre-reading, and dividing the total size d of the data in the last pre-reading by t to obtain the approximate constant reading speed of the application programThe ratio v: v ═ d/(t)2–t1) (ii) a Multiplying the reading speed v of the application program by T to obtain the size of the data volume which should be read by the pre-reading: d ═ v × T. It should be noted that the technical solution of the embodiment of the present invention supports reading of all files at the same rate, and also supports reading of each file at a different rate.
In summary, in the technical solution of the embodiment of the present invention, if the application program increases or decreases the constant reading rate during the reading process of the file, the change can be sensed and the amount of the pre-read data can be adjusted in time. The faster the constant read rate of the file is, the more cache is occupied, and the number of files that can be read simultaneously by the system is reduced, and vice versa.
The specific processing of each module in the embodiment of the present invention can be understood with reference to fig. 4 and the corresponding description in the method embodiment, which are not described herein again.
In summary, the technical solution of the embodiments of the present invention optimizes the cache read-ahead strategy for the application mode of reading files at a constant rate, and can dynamically adjust the size of the read-ahead data amount as required. The technical scheme of the embodiment of the invention can be adopted in all file systems, and the modifiable configuration parameter added in the embodiment of the invention can be adjusted and optimized according to different application scenes, so that the method is more elastic. In the process of reading the file, if the application program increases or decreases the constant reading rate, the strategy provided by the technical scheme of the embodiment of the invention can sense the change and adjust the pre-reading data volume.
Device embodiment II
An embodiment of the present invention provides a pre-reading apparatus for reading a file at a constant rate, as shown in fig. 6, including: a memory 60, a processor 62 and a computer program stored on the memory 60 and executable on the processor 62, which computer program, when executed by the processor 62, carries out the following method steps:
step 201, when reading a file at a constant rate for pre-reading, judging whether the current pre-reading is the first pre-reading;
step 202, if the current pre-reading is the first pre-reading, the current pre-reading is carried out according to the preset pre-reading data size set in the configuration file; and if the current pre-reading is not the first pre-reading, calculating the size of the current pre-reading data, and performing the current pre-reading according to the calculated size of the current pre-reading data.
In step 201, calculating the size of the current pre-read data specifically includes:
step 2021, calculating a constant reading rate of the current file read at a constant rate; specifically, a time difference is calculated according to the end time of the previous pre-reading and the current time, wherein the time difference represents the time taken by the application program to finish reading all the data in the cache in the previous pre-reading; and then dividing the total size of the data read in advance last time by the time difference to calculate the constant reading rate of the current file read at the constant rate.
Step 2022, determine the size of the current pre-read data according to the constant read rate. Specifically, the calculated constant reading rate is multiplied by a preset time parameter to obtain the size of the current pre-read data, wherein the time parameter represents the time required by the application program to completely read the data pre-read into the cache each time.
That is to say, in the embodiment of the present invention, as shown in fig. 3, the technical solution of the embodiment of the present invention is only for constant rate file reading. Firstly, a parameter T (i.e. the time parameter) needs to be defined, where T represents that data pre-read into the cache at each time should be completely read by the application program within T seconds, and the parameter value may be defined in the configuration file before the file system is started, for example, 10 seconds; t is an important parameter, needs to be set before the file system is started, cannot be changed in the operation process of the file system, and can be adjusted to an optimal value according to a test result. Reducing the value of T can correspondingly reduce the cache occupied by the pre-read data of each file, thereby increasing the number of files which can be simultaneously read and supported by a system, but the disadvantages are that the pre-read frequency is increased, and the file read request often has high delay; conversely, increasing the value of T will decrease the system support whileThe number of files to be read, but the frequency of high delay of file read requests is reduced; how to take the value of T is adjusted according to a specific application scenario. After T is set, during the first pre-reading, using the pre-reading amount (i.e. the size of the pre-read data) preset in the configuration file, for example, 100 MB; when the pre-reading is not performed for the first time, the time t of the end of the previous pre-reading can be used1And the current time t2Calculating a time difference t, wherein t represents the time taken by the application program to finish reading all the data in the cache in the previous pre-reading, and dividing the total size d of the data in the previous pre-reading by t to obtain an approximate constant reading speed v of the application program: v ═ d/(t)2–t1) (ii) a Multiplying the reading speed v of the application program by T to obtain the size of the data volume which should be read by the pre-reading: d ═ v × T. It should be noted that the technical solution of the embodiment of the present invention supports reading of all files at the same rate, and also supports reading of each file at a different rate.
In summary, in the technical solution of the embodiment of the present invention, if the application program increases or decreases the constant reading rate during the reading process of the file, the change can be sensed and the amount of the pre-read data can be adjusted in time. The faster the constant read rate of the file is, the more cache is occupied, and the number of files that can be read simultaneously by the system is reduced, and vice versa.
The above technical solutions of the embodiments of the present invention are explained below with reference to the drawings.
Fig. 4 is a detailed flowchart of a pre-reading method for reading a file at a constant rate according to an embodiment of the present invention, and as shown in fig. 4, the method specifically includes the following steps:
step 401, an application program reads files in a constant rate sequence, wherein all files can be read at the same rate, and each file can be read at different rates;
step 402, judging whether the data required to be read is stored in a cache, if so, executing step 401, otherwise, executing step 403;
step 403, preparing to perform pre-reading from a disk;
step 404, judging whether the current pre-reading is the first pre-reading, if so, executing step 406, otherwise, executing step 405;
step 405, calculating the size of the data to be read in advance by using a read-ahead algorithm, wherein the read-ahead algorithm specifically comprises: according to the ending time t of the last preread1And the current time t2Calculating a time difference t, wherein t represents the time taken by the application program to finish reading all the data in the cache in the previous pre-reading, and dividing the total size d of the data in the previous pre-reading by t to obtain an approximate constant reading speed v of the application program: v ═ d/(t)2–t1) (ii) a Multiplying the reading speed v of the application program by T to obtain the size of the data volume which should be read by the pre-reading: d ═ v × T, the size D of the finally obtained pre-read data, step 407 is executed;
step 406, obtaining the size of the first pre-read data according to the configuration file;
step 407; a pre-read is made from disk and step 402 is performed.
In summary, the technical solution of the embodiments of the present invention optimizes the cache read-ahead strategy for the application mode of reading files at a constant rate, and can dynamically adjust the size of the read-ahead data amount as required. The technical scheme of the embodiment of the invention can be adopted in all file systems, and the modifiable configuration parameter added in the embodiment of the invention can be adjusted and optimized according to different application scenes, so that the method is more elastic. In the process of reading the file, if the application program increases or decreases the constant reading rate, the strategy provided by the technical scheme of the embodiment of the invention can sense the change and adjust the pre-reading data volume.
Device embodiment III
The embodiment of the present invention provides a computer-readable storage medium, on which an implementation program for information transmission is stored, and when being executed by a processor 62, the implementation program implements the following method steps:
step 201, when reading a file at a constant rate for pre-reading, judging whether the current pre-reading is the first pre-reading;
step 202, if the current pre-reading is the first pre-reading, the current pre-reading is carried out according to the preset pre-reading data size set in the configuration file; and if the current pre-reading is not the first pre-reading, calculating the size of the current pre-reading data, and performing the current pre-reading according to the calculated size of the current pre-reading data.
In step 201, calculating the size of the current pre-read data specifically includes:
step 2021, calculating a constant reading rate of the current file read at a constant rate; specifically, a time difference is calculated according to the end time of the previous pre-reading and the current time, wherein the time difference represents the time taken by the application program to finish reading all the data in the cache in the previous pre-reading; and then dividing the total size of the data read in advance last time by the time difference to calculate the constant reading rate of the current file read at the constant rate.
Step 2022, determine the size of the current pre-read data according to the constant read rate. Specifically, the calculated constant reading rate is multiplied by a preset time parameter to obtain the size of the current pre-read data, wherein the time parameter represents the time required by the application program to completely read the data pre-read into the cache each time.
That is to say, in the embodiment of the present invention, as shown in fig. 3, the technical solution of the embodiment of the present invention is only for constant rate file reading. Firstly, a parameter T (i.e. the time parameter) needs to be defined, where T represents that data pre-read into the cache at each time should be completely read by the application program within T seconds, and the parameter value may be defined in the configuration file before the file system is started, for example, 10 seconds; t is an important parameter, needs to be set before the file system is started, cannot be changed in the operation process of the file system, and can be adjusted to an optimal value according to a test result. Reducing the value of T can correspondingly reduce the cache occupied by the pre-read data of each file, thereby increasing the number of files which can be simultaneously read and supported by a system, but the disadvantages are that the pre-read frequency is increased, and the file read request often has high delay; conversely, increasing the value of T decreases the number of simultaneous reads that the system can supportVolume, but the frequency of high latency of file read requests decreases; how to take the value of T is adjusted according to a specific application scenario. After T is set, during the first pre-reading, using the pre-reading amount (i.e. the size of the pre-read data) preset in the configuration file, for example, 100 MB; when the pre-reading is not performed for the first time, the time t of the end of the previous pre-reading can be used1And the current time t2Calculating a time difference t, wherein t represents the time taken by the application program to finish reading all the data in the cache in the previous pre-reading, and dividing the total size d of the data in the previous pre-reading by t to obtain an approximate constant reading speed v of the application program: v ═ d/(t)2–t1) (ii) a Multiplying the reading speed v of the application program by T to obtain the size of the data volume which should be read by the pre-reading: d ═ v × T. It should be noted that the technical solution of the embodiment of the present invention supports reading of all files at the same rate, and also supports reading of each file at a different rate.
In summary, in the technical solution of the embodiment of the present invention, if the application program increases or decreases the constant reading rate during the reading process of the file, the change can be sensed and the amount of the pre-read data can be adjusted in time. The faster the constant read rate of the file is, the more cache is occupied, and the number of files that can be read simultaneously by the system is reduced, and vice versa.
The above technical solutions of the embodiments of the present invention are explained below with reference to the drawings.
Fig. 4 is a detailed flowchart of a pre-reading method for reading a file at a constant rate according to an embodiment of the present invention, and as shown in fig. 4, the method specifically includes the following steps:
step 401, an application program reads files in a constant rate sequence, wherein all files can be read at the same rate, and each file can be read at different rates;
step 402, judging whether the data required to be read is stored in a cache, if so, executing step 401, otherwise, executing step 403;
step 403, preparing to perform pre-reading from a disk;
step 404, judging whether the current pre-reading is the first pre-reading, if so, executing step 406, otherwise, executing step 405;
step 405, calculating the size of the data to be read in advance by using a read-ahead algorithm, wherein the read-ahead algorithm specifically comprises: according to the ending time t of the last preread1And the current time t2Calculating a time difference t, wherein t represents the time taken by the application program to finish reading all the data in the cache in the previous pre-reading, and dividing the total size d of the data in the previous pre-reading by t to obtain an approximate constant reading speed v of the application program: v ═ d/(t)2–t1) (ii) a Multiplying the reading speed v of the application program by T to obtain the size of the data volume which should be read by the pre-reading: d ═ v × T, the size D of the finally obtained pre-read data, step 407 is executed;
step 406, obtaining the size of the first pre-read data according to the configuration file;
step 407; a pre-read is made from disk and step 402 is performed.
In summary, the technical solution of the embodiments of the present invention optimizes the cache read-ahead strategy for the application mode of reading files at a constant rate, and can dynamically adjust the size of the read-ahead data amount as required. The technical scheme of the embodiment of the invention can be adopted in all file systems, and the modifiable configuration parameter added in the embodiment of the invention can be adjusted and optimized according to different application scenes, so that the method is more elastic. In the process of reading the file, if the application program increases or decreases the constant reading rate, the strategy provided by the technical scheme of the embodiment of the invention can sense the change and adjust the pre-reading data volume.
The computer-readable storage medium of this embodiment includes, but is not limited to: ROM, RAM, magnetic or optical disks, and the like.
It will be apparent to those skilled in the art that the modules or steps of the present invention described above may be implemented by a general purpose computing device, they may be centralized on a single computing device or distributed across a network of multiple computing devices, and alternatively, they may be implemented by program code executable by a computing device, such that they may be stored in a storage device and executed by a computing device, and in some cases, the steps shown or described may be performed in an order different than that described herein, or they may be separately fabricated into individual integrated circuit modules, or multiple ones of them may be fabricated into a single integrated circuit module. Thus, the present invention is not limited to any specific combination of hardware and software.
The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
1. A method of pre-reading a constant rate read file, comprising:
when reading a file at a constant speed for pre-reading, judging whether the current pre-reading is the first pre-reading;
if the current pre-reading is the first pre-reading, performing the current pre-reading according to the preset pre-reading data size set in the configuration file; and if the current pre-reading is not the first pre-reading, calculating the size of the current pre-reading data, and performing the current pre-reading according to the calculated size of the current pre-reading data.
2. The method of claim 1, wherein calculating the current pre-read data size specifically comprises:
calculating a constant reading rate of a current file read at a constant rate;
and determining the size of the current pre-reading data according to the constant reading speed.
3. The method of claim 2, wherein calculating the constant read rate at which the file is currently read at the constant rate specifically comprises:
calculating a time difference according to the ending time of the last pre-reading and the current time, wherein the time difference represents the time spent by the application program for reading all the data in the cache in the last pre-reading;
and dividing the total size of the data read in advance last time by the time difference to calculate the constant reading rate of the current file read at the constant rate.
4. The method of claim 2, wherein determining a current pre-read data size based on the constant read rate specifically comprises:
and multiplying the calculated constant reading rate by a preset time parameter to obtain the size of the current pre-reading data, wherein the time parameter represents the time required by the application program to completely read the data pre-read into the cache each time.
5. A pre-reading apparatus for reading a file at a constant rate, comprising:
the judging module is used for judging whether the current pre-reading is the first pre-reading when the file is read at a constant speed for pre-reading;
the calculation module is used for performing current pre-reading according to the size of the preset pre-reading data set in the configuration file when the current pre-reading is the first pre-reading; and when the current pre-reading is not the first pre-reading, calculating the size of the current pre-reading data, and performing the current pre-reading according to the calculated size of the current pre-reading data.
6. The apparatus of claim 5, wherein the computing module specifically comprises:
the first calculation module is used for calculating the constant reading rate of the current file read at the constant rate;
and the second calculation module is used for determining the size of the current pre-reading data according to the constant reading rate.
7. The apparatus of claim 6, wherein the first computing module is specifically configured to:
calculating a time difference according to the ending time of the last pre-reading and the current time, wherein the time difference represents the time spent by the application program for reading all the data in the cache in the last pre-reading;
and dividing the total size of the data read in advance last time by the time difference to calculate the constant reading rate of the current file read at the constant rate.
8. The apparatus of claim 6, wherein the second computing module is specifically configured to:
and multiplying the calculated constant reading rate by a preset time parameter to obtain the size of the current pre-reading data, wherein the time parameter represents the time required by the application program to completely read the data pre-read into the cache each time.
9. A device for optimizing caching of constant-rate read files, comprising: memory, processor and computer program stored on the memory and executable on the processor, the computer program when executed by the processor implementing the steps of the method of constant rate read file pre-reading according to any of claims 1 to 4.
10. A computer-readable storage medium, on which an information transfer-implementing program is stored, which, when executed by a processor, implements the steps of the constant-rate read-ahead method for files according to any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010084184.XA CN111309257A (en) | 2020-02-10 | 2020-02-10 | Pre-reading method and device for reading file at constant speed and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010084184.XA CN111309257A (en) | 2020-02-10 | 2020-02-10 | Pre-reading method and device for reading file at constant speed and computer readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111309257A true CN111309257A (en) | 2020-06-19 |
Family
ID=71152731
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010084184.XA Pending CN111309257A (en) | 2020-02-10 | 2020-02-10 | Pre-reading method and device for reading file at constant speed and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111309257A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112181916A (en) * | 2020-09-14 | 2021-01-05 | 星辰天合（北京）数据科技有限公司 | File pre-reading method and device based on user space file system FUSE, and electronic equipment |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101231637A (en) * | 2007-01-22 | 2008-07-30 | 中兴通讯股份有限公司 | Self-adaption pre-reading method base on file system buffer |
| CN104881467A (en) * | 2015-05-26 | 2015-09-02 | 上海交通大学 | Data correlation analysis and pre-reading method based on frequent item set |
| CN105955821A (en) * | 2016-04-21 | 2016-09-21 | 北京小米移动软件有限公司 | Method and device for pre-reading |
| CN107992271A (en) * | 2017-12-21 | 2018-05-04 | 郑州云海信息技术有限公司 | Data pre-head method, device, equipment and computer-readable recording medium |
| CN109542361A (en) * | 2018-12-04 | 2019-03-29 | 郑州云海信息技术有限公司 | A kind of distributed memory system file reading, system and relevant apparatus |
| US20190369912A1 (en) * | 2018-05-30 | 2019-12-05 | Micron Technology, Inc. | Read look ahead data size determination |
-
2020
- 2020-02-10 CN CN202010084184.XA patent/CN111309257A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101231637A (en) * | 2007-01-22 | 2008-07-30 | 中兴通讯股份有限公司 | Self-adaption pre-reading method base on file system buffer |
| CN104881467A (en) * | 2015-05-26 | 2015-09-02 | 上海交通大学 | Data correlation analysis and pre-reading method based on frequent item set |
| CN105955821A (en) * | 2016-04-21 | 2016-09-21 | 北京小米移动软件有限公司 | Method and device for pre-reading |
| CN107992271A (en) * | 2017-12-21 | 2018-05-04 | 郑州云海信息技术有限公司 | Data pre-head method, device, equipment and computer-readable recording medium |
| US20190369912A1 (en) * | 2018-05-30 | 2019-12-05 | Micron Technology, Inc. | Read look ahead data size determination |
| CN109542361A (en) * | 2018-12-04 | 2019-03-29 | 郑州云海信息技术有限公司 | A kind of distributed memory system file reading, system and relevant apparatus |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112181916A (en) * | 2020-09-14 | 2021-01-05 | 星辰天合（北京）数据科技有限公司 | File pre-reading method and device based on user space file system FUSE, and electronic equipment |
| CN112181916B (en) * | 2020-09-14 | 2024-04-09 | 北京星辰天合科技股份有限公司 | File pre-reading method and device based on user space file system FUSE, and electronic equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US6381677B1 (en) | Method and system for staging data into cache | |
| US5381539A (en) | System and method for dynamically controlling cache management | |
| US8892520B2 (en) | Storage device including a file system manager for managing multiple storage media | |
| JP3308554B2 (en) | Control device and control method of control device | |
| EP0848321B1 (en) | Method of data migration | |
| US10540296B2 (en) | Thresholding task control blocks for staging and destaging | |
| JP2986088B2 (en) | Method and associated apparatus for operating a buffer memory | |
| US5226141A (en) | Variable capacity cache memory | |
| JP4429780B2 (en) | Storage control device, control method, and control program. | |
| US8590001B2 (en) | Network storage system with data prefetch and method of operation thereof | |
| CN111258967A (en) | Data reading method and device in file system and computer readable storage medium | |
| CN107870732B (en) | Method and apparatus for flushing pages from solid state storage devices | |
| EP2889776B1 (en) | Data arrangement control program, data arrangement control method and data arrangment control apparatus | |
| CN113377291A (en) | Data processing method, device, equipment and medium of cache equipment | |
| US7660964B2 (en) | Windowing external block translations | |
| CN111930307A (en) | Data reading method, device and equipment and computer readable storage medium | |
| EP1782175B1 (en) | Time budgeting for non-data transfer operations in drive units | |
| CN111309257A (en) | Pre-reading method and device for reading file at constant speed and computer readable storage medium | |
| CN109582233A (en) | A kind of caching method and device of data | |
| US6775745B1 (en) | Method and apparatus for hybrid data caching mechanism | |
| US20040162941A1 (en) | Disk array control apparatus and method | |
| US7421536B2 (en) | Access control method, disk control unit and storage apparatus | |
| KR100974514B1 (en) | A sequential prefetching method in computer system | |
| JPH06243042A (en) | Storage controller | |
| JP2001118365A (en) | System and method for managing storage hierarchy and recording medium with storage hierarchical management program recorded thereon |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200619 |