CN113590560A - Cache optimization method, system, equipment and storage medium of distributed system - Google Patents
Cache optimization method, system, equipment and storage medium of distributed system Download PDFInfo
- Publication number
- CN113590560A CN113590560A CN202110732350.7A CN202110732350A CN113590560A CN 113590560 A CN113590560 A CN 113590560A CN 202110732350 A CN202110732350 A CN 202110732350A CN 113590560 A CN113590560 A CN 113590560A
- Authority
- CN
- China
- Prior art keywords
- cache
- read
- write
- data
- file
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 60
- 238000005457 optimization Methods 0.000 title claims abstract description 38
- 230000015654 memory Effects 0.000 claims abstract description 62
- 230000007717 exclusion Effects 0.000 claims abstract description 27
- 238000011010 flushing procedure Methods 0.000 claims abstract description 15
- 238000004590 computer program Methods 0.000 claims description 14
- 230000008569 process Effects 0.000 claims description 12
- 238000002360 preparation method Methods 0.000 claims description 7
- 230000001680 brushing effect Effects 0.000 claims description 2
- 230000001360 synchronised effect Effects 0.000 description 6
- 230000000694 effects Effects 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 238000004891 communication Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 230000004075 alteration Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/172—Caching, prefetching or hoarding of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/176—Support for shared access to files; File sharing support
- G06F16/1767—Concurrency control, e.g. optimistic or pessimistic approaches
- G06F16/1774—Locking methods, e.g. locking methods for file systems allowing shared and concurrent access to files
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a cache optimization method, a cache optimization system, cache optimization equipment and a cache optimization storage medium for a distributed system, wherein the method comprises the steps of operating the distributed system and a client, applying for a memory space to cache client data from a memory, and starting a cache management thread to manage the client; applying for file authority, reading and writing files in a read-write cache mode after the application is successful, and calculating a file object corresponding to a read-write range; performing read-write operation after the file object is successfully locked, and unlocking after the operation is completed; when the data to be refreshed in the cache reaches a threshold value, applying for a cache global exclusive lock, and after the addition is successful, adding an object lock to the file object and merging the data in the object range; and after the combination is successful, the data is flushed to the storage medium until the mutual exclusion lock is released after the flushing is finished. The method also provides a cache optimization system, equipment and a storage medium, and increases concurrency among objects and improves the throughput rate and the read-write performance of the cluster by performing fine granularity on the global exclusive lock in the cache to the object lock.
Description
Technical Field
The invention belongs to the technical field of distributed system storage, and particularly relates to a cache optimization method, a cache optimization system, cache optimization equipment and a cache optimization storage medium for a distributed system.
Background
In a distributed system, a client cache based on a memory realizes a cache read-write function, and the method in the prior art comprises the following steps: after the client is started, a fixed-size capacity is allocated from the memory to serve as a client read-write cache, and the client needs to apply corresponding read-write permission and read-write cache permission to the metadata server before reading and writing the file, namely, the client applies read permission and read-write cache permission when reading the file and applies write permission and write-cache permission when writing the file. After applying for the read-write cache authority, the read-write data can be put into a client for caching; when the client reads and writes the file, the file is divided into a plurality of equal parts according to the file object so as to increase the reading and writing concurrency of the file; the method comprises the steps of calculating which file object is according to the read-write range when a file is read and written each time, reading and writing the object in a client cache, wherein the client cache is added with a global exclusive lock for ensuring the validity of data under the condition of multithreading concurrency, and the reading and writing of any object in the cache need to obtain the lock first, namely, the lock is firstly added.
When the write-back mode is used, the cache refreshing thread also applies for locking before writing back dirty data to the storage medium, and after the locking is successful, the exclusive lock is held, and at the moment, the reading and writing of other files to the object in the cache are blocked due to the failure of obtaining the lock, so that the reading and writing delay is increased, and the throughput rate is limited; the file is divided into file objects to carry out read-write operation, the read-write concurrency is increased, the read-write performance is improved, but the read-write performance is influenced by serial execution of a lock mechanism in the cache, the original purpose of dividing the file objects is violated, even if a single file is read and written, the read-write performance is influenced by the cache mutual exclusion lock, and the read-write performance, the cache throughput rate and the utilization rate are seriously influenced by the excessively coarse lock granularity of the mutual exclusion lock in the cache.
Disclosure of Invention
In order to solve the technical problems, the invention provides a cache optimization method, a cache optimization system, cache optimization equipment and a cache optimization storage medium for a distributed system, wherein global mutual exclusion locks in client caches are subjected to fine granularity to object locks with file objects as the granularity, so that the throughput rate and the read-write performance of a cluster are greatly improved.
In order to achieve the purpose, the invention adopts the following technical scheme:
a cache optimization method of a distributed system comprises the following steps:
running a distributed system and a client, applying a memory space from a memory to cache client data, and starting a cache management thread to manage the client data;
applying for file read-write permission and read-write caching permission at a client, reading and writing the file in a read-write caching mode after the application is successful, and calculating a file object corresponding to a read-write range; performing read-write operation after the file object is successfully locked, and performing unlocking after the operation is completed;
when the data to be refreshed in the cache reaches a threshold value, applying for caching a global exclusive lock by a cache management thread, and after the exclusive lock is successfully added, adding an object lock to a file object and merging the data in an object range; and after the combination is successful, the data is flushed to the storage medium until the mutual exclusion lock is released after the flushing is finished.
Further, the process of applying for the file read-write permission and the read-write cache permission at the client and performing file read-write through a read-write cache mode after successful application is as follows:
when a client reads and writes a file, sending a message to a metadata server side to apply for read-write permission and read-write cache permission, and if the application is successful, reading and writing the file in a read-write cache mode;
and if the application is not successful, performing data read-write operation by directly reading the storage medium or directly writing the storage medium.
Further, the sending the message to the metadata server side to apply for the read-write permission and the read-write caching permission includes:
when reading the file, applying for read permission and read cache permission, if the application is successful, putting the read data into the cache to increase the read performance;
and when writing the file, applying for writing authority and writing cache authority, and if the application is successful, writing the data to be written into the cache.
Further, the process of calculating the file object corresponding to the read-write range includes: and calculating the file object corresponding to the range through an algorithm according to the initial position and the data length of reading and writing.
Further, the operation is performed after the file object is successfully locked, and the process of unlocking after the operation is completed is as follows:
applying for locking the file object, and waiting if other threads hold the lock of the file object;
after locking is successful, increasing the reference count of the read-write cache permission of the file object, after the reference count is increased successfully, performing read-write operation on the file object, and after the operation is completed, unlocking the file object;
and performing read-write operation after other threads holding the file object lock apply for the lock.
Further, the step of when the data to be flushed in the cache reaches the threshold includes: when the data to be brushed in the cache reaches half of the total amount of the cache or the time remaining in the cache exceeds a time threshold.
Further, the flushing data to the storage medium after the merging is successful until the mutual exclusion lock is released after the flushing is completed includes: and after the data are successfully merged, downloading and brushing the data to a storage medium, releasing an object lock after the data are successfully downloaded and subtracting one from the reference count of the file object read-write cache authority until all the file object data to be downloaded in the cache are completely printed to the storage medium, and then releasing a global mutual exclusion lock.
The invention also provides a cache optimization system of the distributed system, which comprises a preparation module, a first operation module and a second operation module;
the preparation module is used for operating the distributed system and the client, applying for memory space to cache client data from the memory, and starting a cache management thread to manage the client data;
the first operation module is used for applying for file read-write permission and read-write caching permission at the client, reading and writing files in a read-write caching mode after the application is successful, and calculating a file object corresponding to a read-write range; performing read-write operation after the file object is successfully locked, and performing unlocking after the operation is completed;
the second operation module is used for applying for caching a global mutual exclusion lock by a cache management thread when the data to be refreshed in the cache reaches a threshold value, and adding an object lock to a file object and merging data in an object range after the mutual exclusion lock is successfully added; and after the combination is successful, the data is flushed to the storage medium until the mutual exclusion lock is released after the flushing is finished.
The invention also proposes a device comprising:
a memory for storing a computer program;
a processor for implementing the method steps when executing the computer program.
The invention also proposes a readable storage medium having stored thereon a computer program which, when being executed by a processor, carries out the method steps.
The effect provided in the summary of the invention is only the effect of the embodiment, not all the effects of the invention, and one of the above technical solutions has the following advantages or beneficial effects:
the invention has proposed a buffer memory optimization method, system, apparatus and storage medium of the distributed system, said method comprises operating distributed system and customer end, apply for the memory space from the memory and buffer memory customer end data, and start the buffer memory and manage the thread and manage the customer end data; applying for file read-write permission and read-write caching permission at a client, reading and writing the file in a read-write caching mode after the application is successful, and calculating a file object corresponding to a read-write range; performing read-write operation after the file object is successfully locked, and performing unlocking after the operation is completed; when the data to be refreshed in the cache reaches a threshold value, applying for caching a global exclusive lock by a cache management thread, and after the exclusive lock is successfully added, adding an object lock to a file object and merging the data in an object range; and after the combination is successful, the data is flushed to the storage medium until the mutual exclusion lock is released after the flushing is finished. The invention granularizes the global mutual exclusion lock in the client cache to the object lock taking the file object as the granularity, each object has one object lock, and when the read-write operation of the object is carried out in the cache, only the object lock of the corresponding object is needed to be obtained, but the cache global mutual exclusion lock is not needed to be obtained. And the concurrency among the objects is greatly increased because the object lock adopts a read-write lock, the read-write parallelism of the same object is increased, the read-write performance of the client is improved by 20 percent compared with that before the optimization under high concurrency, and the throughput rate and the read-write performance of the cluster are greatly improved.
Based on the cache optimization method of the distributed system provided by the invention, a cache optimization system, equipment and a storage medium of the distributed system are also provided, and the technical effects of the method are also achieved, which are not described herein again.
Drawings
Fig. 1 is a flowchart of a cache optimization method for a distributed system according to embodiment 1 of the present invention;
fig. 2 is a schematic diagram of a cache optimization system of a distributed system according to embodiment 1 of the present invention.
Detailed Description
In order to clearly explain the technical features of the present invention, the following detailed description of the present invention is provided with reference to the accompanying drawings. The following disclosure provides many different embodiments, or examples, for implementing different features of the invention. To simplify the disclosure of the present invention, the components and arrangements of specific examples are described below. Furthermore, the present invention may repeat reference numerals and/or letters in the various examples. This repetition is for the purpose of simplicity and clarity and does not in itself dictate a relationship between the various embodiments and/or configurations discussed. It should be noted that the components illustrated in the figures are not necessarily drawn to scale. Descriptions of well-known components and processing techniques and procedures are omitted so as to not unnecessarily limit the invention.
Example 1
Embodiment 1 of the present invention provides a cache optimization method for a distributed system, which performs fine-grained on a global exclusive lock in a client cache to an object lock with a file object as a granularity, where each object has an object lock, and when performing read-write operation on an object in the cache, only the object lock of the corresponding object needs to be acquired, and the cache global exclusive lock does not need to be acquired. The cache global mutual exclusion lock is acquired only when the global data structure is accessed, so that the concurrency among objects is greatly increased, and the read-write lock is adopted as the object lock, so that the read-parallel on the same object is increased.
Fig. 1 shows a flowchart of a cache optimization method for a distributed system according to embodiment 1 of the present invention.
In step S101, the distributed system and the client are operated, a memory space is applied from the memory to cache the client data, and a cache management thread is started to manage the client data.
When a client program is started, a memory space with a set size is applied from a memory to store client cache data, and a corresponding timed thread flush _ timer is started to manage the cache quantity.
In step S102, the client applies for the file read-write permission and the read-write cache permission, and reads and writes the file in a read-write cache manner after the application is successful.
Sending a message to a metadata server side to apply for read-write permission and read-write cache permission, applying for read permission and read cache permission when reading a file, and if the application is successful, putting the read data into a cache to increase the read performance; and when writing the file, applying for writing authority and writing cache authority, and if the application is successful, writing the data to be written into the cache.
And if the cache authority is not applied, performing data read-write operation in a direct-reading storage medium or direct-writing storage medium mode.
In step S103, a file object corresponding to the read/write range is calculated. According to the invention, the file object corresponding to the range is calculated by an algorithm according to the read-write initial position (offset) and the data length (Len), namely the read-write range.
In step S104, after the file object is successfully locked, the read/write operation is performed, and after the operation is completed, the file object is unlocked.
Applying for locking (object _ lock) to the file object, if the other threads hold the lock of the file object, waiting;
after the locking is successful, the reference count (cap _ ref [ cb ] ++) of the read-write cache authority of the file object is increased, which indicates that the thread is performing cache read-write operation on the file object. After the reference count is successfully increased, performing read-write operation on the file object, and after the operation is completed, unlocking the file object;
and other threads holding the file object lock perform read-write operation after applying for the lock, so that concurrent read-write operation of different objects of the same file is realized.
In step S105, when the data to be flushed in the cache reaches the threshold, the cache management thread applies for a cache global exclusive lock, and after the exclusive lock is successfully added, an object lock is added to the file object and data in the object range is merged.
Checking at regular time by a cache management thread flush _ timer, applying for a cache global exclusive lock (cache _ lock) when data to be flushed in a cache reaches a certain threshold (1/2 of the total size of the cache) or is reserved for more than a certain time (5s) and needs to be flushed to a storage medium, after the lock is successfully added, adding an object lock to a file object (file _ object) in the cache one by one and merging data in an object range
In step S106, the data is flushed to the storage medium after the merging is successful, until the mutual exclusion lock is released after the flushing is completed. And after the data are successfully merged, the data are flushed to the storage medium, after the data are successfully flushed, the object lock is released, and the reference count of the file read-write cache authority is reduced by one (cap _ ref [ bc ] - -) until all the file object data to be flushed in the cache are flushed to the storage medium, and then the global exclusive lock (uncack) is released.
After the cache global exclusive lock is subjected to fine-grained operation to a specific object lock, the concurrent read-write performance and the cache utilization rate of objects among different files or the same file are increased, the client response capability is improved, and the distributed system performance is improved.
By carrying out fine granularity on the global exclusive lock in the client cache to the object lock taking the file object as the granularity, when the read-write operation of the object is carried out in the cache, only the object lock of the corresponding object needs to be obtained, but the cache global exclusive lock does not need to be obtained, the concurrency among the objects is greatly increased, the read-write lock is adopted for the object lock, the read-parallel of the same object is increased, the read-write performance of the client under high concurrency is improved by 20 percent compared with that before the optimization, and the throughput rate and the read-write performance of the cluster are greatly improved.
Example 2
Based on the method for optimizing the cache of the distributed system provided by embodiment 1 of the present invention, embodiment 2 of the present invention further provides a cache optimization system of the distributed system, and as shown in fig. 2, a schematic diagram of the cache optimization system of the distributed system provided by embodiment 2 of the present invention is provided, where the system includes a preparation module, a first operation module, and a second operation module;
the preparation module is used for operating the distributed system and the client, applying for memory space cache client data from the memory and starting a cache management thread to manage the client data;
the first operation module is used for applying for file read-write permission and read-write caching permission at the client, reading and writing the file in a read-write caching mode after the application is successful, and calculating a file object corresponding to a read-write range; performing read-write operation after the file object is successfully locked, and performing unlocking after the operation is completed;
the second operation module is used for applying for caching a global mutual exclusion lock by the cache management thread when the data to be refreshed in the cache reaches a threshold value, adding an object lock to the file object and merging the data in the object range after the mutual exclusion lock is successfully added; and after the combination is successful, the data is flushed to the storage medium until the mutual exclusion lock is released after the flushing is finished.
The preparation module realizes the following processes: when a client program is started, a memory space with a set size is applied from a memory to store client cache data, and a corresponding timed thread flush _ timer is started to manage the cache quantity.
The first operation module realizes the following processes: sending a message to a metadata server side to apply for read-write permission and read-write cache permission, applying for read permission and read cache permission when reading a file, and if the application is successful, putting the read data into a cache to increase the read performance; and when writing the file, applying for writing authority and writing cache authority, and if the application is successful, writing the data to be written into the cache.
And if the cache authority is not applied, performing data read-write operation in a direct-reading storage medium or direct-writing storage medium mode.
And calculating the file object corresponding to the read-write range. According to the invention, the file object corresponding to the range is calculated by an algorithm according to the read-write initial position (offset) and the data length (Len), namely the read-write range.
And performing read-write operation after the file object is successfully locked, and unlocking after the operation is completed.
Applying for locking (object _ lock) to the file object, if the other threads hold the lock of the file object, waiting;
after the locking is successful, the reference count (cap _ ref [ cb ] ++) of the read-write cache authority of the file object is increased, which indicates that the thread is performing cache read-write operation on the file object. After the reference count is successfully increased, performing read-write operation on the file object, and after the operation is completed, unlocking the file object;
and other threads holding the file object lock perform read-write operation after applying for the lock, so that concurrent read-write operation of different objects of the same file is realized.
The second operation module realizes the following processes: checking at regular time by a cache management thread flush _ timer, applying for a cache global exclusive lock (cache _ lock) when data to be flushed in a cache reaches a certain threshold (1/2 of the total size of the cache) or is reserved for more than a certain time (5s) and needs to be flushed to a storage medium, after the lock is successfully added, adding an object lock to a file object (file _ object) in the cache one by one and merging data in an object range
In step S106, the data is flushed to the storage medium after the merging is successful, until the mutual exclusion lock is released after the flushing is completed. And after the data are successfully merged, the data are flushed to the storage medium, after the data are successfully flushed, the object lock is released, and the reference count of the file read-write cache authority is reduced by one (cap _ ref [ bc ] - -) until all the file object data to be flushed in the cache are flushed to the storage medium, and then the global exclusive lock (uncack) is released.
The invention also proposes a device comprising:
a memory for storing a computer program;
a processor for implementing the method steps when executing the computer program as follows:
in step S101, the distributed system and the client are operated, a memory space is applied from the memory to cache the client data, and a cache management thread is started to manage the client data.
When a client program is started, a memory space with a set size is applied from a memory to store client cache data, and a corresponding timed thread flush _ timer is started to manage the cache quantity.
In step S102, the client applies for the file read-write permission and the read-write cache permission, and reads and writes the file in a read-write cache manner after the application is successful.
Sending a message to a metadata server side to apply for read-write permission and read-write cache permission, applying for read permission and read cache permission when reading a file, and if the application is successful, putting the read data into a cache to increase the read performance; and when writing the file, applying for writing authority and writing cache authority, and if the application is successful, writing the data to be written into the cache.
And if the cache authority is not applied, performing data read-write operation in a direct-reading storage medium or direct-writing storage medium mode.
In step S103, a file object corresponding to the read/write range is calculated. According to the invention, the file object corresponding to the range is calculated by an algorithm according to the read-write initial position (offset) and the data length (Len), namely the read-write range.
In step S104, after the file object is successfully locked, the read/write operation is performed, and after the operation is completed, the file object is unlocked.
Applying for locking (object _ lock) to the file object, if the other threads hold the lock of the file object, waiting;
after the locking is successful, the reference count (cap _ ref [ cb ] ++) of the read-write cache authority of the file object is increased, which indicates that the thread is performing cache read-write operation on the file object. After the reference count is successfully increased, performing read-write operation on the file object, and after the operation is completed, unlocking the file object;
and other threads holding the file object lock perform read-write operation after applying for the lock, so that concurrent read-write operation of different objects of the same file is realized.
In step S105, when the data to be flushed in the cache reaches the threshold, the cache management thread applies for a cache global exclusive lock, and after the exclusive lock is successfully added, an object lock is added to the file object and data in the object range is merged.
Checking at regular time by a cache management thread flush _ timer, applying for a cache global exclusive lock (cache _ lock) when data to be flushed in a cache reaches a certain threshold (1/2 of the total size of the cache) or is reserved for more than a certain time (5s) and needs to be flushed to a storage medium, after the lock is successfully added, adding an object lock to a file object (file _ object) in the cache one by one and merging data in an object range
In step S106, the data is flushed to the storage medium after the merging is successful, until the mutual exclusion lock is released after the flushing is completed. And after the data are successfully merged, the data are flushed to the storage medium, after the data are successfully flushed, the object lock is released, and the reference count of the file read-write cache authority is reduced by one (cap _ ref [ bc ] - -) until all the file object data to be flushed in the cache are flushed to the storage medium, and then the global exclusive lock (uncack) is released.
Need to explain: the technical solution of the present invention also provides an electronic device, including: the communication interface can carry out information interaction with other equipment such as network equipment and the like; the processor is connected with the communication interface to realize information interaction with other equipment, and is used for executing the cache optimization method of the distributed system provided by one or more technical schemes when running a computer program, and the computer program is stored on the memory. Of course, in practice, the various components in an electronic device are coupled together by a bus system. It will be appreciated that a bus system is used to enable communications among the components. The bus system includes a power bus, a control bus, and a status signal bus in addition to a data bus. The memory in the embodiments of the present application is used to store various types of data to support the operation of the electronic device. Examples of such data include: any computer program for operating on an electronic device. It will be appreciated that the memory can be either volatile memory or nonvolatile memory, and can include both volatile and nonvolatile memory. Among them, the nonvolatile Memory may be a Read Only Memory (ROM), a Programmable Read Only Memory (PROM), an Erasable Programmable Read-Only Memory (EPROM), an Electrically Erasable Programmable Read-Only Memory (EEPROM), a magnetic random access Memory (FRAM), a Flash Memory (Flash Memory), a magnetic surface Memory, an optical disk, or a Compact Disc Read-Only Memory (CD-ROM); the magnetic surface storage may be disk storage or tape storage. Volatile memory can be Random Access Memory (RAM), which acts as external cache memory. By way of illustration and not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), Enhanced Synchronous Dynamic Random Access Memory (Enhanced DRAM), Synchronous Dynamic Random Access Memory (SLDRAM), Direct Memory (DRmb Access), and Random Access Memory (DRAM). The memories described in the embodiments of the present application are intended to comprise, without being limited to, these and any other suitable types of memory. The method disclosed in the embodiments of the present application may be applied to a processor, or may be implemented by a processor. The processor may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware in a processor or instructions in the form of software. The processor may be a general purpose processor, a DSP (Digital Signal Processing, i.e., a chip capable of implementing Digital Signal Processing technology), or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, etc. The processor may implement or perform the methods, steps, and logic blocks disclosed in the embodiments of the present application. A general purpose processor may be a microprocessor or any conventional processor or the like. The steps of the method disclosed in the embodiments of the present application may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium located in a memory where a processor reads the programs in the memory and in combination with its hardware performs the steps of the method as previously described. When the processor executes the program, corresponding processes in the methods of the embodiments of the present application are implemented, and for brevity, are not described herein again.
The invention also proposes a readable storage medium on which a computer program is stored, which, when executed by a processor, implements the method steps of:
in step S101, the distributed system and the client are operated, a memory space is applied from the memory to cache the client data, and a cache management thread is started to manage the client data.
When a client program is started, a memory space with a set size is applied from a memory to store client cache data, and a corresponding timed thread flush _ timer is started to manage the cache quantity.
In step S102, the client applies for the file read-write permission and the read-write cache permission, and reads and writes the file in a read-write cache manner after the application is successful.
Sending a message to a metadata server side to apply for read-write permission and read-write cache permission, applying for read permission and read cache permission when reading a file, and if the application is successful, putting the read data into a cache to increase the read performance; and when writing the file, applying for writing authority and writing cache authority, and if the application is successful, writing the data to be written into the cache.
And if the cache authority is not applied, performing data read-write operation in a direct-reading storage medium or direct-writing storage medium mode.
In step S103, a file object corresponding to the read/write range is calculated. According to the invention, the file object corresponding to the range is calculated by an algorithm according to the read-write initial position (offset) and the data length (Len), namely the read-write range.
In step S104, after the file object is successfully locked, the read/write operation is performed, and after the operation is completed, the file object is unlocked.
Applying for locking (object _ lock) to the file object, if the other threads hold the lock of the file object, waiting;
after the locking is successful, the reference count (cap _ ref [ cb ] ++) of the read-write cache authority of the file object is increased, which indicates that the thread is performing cache read-write operation on the file object. After the reference count is successfully increased, performing read-write operation on the file object, and after the operation is completed, unlocking the file object;
and other threads holding the file object lock perform read-write operation after applying for the lock, so that concurrent read-write operation of different objects of the same file is realized.
In step S105, when the data to be flushed in the cache reaches the threshold, the cache management thread applies for a cache global exclusive lock, and after the exclusive lock is successfully added, an object lock is added to the file object and data in the object range is merged.
Checking at regular time by a cache management thread flush _ timer, applying for a cache global exclusive lock (cache _ lock) when data to be flushed in a cache reaches a certain threshold (1/2 of the total size of the cache) or is reserved for more than a certain time (5s) and needs to be flushed to a storage medium, after the lock is successfully added, adding an object lock to a file object (file _ object) in the cache one by one and merging data in an object range
In step S106, the data is flushed to the storage medium after the merging is successful, until the mutual exclusion lock is released after the flushing is completed. And after the data are successfully merged, the data are flushed to the storage medium, after the data are successfully flushed, the object lock is released, and the reference count of the file read-write cache authority is reduced by one (cap _ ref [ bc ] - -) until all the file object data to be flushed in the cache are flushed to the storage medium, and then the global exclusive lock (uncack) is released.
Embodiments of the present application further provide a storage medium, that is, a computer storage medium, specifically, a computer-readable storage medium, for example, a memory storing a computer program, where the computer program is executable by a processor to perform the steps of the foregoing method. The computer readable storage medium may be Memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface Memory, optical disk, or CD-ROM.
Those of ordinary skill in the art will understand that: all or part of the steps for implementing the method embodiments may be implemented by hardware related to program instructions, and the program may be stored in a computer readable storage medium, and when executed, the program performs the steps including the method embodiments; and the aforementioned storage medium includes: a removable storage device, a ROM, a RAM, a magnetic or optical disk, or various other media that can store program code. Alternatively, the integrated units described above in the present application may be stored in a computer-readable storage medium if they are implemented in the form of software functional modules and sold or used as independent products. Based on such understanding, the technical solutions of the embodiments of the present application may be essentially implemented or portions thereof that contribute to the prior art may be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling an electronic device (which may be a personal computer, a server, or a network device) to execute all or part of the methods described in the embodiments of the present application. And the aforementioned storage medium includes: a removable storage device, a ROM, a RAM, a magnetic or optical disk, or various other media that can store program code.
For a description of a relevant part in a device and a storage medium for cache optimization of a distributed system provided in an embodiment of the present application, reference may be made to a detailed description of a corresponding part in a cache optimization method of a distributed system provided in embodiment 1 of the present application, and details are not described here again.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Furthermore, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include elements inherent in the list. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element. In addition, parts of the above technical solutions provided in the embodiments of the present application, which are consistent with the implementation principles of corresponding technical solutions in the prior art, are not described in detail so as to avoid redundant description.
Although the embodiments of the present invention have been described with reference to the accompanying drawings, the scope of the present invention is not limited thereto. Various modifications and alterations will occur to those skilled in the art based on the foregoing description. And are neither required nor exhaustive of all embodiments. On the basis of the technical scheme of the invention, various modifications or changes which can be made by a person skilled in the art without creative efforts are still within the protection scope of the invention.
Claims (10)
1. A cache optimization method of a distributed system is characterized by comprising the following steps:
running a distributed system and a client, applying a memory space from a memory to cache client data, and starting a cache management thread to manage the client data;
applying for file read-write permission and read-write caching permission at a client, reading and writing the file in a read-write caching mode after the application is successful, and calculating a file object corresponding to a read-write range; performing read-write operation after the file object is successfully locked, and performing unlocking after the operation is completed;
when the data to be refreshed in the cache reaches a threshold value, applying for caching a global exclusive lock by a cache management thread, and after the exclusive lock is successfully added, adding an object lock to a file object and merging the data in an object range; and after the combination is successful, the data is flushed to the storage medium until the mutual exclusion lock is released after the flushing is finished.
2. The cache optimization method of the distributed system according to claim 1, wherein the process of applying for the file read-write permission and the read-write cache permission at the client, and performing the file read-write through the read-write cache mode after the application is successful, is as follows:
when a client reads and writes a file, sending a message to a metadata server side to apply for read-write permission and read-write cache permission, and if the application is successful, reading and writing the file in a read-write cache mode;
and if the application is not successful, performing data read-write operation by directly reading the storage medium or directly writing the storage medium.
3. The cache optimization method of the distributed system according to claim 2, wherein the sending the message to the metadata server for applying for the read-write permission and the read-write cache permission comprises:
when reading the file, applying for read permission and read cache permission, if the application is successful, putting the read data into the cache to increase the read performance;
and when writing the file, applying for writing authority and writing cache authority, and if the application is successful, writing the data to be written into the cache.
4. The cache optimization method of the distributed system according to claim 1, wherein the process of calculating the file object corresponding to the read-write range is as follows: and calculating the file object corresponding to the range through an algorithm according to the initial position and the data length of reading and writing.
5. The cache optimization method of the distributed system according to claim 1, wherein the operation is performed after the file object is successfully locked, and the process of unlocking after the operation is completed is as follows:
applying for locking the file object, and waiting if other threads hold the lock of the file object;
after locking is successful, increasing the reference count of the read-write cache permission of the file object, after the reference count is increased successfully, performing read-write operation on the file object, and after the operation is completed, unlocking the file object;
and performing read-write operation after other threads holding the file object lock apply for the lock.
6. The method according to claim 1, wherein when the data to be flushed in the cache reaches a threshold, the method comprises: when the data to be brushed in the cache reaches half of the total amount of the cache or the time remaining in the cache exceeds a time threshold.
7. The cache optimization method of claim 6, wherein the flushing data to the storage medium after the merging is successful until the mutual exclusion lock is released after the flushing is completed comprises: and after the data are successfully merged, downloading and brushing the data to a storage medium, releasing an object lock after the data are successfully downloaded and subtracting one from the reference count of the file object read-write cache authority until all the file object data to be downloaded in the cache are completely printed to the storage medium, and then releasing a global mutual exclusion lock.
8. The cache optimization system of the distributed system is characterized by comprising a preparation module, a first operation module and a second operation module;
the preparation module is used for operating the distributed system and the client, applying for memory space to cache client data from the memory, and starting a cache management thread to manage the client data;
the first operation module is used for applying for file read-write permission and read-write caching permission at the client, reading and writing files in a read-write caching mode after the application is successful, and calculating a file object corresponding to a read-write range; performing read-write operation after the file object is successfully locked, and performing unlocking after the operation is completed;
the second operation module is used for applying for caching a global mutual exclusion lock by a cache management thread when the data to be refreshed in the cache reaches a threshold value, and adding an object lock to a file object and merging data in an object range after the mutual exclusion lock is successfully added; and after the combination is successful, the data is flushed to the storage medium until the mutual exclusion lock is released after the flushing is finished.
9. An apparatus, comprising:
a memory for storing a computer program;
a processor for implementing the method steps of any one of claims 1 to 7 when executing the computer program.
10. A readable storage medium, characterized in that the readable storage medium has stored thereon a computer program which, when being executed by a processor, carries out the method steps of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110732350.7A CN113590560A (en) | 2021-06-29 | 2021-06-29 | Cache optimization method, system, equipment and storage medium of distributed system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110732350.7A CN113590560A (en) | 2021-06-29 | 2021-06-29 | Cache optimization method, system, equipment and storage medium of distributed system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113590560A true CN113590560A (en) | 2021-11-02 |
Family
ID=78245066
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110732350.7A Pending CN113590560A (en) | 2021-06-29 | 2021-06-29 | Cache optimization method, system, equipment and storage medium of distributed system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113590560A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115951844A (en) * | 2023-03-13 | 2023-04-11 | 浪潮电子信息产业股份有限公司 | File lock management method, device and medium for distributed file system |
Citations (23)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5615373A (en) * | 1993-08-26 | 1997-03-25 | International Business Machines Corporation | Data lock management in a distributed file server system determines variable lock lifetime in response to request to access data object |
| US20130254246A1 (en) * | 2012-03-21 | 2013-09-26 | Todd Lipcon | Data processing performance enhancement in a distributed file system |
| CN104113587A (en) * | 2014-06-23 | 2014-10-22 | 华中科技大学 | Client metadata buffer optimization method of distributed file system |
| CN104331453A (en) * | 2014-10-30 | 2015-02-04 | 北京思特奇信息技术股份有限公司 | Distributed file system and constructing method thereof |
| CN105183378A (en) * | 2015-08-31 | 2015-12-23 | 北京神州云科数据技术有限公司 | Adaptive cache mixed reading/writing method |
| US20160004718A1 (en) * | 2014-07-02 | 2016-01-07 | Panzura, Inc. | Using byte-range locks to manage multiple concurrent accesses to a file in a distributed filesystem |
| CN106021381A (en) * | 2016-05-11 | 2016-10-12 | 北京搜狐新媒体信息技术有限公司 | Data access/storage method and device for cloud storage service system |
| CN106446037A (en) * | 2016-08-31 | 2017-02-22 | 南威软件股份有限公司 | Method for realizing consistency of Redis and MYSQL data based on distributed lock |
| CN106469150A (en) * | 2015-08-14 | 2017-03-01 | 阿里巴巴集团控股有限公司 | File read/write method, device and system |
| CN106776798A (en) * | 2016-11-23 | 2017-05-31 | 深圳市中博睿存科技有限公司 | A kind of cluster file system is client-based to propagate caching method |
| CN106897440A (en) * | 2017-02-28 | 2017-06-27 | 郑州云海信息技术有限公司 | A kind of distributed file system file read-write processing method |
| WO2017107948A1 (en) * | 2015-12-23 | 2017-06-29 | 中兴通讯股份有限公司 | File write-aggregation and read-aggregation method and system, and client |
| CN107632945A (en) * | 2016-07-18 | 2018-01-26 | 大唐移动通信设备有限公司 | The data read-write method and device of a kind of shared drive |
| CN107832423A (en) * | 2017-11-13 | 2018-03-23 | 中山大学 | A kind of file read/write method for distributed file system |
| CN107888687A (en) * | 2017-11-15 | 2018-04-06 | 长沙证通云计算有限公司 | A kind of agent client storage accelerated method and system based on distributed memory system |
| CN109597769A (en) * | 2018-12-04 | 2019-04-09 | 郑州云海信息技术有限公司 | A kind of data cached write-back method, system, device and readable storage medium storing program for executing |
| CN110750507A (en) * | 2019-09-30 | 2020-02-04 | 华中科技大学 | Client persistent caching method and system under global namespace facing DFS |
| CN112099977A (en) * | 2020-09-30 | 2020-12-18 | 浙江工商大学 | Real-time data analysis engine of distributed tracking system |
| CN112099962A (en) * | 2020-11-06 | 2020-12-18 | 成都新希望金融信息有限公司 | Distributed lock implementation method and device and electronic equipment |
| CN112115515A (en) * | 2020-10-29 | 2020-12-22 | 苏州浪潮智能科技有限公司 | Data protection method and device, electronic equipment and storage medium |
| CN112416556A (en) * | 2020-11-23 | 2021-02-26 | 西安西热电站信息技术有限公司 | Data read-write priority balancing method, system, device and storage medium |
| CN112650720A (en) * | 2020-12-18 | 2021-04-13 | 深圳市佳创视讯技术股份有限公司 | Cache system management method and device and computer readable storage medium |
| CN113031864A (en) * | 2021-03-19 | 2021-06-25 | 上海众源网络有限公司 | Data processing method and device, electronic equipment and storage medium |
-
2021
- 2021-06-29 CN CN202110732350.7A patent/CN113590560A/en active Pending
Patent Citations (23)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5615373A (en) * | 1993-08-26 | 1997-03-25 | International Business Machines Corporation | Data lock management in a distributed file server system determines variable lock lifetime in response to request to access data object |
| US20130254246A1 (en) * | 2012-03-21 | 2013-09-26 | Todd Lipcon | Data processing performance enhancement in a distributed file system |
| CN104113587A (en) * | 2014-06-23 | 2014-10-22 | 华中科技大学 | Client metadata buffer optimization method of distributed file system |
| US20160004718A1 (en) * | 2014-07-02 | 2016-01-07 | Panzura, Inc. | Using byte-range locks to manage multiple concurrent accesses to a file in a distributed filesystem |
| CN104331453A (en) * | 2014-10-30 | 2015-02-04 | 北京思特奇信息技术股份有限公司 | Distributed file system and constructing method thereof |
| CN106469150A (en) * | 2015-08-14 | 2017-03-01 | 阿里巴巴集团控股有限公司 | File read/write method, device and system |
| CN105183378A (en) * | 2015-08-31 | 2015-12-23 | 北京神州云科数据技术有限公司 | Adaptive cache mixed reading/writing method |
| WO2017107948A1 (en) * | 2015-12-23 | 2017-06-29 | 中兴通讯股份有限公司 | File write-aggregation and read-aggregation method and system, and client |
| CN106021381A (en) * | 2016-05-11 | 2016-10-12 | 北京搜狐新媒体信息技术有限公司 | Data access/storage method and device for cloud storage service system |
| CN107632945A (en) * | 2016-07-18 | 2018-01-26 | 大唐移动通信设备有限公司 | The data read-write method and device of a kind of shared drive |
| CN106446037A (en) * | 2016-08-31 | 2017-02-22 | 南威软件股份有限公司 | Method for realizing consistency of Redis and MYSQL data based on distributed lock |
| CN106776798A (en) * | 2016-11-23 | 2017-05-31 | 深圳市中博睿存科技有限公司 | A kind of cluster file system is client-based to propagate caching method |
| CN106897440A (en) * | 2017-02-28 | 2017-06-27 | 郑州云海信息技术有限公司 | A kind of distributed file system file read-write processing method |
| CN107832423A (en) * | 2017-11-13 | 2018-03-23 | 中山大学 | A kind of file read/write method for distributed file system |
| CN107888687A (en) * | 2017-11-15 | 2018-04-06 | 长沙证通云计算有限公司 | A kind of agent client storage accelerated method and system based on distributed memory system |
| CN109597769A (en) * | 2018-12-04 | 2019-04-09 | 郑州云海信息技术有限公司 | A kind of data cached write-back method, system, device and readable storage medium storing program for executing |
| CN110750507A (en) * | 2019-09-30 | 2020-02-04 | 华中科技大学 | Client persistent caching method and system under global namespace facing DFS |
| CN112099977A (en) * | 2020-09-30 | 2020-12-18 | 浙江工商大学 | Real-time data analysis engine of distributed tracking system |
| CN112115515A (en) * | 2020-10-29 | 2020-12-22 | 苏州浪潮智能科技有限公司 | Data protection method and device, electronic equipment and storage medium |
| CN112099962A (en) * | 2020-11-06 | 2020-12-18 | 成都新希望金融信息有限公司 | Distributed lock implementation method and device and electronic equipment |
| CN112416556A (en) * | 2020-11-23 | 2021-02-26 | 西安西热电站信息技术有限公司 | Data read-write priority balancing method, system, device and storage medium |
| CN112650720A (en) * | 2020-12-18 | 2021-04-13 | 深圳市佳创视讯技术股份有限公司 | Cache system management method and device and computer readable storage medium |
| CN113031864A (en) * | 2021-03-19 | 2021-06-25 | 上海众源网络有限公司 | Data processing method and device, electronic equipment and storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 余洪春: "Linux集群之美", 31 January 2021, 北京：机械工业出版社, pages: 95 - 96 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115951844A (en) * | 2023-03-13 | 2023-04-11 | 浪潮电子信息产业股份有限公司 | File lock management method, device and medium for distributed file system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20240095233A1 (en) | Persistent memory management | |
| Kim et al. | PACTree: A high performance persistent range index using PAC guidelines | |
| KR101288408B1 (en) | A method and system for facilitating fast wake-up of a flash memory system | |
| US10360149B2 (en) | Data structure store in persistent memory | |
| US8984239B2 (en) | Hardware and operating system support for persistent memory on a memory bus | |
| EP2972891B1 (en) | Multiversioned nonvolatile memory hierarchy for persistent memory | |
| EP3805944B1 (en) | Data page access method, storage engine, and computer readable storage medium | |
| Gao et al. | PCMLogging: Reducing transaction logging overhead with PCM | |
| CN108595347B (en) | Cache control method and device and computer readable storage medium | |
| US11449402B2 (en) | Handling of offline storage disk | |
| Gao et al. | Pcmlogging: Optimizing transaction logging and recovery performance with pcm | |
| Lersch et al. | Persistent buffer management with optimistic consistency | |
| CN110287129B (en) | L2P table updating and writing management method and device based on solid state disk | |
| US20060277221A1 (en) | Transactional file system with client partitioning | |
| CN113590560A (en) | Cache optimization method, system, equipment and storage medium of distributed system | |
| US10528436B2 (en) | Micro-journal based transaction logging | |
| US10445308B2 (en) | Commit coalescing for micro-journal based transaction logging | |
| Kwon et al. | DeNOVA: Deduplication extended nova file system | |
| Son et al. | A log-structured buffer for database systems using non-volatile memory | |
| CN111611223A (en) | Nonvolatile data access method, system, electronic device and medium | |
| CN115048046B (en) | Log file system and data management method | |
| Deng et al. | Efficient hardware-assisted out-place update for persistent memory | |
| CN112214171B (en) | SQLite database-oriented non-volatile memory buffer area design method | |
| Son et al. | Nonvolatile write buffer-based journaling bypass for storage write reduction in mobile devices | |
| Kim et al. | A scalable and persistent key-value store using non-volatile memory |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |