CN114063888A - Data storage system, data processing method, terminal and storage medium - Google Patents
Data storage system, data processing method, terminal and storage medium Download PDFInfo
- Publication number
- CN114063888A CN114063888A CN202010763506.3A CN202010763506A CN114063888A CN 114063888 A CN114063888 A CN 114063888A CN 202010763506 A CN202010763506 A CN 202010763506A CN 114063888 A CN114063888 A CN 114063888A
- Authority
- CN
- China
- Prior art keywords
- data
- type
- memory
- hard disk
- class
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003860 storage Methods 0.000 title claims abstract description 92
- 238000013500 data storage Methods 0.000 title claims abstract description 39
- 238000003672 processing method Methods 0.000 title claims abstract description 14
- 230000015654 memory Effects 0.000 claims abstract description 175
- 238000000034 method Methods 0.000 claims description 37
- 239000007787 solid Substances 0.000 claims description 18
- 238000004220 aggregation Methods 0.000 claims description 13
- 230000002776 aggregation Effects 0.000 claims description 11
- 238000004590 computer program Methods 0.000 claims description 11
- 230000004931 aggregating effect Effects 0.000 claims description 4
- 239000000725 suspension Substances 0.000 claims description 3
- 238000012546 transfer Methods 0.000 claims description 3
- 238000012545 processing Methods 0.000 abstract description 13
- 238000010586 diagram Methods 0.000 description 12
- 230000008569 process Effects 0.000 description 6
- 230000004044 response Effects 0.000 description 6
- 238000004422 calculation algorithm Methods 0.000 description 5
- 230000008859 change Effects 0.000 description 4
- 238000004519 manufacturing process Methods 0.000 description 4
- 230000011218 segmentation Effects 0.000 description 4
- 238000004891 communication Methods 0.000 description 3
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 238000009826 distribution Methods 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 238000012163 sequencing technique Methods 0.000 description 2
- 238000000638 solvent extraction Methods 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 230000000903 blocking effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000013523 data management Methods 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000005055 memory storage Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000005192 partition Methods 0.000 description 1
- 230000000737 periodic effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2219—Large Object storage; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0629—Configuration or reconfiguration of storage systems
- G06F3/0631—Configuration or reconfiguration of storage systems by allocating resources to storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/064—Management of blocks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present disclosure relates to a data storage system, a data processing method, a terminal, and a storage medium. The data storage system includes: at least one working node; the working node includes: the memory is used for storing the first type of data; the first type hard disk is used for storing part of the first type data and/or the second type data; the reading and writing frequency of the second type of data is lower than that of the first type of data; the second type of hard disk is used for storing third type of data; the reading rate of the first type of hard disk is higher than that of the second type of hard disk; the read-write frequency of the third type of data is lower than that of the second type of data. Therefore, in data application, the data frequently called for high-frequency application can be conveniently called directly from the memory, and the processing efficiency of the system on the hot spot data is improved.
Description
Technical Field
The present invention relates to the field of data storage technologies, and in particular, to a data storage system, a data processing method, a terminal, and a storage medium.
Background
With the development of the internet era, the data which people need to acquire is exponentially increased. A large amount of data is used in engineering. At the same time, storage of large amounts of data is becoming an issue that needs to be addressed in engineering applications. According to the traditional data storage mode, the storage capacity of a computer file system is expanded by simply increasing the number of hard disks, and the change of the capacity size, the capacity increase speed and the like is very strong. Especially when a large amount of data is stored, the efficiency is low when frequently used key data is called according to engineering requirements.
Disclosure of Invention
In view of the above, embodiments of the present disclosure are intended to provide a data storage system, a data processing method, a terminal, and a storage medium.
The technical scheme of the disclosure is realized as follows:
the present disclosure provides a data storage system.
The data storage system provided by the embodiment of the disclosure comprises:
at least one working node;
the work node includes:
the memory is used for storing the first type of data;
the first type hard disk is used for storing part of the first type data and/or the second type data; the reading and writing frequency of the second type of data is lower than that of the first type of data;
the second type of hard disk is used for storing third type of data; the reading rate of the first type of hard disk is higher than that of the second type of hard disk; and the read-write frequency of the third type of data is lower than that of the second type of data.
In some embodiments, the first type of hard disk includes: a solid state disk; the second type of hard disk comprises: mechanical hard disk.
In some embodiments, the first type of hard disk is specifically configured to store the first type of data and the second type of data, where the data amount of the first type of data and the second type of data is greater than a first threshold, where the first threshold is a preset value determined according to a capacity of a memory space of the working node.
In some embodiments, the system further comprises:
the scheduling node is used for dynamically dividing data into the first class of data, the second class of data and the third class of data according to the reading and writing frequency of the data stored in the storage system; and scheduling the memory, the first class of hard disk and the second class of hard disk of one or more working nodes to store the first class of data, the second class of data and the third class of data respectively according to the data types divided by control.
In some embodiments, the scheduling node establishes a network connection with a client, and is configured to receive a read request sent by the client; when the client requests to read one or more complete data blocks according to the reading request, selecting an idle working node to respond to the reading request according to the load rate of the working node; and/or when the client requests to read partial data in one data block according to the reading request, sending a pause instruction and a reading instruction to a working node responding to the reading request; and the pause instruction is used for controlling the working node to pause the caching of the data in the memory and returning the read data to the client side corresponding to the read instruction.
In some embodiments, the scheduling node is further configured to receive a request for applying for a storage space from the client; according to the surplus of the memory in the working node, when the memory space amount in the working node is determined to be not less than the requested space capacity, the surplus memory resources are allocated to the client; and/or when the memory space amount in the working node is determined to be smaller than the requested space capacity, transferring the first type of data with the lowest memory access frequency to the first type of hard disk, releasing part of the memory space of the memory, and allocating the memory space released by the memory to the client.
In some embodiments, the scheduling node is further configured to split a plurality of data blocks of a super data block whose data size is greater than a second threshold, and schedule memories in a plurality of working nodes to store the different split data blocks respectively.
In some embodiments, the scheduling node is further configured to, when receiving a read request for reading the super data block by the client, split the read request into a plurality of subtasks, and schedule the plurality of working nodes to respond to the subtasks respectively; and aggregating the data blocks read by the multiple working nodes in an aggregation queue and returning the aggregated data blocks to the client.
On the other hand, the present disclosure also provides a data processing method.
The data processing method provided by the embodiment of the disclosure is applied to a scheduling node of the data storage system, and the method comprises the following steps:
dividing the data into first class data, second class data and third class data according to the reading and writing frequency of the data; the data volume of the second type of data is larger than that of the first type of data; the read-write frequency of the third type of data is lower than that of the second type of data;
the memory of the scheduling working node stores the first type of data;
scheduling a first type hard disk of the working node to store part of the first type data and/or the second type data;
scheduling a second type of hard disk of the working node to store the third type of data; and the reading rate of the first type of hard disk is higher than that of the second type of hard disk.
In some embodiments, said scheduling the first type of hard disk of the working node to store the portion of the first type of data and/or the second type of data comprises:
storing the first type of data with the stored data volume larger than a first threshold value into the first type of hard disk, wherein the first threshold value is a preset value determined according to the capacity of the memory space of the working node;
and storing the second type of data to the first type of hard disk.
In some embodiments, the dividing the data into a first type of data, a second type of data, and a third type of data according to the read-write frequency of the data includes:
the scheduling node dynamically divides the data into the first class of data, the second class of data and the third class of data according to the reading and writing frequency of the data stored in the storage system;
the method further comprises the following steps:
and the scheduling node schedules the memory, the first class of hard disk and the second class of hard disk of one or more working nodes according to the data type divided by control, and stores the first class of data, the second class of data and the third class of data respectively.
In some embodiments, further comprising:
the scheduling node receives a reading request sent by a client;
when the client requests to read one or more complete data blocks according to the reading request, selecting an idle working node to respond to the reading request according to the load rate of the working node;
and/or the presence of a gas in the gas,
when the client requests to read partial data in one data block according to the reading request, sending a pause instruction and a reading instruction to a working node responding to the reading request; the suspension instruction is used for controlling the working node to suspend caching data in the memory, and returning the read data to the client side corresponding to the reading instruction.
In some embodiments, further comprising:
receiving a request for applying for a storage space sent by the client;
according to the surplus of the memory in the working node, when the memory space amount in the working node is determined to be not less than the requested space capacity, the surplus memory resources are allocated to the client;
and/or the presence of a gas in the gas,
and according to the residual amount of the memory in the working node, when the memory space amount in the working node is determined to be smaller than the requested space capacity, controlling the working node to transfer the first type of data with the lowest access frequency in the memory to the first type of hard disk, releasing part of the storage space of the memory, and distributing the storage space released by the memory to the client.
In another aspect, the present disclosure also provides a terminal.
The terminal provided by the embodiment of the disclosure comprises: a processor and a memory for storing a computer program capable of running on the processor, wherein the processor is configured to execute steps of a data processing method provided by another aspect of the present disclosure when the computer program is run.
In yet another aspect, the present disclosure also provides a computer-readable storage medium.
The computer readable storage medium provided by the embodiments of the present disclosure has a computer program stored thereon, and the computer program, when executed by a processor, implements the steps of the data processing method provided by the embodiments of the present disclosure in another aspect. The data classification storage method and the data classification storage device perform data classification storage on a memory, a first class hard disk and a second class hard disk in a working node, store first class data in the memory according to the read-write frequency of a system, store the first class data and/or the second class data in the first class hard disk, and store third class data in the second class hard disk, wherein the read-write frequency of the third class data is lower than that of the second class data, and the read-write frequency of the second class data is lower than that of the first class data. Because the system directly reads and writes the data in the memory when reading and writing the data, the data stored in the hard disk is called based on the memory, and then the data is read and written. Therefore, the first type of data which is frequently read and written and has high frequency is stored in the memory, so that when the system reads and writes a large amount of data in the data application, the data which is frequently called and applied with high frequency can be conveniently called from the memory directly, and the processing efficiency of the system on the hot spot data is improved.
Drawings
FIG. 1 is a schematic diagram of a data storage system architecture shown in accordance with an exemplary embodiment;
FIG. 2 is a flow diagram illustrating a method of data processing in accordance with an exemplary embodiment;
FIG. 3 is a flowchart illustrating a method of selective hotspot splitting cache scheduling in accordance with an exemplary embodiment;
FIG. 4 is a graph illustrating data size and probability distribution of cold and hot data in a production cluster using statistics of the present technique, according to an exemplary embodiment;
FIG. 5 is a schematic diagram of a tiered big data storage system architecture shown in accordance with an exemplary embodiment;
FIG. 6 is a schematic diagram illustrating a hierarchical big data storage system cache scheduling, according to an example embodiment;
FIG. 7 is a schematic diagram illustrating file splitting in accordance with an exemplary embodiment;
FIG. 8 is a diagram illustrating a fork-aggregate queue model in accordance with an exemplary embodiment.
Detailed Description
The technical solution of the present invention is further described in detail with reference to the drawings and the specific embodiments of the specification. Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the invention, as detailed in the appended claims.
The big data storage System may be a Distributed File System HDFS (Hadoop Distributed File System) based on disk storage. HDFS has particular advantages in storing large numbers of files and a single large file, and supports streaming file access, where files can be read and accessed many times with only one write. When the cluster is arranged on different machines, a plurality of backups are stored in data, and a system which is down at one node or has an error in reading a document can call backup files on other nodes to ensure that a program runs smoothly. The dynamic data movement among the nodes of the system enables the data to be processed concurrently among the nodes, ensures the dynamic balance of the nodes, and completes the expansion of the cluster through the expansion of the nodes. In the distributed system, the database mostly adopts an LIRS (Low Inter-reference Reference Set) algorithm to realize a data caching mechanism. The LIRS algorithm mainly realizes data caching through two parameters, namely IRR and Recence, the IRR parameter is used for representing the latest two-time access interval of one page, and the Recence parameter is used for representing how many other pages are accessed so far in the last access. A caching policy is given for recent iterations of data. The LIRS algorithm determines which data blocks are replaced by the cache by comparing IRR, and realizes a cache replacement algorithm with low overhead and without additional parameter setting through the LIRS. Wherein, IRR (Inter-Reference) represents the number of non-repeated blocks accessed in two consecutive access periods of the same block.
Due to limited memory resources, for a large amount of data stored locally, the situation that the queried data cannot be loaded into the memory all at once may occur in the working nodes in the distributed cluster, and multiple times of data transmission between the memory and the disk will increase the execution time of the query task, thereby reducing the query efficiency. For some light-weight data management services, database query requests are mostly frequent query processing of some key data attributes. In addition, the speed of a modern data center network is greatly increased, the read-write operation of the HDFS hard disk-based file data storage system in a large-scale data scene becomes a performance bottleneck, additional read-write and network communication expenses are brought by a plurality of sets of local mechanisms, and the application requirements of real-time or quasi-real-time large data analysis and processing cannot be met.
In order to overcome the performance deficiency of a distributed file system based on a disk, the method introduces memory calculation and storage to the file system layer on the basis of using a layered storage architecture so as to fully utilize the distributed memory storage to change the data read-write performance, and can obtain performance improvement of order of magnitude compared with the traditional solution using the disk in the data read-write intensive application scene by caching the data object in the memory. In the process of loading the memory with the data, the frequently processed attribute data is loaded into the memory as much as possible to serve as a main target, the query opportunity of the frequent data can be increased, and cache data with a higher hit rate compared with an LIRS algorithm can be obtained, so that the processing efficiency of the hot data is improved.
The present disclosure provides a data storage system. FIG. 1 is a schematic diagram of a data storage system architecture shown in accordance with an exemplary embodiment. As shown in fig. 1, the data storage system includes:
at least one working node;
the work node includes:
a memory 11 for storing a first type of data;
the first type hard disk 12 is used for storing part of the first type data and/or the second type data; the reading and writing frequency of the second type of data is lower than that of the first type of data;
the second type hard disk 13 is used for storing third type data; the reading rate of the first type of hard disk is higher than that of the second type of hard disk; and the read-write frequency of the third type of data is lower than that of the second type of data.
In the present exemplary embodiment, the data storage system may be divided into a three-tier storage system of memory-first class hard disks-second class hard disks. The three-layer storage system preferentially stores the three types of data.
In some embodiments, the first type of hard disk includes: a solid state disk; the second type of hard disk comprises: mechanical hard disk.
In this exemplary embodiment, the first type of hard disk may be a solid state hard disk with a relatively high number of data reads and writes. The data read-write speed of the solid state disk can be more than 500 MB/S. The first type of hard disk may be a mechanical hard disk.
In the present exemplary embodiment, data classification storage is performed on the memory, the first type of hard disk, and the second type of hard disk in the working node. Storing first class data in a memory according to the read-write frequency of the system to the data, storing the first class data and/or second class data in a first class hard disk, and storing third class data in a second class hard disk, wherein the read-write frequency of the third class data is lower than that of the second class data, and the read-write frequency of the second class data is lower than that of the first class data.
Because the system directly reads and writes the data in the memory when reading and writing the data, the data stored in the hard disk is called based on the memory, and then the data is read and written. Therefore, the first type of data which is frequently read and written and has high frequency is stored in the memory, so that when the system reads and writes a large amount of data in data application, the data which is frequently used by high frequency can be directly called from the memory conveniently, and the processing efficiency of the system on the hot spot data is improved.
In the present exemplary embodiment, the first type data may be hotspot data.
In this exemplary embodiment, the data storage system further includes an underlying file system for storing a fourth type of data other than the above-mentioned three types of data, where the fourth type of data is underlying data with the lowest frequency of application.
In some embodiments, the first type of hard disk is specifically configured to store the first type of data and the second type of data, where the data amount of the first type of data and the second type of data is greater than a first threshold, where the first threshold is a preset value determined according to a capacity of a memory space of the working node.
In the present exemplary embodiment, since the storage capacity of the memory is limited, when the data amount of the first type data is greater than the first threshold, the first type data that is not stored in the memory may be stored in the first type hard disk. The first threshold is a preset value determined according to the capacity of the memory space of the working node. This preset value can be set within the system. When the data quantity is compared, the system can automatically call the threshold value, so that the storage space allocation of the data is convenient for data storage through threshold value comparison.
The first threshold value and the capacity of the memory space may have a certain proportional relationship. For example, the first threshold may be a capacity value that is less than 1/3 or less than 1/4 memory capacity of a working node.
In some embodiments, the system further comprises:
the scheduling node is used for dynamically dividing data into the first class of data, the second class of data and the third class of data according to the reading and writing frequency of the data stored in the storage system; and scheduling the memory, the first class of hard disk and the second class of hard disk of one or more working nodes to store the first class of data, the second class of data and the third class of data respectively according to the data types divided by control.
In the present exemplary embodiment, a scheduling node, which may be a controller or become a master, is also included in the data storage system. The scheduling node divides the data into a first class of data, a second class of data and a third class of data according to the read-write frequency of the stored data, and stores the data of each class in the memory, the first class of hard disk and the second class of hard disk respectively. For example, the first type of data is stored in the memory, the second type of data and the first type of data that is not stored in the memory are stored in the first type of hard disk, and the third type of data is stored in the third type of hard disk.
In this exemplary embodiment, dynamically dividing data into the first type of data, the second type of data, and the third type of data according to the read-write frequency of data stored in the storage system includes:
continuously monitoring the reading and writing frequency of the data stored in the storage system, taking a preset time period as a cycle, and performing periodic data division to obtain the first type data, the second type data and the third type data, or,
and when the change of the data reading and writing frequency exceeds the preset percentage of the reading and writing frequency when the data is divided last time, carrying out re-division adjustment on the data. For example, the read-write frequency of a certain data before the data division is s, and when the read-write frequency of the data is monitored to be 50% or 200% of s, the data is subjected to repartitioning adjustment. By dynamically dividing the data types, the data types can be adjusted to be correspondingly stored according to the data reading and writing frequency change in time, so that the data with high reading and writing frequency can be stored in the memory in time, reading and writing are facilitated, and the processing efficiency of the system on the hot data is improved.
In some embodiments, the method for counting the reading and writing frequency or heat of data comprises the following steps:
collecting all logs in a cluster server, and acquiring access records of all data files within a certain time period;
acquiring the access frequency of each data file based on the access record of each data file;
sequencing the access frequency of each data file from high to low to form a data read-write frequency arrangement table, wherein the arrangement table comprises a plurality of read-write frequencies, and each read-write frequency corresponds to a corresponding access file;
and acquiring the average size of the file corresponding to each read-write frequency based on the access file corresponding to each read-write frequency.
In some embodiments, further comprising:
sorting the highest frequency corresponding to the data of each file in the arrangement table, and acquiring the file size corresponding to each read-write frequency data and the total file size in the arrangement table;
and acquiring the data heat corresponding to each read-write frequency data based on the file size and the total file size corresponding to each read-write frequency data. For example, if the file sizes corresponding to the frequency data are M1, M2, M3 and M4 … … Mn, and the total file size is Ma, the heat of the frequency data is Ri＝Mi/Ma。
In the present exemplary embodiment, the work node may be one or more. Each working node comprises a memory, a first type of hard disk and a second type of hard disk. In a specific application, a plurality of working nodes can be communicated with each other. When data is read and written, the hard disks and the memories between different working points can mutually perform data transmission, and the system can simultaneously read and write data stored in the memories of a plurality of working nodes. For example, when it is necessary to read and write second type data stored in a second type hard disk of one working node, where the data volume of the second type data is much larger than the storage capacity of the memory, part of the second type data in the second type hard disk may be synchronously fetched to the memory in another working node for reading and writing operations, so that the data processing efficiency of the entire system may be improved in a unit time.
In some embodiments, the scheduling node establishes a network connection with a client, and is configured to receive a read request sent by the client; when the client requests to read one or more complete data blocks according to the reading request, selecting an idle working node to respond to the reading request according to the load rate of the working node; and/or when the client requests to read partial data in one data block according to the reading request, sending a pause instruction and a reading instruction to a working node responding to the reading request; and the pause instruction is used for controlling the working node to pause the caching of the data in the memory and returning the read data to the client side corresponding to the read instruction.
In this exemplary embodiment, when the scheduling node obtains a plurality of read requests sent by the client, the read requests may be sorted from high to low according to the read-write frequency of data to be read in the read request instruction, and then the scheduling node sequentially reads corresponding data according to the sorted read request instruction. In the storage system, data are classified and stored according to the read-write frequency, and the data with high read-write frequency are preferentially stored in the memory. Therefore, when reading is performed in the order of the read-write frequency from high to low, data can be read directly from the memory preferentially, and the data reading efficiency of the system can be improved as a whole.
In the exemplary embodiment, the scheduling node may establish a network connection with the client, and receive the read request sent by the client through the network connection. The read requests sent by the client may include a read request to read one or more complete data blocks, and a read request to read partial data in one data block.
In this exemplary embodiment, when the scheduling node receives a read request of one or more complete data blocks, the scheduling node analyzes the data load rate of each working node, and selects an idle working node to respond to the read request, or a working node with a larger amount of idle data storage space in the same load rate to respond to the read request, so as to implement load balancing of each working node while completing the response of the read request.
In the present exemplary embodiment, further includes: when the reading request determines that the client requests to read part of data in one data block or reads the memory in a sequence other than a preset sequence, the working node is controlled to pause the cache data in the memory through a pause instruction, and the read data is returned to the client in response to the reading instruction. For example, when a working node is caching data in a memory, and receives a request sent by a client for reading the memory from a data block or in an order other than a predetermined order, the working node preferentially processes the request of the client and suspends an ongoing storage task, so that the client can timely process the request for reading the memory from the data block or in the order other than the predetermined order.
In some embodiments, the scheduling node is further configured to receive a request for applying for a storage space from the client; according to the surplus of the memory in the working node, when the memory space amount in the working node is determined to be not less than the requested space capacity, the surplus memory resources are allocated to the client; and/or when the memory space amount in the working node is determined to be smaller than the requested space capacity, transferring the first type of data with the lowest memory access frequency to the first type of hard disk, releasing part of the memory space of the memory, and allocating the memory space released by the memory to the client.
In this exemplary embodiment, the scheduling node may further receive a request for applying for the storage space sent by the client. After receiving the request for applying the storage space, the scheduling node compares the residual storage space amount in the memory space of each working node with the application amount of the storage space, and preferentially allocates the memory space, which is not occupied by the data, of the working node meeting the application amount of the storage space to the client. If the residual storage spaces of the memory spaces in all the working nodes are not satisfied, transferring the first class data with the lowest access frequency in the memory of one of the working nodes to the first class hard disk, releasing the storage space of part of the memory, and allocating the storage space released by the memory to the client so as to satisfy the storage space application of the client.
In some embodiments, the scheduling node is further configured to split a plurality of data blocks of a super data block whose data size is greater than a second threshold, and schedule memories in a plurality of working nodes to store the different split data blocks respectively.
In this exemplary embodiment, the second threshold is a preset data splitting value determined according to the capacity of the memory space of the working node. And judging the data quantity of the data block to be a super data block when the data quantity of the data block exceeds a second threshold value. At this time, the super data block is divided into a plurality of data blocks with small data volume, and the plurality of data blocks are stored in the memory space of a plurality of working nodes, so that the super data block with large data volume is stored.
The data blocks can be stored in the memory spaces of the working nodes by performing queue sorting on the data blocks according to mutual connection of the data, and the data blocks are sequentially and respectively stored in the storage spaces of the working nodes with the same queue sorting according to the sequence. The storage space of the queue-sorted working node may be a memory space adjacent to each other in the plurality of working nodes.
In some embodiments, the scheduling node is further configured to, when receiving a read request for reading the super data block by the client, split the read request into a plurality of subtasks, and schedule the plurality of working nodes to respond to the subtasks respectively; and aggregating the data blocks read by the multiple working nodes in an aggregation queue and returning the aggregated data blocks to the client.
In this exemplary embodiment, the scheduling node may also receive a read request sent by the client to read the super data block. And meanwhile, the reading request is split into a plurality of subtasks, a plurality of work node response subtasks are respectively scheduled according to each subtask so as to read the data to be read in the storage space of each corresponding work node, and the read data are aggregated through an aggregation queue so as to realize the response of the reading request of the super data block sent by the client. Wherein the aggregation queue may be a fork-aggregation queue. Therefore, the reading request is branched into a plurality of subtasks, the subtasks are distributed to a plurality of working nodes, all reading results are combined into a source file to finish reading of the super-large data, and the reading efficiency of the super-data blocks is improved by ensuring the reading efficiency of the data blocks with various sizes.
The splitting and the aggregation of the super data blocks are a forward and reverse process, and the splitting of the super data blocks can be sequentially and directly split into a plurality of data blocks according to the data arrangement sequence in the super data for orderly storage. Meanwhile, when the data are called and aggregated through a plurality of subtasks, ordered aggregation can be performed through the splitting sequence of the super data blocks. For example, one super block is sequentially divided into first to fifth blocks in order, and when data is aggregated, sequential aggregation may be performed in order from first to fifth blocks.
The present disclosure also provides a data processing method.
FIG. 2 is a flow diagram illustrating a method of data processing according to an example embodiment. As shown in fig. 2, a data processing method provided in the embodiment of the present disclosure is applied to a scheduling node of the data storage system, and the method includes:
In the exemplary embodiment, data is divided into first type data, second type data and third type data according to the read-write frequency of the data, and the first type data is stored in the memory of the working node; storing part of the first type data and/or the second type data to a first type hard disk of the working node; and storing the third type of data to a second type of hard disk of the working node, wherein the reading and writing frequency of the third type of data is lower than that of the second type of data, and the reading and writing frequency of the second type of data is lower than that of the first type of data.
Because the system directly reads and writes the data in the memory when reading and writing the data, the data stored in the hard disk is called based on the memory, and then the data is read and written. Therefore, the first type of data which is frequently read and written and has high frequency is stored in the memory, so that when the system reads and writes a large amount of data in the data application, the data which is frequently called and applied with high frequency can be conveniently called from the memory directly, and the processing efficiency of the system on the hot spot data is improved.
In some embodiments, said scheduling the first type of hard disk of the working node to store the portion of the first type of data and/or the second type of data comprises:
storing the first type of data with the stored data volume larger than a first threshold value into the first type of hard disk, wherein the first threshold value is a preset value determined according to the capacity of the memory space of the working node;
and storing the second type of data to the first type of hard disk.
In the present exemplary embodiment, since the storage capacity of the memory is limited, when the data amount of the first type data is greater than the first threshold, the first type data that is not stored in the memory may be stored in the first type hard disk. The first threshold is a preset value determined according to the capacity of the memory space of the working node. This preset value can be set within the system. When the data quantity is compared, the system can automatically call the threshold value, so that the storage space allocation of the data is convenient for data storage through threshold value comparison.
In some embodiments, the dividing the data into a first type of data, a second type of data, and a third type of data according to the read-write frequency of the data includes:
the scheduling node dynamically divides the data into the first class of data, the second class of data and the third class of data according to the reading and writing frequency of the data stored in the storage system;
the method further comprises the following steps:
and the scheduling node schedules the memory, the first class of hard disk and the second class of hard disk of one or more working nodes according to the data type divided by control, and stores the first class of data, the second class of data and the third class of data respectively.
In the present exemplary embodiment, a scheduling node, which may be a controller, is also included in the data storage system. The scheduling node divides the data into a first class of data, a second class of data and a third class of data according to the read-write frequency of the stored data, and stores the data of each class in the memory, the first class of hard disk and the second class of hard disk respectively. For example, the first type of data is stored in the memory, the second type of data and the first type of data that is not stored in the memory are stored in the first type of hard disk, and the third type of data is stored in the third type of hard disk.
In some embodiments, further comprising:
the scheduling node receives a reading request sent by a client;
when the client requests to read one or more complete data blocks according to the reading request, selecting an idle working node to respond to the reading request according to the load rate of the working node;
and/or the presence of a gas in the gas,
when the client requests to read partial data in one data block according to the reading request, sending a pause instruction and a reading instruction to a working node responding to the reading request; the suspension instruction is used for controlling the working node to suspend caching data in the memory, and returning the read data to the client side corresponding to the reading instruction.
In the exemplary embodiment, the scheduling node may establish a network connection with the client, and receive the read request sent by the client through the network connection. The read requests sent by the client may include read requests to read one or more complete data blocks, read requests to read partial data in one data block, and/or a predetermined order of data read requests.
In the exemplary embodiment, when the scheduling node receives a read request of one or more complete data blocks, the scheduling node analyzes the data load rate of each working node, selects an idle working node to respond to the read request, or selects a working node with a larger data storage space redundancy in the same load rate to respond to the read request, so as to improve the processing efficiency of the system on the client data read request.
In some embodiments, further comprising:
receiving a request for applying for a storage space sent by the client;
according to the surplus of the memory in the working node, when the memory space amount in the working node is determined to be not less than the requested space capacity, the surplus memory resources are allocated to the client;
and/or the presence of a gas in the gas,
and according to the residual amount of the memory in the working node, when the memory space amount in the working node is determined to be smaller than the requested space capacity, controlling the working node to transfer the first type of data with the lowest access frequency in the memory to the first type of hard disk, releasing part of the storage space of the memory, and distributing the storage space released by the memory to the client.
In this exemplary embodiment, the scheduling node may further receive a request for applying for the storage space sent by the client. After receiving the request for applying the storage space, the scheduling node compares the residual storage space amount in the memory space of each working node with the application amount of the storage space, and preferentially allocates the memory space, which is not occupied by the data, of the working node meeting the application amount of the storage space to the client. If the residual storage spaces of the memory spaces in all the working nodes are not satisfied, transferring the first class data with the lowest access frequency in the memory of one of the working nodes to the first class hard disk, releasing the storage space of part of the memory, and allocating the storage space released by the memory to the client so as to satisfy the storage space application of the client.
In some embodiments, further comprising:
splitting a plurality of data blocks of the super data blocks with the data volume larger than a second threshold value;
and scheduling the memories in the working nodes to respectively store the different split data blocks.
In this exemplary embodiment, the second threshold is a preset data splitting value determined according to the capacity of the memory space of the working node. And judging the data quantity of the data block to be a super data block when the data quantity of the data block exceeds a second threshold value. At this time, the super data block is divided into a plurality of data blocks with small data volume, and the plurality of data blocks are stored in the memory space of a plurality of working nodes, so that the super data block with large data volume is stored.
The data blocks can be stored in the memory spaces of the working nodes by performing queue sorting on the data blocks according to mutual connection of the data, and the data blocks are sequentially and respectively stored in the storage spaces of the working nodes with the same queue sorting according to the sequence. The storage space of the queue-sorted working node may be a memory space adjacent to each other in the plurality of working nodes.
In some embodiments, further comprising:
when a reading request for reading the super data block by the client is received, splitting the reading request into a plurality of subtasks;
respectively scheduling a plurality of the working nodes to respond to the subtasks;
and aggregating the data blocks read by the multiple working nodes in an aggregation queue, and returning the aggregated data blocks to the client.
In this exemplary embodiment, the scheduling node may also receive a read request sent by the client to read the super data block. And meanwhile, the reading request is split into a plurality of subtasks, a plurality of work node response subtasks are respectively scheduled according to each subtask so as to read the data to be read in the storage space of each corresponding work node, and the read data are aggregated through an aggregation queue so as to realize the response of the reading request of the super data block sent by the client. Wherein the aggregation queue may be a fork-aggregation queue. Therefore, the reading request is branched into a plurality of subtasks, the subtasks are distributed to a plurality of working nodes, all reading results are combined into a source file to finish reading of the super-large data, and the reading efficiency of the super-data blocks is improved by ensuring the reading efficiency of the data blocks with various sizes.
The present disclosure also provides a terminal. The terminal provided by the embodiment of the disclosure comprises: a processor and a memory for storing a computer program capable of running on the processor, wherein the processor is configured to execute steps of a data processing method provided by another aspect of the present disclosure when the computer program is run.
The present disclosure also provides a computer-readable storage medium. The computer readable storage medium provided by the embodiments of the present disclosure has a computer program stored thereon, and the computer program, when executed by a processor, implements the steps of the data processing method provided by the embodiments of the present disclosure in another aspect.
The disclosure also provides a selective hotspot segmentation cache scheduling method. Fig. 3 is a flowchart illustrating a method of selective hotspot splitting cache scheduling in accordance with an exemplary embodiment. As shown in fig. 3, the method includes:
step 1: collecting all logs in the cluster server, reading information of data access by setting specific span date, sorting log records according to categories, filtering out logs irrelevant to read-write operation, and only keeping operation logs for data read-write.
Step 2: and counting the read-write operation logs of all the data in the cluster, and recording the size of the high-frequency read-write access data. FIG. 4 is a graph illustrating data size and probability distribution of cold and hot data in a production cluster using statistics of the present technique, according to an exemplary embodiment. As shown, the frequency read and write probability distribution of the hot and cold data in the production cluster is shown.
And step 3: and analyzing the production cluster logs to obtain frequently read-write access files, and sequencing and numbering the frequently read-write access files according to the data size. All data cannot be simultaneously placed in the memory for quick reading, and data storage is carried out in a mode of layering physical storage media.
In this embodiment, a hierarchical big data storage system can be configured. FIG. 5 is a schematic diagram illustrating a tiered big data storage system architecture, according to an example embodiment. As shown in fig. 5, each working node in the big data storage system server is divided into a three-layer storage system of memory-solid state disk-hard disk, hotter data is preferentially placed in the 3-layer storage system, and the rest of the data is placed in the underlying file system. The hard disk here may be a mechanical hard disk.
Setting the data with higher occurrence frequency as hot spot data according to the statistics of the data reading and writing frequency in the step 2, sorting the files according to the occurrence frequency counted in the step 2, wherein the files are ranked according to the { N1, N2 and N3 … … Nn }, generating a file probability table Tn, and counting the total size Na of all N files so as to obtain the total size Na of all N files
And calculating the heat probability of each file as { P1, P2, P3 … … Pn }, and generating a data heat probability table Tp, wherein the higher the heat probability represents the higher the reading and writing frequency of the data application. The hot data read and written in the memory, the solid state disk and the hard disk execute the following rules:
rule 1: the memory layer preferentially places the data of the type which is arranged earlier in the heat probability table Tn, when the data of the type is read and written in a large quantity and the head of the data block is overlarge, the file probability table Tn in the step 2 is updated, and if the head of the data block of the rearranged hot spot data queue is larger than the head of the data block before updating, the step 4 is executed to segment the hot spot data;
rule 2: the SSD stores two types of data, the first type is data with a front occurrence frequency but not placed in the rule 1, and the second type is data with an overlarge data block, which cannot be read and written into the memory at one time and cannot meet the requirement of executing segmentation in the step 4.
With the continuous reading and writing of data, when the access frequency of the head data in the memory is very low, the data is stored and transferred to the solid state disk continuously in the rear of the file probability table Tn, and the file probability table Tn sequences the data padding bits in the solid state disk to enter the memory, so that the replacement of cold data in the memory and hot data in the solid state disk is completed.
Rule 3: the same as the rule 2, the data with lower heat are stored in the hard disk, and when the frequency of the data in the solid state disk is lower, the data are transferred to the hard disk. Meanwhile, the hard disk constantly receives data which only needs to be read and is not written in the memory, the hard disk layer interacts with the bottom file system, data synchronization among different Worker is completed, and the consistency of bottom data is guaranteed.
And 4, step 4: the file segmentation follows the following rules:
rule 1: and setting different data partition units according to the heat probability prediction of data in the memory-solid state disk-hard disk drive. For example, the file a has a file size SA, PA is the statistical probability of the data heat probability table Tp in step 3, and the expected load amount of the file a is set to LA × SA. LA is taken as an integral data block as the minimum data unit of the file A, and the file A is divided by LA data size.
Rule 2: and (3) requesting the underlying storage system to read the data block into the memory by the Client (Client) for the data which is not in the memory-solid state disk-hard disk drive, assigning a probability P1 to the data, and performing the partitioning operation according to the method of the rule 1 in the step 4.
And 5: the method for transferring the cache overhead from the Client to the Worker is used, the Client reads data preferentially, and a mode that the Worker reads the data in parallel with the Client is not used, so that the problem that the load of sending the data to the Worker is too high when the Client reads the data is solved, and the use rule is as follows:
rule 1: when the Client reads a complete data block, the Master node judges an idle Worker node according to the data heat probability table Tp, assists the Client to accumulate complete data from the memory-solid state disk-hard disk drive and the bottom file system, and the Worker node directly releases the data block after the Client reads the whole data block in sequence.
Rule 2: when the Client reads only a part of data or reads data in a memory-solid state disk-hard disk drive in a non-sequential mode, a Worker related to the Client to read the data abandons the data caching work, a completion command is sent to the corresponding Worker after the Client completely reads the data, and the Worker node continues to perform the data caching work or request the data work from the Worker.
Step 6: when the Client reads the data in the step 5, the Worker only performs asynchronous data reading and caching operation, and no matter a complete or partial data block is read, the performance of the Client is not influenced in the memory-solid state disk-hard disk drive. In order to ensure that the situation of insufficient resource space does not occur in the memory layer, the read-write of the hot data in the memory is monitored in real time. FIG. 6 is a schematic diagram illustrating a hierarchical big data storage system cache scheduling, according to an example embodiment. As shown in fig. 6, the following rules are observed when the Client initiates a request for space application to the Worker:
rule 1: and if the memory layer of the Worker has enough residual storage space, directly allocating space to the Client to be used by the Client in a scheduling mode.
Rule 2: the Worker's memory layer does not have enough space left, and replaces part of the data blocks to the solid state hard disk layer according to the data heat probability table Tp, and the solid state hard disk layer and the hard disk drive simultaneously obey the rule.
And 7: when the segmentation operation in the step 4 is executed, the fact that the file is not suitable for being infinitely segmented into small blocks with any granularity in practical engineering application is considered, and when the erasure codes are used for coding a large file into a plurality of small data blocks (including information blocks and check blocks), the Master monitors in real time to limit the segmented file. FIG. 7 is a file splitting diagram shown in accordance with an example embodiment. As shown in fig. 7, the divided minimum file may not be smaller than 10 MB. The partitioned data are stored in the adjacent storage space of the original data, and consistency is guaranteed when the partitioned data are read in a blocking mode. For example, when the first to fourth storage spaces adjacent to each other store data files, a 1000Mbps file needs to be stored in the first storage space, and a 300Mbps file needs to be stored in the second storage space. Due to the limitation of the capacity of the storage space, the 1000Mbps file cannot be stored or is difficult to store. At this time, the 1000Mbps file may be divided into four 250 Mbps-sized files, which are stored in the first to fourth storage spaces adjacent to each other, respectively. Since the file of 300Mbps is relatively small, it can be directly stored in the second storage space. When storing, the second storage space may store the 300Mbps file first, and then store the 250Mbps split file.
And 8: FIG. 8 is a diagram illustrating a fork-aggregate queue model in accordance with an exemplary embodiment. As shown in fig. 8, when a data block is very large, and cannot be read by one memory alone, and multiple memories need to be read cooperatively, a large file is divided and then processed in a form of a fork-aggregate queue.
The Client reading request is branched into a plurality of subtasks, the subtasks are distributed to the memory nodes for storing the files, the memory nodes combine all reading results into a source file to finish reading of the extra-large data, and the reading efficiency of data blocks of various sizes is guaranteed.
The layered data storage system of the present disclosure has the following advantages:
1. the problem of read-write operation bottleneck of a distributed file system (HDFS) based on disk storage in a large-scale data scene is solved, hot data are preferentially placed in a memory according to statistical probability by using a hierarchical data storage structure, and real-time access efficiency of upper-layer application operation data is guaranteed.
2. An extensible hierarchical data use method is provided on the basis of a data storage system, frequently processed data are loaded into a memory to serve as a main target of data reading and writing, a data heat degree reference table is generated through data reading and writing frequency, a hot data partitioning strategy is selectively carried out, and big data application of an upper layer is accelerated.
The Master-Worker starts two processes to work together: the system comprises a Master and a Worker process, wherein the Master is responsible for receiving and distributing tasks, the Worker is responsible for executing specific subtasks, each Worker returns a result to the Master after executing the tasks, and the Master collects the results.
And the Client is used for creating tasks to the Master.
The Master accepts the tasks and assigns the tasks to the Worker.
Master and Worker are classes used in Spark independent clusters, Master is a controller of each independent cluster, and Worker is an executor, and one Spark independent cluster needs to start one Master and a plurality of workers.
The data block is the smallest data storage unit in the bai database, and dao is read from the data block when the user du performs input and output operations on the database zhi. In oracle11g, the data block is divided into a header information area and a storage area, the header information area does not store data, but the storage area really stores data records as a guide to store related information about table data and index data in the storage area.
In the several embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. The above-described device embodiments are merely illustrative, for example, the division of the unit is only a logical functional division, and there may be other division ways in actual implementation, such as: multiple units or components may be combined, or may be integrated into another system, or some features may be omitted, or not implemented. In addition, the coupling, direct coupling or communication connection between the components shown or discussed may be through some interfaces, and the indirect coupling or communication connection between the devices or units may be electrical, mechanical or other forms.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed on a plurality of network units; some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, all the functional units in the embodiments of the present invention may be integrated into one processing module, or each unit may be separately used as one unit, or two or more units may be integrated into one unit; the integrated unit can be realized in a form of hardware, or in a form of hardware plus a software functional unit.
In some cases, any two of the above technical features may be combined into a new method solution without conflict.
In some cases, any two of the above technical features may be combined into a new device solution without conflict.
Those of ordinary skill in the art will understand that: all or part of the steps for implementing the method embodiments may be implemented by hardware related to program instructions, and the program may be stored in a computer readable storage medium, and when executed, the program performs the steps including the method embodiments; and the aforementioned storage medium includes: various media capable of storing program codes, such as a removable Memory device, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, and an optical disk.
The above description is only for the specific embodiments of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present invention, and all the changes or substitutions should be covered within the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the appended claims.
Claims (15)
1. A data storage system, the system comprising:
at least one working node;
the work node includes:
the memory is used for storing the first type of data;
the first type hard disk is used for storing part of the first type data and/or the second type data; the reading and writing frequency of the second type of data is lower than that of the first type of data;
the second type of hard disk is used for storing third type of data; the reading rate of the first type of hard disk is higher than that of the second type of hard disk; and the read-write frequency of the third type of data is lower than that of the second type of data.
2. The system of claim 1, wherein the first type of hard disk comprises: a solid state disk; the second type of hard disk comprises: mechanical hard disk.
3. The system according to claim 1 or 2, wherein the first type of hard disk is specifically configured to store the first type of data and the second type of data, the data amount of which is greater than a first threshold, where the first threshold is a preset value determined according to a capacity of a memory space of the working node.
4. The system according to claim 1 or 2, characterized in that the system further comprises:
the scheduling node is used for dynamically dividing data into the first class of data, the second class of data and the third class of data according to the reading and writing frequency of the data stored in the storage system; and scheduling the memory, the first class of hard disk and the second class of hard disk of one or more working nodes to store the first class of data, the second class of data and the third class of data respectively according to the data types divided by control.
5. The system of claim 4,
the scheduling node is connected with a client through a network and used for receiving a reading request sent by the client; when the client requests to read one or more complete data blocks according to the reading request, selecting an idle working node to respond to the reading request according to the load rate of the working node; and/or when the client requests to read partial data in one data block according to the reading request, sending a pause instruction and a reading instruction to a working node responding to the reading request; and the pause instruction is used for controlling the working node to pause the caching of the data in the memory and returning the read data to the client side corresponding to the read instruction.
6. The system of claim 4, wherein the scheduling node is further configured to receive a request for storage space from the client; according to the surplus of the memory in the working node, when the memory space amount in the working node is determined to be not less than the requested space capacity, the surplus memory resources are allocated to the client; and/or when the memory space amount in the working node is determined to be smaller than the requested space capacity, transferring the first type of data with the lowest memory access frequency to the first type of hard disk, releasing part of the memory space of the memory, and allocating the memory space released by the memory to the client.
7. The system according to claim 4, wherein the scheduling node is further configured to split a plurality of data blocks of the super data block whose data size is greater than the second threshold, and schedule memories in a plurality of working nodes to store different split data blocks respectively.
8. The system according to claim 7, wherein the scheduling node is further configured to, when receiving a read request for reading the super data block by the client, split the read request into a plurality of subtasks, and schedule the plurality of working nodes to respond to the subtasks respectively; and aggregating the data blocks read by the multiple working nodes in an aggregation queue and returning the aggregated data blocks to the client.
9. A data processing method applied to a scheduling node of a data storage system, the method comprising:
dividing the data into first class data, second class data and third class data according to the reading and writing frequency of the data; the reading and writing frequency of the second type of data is lower than that of the first type of data; the read-write frequency of the third type of data is lower than that of the second type of data;
the memory of the scheduling working node stores the first type of data;
scheduling a first type hard disk of the working node to store part of the first type data and/or the second type data;
scheduling a second type of hard disk of the working node to store the third type of data; and the reading rate of the first type of hard disk is higher than that of the second type of hard disk.
10. The method of claim 9, wherein said scheduling the first type of hard disk of the worker node to store a portion of the first type of data and/or the second type of data comprises:
storing the first type of data with the stored data volume larger than a first threshold value into the first type of hard disk, wherein the first threshold value is a preset value determined according to the capacity of the memory space of the working node;
and storing the second type of data to the first type of hard disk.
11. The method according to claim 9 or 10, wherein the dividing the data into the first type of data, the second type of data, and the third type of data according to the read-write frequency of the data comprises:
the scheduling node dynamically divides the data into the first class of data, the second class of data and the third class of data according to the reading and writing frequency of the data stored in the storage system;
the method further comprises the following steps:
and the scheduling node schedules the memory, the first class of hard disk and the second class of hard disk of one or more working nodes according to the data type divided by control, and stores the first class of data, the second class of data and the third class of data respectively.
12. The method of claim 11, further comprising:
the scheduling node receives a reading request sent by a client;
when the client requests to read one or more complete data blocks according to the reading request, selecting an idle working node to respond to the reading request according to the load rate of the working node;
and/or the presence of a gas in the gas,
when the client requests to read partial data in one data block according to the reading request, sending a pause instruction and a reading instruction to a working node responding to the reading request; the suspension instruction is used for controlling the working node to suspend caching data in the memory, and returning the read data to the client side corresponding to the reading instruction.
13. The method of claim 11, further comprising:
receiving a request for applying for a storage space sent by the client;
according to the surplus of the memory in the working node, when the memory space amount in the working node is determined to be not less than the requested space capacity, the surplus memory resources are allocated to the client;
and/or the presence of a gas in the gas,
and according to the residual amount of the memory in the working node, when the memory space amount in the working node is determined to be smaller than the requested space capacity, controlling the working node to transfer the first type of data with the lowest access frequency in the memory to the first type of hard disk, releasing part of the storage space of the memory, and distributing the storage space released by the memory to the client.
14. A terminal, comprising: a processor and a memory for storing a computer program operable on the processor, wherein the processor is operable to perform the steps of the method of claims 9 to 13 when executing the computer program.
15. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the method of claims 9 to 13.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010763506.3A CN114063888A (en) | 2020-07-31 | 2020-07-31 | Data storage system, data processing method, terminal and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010763506.3A CN114063888A (en) | 2020-07-31 | 2020-07-31 | Data storage system, data processing method, terminal and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114063888A true CN114063888A (en) | 2022-02-18 |
Family
ID=80227934
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010763506.3A Pending CN114063888A (en) | 2020-07-31 | 2020-07-31 | Data storage system, data processing method, terminal and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114063888A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116303119A (en) * | 2023-05-19 | 2023-06-23 | 珠海妙存科技有限公司 | Method, system and storage medium for identifying cold and hot data |
| CN116339561A (en) * | 2023-03-29 | 2023-06-27 | 郑州裕中能源有限责任公司 | Three-dimensional visualization method, system and equipment for boiler monitoring |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN2791752Y (en) * | 2005-04-04 | 2006-06-28 | 苏州鹞鹰数据技术有限公司 | High-speed data storage apparatus |
| CN1852318A (en) * | 2006-04-19 | 2006-10-25 | 华中科技大学 | Distributed multi-stage buffer storage system suitable for object network storage |
| CN106095572A (en) * | 2016-06-08 | 2016-11-09 | 东方网力科技股份有限公司 | The Dispatching System of a kind of big data process and method |
| CN106648442A (en) * | 2015-10-29 | 2017-05-10 | 阿里巴巴集团控股有限公司 | Metadata node internal memory mirroring method and device |
| CN106776368A (en) * | 2016-11-29 | 2017-05-31 | 郑州云海信息技术有限公司 | Buffer memory management method, apparatus and system during a kind of digital independent |
| CN107273040A (en) * | 2016-04-08 | 2017-10-20 | 北京优朋普乐科技有限公司 | data cache method and device |
| CN107506146A (en) * | 2017-08-29 | 2017-12-22 | 郑州云海信息技术有限公司 | A kind of data-storage system |
| CN109614372A (en) * | 2018-10-26 | 2019-04-12 | 阿里巴巴集团控股有限公司 | A kind of object storage, read method, device and service server |
| CN111176584A (en) * | 2019-12-31 | 2020-05-19 | 曙光信息产业(北京)有限公司 | Data processing method and device based on hybrid memory |
| CN111208952A (en) * | 2020-02-13 | 2020-05-29 | 河南文正电子数据处理有限公司 | Storage system capacity expansion method, readable storage medium and computing device |
-
2020
- 2020-07-31 CN CN202010763506.3A patent/CN114063888A/en active Pending
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN2791752Y (en) * | 2005-04-04 | 2006-06-28 | 苏州鹞鹰数据技术有限公司 | High-speed data storage apparatus |
| CN1852318A (en) * | 2006-04-19 | 2006-10-25 | 华中科技大学 | Distributed multi-stage buffer storage system suitable for object network storage |
| CN106648442A (en) * | 2015-10-29 | 2017-05-10 | 阿里巴巴集团控股有限公司 | Metadata node internal memory mirroring method and device |
| CN107273040A (en) * | 2016-04-08 | 2017-10-20 | 北京优朋普乐科技有限公司 | data cache method and device |
| CN106095572A (en) * | 2016-06-08 | 2016-11-09 | 东方网力科技股份有限公司 | The Dispatching System of a kind of big data process and method |
| CN106776368A (en) * | 2016-11-29 | 2017-05-31 | 郑州云海信息技术有限公司 | Buffer memory management method, apparatus and system during a kind of digital independent |
| CN107506146A (en) * | 2017-08-29 | 2017-12-22 | 郑州云海信息技术有限公司 | A kind of data-storage system |
| CN109614372A (en) * | 2018-10-26 | 2019-04-12 | 阿里巴巴集团控股有限公司 | A kind of object storage, read method, device and service server |
| CN111176584A (en) * | 2019-12-31 | 2020-05-19 | 曙光信息产业(北京)有限公司 | Data processing method and device based on hybrid memory |
| CN111208952A (en) * | 2020-02-13 | 2020-05-29 | 河南文正电子数据处理有限公司 | Storage system capacity expansion method, readable storage medium and computing device |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116339561A (en) * | 2023-03-29 | 2023-06-27 | 郑州裕中能源有限责任公司 | Three-dimensional visualization method, system and equipment for boiler monitoring |
| CN116339561B (en) * | 2023-03-29 | 2024-03-15 | 郑州裕中能源有限责任公司 | Three-dimensional visualization method, system and equipment for boiler monitoring |
| CN116303119A (en) * | 2023-05-19 | 2023-06-23 | 珠海妙存科技有限公司 | Method, system and storage medium for identifying cold and hot data |
| CN116303119B (en) * | 2023-05-19 | 2023-08-11 | 珠海妙存科技有限公司 | Method, system and storage medium for identifying cold and hot data |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2021208546A1 (en) | Multi-dimensional resource scheduling method in kubernetes cluster architecture system | |
| US10185592B2 (en) | Network storage device using dynamic weights based on resource utilization | |
| US10534542B2 (en) | Dynamic core allocation for consistent performance in a non-preemptive scheduling environment | |
| US10853139B2 (en) | Dynamic workload management based on predictive modeling and recommendation engine for storage systems | |
| CN100407152C (en) | Methods and systems for multi-policy resource scheduling | |
| JP5500256B2 (en) | Storage system | |
| EP4160405A1 (en) | Task execution method and storage device | |
| CN108932150B (en) | Caching method, device and medium based on SSD and disk hybrid storage | |
| CN108595254B (en) | Query scheduling method | |
| US20170010919A1 (en) | Dynamic weight accumulation for fair allocation of resources in a scheduler hierarchy | |
| CN114063888A (en) | Data storage system, data processing method, terminal and storage medium | |
| CN111737168A (en) | Cache system, cache processing method, device, equipment and medium | |
| CN114880130A (en) | Method, system, device and storage medium for breaking memory limitation in parallel training | |
| US20210390405A1 (en) | Microservice-based training systems in heterogeneous graphic processor unit (gpu) cluster and operating method thereof | |
| WO2021115082A1 (en) | Job scheduling method and job scheduling apparatus | |
| Wang et al. | An Improved Memory Cache Management Study Based on Spark. | |
| WO2011078162A1 (en) | Scheduling device, scheduling method and program | |
| Kumar et al. | An extended approach to Non-Replicated dynamic fragment allocation in distributed database systems | |
| Soosai et al. | Dynamic replica replacement strategy in data grid | |
| Mathiason et al. | Virtual full replication by adaptive segmentation | |
| CN111338782A (en) | Node allocation method based on competition perception and oriented to shared burst data caching | |
| JP2012038275A (en) | Transaction calculation simulation system, method, and program | |
| Xiu et al. | A memory management strategy based on task requirement for in-memory computing | |
| Luo et al. | Towards efficiently supporting database as a service with QoS guarantees | |
| CN115834669B (en) | Distributed AI service engine system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |