CN111562884B - Data storage method and device and electronic equipment - Google Patents
Data storage method and device and electronic equipment Download PDFInfo
- Publication number
- CN111562884B CN111562884B CN202010348430.8A CN202010348430A CN111562884B CN 111562884 B CN111562884 B CN 111562884B CN 202010348430 A CN202010348430 A CN 202010348430A CN 111562884 B CN111562884 B CN 111562884B
- Authority
- CN
- China
- Prior art keywords
- storage medium
- data
- candidate
- weight
- state information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 50
- 238000013500 data storage Methods 0.000 title claims abstract description 27
- 230000002596 correlated effect Effects 0.000 claims abstract description 16
- 238000004891 communication Methods 0.000 claims description 18
- 238000004364 calculation method Methods 0.000 claims description 13
- 238000004590 computer program Methods 0.000 claims description 9
- 238000007405 data analysis Methods 0.000 claims description 5
- 239000002699 waste material Substances 0.000 abstract description 6
- 238000004422 calculation algorithm Methods 0.000 description 6
- 230000000875 corresponding effect Effects 0.000 description 6
- 230000008569 process Effects 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 238000012545 processing Methods 0.000 description 3
- 230000003247 decreasing effect Effects 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 238000013459 approach Methods 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 230000007423 decrease Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000000802 evaporation-induced self-assembly Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 230000005012 migration Effects 0.000 description 1
- 238000013508 migration Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0614—Improving the reliability of storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0629—Configuration or reconfiguration of storage systems
- G06F3/0631—Configuration or reconfiguration of storage systems by allocating resources to storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention provides a data storage method and device and electronic equipment. Wherein the method comprises the following steps: acquiring state information of each candidate storage medium in a storage system; setting weights of the respective storage media according to the usage rates indicated by the state information of the respective candidate storage media, wherein the weights are inversely related to the usage rates indicated by the state information; selecting one candidate storage medium from the plurality of candidate storage media, wherein the probability of each candidate storage medium being selected is positively correlated with the weight of the candidate storage medium; and storing the data to be stored into the selected candidate storage medium. The probability that the stored data to be stored is distributed to the storage medium for storage can be dynamically adjusted according to the utilization rate of the storage space of each storage medium, so that the stored data of each storage medium is close to balance, further load balancing among the storage media in the storage system is realized, and waste of storage resources is reduced.
Description
Technical Field
The present invention relates to the field of cloud storage technologies, and in particular, to a data storage method, a data storage device, and an electronic device.
Background
The storage system may include a plurality of storage media in which data is stored, respectively. For example, when new data needs to be stored, a preset random algorithm may be used to randomly select a storage medium from a plurality of disks in the storage system, and store the new data into the selected storage medium.
However, the limited randomness of the restricted random algorithm may result in a probability that a portion of the storage medium is selected higher than other storage media, thus resulting in a greater amount of data stored in the portion of the storage medium than in the other storage media. In the related art, the storage system often uses the utilization rate of the storage medium with the highest utilization rate as the utilization rate of the storage space in the storage system. Therefore, the data is unevenly distributed in each storage medium, which may cause that a storage medium with more residual storage space exists in the storage system, but new data cannot be continuously stored, namely, the waste of storage resources is caused.
Disclosure of Invention
The embodiment of the invention aims to provide a data storage method, a data storage device and electronic equipment, so as to realize load balancing among storage media in a storage system and reduce waste of storage resources. The specific technical scheme is as follows:
in a first aspect of an embodiment of the present invention, there is provided a data storage method, the method including:
acquiring state information of each candidate storage medium in a storage system, wherein the state information is used for representing the utilization rate of the storage medium;
setting weights of the respective storage media according to the usage rates indicated by the state information of the respective candidate storage media, wherein the weights are inversely related to the usage rates indicated by the state information;
selecting one candidate storage medium from the plurality of candidate storage media, wherein the probability of each candidate storage medium being selected is positively correlated with the weight of the candidate storage medium;
and storing the data to be stored into the selected candidate storage medium.
In a possible embodiment, before said setting of the weight of each of said storage media according to the usage indicated by said status information of each of said candidate storage media, said method further comprises:
calculating the data unbalance degree of the storage system according to the utilization rate represented by the state information of each candidate storage medium, wherein the data unbalance degree is used for representing the deviation degree between the utilization rates of each storage medium;
the setting of the weight of each storage medium according to the usage rate indicated by the status information of each storage medium candidate includes:
and if the data unbalance degree is larger than a preset unbalance degree threshold value, setting the weight of each storage medium according to the use rate indicated by the state information of each candidate storage medium.
In one possible embodiment, the calculating the data unbalance degree of the storage system according to the usage rate indicated by the status information of each of the candidate storage mediums includes:
and calculating a difference between a maximum value and an average value of the usage rates indicated by the state information of the candidate storage media according to the usage rates indicated by the state information of the candidate storage media, and taking the difference as the data unbalance degree of the storage system.
In one possible embodiment, the method further comprises:
and if the data unbalance degree is not greater than a preset unbalance degree threshold value, selecting one candidate storage medium from the plurality of candidate storage media, wherein the probability of each candidate storage medium being selected is positively correlated with the preset weight of the candidate storage medium.
In a possible embodiment, the status information is further used to represent an access frequency of the storage medium;
the setting of the weight of each storage medium according to the usage rate indicated by the status information of each storage medium candidate includes:
a weight of each of the storage media is set in accordance with the usage rate and the access frequency indicated by the state information of each of the candidate storage media, the weight being inversely related to the usage rate indicated by the state information, and the weight being inversely related to the access frequency indicated by the state information.
In a possible embodiment, before the setting of the weight of each of the storage media according to the usage rate and the access frequency indicated by the status information of each of the candidate storage media, the method further comprises:
determining whether the data to be stored is hot spot data;
the setting of the weight of each storage medium according to the usage rate and the access frequency indicated by the status information of each storage medium candidate includes:
and if the data to be stored is hot spot data, setting the weight of each storage medium according to the use rate and the access frequency indicated by the state information of each candidate storage medium.
In a second aspect of embodiments of the present invention, there is provided a data storage device, the device comprising:
the data acquisition module is used for acquiring state information of each candidate storage medium in the storage system, wherein the state information is used for representing the utilization rate of the storage medium;
a data calculation module, configured to set a weight of each storage medium according to a usage rate indicated by the status information of each candidate storage medium, where the weight is inversely related to the usage rate indicated by the status information;
a medium selection module, configured to select one candidate storage medium from the plurality of candidate storage media, where a probability of each candidate storage medium being selected is positively correlated with the weight of the candidate storage medium;
and the data storage module is used for storing the data to be stored into the selected candidate storage medium.
In a possible embodiment, the apparatus further includes a data analysis module, configured to calculate, according to the usage rates indicated by the status information of the candidate storage media, a data imbalance degree of the storage system, where the data imbalance degree is used to indicate a degree of deviation between the usage rates of the storage media;
the data calculation module is specifically configured to set a weight of each storage medium according to a usage rate indicated by the status information of each candidate storage medium if the data unbalance degree is greater than a preset unbalance degree threshold.
In one possible embodiment, the data analysis module is specifically configured to calculate, according to the usage rates indicated by the status information of the candidate storage media, a difference between a maximum value and an average value of the usage rates indicated by the status information of the candidate storage media, as the data imbalance degree of the storage system.
In a possible embodiment, the medium selection module is further configured to select one candidate storage medium from the plurality of candidate storage mediums if the data imbalance degree is not greater than a preset imbalance degree threshold, where a probability that each candidate storage medium is selected is positively correlated with a preset weight of the candidate storage medium.
In a possible embodiment, the status information is further used to represent an access frequency of the storage medium;
the data calculation module is specifically configured to set a weight of each storage medium according to a usage rate and an access frequency indicated by the state information of each candidate storage medium, where the weight is inversely related to the usage rate indicated by the state information, and the weight is inversely related to the access frequency indicated by the state information.
In a possible embodiment, the apparatus further includes a hotspot determining module, configured to determine whether the data to be stored is hotspot data;
the data calculation module is specifically configured to set a weight of each storage medium according to a usage rate and an access frequency indicated by the status information of each candidate storage medium if the data to be stored is hot spot data.
In a third aspect of the embodiments of the present invention, there is provided an electronic device including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory perform communication with each other through the communication bus;
a memory for storing a computer program;
a processor for implementing the method steps of any of the above first aspects when executing a program stored on a memory.
In a fourth aspect of embodiments of the present invention, there is provided a computer readable storage medium having stored therein a computer program which when executed by a processor implements the method steps of any of the first aspects described above.
According to the data storage method, the data storage device and the electronic equipment, the probability that the stored data to be stored are distributed to the storage media for storage can be dynamically adjusted according to the utilization rate of the storage space of the storage media, so that the stored data of the storage media are close to balance, load balance among the storage media in a storage system is further achieved, and waste of storage resources is reduced. Of course, it is not necessary for any one product or method of practicing the invention to achieve all of the advantages set forth above at the same time.
Drawings
In order to more clearly illustrate the embodiments of the invention or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic flow chart of a data storage method according to an embodiment of the present invention;
FIG. 2 is a schematic flow chart of another data storage method according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of a data storage device according to an embodiment of the present invention;
fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
In order to more clearly describe the data storage method provided by the embodiment of the present invention, an exemplary description will be given below of one possible application scenario of the data storage method provided by the embodiment of the present invention, it will be understood that this application scenario is only one possible application scenario of the data storage method provided by the embodiment of the present invention, and in other possible embodiments, the data storage method provided by the embodiment of the present invention may also be applied to other possible application scenarios, which is not limited in this example.
Ceph is an open source distributed storage system, and is commonly used in cloud storage systems, where Ceph includes multiple PGs (Placement Grouops, a type of logic slicing), and each PG includes multiple OSD (Object Storage Device, a process in Ceph responsible for returning specific data in response to a client request). When the Ceph stores new data, hash calculation can be performed on the new data, 1 PG is selected from the multiple PGs according to the calculated hash value, for example, if 10 PGs are total, the hash value may be divided by 10 to obtain a remainder, and the remainder PG is selected. And selecting an OSD from the selected PG randomly by using CRUSH (a random algorithm), and storing new data into a disk corresponding to the OSD by using the OSD.
However, the CRUSH algorithm is limited to the design of the algorithm, has limited randomness, and cannot realize theoretical true randomness, so that the probability of selecting partial OSD is larger than the probability of selecting other OSD. The usage rate of the disk corresponding to the partial OSD is higher than the usage rate of the disk corresponding to other OSDs.
On the one hand, the use rate of the magnetic disk corresponding to the OSD is too high, which may result in higher use rate of Ceph, but in reality, more storage space may remain in the magnetic disk corresponding to other OSDs in Ceph, and at this time, new data may not be correctly stored due to higher use rate of Ceph, that is, although more storage space remains in Ceph, data cannot be stored, which results in waste of storage resources.
On the other hand, if the OSD fails, the data amount of the data to be transferred is large, which takes much time, resulting in poor stability of Ceph.
Based on this, an embodiment of the present invention provides a data storage method, which may be shown in fig. 1, including:
s101, acquiring state information of each candidate storage medium in the storage system.
S102, setting the weight of each storage medium according to the utilization rate indicated by the state information of each candidate storage medium.
S103, selecting one candidate storage medium from a plurality of candidate storage mediums, wherein the probability of each candidate storage medium being selected is positively correlated with the weight of the candidate storage medium.
S104, storing the data to be stored into the selected candidate storage medium.
By adopting the embodiment, on one hand, the probability that the stored data to be stored is distributed to the storage medium for storage can be dynamically adjusted according to the utilization rate of the storage space of each storage medium, so that the stored data of each storage medium is close to balance, further, the load balance among the storage media in the storage system is realized, and the waste of storage resources is reduced.
On the other hand, the data stored in each storage medium can be made to approach to balance, so that when the OSD in the Ceph fails, the data amount of the data to be migrated is not excessively large, the possibility of long-time data migration of the Ceph caused by the failure of the OSD is reduced, and the stability of the Ceph is improved.
In S101, the status information is used to indicate the usage of the storage medium. Depending on the application scenario, the state information may be represented in different manners, for example, the state information may be represented directly by using the usage rate, or the state information may be represented by using the total memory space and the already used memory space (or the remaining memory space). It will be appreciated that, since the usage rate and the idle rate may be converted from each other, in some possible application scenarios, the idle rate may also be used to represent the status information.
The status information may be used to indicate information other than the usage rate of the storage medium, for example, an access rating rate, access delay, and the like of the storage medium, depending on the application scenario.
In S102, the weight is inversely related to the usage rate indicated by the status information, and the negative correlation means that the weight decreases as the usage rate increases, while other factors affecting the weight are unchanged. The weight may be monotonically decreasing with the increase of the usage rate, or may be decreasing with the increase of the usage rate, and the present embodiment does not limit the correspondence between the usage rate and the weight, but should satisfy the weight of the storage medium with smaller usage rate indicated by the state information, and not smaller than the weight of the storage medium with larger usage rate indicated by the state information.
In S103, the selection may be performed by using any random algorithm, but because the probability of selecting each candidate storage medium is positively correlated with the weight of the candidate storage medium when it is satisfied, that is, the probability of selecting a storage medium with a higher weight is higher. In Ceph, an OSD may be selected according to the weight of the storage medium, and the probability of each OSD being selected is positively correlated with the weight of the storage medium corresponding to the OSD.
In one possible embodiment, as shown in fig. 2, before S102, the method may further include:
s105, calculating the data unbalance degree of the storage system according to the utilization rate indicated by the state information of each candidate storage medium, wherein the data unbalance degree is used for indicating the deviation degree between the utilization rates of each storage medium.
S102 may specifically be to set the weight of each storage medium according to the usage rate indicated by the status information of each candidate storage medium if the data unbalance degree is greater than the preset unbalance degree threshold.
By adopting the embodiment, the unbalance degree of the data stored among all storage media in the storage system can be measured through the data unbalance degree, when the unbalance degree is large, the stored data is balanced in a restoration mode through adjusting the weight, and when the unbalance degree is in an acceptable range, the weight is not adjusted any more, so that the calculation amount required for storing the data is saved.
In one possible embodiment, calculating the data imbalance degree of the storage system according to the usage rate represented by the state information of each candidate storage medium may include:
according to the utilization rate represented by the state information of each candidate storage medium, calculating the difference between the maximum value and the average value of the utilization rate represented by the state information of each candidate storage medium as the data unbalance degree of the storage system.
In one possible embodiment, the method further comprises:
and if the data unbalance degree is not greater than the preset unbalance degree threshold value, selecting one candidate storage medium from a plurality of candidate storage mediums, wherein the probability of selecting each candidate storage medium is positively correlated with the preset weight of the candidate storage medium.
In a possible embodiment, the status information is also used to indicate the access frequency of the storage medium, which may be used to indicate the number of times the storage medium is accessed per unit time.
Setting weights of the respective storage media according to the usage rates indicated by the status information of the respective candidate storage media, comprising:
according to the usage rate and the access frequency indicated by the state information of each candidate storage medium, a weight of each storage medium is set, the weight is inversely related to the usage rate indicated by the state information, and the weight is inversely related to the access frequency indicated by the state information.
It can be understood that the bandwidth of each storage medium is often limited, so that the maximum number of times that a storage medium can be accessed in a unit time is limited, if more data is stored in a storage medium, the probability that the storage medium is accessed is higher, therefore, with this embodiment, the weight of each storage medium can be dynamically adjusted according to the access frequency of the storage medium, so as to balance the access frequency of each storage medium, and avoid that the bandwidth pressure is higher due to the overhigh access frequency of part of storage media, and further delay of data reading and writing of the storage system is caused.
In one possible embodiment, before setting the weight of each storage medium according to the usage rate and the access frequency indicated by the status information of each candidate storage medium, the method further includes:
determining whether the data to be stored is hot spot data;
setting weights of the respective storage media according to the usage rate and the access frequency indicated by the status information of the respective candidate storage media, comprising:
and if the data to be stored is hot spot data, setting the weight of each storage medium according to the use rate and the access frequency indicated by the state information of each candidate storage medium.
It can be understood that the higher the possibility that the hot spot data is accessed compared with the non-hot spot data, therefore, by adopting the embodiment, the hot spot data can be uniformly distributed in each storage medium, and the problem that the bandwidth pressure is high due to the fact that too much hot spot data is stored in part of storage media, and further the delay of the storage system for reading and writing the hot spot data is caused is avoided.
Referring to fig. 3, fig. 3 is a schematic structural diagram of a data storage device according to an embodiment of the present invention, which may include:
the data acquisition module 301 is configured to acquire status information of each candidate storage medium in the storage system, where the status information is used to represent a usage rate of the storage medium;
a data calculation module 302, configured to set a weight of each storage medium according to the usage rate indicated by the status information of each candidate storage medium, where the weight is inversely related to the usage rate indicated by the status information;
a medium selection module 303, configured to select one candidate storage medium from a plurality of candidate storage media, where a probability of each candidate storage medium being selected is positively correlated with a weight of the candidate storage medium;
the data storing module 304 is configured to store the data to be stored in the selected candidate storage medium.
In a possible embodiment, the apparatus further includes a data analysis module, configured to calculate a data imbalance degree of the storage system according to the usage rates represented by the status information of the respective candidate storage media, where the data imbalance degree is used to represent a degree of deviation between the usage rates of the respective storage media;
the data calculation module is specifically configured to set the weight of each storage medium according to the usage rate indicated by the status information of each candidate storage medium if the data unbalance is greater than the preset unbalance threshold.
In one possible embodiment, the data analysis module is specifically configured to calculate, as the data imbalance degree of the storage system, a difference between a maximum value and an average value of the usage rates indicated by the status information of the candidate storage media according to the usage rates indicated by the status information of the candidate storage media.
In one possible embodiment, the medium selection module is further configured to select one candidate storage medium from a plurality of candidate storage mediums if the data imbalance is not greater than a preset imbalance threshold, where a probability that each candidate storage medium is selected is positively correlated with a preset weight of the candidate storage medium.
In a possible embodiment, the status information is further used to represent the frequency of access of the storage medium;
the data calculation module is specifically configured to set a weight of each storage medium according to the usage rate and the access frequency indicated by the state information of each candidate storage medium, wherein the weight is inversely related to the usage rate indicated by the state information, and the weight is inversely related to the access frequency indicated by the state information.
In a possible embodiment, the apparatus further includes a hotspot determining module, configured to determine whether the data to be stored is hotspot data;
the data calculation module is specifically configured to set the weight of each storage medium according to the usage rate and the access frequency indicated by the status information of each candidate storage medium if the data to be stored is hot spot data.
The embodiment of the invention also provides an electronic device, as shown in fig. 4, which comprises a processor 401, a communication interface 402, a memory 403 and a communication bus 404, wherein the processor 401, the communication interface 402 and the memory 403 complete communication with each other through the communication bus 404,
a memory 403 for storing a computer program;
the processor 401, when executing the program stored in the memory 403, implements the following steps:
acquiring state information of each candidate storage medium in the storage system, wherein the state information is used for representing the utilization rate of the storage medium;
setting weights of the storage media according to the utilization rate represented by the state information of the candidate storage media, wherein the weights are inversely related to the utilization rate represented by the state information;
selecting one candidate storage medium from a plurality of candidate storage media, wherein the probability of each candidate storage medium being selected is positively correlated with the weight of the candidate storage medium;
and storing the data to be stored into the selected candidate storage medium.
In a possible embodiment, before setting the weight of each storage medium according to the usage rate indicated by the status information of each candidate storage medium, the method further comprises:
calculating the data unbalance degree of the storage system according to the utilization rate represented by the state information of each candidate storage medium, wherein the data unbalance degree is used for representing the deviation degree between the utilization rates of each storage medium;
setting weights of the respective storage media according to the usage rates indicated by the status information of the respective candidate storage media, comprising:
if the data unbalance is larger than a preset unbalance threshold, setting the weight of each storage medium according to the utilization rate indicated by the state information of each candidate storage medium.
In one possible embodiment, calculating the data imbalance of the storage system according to the usage rate indicated by the status information of each candidate storage medium includes:
according to the utilization rate represented by the state information of each candidate storage medium, calculating the difference between the maximum value and the average value of the utilization rate represented by the state information of each candidate storage medium as the data unbalance degree of the storage system.
In one possible embodiment, the method further comprises:
and if the data unbalance degree is not greater than the preset unbalance degree threshold value, selecting one candidate storage medium from a plurality of candidate storage mediums, wherein the probability of selecting each candidate storage medium is positively correlated with the preset weight of the candidate storage medium.
In a possible embodiment, the status information is further used to represent the frequency of access of the storage medium;
setting weights of the respective storage media according to the usage rates indicated by the status information of the respective candidate storage media, comprising:
according to the usage rate and the access frequency indicated by the state information of each candidate storage medium, a weight of each storage medium is set, the weight is inversely related to the usage rate indicated by the state information, and the weight is inversely related to the access frequency indicated by the state information.
In one possible embodiment, before setting the weight of each storage medium according to the usage rate and the access frequency indicated by the status information of each candidate storage medium, the method further includes:
determining whether the data to be stored is hot spot data;
setting weights of the respective storage media according to the usage rate and the access frequency indicated by the status information of the respective candidate storage media, comprising:
and if the data to be stored is hot spot data, setting the weight of each storage medium according to the use rate and the access frequency indicated by the state information of each candidate storage medium.
The communication bus mentioned above for the electronic devices may be a peripheral component interconnect standard (Peripheral Component Interconnect, PCI) bus or an extended industry standard architecture (Extended Industry Standard Architecture, EISA) bus, etc. The communication bus may be classified as an address bus, a data bus, a control bus, or the like. For ease of illustration, the figures are shown with only one bold line, but not with only one bus or one type of bus.
The communication interface is used for communication between the electronic device and other devices.
The Memory may include random access Memory (Random Access Memory, RAM) or may include Non-Volatile Memory (NVM), such as at least one disk Memory. Optionally, the memory may also be at least one memory device located remotely from the aforementioned processor.
The processor may be a general-purpose processor, including a central processing unit (Central Processing Unit, CPU), a network processor (Network Processor, NP), etc.; but also digital signal processors (Digital Signal Processing, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), field programmable gate arrays (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
In yet another embodiment of the present invention, a computer readable storage medium having instructions stored therein, which when run on a computer, cause the computer to perform any of the data storage methods of the above embodiments is also provided.
In yet another embodiment of the present invention, there is also provided a computer program product containing instructions that, when run on a computer, cause the computer to perform any of the data storage methods of the above embodiments.
In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, produces a flow or function in accordance with embodiments of the present invention, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions may be stored in or transmitted from one computer-readable storage medium to another, for example, by wired (e.g., coaxial cable, optical fiber, digital Subscriber Line (DSL)), or wireless (e.g., infrared, wireless, microwave, etc.). The computer readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that contains an integration of one or more available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid State Disk (SSD)), etc.
It is noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
In this specification, each embodiment is described in a related manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for embodiments of the apparatus, electronic device, computer readable storage medium, computer program product, the description is relatively simple as it is substantially similar to the method embodiments, where relevant see also part of the description of the method embodiments.
The foregoing description is only of the preferred embodiments of the present invention and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.
Claims (9)
1. A method of data storage, the method comprising:
acquiring state information of each candidate storage medium in a storage system, wherein the state information is used for representing the utilization rate of the storage medium;
setting weights of the respective storage media according to the usage rates indicated by the state information of the respective candidate storage media, wherein the weights are inversely related to the usage rates indicated by the state information;
selecting one candidate storage medium from the candidate storage mediums, wherein the probability of each candidate storage medium being selected is positively correlated with the weight of the candidate storage medium;
storing the data to be stored into the selected candidate storage medium;
the status information is also used to represent the access frequency of the storage medium;
the setting of the weight of each storage medium according to the usage rate indicated by the status information of each storage medium candidate includes:
a weight of each of the storage media is set in accordance with the usage rate and the access frequency indicated by the state information of each of the candidate storage media, the weight being inversely related to the usage rate indicated by the state information, and the weight being inversely related to the access frequency indicated by the state information.
2. The method of claim 1, wherein prior to said setting of the weight of each of said storage media in accordance with the usage rate indicated by said status information of each of said candidate storage media, said method further comprises:
calculating the data unbalance degree of the storage system according to the utilization rate represented by the state information of each candidate storage medium, wherein the data unbalance degree is used for representing the deviation degree between the utilization rates of each storage medium;
the setting of the weight of each storage medium according to the usage rate indicated by the status information of each storage medium candidate includes:
and if the data unbalance degree is larger than a preset unbalance degree threshold value, setting the weight of each storage medium according to the use rate indicated by the state information of each candidate storage medium.
3. The method of claim 2, wherein calculating the degree of data imbalance of the storage system according to the usage rate indicated by the status information of each of the candidate storage mediums comprises:
and calculating a difference between a maximum value and an average value of the usage rates indicated by the state information of the candidate storage media according to the usage rates indicated by the state information of the candidate storage media, and taking the difference as the data unbalance degree of the storage system.
4. The method according to claim 2, wherein the method further comprises:
and if the data unbalance degree is not greater than a preset unbalance degree threshold value, selecting one candidate storage medium from the candidate storage mediums, wherein the probability of selecting each candidate storage medium is positively correlated with the preset weight of the candidate storage medium.
5. The method of claim 1, wherein prior to said setting of the weight of each of said storage media in accordance with the usage and access frequency indicated by said status information of each of said candidate storage media, said method further comprises:
determining whether the data to be stored is hot spot data;
the setting of the weight of each storage medium according to the usage rate and the access frequency indicated by the status information of each storage medium candidate includes:
and if the data to be stored is hot spot data, setting the weight of each storage medium according to the use rate and the access frequency indicated by the state information of each candidate storage medium.
6. A data storage device, the device comprising:
the data acquisition module is used for acquiring state information of each candidate storage medium in the storage system, wherein the state information is used for representing the utilization rate of the storage medium; the status information is also used to represent the access frequency of the storage medium;
a data calculation module, configured to set a weight of each storage medium according to a usage rate indicated by the status information of each candidate storage medium, where the weight is inversely related to the usage rate indicated by the status information;
a medium selection module, configured to select one candidate storage medium from the candidate storage mediums, where a probability of each candidate storage medium being selected is positively correlated with the weight of the candidate storage medium;
the data storage module is used for storing the data to be stored into the selected candidate storage medium;
the data calculation module is specifically configured to set a weight of each storage medium according to a usage rate and an access frequency indicated by the state information of each candidate storage medium, where the weight is inversely related to the usage rate indicated by the state information, and the weight is inversely related to the access frequency indicated by the state information.
7. The apparatus of claim 6, further comprising a data analysis module for calculating a data imbalance degree of the storage system according to the usage rate indicated by the status information of each of the candidate storage mediums, the data imbalance degree being indicative of a degree of deviation between the usage rates of each of the storage mediums;
the data calculation module is specifically configured to set a weight of each storage medium according to a usage rate indicated by the status information of each candidate storage medium if the data unbalance degree is greater than a preset unbalance degree threshold.
8. The electronic equipment is characterized by comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory are communicated with each other through the communication bus;
a memory for storing a computer program;
a processor for carrying out the method steps of any one of claims 1-5 when executing a program stored on a memory.
9. A computer-readable storage medium, characterized in that the computer-readable storage medium has stored therein a computer program which, when executed by a processor, implements the method steps of any of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010348430.8A CN111562884B (en) | 2020-04-28 | 2020-04-28 | Data storage method and device and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010348430.8A CN111562884B (en) | 2020-04-28 | 2020-04-28 | Data storage method and device and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111562884A CN111562884A (en) | 2020-08-21 |
| CN111562884B true CN111562884B (en) | 2023-10-27 |
Family
ID=72074367
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010348430.8A Active CN111562884B (en) | 2020-04-28 | 2020-04-28 | Data storage method and device and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111562884B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112905119B (en) * | 2021-02-19 | 2022-10-28 | 山东英信计算机技术有限公司 | Data write-in control method, device and equipment of distributed storage system |
| CN114356211B (en) * | 2021-11-19 | 2023-08-29 | 苏州浪潮智能科技有限公司 | Storage space processing method and device, electronic equipment and storage medium |
| CN116737380B (en) * | 2023-06-16 | 2024-02-09 | 深圳市青葡萄科技有限公司 | Balanced storage method, device and equipment for distributed memory and storage medium |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109343801A (en) * | 2018-10-23 | 2019-02-15 | 深圳前海微众银行股份有限公司 | Date storage method, equipment and computer readable storage medium |
| CN110636122A (en) * | 2019-09-11 | 2019-12-31 | 中移（杭州）信息技术有限公司 | Distributed storage method, server, system, electronic device, and storage medium |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP6283770B2 (en) * | 2015-04-22 | 2018-02-21 | 株式会社日立製作所 | Storage system |
-
2020
- 2020-04-28 CN CN202010348430.8A patent/CN111562884B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109343801A (en) * | 2018-10-23 | 2019-02-15 | 深圳前海微众银行股份有限公司 | Date storage method, equipment and computer readable storage medium |
| CN110636122A (en) * | 2019-09-11 | 2019-12-31 | 中移（杭州）信息技术有限公司 | Distributed storage method, server, system, electronic device, and storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 云环境下分布式文件***负载均衡研究;吴瑶瑶 等;计算机工程与应用;第55卷(第10期);正文第67-72,224页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111562884A (en) | 2020-08-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111562884B (en) | Data storage method and device and electronic equipment | |
| CN110768912B (en) | API gateway current limiting method and device | |
| US10789085B2 (en) | Selectively providing virtual machine through actual measurement of efficiency of power usage | |
| US11537304B2 (en) | Data verification method and apparatus, and storage medium | |
| US8572621B2 (en) | Selection of server for relocation of application program based on largest number of algorithms with identical output using selected server resource criteria | |
| CN112506643A (en) | Load balancing method and device of distributed system and electronic equipment | |
| CN114157666A (en) | Load balancing system, method, medium and device based on intelligent cloud platform | |
| EP3817333B1 (en) | Method and system for processing requests in a consortium blockchain | |
| CN109981702B (en) | File storage method and system | |
| EP3816914A1 (en) | Credibility verification method of alliance chain, system, device and apparatus | |
| CN111367651A (en) | Service current limiting system, method and device and electronic equipment | |
| CN105868002B (en) | Method and device for processing retransmission request in distributed computing | |
| CN110311862B (en) | Service chain mapping method and device | |
| CN114528231A (en) | Data dynamic storage method and device, electronic equipment and storage medium | |
| CN115190010B (en) | Distributed recommendation method and device based on software service dependency relationship | |
| CN108845860B (en) | Method and device for managing quota and electronic equipment | |
| CN108574610B (en) | Pressure testing method and device, electronic equipment and medium | |
| CN116010450A (en) | Method, apparatus, electronic device and computer readable medium for resource allocation | |
| CN112910936B (en) | Data processing method, device and system, electronic equipment and readable storage medium | |
| CN109522118B (en) | Service processing method, device, server and readable storage medium | |
| US9479579B2 (en) | Grouping processing method and system | |
| EP3556084A1 (en) | Automated server deployment platform | |
| CN116450054B (en) | IO request processing method, device, host and computer readable storage medium | |
| CN111614747B (en) | Information processing method and device | |
| CN108881057B (en) | Method for selecting back source line and flow distributor |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |