WO2017036428A2 - 容量变更建议方法及装置 - Google Patents

容量变更建议方法及装置 Download PDF

Info

Publication number
WO2017036428A2
WO2017036428A2 PCT/CN2016/104851 CN2016104851W WO2017036428A2 WO 2017036428 A2 WO2017036428 A2 WO 2017036428A2 CN 2016104851 W CN2016104851 W CN 2016104851W WO 2017036428 A2 WO2017036428 A2 WO 2017036428A2
Authority
WO
WIPO (PCT)
Prior art keywords
resource pool
storage resource
data
storage
migration
Prior art date
Application number
PCT/CN2016/104851
Other languages
English (en)
French (fr)
Other versions
WO2017036428A3 (zh
Inventor
黄升旗
韩银俊
陆平
赵培
Original Assignee
中兴通讯股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 中兴通讯股份有限公司 filed Critical 中兴通讯股份有限公司
Publication of WO2017036428A2 publication Critical patent/WO2017036428A2/zh
Publication of WO2017036428A3 publication Critical patent/WO2017036428A3/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0629—Configuration or reconfiguration of storage systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F12/00—Accessing, addressing or allocating within memory systems or architectures
    • G06F12/02—Addressing or allocation; Relocation
    • G06F12/08—Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
    • G06F3/0647—Migration mechanisms

Definitions

  • the present invention relates to the field of storage, and in particular to a method and apparatus for recommending a capacity change.
  • the traditional hierarchical storage introduces a certain amount of time to the storage system by using a high-performance (read performance) storage medium with a certain proportion of space on the basis of the original massive low-performance storage medium.
  • the historical statistics are compared with the manually preset storage grading strategy, and the content stored on the low-level (low-performance) medium and the read access heat exceeds the set rising heat threshold is raised to the high-level (high-performance) storage medium.
  • Migration simultaneous down-migration to low-level (low-performance) storage media for content stored on high-level (high-performance) media with read access heat below a set thermal descent threshold.
  • the thermal data resides on the high-performance layer media as much as possible, and the cold data resides on the low-performance layer storage medium as much as possible.
  • the heat of the content changes with the access of the application, and becomes hot or changed when the statistical period is set.
  • the migration corresponding to cold or constant, hot and cold data is transparent to the application.
  • the biggest problem in the traditional way is that the migration strategy of the data fragmentation is manually preset.
  • the hierarchical structure of the resource pools at different levels is static, and the dynamic expansion/contraction of different hierarchical resource pools cannot be real-time perceived and synchronized. It is impossible to dynamically adapt to the range change of the heat threshold in the upper-layer application storage access model, which may result in waste that cannot be fully utilized by high-level (high-performance) storage media or frequent content migration and migration resulting in reduced access to actual services. .
  • the embodiment of the invention provides a method and a device for recommending capacity change, so as to at least solve the problem that the access performance of the migration policy is reduced by manual static setting.
  • a method for recommending a capacity change including: counting that the number of two-way migration between the first storage resource pool and the second storage resource pool on the data storage server cluster exceeds the pre-predetermined time period And a third data fragment of the migration number is set; and according to the data volume of the third data fragment, a capacity change suggestion for the first storage resource pool is reported.
  • the method further includes: reporting alarm information for the third data fragment, where the alarm information includes: an alarm used to indicate that the current migration policy performance is low, and/or And an alarm indicating that the third data fragment is frequently migrated.
  • the method further includes: Activating a timer; if the counter times out and does not receive the response message of the capacity change recommendation, raising a first access heat threshold, and/or raising a second access heat threshold, wherein the first access The heat threshold is a minimum access heat value of the first data fragment that needs to be migrated from the second storage resource pool to the first storage resource pool, and the second access heat threshold is required by the first storage resource pool. The maximum access heat value of the second data fragment migrated to the second storage resource pool.
  • a capacity change suggesting apparatus comprising: a third data fragmentation statistics module, configured to count the first storage resource pool and the first on the data storage server cluster within a preset time period The second data fragment of the second storage resource pool exceeds the preset number of migrations; the capacity change recommendation reporting module is configured to report the first storage resource pool according to the data volume of the third data fragment. Capacity change recommendations.
  • the device further includes: an alarm information reporting module, configured to report alarm information for the third data fragment, wherein the alarm information includes: an alarm used to indicate that the performance of the current migration policy is low, and / or an alarm for indicating that the third data fragment is frequently migrated.
  • an alarm information reporting module configured to report alarm information for the third data fragment, wherein the alarm information includes: an alarm used to indicate that the performance of the current migration policy is low, and / or an alarm for indicating that the third data fragment is frequently migrated.
  • the apparatus further comprises: a timer starting module, configured to start a timer; a first access heat threshold and/or a second access heat threshold boosting module, configured to time out the counter and not receive the capacity
  • a timer starting module configured to start a timer
  • a first access heat threshold and/or a second access heat threshold boosting module configured to time out the counter and not receive the capacity
  • the first access heat threshold is raised, and/or the second access heat threshold is raised, wherein the first access heat threshold needs to be migrated from the second storage resource pool to the a minimum access heat value of the first data fragment of the first storage resource pool, where the second access heat threshold is a second data fragment that needs to be migrated from the first storage resource pool to the second storage resource pool Maximum access heat value.
  • a computer storage medium is further provided, and the computer storage medium may store an execution instruction for performing the implementation of the capacity change suggestion method in the foregoing embodiment.
  • the third data fragment whose number of bidirectional migrations between the first storage resource pool and the second storage resource pool on the data storage server cluster exceeds the preset migration number is counted in the preset time period;
  • the data volume of the data fragment is reported in the manner of the capacity change of the first storage resource pool, which solves the problem that the access performance of the migration policy is reduced due to manual static setting, and improves the access performance of the service.
  • FIG. 1 is a flowchart of a migration policy adjustment method according to an embodiment of the present invention.
  • FIG. 2 is a flow chart of a method for recommending a capacity change according to an embodiment of the present invention
  • FIG. 3 is a schematic structural diagram of a migration policy adjustment apparatus according to an embodiment of the present invention.
  • FIG. 4 is a first schematic structural diagram 1 of a migration policy adjustment apparatus according to an embodiment of the present invention.
  • FIG. 5 is a second schematic structural diagram of a migration policy adjustment apparatus according to an embodiment of the present invention.
  • FIG. 6 is a third schematic structural diagram of a migration policy adjustment apparatus according to an embodiment of the present invention.
  • FIG. 7 is a fourth schematic structural diagram of a migration policy adjusting apparatus according to an embodiment of the present invention.
  • FIG. 8 is a schematic diagram 5 of a preferred structure of a migration policy adjustment apparatus according to an embodiment of the present invention.
  • FIG. 9 is a schematic structural diagram of a capacity change suggesting apparatus according to an embodiment of the present invention.
  • FIG. 10 is a first schematic structural diagram 1 of a capacity change suggesting apparatus according to an embodiment of the present invention.
  • FIG. 11 is a second schematic structural diagram of a capacity change suggesting apparatus according to an embodiment of the present invention.
  • FIG. 12 is a schematic structural diagram of a smart hierarchical storage system according to a preferred embodiment of the present invention.
  • FIG. 13 is a flow chart of a smart hierarchical storage method in accordance with a preferred embodiment of the present invention.
  • FIG. 1 is a flowchart of a migration policy adjustment method according to an embodiment of the present invention. As shown in FIG. 1, the process includes the following steps:
  • Step S102 collecting the capacity information of the first storage resource pool on the data storage server cluster in the preset time period, where the first storage resource pool is used to store the data fragment whose access heat value exceeds the preset access heat threshold;
  • Step S104 Determine, according to the capacity information, whether a storage capacity of the first storage resource pool changes.
  • step S106 when it is determined that the storage capacity changes, the migration strategy of the data fragment of the data storage server cluster is adjusted.
  • the migration policy of the data fragment is adjusted according to the change of the storage capacity of the first storage resource pool; for example, in the case that the first storage resource pool is a high-performance storage resource pool, the foregoing steps may be dynamically
  • the change of the storage capacity of the high-performance storage resource pool adjusts the migration policy.
  • the high-performance storage medium waste and service access performance caused by manual static setting of the migration policy are solved. Reduced problems can improve the utilization of high-performance storage media and improve service access performance.
  • the method further includes: counting access information of the data fragments on the metadata server cluster in the preset time period; and collecting, according to the access information, the access heat value of the data fragments in the preset time period; The value determines the data fragment to be migrated in the data fragment, and the data fragment to be migrated is migrated.
  • the above-mentioned access heat value refers to the frequency at which data fragments are accessed. The higher the frequency of access, the higher the access heat value.
  • the specific access heat value statistics mode can be designed according to actual needs, and the statistical mode is not limited in the embodiment of the present invention.
  • Data fragmentation is generally stored in multiple In a storage resource pool, for example, according to storage performance classification, it can be divided into a high-performance storage resource pool (such as a storage resource pool of SSD media) and a low-performance storage resource pool (for example, a Serial Advanced Technology Attachment (Serial Advanced Technology Attachment, Referred to as SATA)).
  • the data fragment stored in the high-performance storage resource pool is a data fragment with a large access value. Because these data fragments are accessed frequently, the read performance is high; otherwise, the low-performance storage resource pool is stored.
  • the data fragmentation is to access data fragments with lower heat values.
  • the remaining storage space of the first storage resource pool (for example, a high performance storage resource pool) may be used. And the amount of data to be migrated to the first storage resource pool is processed.
  • the first storage resource pool (eg, the low performance storage resource pool) needs to be migrated to the first storage resource pool according to the access heat value.
  • a first data amount of the data fragment determining whether the remaining storage space of the first storage resource pool satisfies the storage requirement of the first data fragment; and determining that the remaining storage space of the first storage resource pool satisfies the first data fragment
  • the first data fragment is migrated to the first storage resource pool.
  • the storage resource pool determines, according to the remaining storage space of the first storage resource pool and the first data amount, that the first needs to be determined by the first And storing, by the storage resource pool, a second data quantity of the second data fragment of the second storage resource pool, where the second data quantity is greater than or equal to a difference between the first data quantity and the remaining storage space;
  • the two data fragments are migrated to the second storage resource pool; the first data fragment of the first data amount is migrated to the first storage resource pool.
  • the minimum data volume migration between the high performance storage resource pool and the low performance storage resource pool is implemented, thereby avoiding data.
  • a large number of shards are migrated.
  • the utilization of the high-performance storage resource pool is also improved, and the access performance of the data shards is improved.
  • step S106 when the migration policy is adjusted, the migration information of the data fragment in the preset time period may be counted; and according to the migration information, the first storage resource pool and the second storage resource pool are determined to be in the preset time period.
  • the fragment resides in the first storage resource pool; or determines that the number of bidirectional migrations between the first storage resource pool and the second storage resource pool in the preset time period is lower than the fourth data fragment of the preset migration number;
  • the fourth data fragment resides in the second storage resource pool within a preset multiple access popularity value statistics period after the current access heat value statistics period.
  • the migration policy of adjusting the data fragment of the data storage server cluster may further include at least one of: determining the first storage When the capacity of the resource pool is increased, the first access heat threshold is lowered; when it is determined that the capacity of the first storage resource pool is increased, the second access heat threshold is decreased; and when it is determined that the capacity of the first storage resource pool is decreased, Increase the first access heat threshold And determining, in the case that the first storage resource pool is reduced in capacity, the second access heat threshold is increased; wherein the first access heat threshold is a first data point that needs to be migrated from the second storage resource pool to the first storage resource pool.
  • the minimum access heat value of the slice, and the second access heat threshold is a maximum access heat value of the second data slice that needs to be migrated from the first storage resource pool to the second storage resource pool.
  • the access heat threshold of the uplink/downward migration can be adjusted according to the capacity change of the high-performance storage resource pool, so that the data volume of the hot data fragment can be expanded according to the capacity change of the high-performance storage resource pool or shrink.
  • the utilization rate of the first storage resource pool on the data storage service cluster may be determined according to the capacity information.
  • the utilization rate is increased.
  • the access heat value statistics period is increased, and/or the second is decreased.
  • the access popularity threshold wherein the second access heat threshold is a maximum access heat value of the second data fragment that needs to be migrated from the first storage resource pool to the second storage resource pool.
  • the method further includes: counting a third data fragment whose number of bidirectional migrations between the first storage resource pool and the second storage resource pool exceeds a preset migration number within a preset time period; reporting the third data fragmentation
  • the alarm information includes: an alarm for indicating that the performance of the current migration policy is low, and/or an alarm for indicating that the third data fragment is frequently migrated.
  • the capacity change suggestion for the first storage resource pool may be reported according to the data amount of the third data fragment.
  • the high-level follow-up can adjust the capacity of the high-performance storage resource pool according to the capacity change proposal.
  • the timer may also be started; if the counter expires and the response message of the capacity change recommendation is not received, the first access heat threshold is raised, and/or And raising a second access heat threshold, where the first access heat threshold is a minimum access heat value of the first data fragment that needs to be migrated from the second storage resource pool to the first storage resource pool, and the second access heat threshold is required by The maximum access heat value of the second data fragment of the first storage resource pool migrated to the second storage resource pool.
  • the third data fragment can be frequently migrated by adjusting the access heat threshold.
  • first data fragment “first data fragment”, “second data fragmentation”, “third data fragmentation” or “fourth data fragmentation” referred to in the embodiment of the present invention may be one. Or multiple, and, in general, due to the large amount of data in the storage system, the migrated data is generally also a data slice set composed of a plurality of data fragments.
  • FIG. 2 is a flowchart of a capacity change suggestion method according to an embodiment of the present invention. As shown in FIG. 2, the process includes the following steps:
  • Step S202 the third data fragment whose number of bidirectional migrations between the first storage resource pool and the second storage resource pool on the data storage server cluster exceeds the preset migration number in the preset time period is counted;
  • Step S204 Report a capacity change suggestion for the first storage resource pool according to the data volume of the third data fragment.
  • the capacity change proposal of the first storage resource pool (for example, a high-performance storage resource pool) may be reported according to the migration of the data fragment, so that the upper layer can automatically recommend the capacity of the first storage resource pool according to the capacity change proposal.
  • the problem is that the high-performance storage medium is wasted and the access performance of the service is reduced due to the static setting of the migration policy.
  • the utilization of the high-performance storage medium can be improved and the access performance of the service can be improved.
  • the foregoing may also be reported to The alarm information of the three data fragments, wherein the alarm information includes: an alarm for indicating that the performance of the current migration policy is low, and/or an alarm for indicating that the third data fragment is frequently migrated.
  • the timer may be started; if the counter expires and the response message of the capacity change recommendation is not received, the first access heat threshold is raised, and/or, The second access heat threshold is increased, where the first access heat threshold is a minimum access heat value of the first data fragment that needs to be migrated from the second storage resource pool to the first storage resource pool, and the second access heat threshold is required to be The maximum access heat value of the second data fragment of the storage resource pool migrated to the second storage resource pool.
  • the third data fragment can be frequently migrated by adjusting the access heat threshold.
  • first data fragment “first data fragment”, “second data fragmentation”, “third data fragmentation” or “fourth data fragmentation” referred to in the embodiment of the present invention may be one. Or multiple, and, in general, due to the large amount of data in the storage system, the migrated data is generally also a data slice set composed of a plurality of data fragments.
  • the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, by hardware, but in many cases, the former is A better implementation.
  • the technical solution of the present invention which is essential or contributes to the prior art, may be embodied in the form of a software product stored in a storage medium (such as ROM/RAM, disk,
  • the optical disc includes a number of instructions for causing a terminal device (which may be a cell phone, a computer, a server, or a network device, etc.) to perform the methods of various embodiments of the present invention.
  • a migration policy adjustment apparatus is further provided to implement the foregoing embodiments and preferred implementation manners, which are not described again.
  • the term "module” may implement a combination of software and/or hardware of a predetermined function.
  • the apparatus described in the following embodiments is preferably implemented in software, hardware, or a combination of software and hardware, is also possible and contemplated.
  • FIG. 3 is a schematic structural diagram of a migration policy adjustment apparatus according to an embodiment of the present invention.
  • the apparatus includes: a capacity information statistics module 32, a capacity information determination module 34, and a migration policy adjustment module 36, wherein capacity information is counted.
  • the module 32 is configured to collect the capacity information of the first storage resource pool on the data storage server cluster in the preset time period, where the first storage resource pool is used to store the data fragment whose access heat value exceeds the preset access heat threshold.
  • the capacity information judging module 34 is configured to determine whether the storage capacity of the first storage resource pool changes according to the capacity information; and the migration policy adjustment module 36 is configured to adjust the data storage server cluster when it is determined that the storage capacity changes.
  • the data Fragmentation migration strategy is configured to collect the capacity information of the first storage resource pool on the data storage server cluster in the preset time period, where the first storage resource pool is used to store the data fragment whose access heat value exceeds the preset access heat threshold.
  • the capacity information judging module 34 is configured to
  • FIG. 4 is a first schematic structural diagram of a migration policy adjustment apparatus according to an embodiment of the present invention.
  • the capacity information statistics module 32 is further configured to: count statistics on a cluster of metadata servers within a preset time period.
  • the access information of the data fragmentation device includes: an access heat value statistics module 42 coupled to the capacity information statistics module 32, configured to count the access heat value of the data fragments in the preset time period according to the access information;
  • the migration module 44 is coupled to the access heat value statistics module 42 and configured to determine data fragments to be migrated in the data fragment according to the access heat value, and migrate the data fragments to be migrated.
  • the data fragment migration module 44 includes: a first data amount determining unit 442, configured to determine, according to the access heat value, that the second storage is required, if the occupied ratio of the first storage resource pool does not reach a preset full threshold.
  • the resource pool is migrated to the first data volume of the first data fragment of the first storage resource pool;
  • the storage requirement determining unit 444 is coupled to the first data amount determining unit 442, and configured to determine whether the remaining storage space of the first storage resource pool is Satisfying the storage requirement of the first data fragment;
  • the first data fragment migration unit 446 is coupled to the storage requirement determining unit 444, and configured to determine that the remaining storage space of the first storage resource pool satisfies the storage requirement of the first data fragment In the case of the first data fragment, the first data fragment is migrated to the first storage resource pool.
  • the data fragment migration module 44 further includes: a second data amount determining unit 448 coupled to the storage requirement determining unit 444, configured to determine that the remaining storage space of the first storage resource pool cannot satisfy the first data fragmentation
  • the second data amount of the second data fragment that needs to be migrated from the first storage resource pool to the second storage resource pool is determined according to the remaining storage space of the first storage resource pool and the first data amount, where The second data amount is greater than or equal to the difference between the first data amount and the remaining storage space
  • the second data fragment migration unit 449 is coupled to the second data amount determining unit 448 and configured to set the second data amount of the second data.
  • the fragment is migrated to the second storage resource pool; wherein, the first data fragment migration unit 446 is further configured to: after migrating the second data fragment of the second data amount to the second storage resource pool, the first data volume The first data fragment is migrated to the first storage resource pool.
  • the migration policy adjustment module 36 includes: a migration information statistics unit 362 configured to count migration information of data fragments in a preset time period; a third data fragment determination unit 364 coupled to the migration information statistics unit 362, configured And determining, according to the migration information, a third data fragment whose number of bidirectional migrations between the first storage resource pool and the second storage resource pool exceeds a preset migration number within a preset time period; and/or, the fourth data fragmentation is determined
  • the unit 366 is coupled to the migration information statistics unit 362, and configured to determine, according to the migration information, that the number of bidirectional migrations between the first storage resource pool and the second storage resource pool is lower than the preset migration number in the preset time period.
  • a third data fragment resident unit 365 coupled to the third data fragment determining unit 364, configured to preset a plurality of accesses after the current access heat value statistical period in the case of determining that the storage capacity is increased
  • the third data fragment resides in the first storage resource pool during the heat value statistics period; and/or the fourth data fragment resident unit 367 is coupled to the fourth data point
  • the slice determining unit 366 is configured to: in the case that it is determined that the capacity of the first storage resource pool is reduced, the fourth data fragment resides in the preset multiple access heat value statistics period after the current access heat value statistics period Two storage resource pools.
  • the migration policy adjustment module 36 further includes at least one of the following: the first access heat threshold reduction unit is configured to reduce the first access heat threshold in the case that the first storage resource pool capacity is determined to be increased; the second access heat a threshold reduction unit, configured to decrease a second access heat threshold when it is determined that the capacity of the first storage resource pool increases; An access heat threshold raising unit is configured to: when it is determined that the capacity of the first storage resource pool is reduced, the first access heat threshold is increased; and the second access heat threshold increasing unit is configured to determine that the first storage resource pool is reduced in capacity.
  • the first access heat threshold reduction unit is configured to reduce the first access heat threshold in the case that the first storage resource pool capacity is determined to be increased
  • the second access heat a threshold reduction unit configured to decrease a second access heat threshold when it is determined that the capacity of the first storage resource pool increases
  • An access heat threshold raising unit is configured to: when it is determined that the capacity of the first storage resource pool is reduced, the first access heat threshold is increased
  • the second access heat threshold increasing unit is configured to determine that
  • the second access heat threshold is raised, where the first access heat threshold is a minimum access heat value of the first data fragment that needs to be migrated from the second storage resource pool to the first storage resource pool, and the second access heat threshold is used.
  • FIG. 5 is a schematic diagram of a preferred structure of a migration policy adjustment apparatus according to an embodiment of the present invention.
  • the apparatus further includes: a utilization determination module 52 coupled to the capacity information statistics module 32, configured to determine Whether the utilization of the first storage resource pool in the continuous multiple access popularity value statistics period is lower than the preset utilization rate; the access heat value statistics period and/or the second access heat threshold adjustment module 54 is coupled to the utilization determination module 52.
  • the access popularity threshold is a maximum access heat value of the second data fragment that needs to be migrated from the first storage resource pool to the second storage resource pool.
  • FIG. 6 is a schematic diagram of a preferred structure of a migration policy adjustment apparatus according to an embodiment of the present invention.
  • the capacity information statistics module 32 is further configured to: count the first storage resource pool in a preset time period and The third data fragment of the second storage resource pool exceeds the preset number of migrations.
  • the device further includes: an alarm information reporting module 62 coupled to the capacity information statistics module 32, configured to report the third data fragmentation.
  • the alarm information includes: an alarm for indicating that the performance of the current migration policy is low, and/or an alarm for indicating that the third data fragment is frequently migrated.
  • FIG. 7 is a schematic diagram of a preferred structure of a migration policy adjustment apparatus according to an embodiment of the present invention.
  • the apparatus further includes: a capacity change suggestion reporting module 72 coupled to the capacity information statistics module 32, configured to be configured according to The amount of data of the third data fragment is reported to suggest a capacity change of the first storage resource pool.
  • FIG. 8 is a schematic diagram of a preferred structure of a migration policy adjustment apparatus according to an embodiment of the present invention. As shown in FIG. 8, the apparatus further includes: a timer activation module 82 coupled to the capacity change suggestion reporting module 72, configured to be activated.
  • a timer a timer; a first access heat threshold and/or a second access heat threshold boost module 84 coupled to the timer activation module 82, configured to raise the first in the event that the counter times out and does not receive a response message to the capacity change recommendation Accessing the popularity threshold, and/or increasing the second access heat threshold, wherein the first access heat threshold is a minimum access heat value of the first data fragment that needs to be migrated from the second storage resource pool to the first storage resource pool,
  • the second access heat threshold is a maximum access heat value of the second data fragment that needs to be migrated from the first storage resource pool to the second storage resource pool.
  • the embodiment also provides a capacity change suggesting device for implementing the above-mentioned embodiments and preferred embodiments, which have not been described again.
  • FIG. 9 is a schematic structural diagram of a capacity change suggesting apparatus according to an embodiment of the present invention.
  • the apparatus includes: a third data fragmentation statistics module 92 and a capacity change suggestion reporting module 94, wherein the third data fragmentation
  • the statistics module 92 is configured to collect a third data fragment whose number of bidirectional migrations between the first storage resource pool and the second storage resource pool exceeds a preset migration number in the data storage server cluster within a preset time period;
  • the module 94 is coupled to the third data fragmentation statistics module 92, and is configured to report a capacity change suggestion for the first storage resource pool according to the data volume of the third data fragment.
  • FIG. 10 is a schematic diagram of a preferred structure of a capacity change suggesting apparatus according to an embodiment of the present invention.
  • the apparatus further includes: an alarm information reporting module 102 coupled to the third data fragmentation counting module 92, and configured.
  • the alarm information includes: an alarm for indicating that the performance of the current migration policy is low, and/or an alarm for indicating that the third data fragment is frequently migrated.
  • FIG. 11 is a schematic diagram of a preferred structure of a capacity change suggesting apparatus according to an embodiment of the present invention. As shown in FIG. 11, the apparatus further includes: a timer starting module 112 coupled to the capacity change suggestion reporting module 94, configured to be activated.
  • a timer a timer; a first access heat threshold and/or a second access heat threshold boost module 114, coupled to the timer activation module 112, configured to raise the first in the event that the counter times out and does not receive a response message to the capacity change recommendation Accessing the popularity threshold, and/or increasing the second access heat threshold, wherein the first access heat threshold is a minimum access heat value of the first data fragment that needs to be migrated from the second storage resource pool to the first storage resource pool,
  • the second access heat threshold is a maximum access heat value of the second data fragment that needs to be migrated from the first storage resource pool to the second storage resource pool.
  • each of the above modules may be implemented by software or hardware.
  • the foregoing may be implemented by, but not limited to, the foregoing modules are all located in the same processor; or, the modules are located in multiple In the processor.
  • Embodiments of the present invention also provide a software for performing the technical solutions described in the above embodiments and preferred embodiments.
  • Embodiments of the present invention also provide a storage medium.
  • the above storage medium may be configured to store program code for performing the following steps:
  • Step S102 collecting the capacity information of the first storage resource pool on the data storage server cluster in the preset time period, where the first storage resource pool is used to store the data fragment whose access heat value exceeds the preset access heat threshold;
  • Step S104 Determine, according to the capacity information, whether a storage capacity of the first storage resource pool changes.
  • step S106 when it is determined that the storage capacity changes, the migration strategy of the data fragment of the data storage server cluster is adjusted.
  • Embodiments of the present invention also provide a storage medium.
  • the above storage medium may be configured to store program code for performing the following steps:
  • Step S202 the third data fragment whose number of bidirectional migrations between the first storage resource pool and the second storage resource pool on the data storage server cluster exceeds the preset migration number in the preset time period is counted;
  • Step S204 Report a capacity change suggestion for the first storage resource pool according to the data volume of the third data fragment.
  • the foregoing storage medium may include, but is not limited to, a USB flash drive, a Read-Only Memory (ROM), and a Random Access Memory (RAM).
  • ROM Read-Only Memory
  • RAM Random Access Memory
  • the hardware resource adjustment suggestions of the intelligent resource pool are given, and the hardware configuration is adapted to the service application performance requirement; or the intelligent dynamic grading strategy adjustment is performed to reduce/eliminate the internal migration jitter and optimize the system access performance.
  • a preferred embodiment of the present invention will be described and illustrated by taking a mass file storage application scenario based on a distributed file system and a Write Once & Read Mostly (WORM) model in the cloud storage domain as an example.
  • a high-level storage resource pool (or a high-performance storage resource pool) is equivalent to the foregoing first storage resource pool; and a low-level storage resource pool (or a low-performance storage resource pool) is equivalent to the second Storage resource pool.
  • an intelligent grading engine embedded in a distributed file storage system, and the intelligent grading engine can sense the dynamic change of the current service hotspot model in real time (the spread and shrinkage of the hot content range) and different levels.
  • Resource pool expansion/contraction dynamic change real-time statistical analysis of data migration direction and data volume for the current period of time (including all data fragmentation heat statistics period and migration period in multiple periods), service access distribution and hierarchical performance improvement benefit statistics
  • Intelligent adaptive adjustment of hierarchical policies such as content rise/fall heat threshold and corresponding migration content in hierarchical storage, and intelligent hardware configuration adjustment suggestions for system administrators for current business hotspot access models; The system optimizes the performance matching of the current business model and optimizes the utilization of the storage system performance.
  • the solution of the preferred embodiment of the present invention includes the following three parts:
  • the first part is the real-time calculation of data migration cost and performance improvement benefit under the current hierarchical configuration of the system:
  • the data migration (two-way) is recorded in the real-time system.
  • the data migration takes precedence over the different levels of storage media groups between the single nodes in the storage server cluster.
  • the records can include: migration direction, migration start time, migration end time, and migration occupation.
  • the second part the business access hotspot model perception and hierarchical strategy intelligent adaptive adjustment:
  • the access type, the access amount, and the access frequency of the stored data fragments are recorded in real time on the data fragmentation unit in the storage server cluster, and the current system is calculated and recorded in the current system.
  • the access to the lower-level storage resource pool is high for the case where the overall occupancy ratio of the upper-level storage resource pool does not reach the full threshold (assuming the capacity corresponding to the full threshold is T1).
  • the migration amount equal to the data fragment content in the high-level storage resource pool (assuming the total amount of migration required is A1), when The ascending migration plus the current data capacity that does not need to be migrated (assumed to be C1) exceeds the high-level storage pool full threshold (ie: C1+A1>T1), first of all, the access heat in the high-level storage resource pool is lower than or equal to low.
  • the upward migration (A1) is upgraded.
  • the space occupancy statistics of different hierarchical storage resource pools (especially high-level storage resource pools) in each cycle are simultaneously increased, when hotspot ranges occur.
  • Shrinking causes the high-level storage resource pool utilization to be below the set maximum value of the full threshold for a continuous period of time (for example, multiple heat statistics periods): 1) automatically adjust the heat statistics period, for example, depending on the hot content The degree of shrinkage can adjust the statistical period to 1.5 or more times, reducing the unnecessary migration consumption in different levels of storage resource pools; 2) automatically lowering the data fragmentation without exceeding the full threshold of the high-level storage resource pool
  • the thermal threshold of the ascending migration ensures a moderate fullness and high utilization of the high-level storage resource pool;
  • the response is divided into two priorities.
  • the data fragmentation content that is frequently bidirectionally migrated in consecutive cycles can stay at the high level (in the case of high-level storage resource pool expansion)/low level (in the high-level storage resource pool) for a certain period of time. In the case of volume reduction) in the storage resource pool.
  • the high-level storage resource pool cannot reach the full threshold (assuming the full threshold corresponds to the capacity T1).
  • the hotspot data in the previous statistical period in the high-level level is continuously replaced by the hotspot data in the new statistical period in the lower level, thereby Two-way frequent data migration between two levels of storage resource pools occurs. This migration will result in a reduction in the system's external service capabilities.
  • the minimum mode bearer refers to the current storage redundancy mode in the system, such as multiple copy or Erasure Coding (EC) mode, and the storage service cluster meets the data volume of the ascending migration (A1) in the high-level storage.
  • the minimum size at which the resource pool resides. For example, in n(n> 1), only one hotspot copy resides in the high-level storage resource pool, and the remaining n-1 copies still reside in the low-level storage resource pool; in EC mode (assumed redundancy)
  • the ratio of n:m) only n copies of data reside in the high-level storage resource pool, and the remaining m copies of data reside in the low-level storage resource pool.
  • the intelligent classification engine gives low performance penalty and frequent migration to the data fragments of frequent bidirectional migration according to the recorded multiple heat statistics and the data fragment migration information and the access heat statistics information in the data migration period. police.
  • high-level storage resource pool expansion proposals according to the number of frequently migrated data fragments, for example, minimum (minimum bearer mode) capacity and optimal capacity two quantity value suggestions.
  • the low-level storage resource pool data fragmentation content rising migration heat threshold and the high-level storage resource pool data fragmentation content drop migration heat threshold may be the same value.
  • the following describes the internal adaptation and adjustment process of intelligent tiered storage by taking the expansion of the storage resource pool and the application hotspot model change (hotspot extension) as an example.
  • FIG. 12 is a schematic structural diagram of a smart hierarchical storage system according to a preferred embodiment of the present invention.
  • the system includes a metadata server cluster, a file access access server cluster, a data storage server cluster, and a smart hierarchical storage engine.
  • the solution of the preferred embodiment of the present invention is implemented by the cooperation of the intelligent hierarchical storage engine and other service clusters.
  • FIG. 13 is a flowchart of a smart tiered storage method according to a preferred embodiment of the present invention.
  • the smart adjustment of high-level storage resource pool expansion, intelligent adjustment of high-level storage resource pool shrinkage, and system hotspot range expansion are described below with reference to FIG. Hardware adjustment suggestions / adaptive adjustments are explained.
  • the smart adjustment for the expansion of the high-level storage resource pool includes the following steps:
  • SSD Solid State Drives
  • the storage resource pool information monitoring and reporting module deployed on the storage server reports the resource pool information of the SSD device that is added to the storage device to the intelligent tiered storage engine, which includes new device information of different hierarchical storage resource pools, including high-level Current storage resource pool current usage capacity, new total capacity, etc.
  • the intelligent tiered storage engine collects the data fragmentation access information reported by the real-time data fragmentation collection and reporting module on the metadata server cluster in the new data fragmentation access statistical period, where the data fragmentation resource is included. The location of the pool, the number of times visited, etc.;
  • the intelligent hierarchical storage engine collects and records the detailed migration information based on the data fragmentation, including the migration direction, reported by the data fragmentation migration and reporting module on the data storage server cluster in the new data fragmentation access statistical period;
  • All data fragmentation information lists LT including current location information and access heat information, which are bidirectionally migrated in all previous consecutive 2 statistical periods recorded according to the method of step (5);
  • the intelligent tiered storage engine collects and counts all the data fragmentation heat information in the system during the new data fragmentation access statistical period, and compares the current migration heat threshold to form a new list of pending migration data fragments LA. And a new migration data fragmentation list LD to be dropped;
  • the high-level resource pool after the expansion is fully utilized, especially for data fragmentation that is frequently bidirectionally migrated before unexpanded, and the jitter migration is effectively suppressed, and the internal migration consumption of data between different hierarchical resource pools is reduced.
  • the performance gain value is increased.
  • the number of hot data fragments included in the high-level storage resource pool is automatically expanded to be more and wider. The system has greatly improved the overall access performance of the upper application.
  • Smart adjustments for high-level storage resource pool shrinkage include the following steps:
  • the storage resource pool information monitoring and reporting module deployed on the storage server reports the new resource pool information of the SSD device abnormality or the SSD device to be extracted to the intelligent hierarchical storage engine, including new devices of different hierarchical storage resource pools.
  • the composition information includes the current usage capacity of the high-level storage resource pool, the new total capacity, and the like.
  • OMM operation and maintenance management
  • the intelligent hierarchical storage engine collects and records the detailed migration information based on the data fragmentation reported by the data fragment migration and reporting module on the data storage server cluster, including the migration direction, in the new data fragmentation access statistical period;
  • the number of hotspot contents of the system does not change when the SSD disk is abnormal (or the SSD disk is removed), but the high-level storage resource pool is caused by the abnormality of the SSD disk (or the SSD disk is pulled out).
  • the capacity becomes smaller, causing the high-level storage resource pool space usage ratio r to rise in each cycle after the change occurs;
  • r1 is less than or equal to the high-level storage resource pool full threshold, the system can do nothing, indicating that the current SSD disk is abnormal (or the SSD disk is unplugged) and does not access the hot content currently stored in the upper-layer storage resource pool. Influencing (declining migration), that is, the performance gain value of the current hot content remains unchanged;
  • r1 is greater than the current threshold of the current high-level resource pool of the system, in order to ensure the normal operation of the high-level resource pool, the system must access the heat record according to the data fragmentation in the new at least two statistical periods, and the high-level resource pool will be The content of the data fragment that occupies the space that exceeds the full threshold of the resource pool is forcibly dropped and migrated to the low-level storage resource pool, resulting in a decrease in system performance improvement revenue;
  • step (10) The content of the bidirectional migration jitter in step (9) will be accurately identified by the data fragmentation access thermal statistics recording module and the data fragmentation migration statistical recording module in the intelligent hierarchical storage engine;
  • Hb may not exist. At this time, it indicates that the data fragmentation capacity that needs to be upgraded in each cycle of the system exceeds the maximum capacity r ⁇ C of the entire upper-level resource pool.
  • the threshold of the system data fragmentation migration is adjusted to: Ha*LA/(r ⁇ C), intelligently completes the thermal threshold adjustment, and dynamically eliminates/reduces the system bidirectional migration jitter caused by the reduction of the high-level resource pool capacity.
  • Intelligent hardware tuning recommendations/adaptive adjustments for system hotspot extensions include the following steps:
  • the hotspot range in the system expands, including the heat statistics period in the time point and subsequent consecutive heat statistics periods to find that the hot content expands, and the high-level storage resource pool is limited.
  • the intelligent tiered storage engine is determined to be frequently swapped out in two or more consecutive periods according to the statistics and records of the bidirectional migration of data fragments in two consecutive periods or more, and the performance improvement gains compared to the decline before the hotspot is expanded.
  • the data fragment set that is swapped in, the required capacity of the slice set is set according to the current storage redundancy mode of the distributed storage system, and the full threshold value is used to calculate the minimum capacity and the optimal capacity that the upper-level storage resource pool needs to expand. ;
  • the intelligent grading engine sets a timeout timer while the feedback of the high-level storage resource pool is extended due to the expansion of the hotspot range. Before the timeout timer expires, the system administrator expands according to the recommendation. The system automatically performs the following steps: "Intelligent adjustment of high-level storage resource pool expansion".
  • Hb may not exist. At this time, it indicates that the data fragmentation capacity that needs to be upgraded in each cycle of the system exceeds the maximum capacity r ⁇ C of the entire upper-level resource pool.
  • the system data fragmentation migration threshold is adjusted to: Ha * LA / (r ⁇ C);
  • modules or steps of the present invention described above can be implemented by a general-purpose computing device that can be centralized on a single computing device or distributed across a network of multiple computing devices. Alternatively, they may be implemented by program code executable by the computing device such that they may be stored in the storage device by the computing device and, in some cases, may be different from the order herein.
  • the steps shown or described are performed, or they are separately fabricated into individual integrated circuit modules, or a plurality of modules or steps thereof are fabricated as a single integrated circuit module.
  • the invention is not limited to any specific combination of hardware and software.
  • the foregoing technical solution provided by the embodiment of the present invention may be applied to the process of establishing a capacity change, and the number of two-way migration between the first storage resource pool and the second storage resource pool on the data storage server cluster in the preset time period is more than
  • the third data fragment of the migration number is set. According to the data volume of the third data fragment, the manner of recommending the capacity change of the first storage resource pool is reported, and the service performance degradation of the migration policy caused by manual static setting is solved. The problem is to improve the access performance of the business.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Debugging And Monitoring (AREA)

Abstract

本发明提供了一种容量变更建议方法及装置。其中，上述方法包括：统计在预设时间段内在数据存储服务器集群上第一存储资源池和第二存储资源池之间双向迁移的次数超过预设迁移数的第三数据分片；根据第三数据分片的数据量，上报对第一存储资源池的容量变更建议。通过本发明，解决了迁移策略由人工进行静态设置导致的业务的访问性能降低的问题，提升业务的访问性能。

Description

容量变更建议方法及装置 技术领域
本发明涉及存储领域，具体而言，涉及一种容量变更建议方法及装置。
背景技术
传统分级存储依据存储访问的局部性突出的应用场景，通过在原有的海量低性能存储介质基础上引入一定空间比例的高性能(读性能)存储介质，将应用对存储***的一定时长的内容访问历史统计信息和手动预设的存储分级策略进行比对，对存储在低层(低性能)介质上且读访问热度超过设定的上升热度阀值的内容向高层(高性能)存储介质上进行上升式迁移；同时对存储在高层(高性能)介质上且读访问热度低于设定的热度下降阀值的内容向低层(低性能)存储介质上进行下降式迁移。使得热数据尽可能驻留在高性能层介质上，而冷数据尽可能驻留在低性能层存储介质上，内容的热度随应用访问的变化而变化，在设定统计周期时长变热或变冷或不变，变热和变冷的数据对应的迁移对应用透明。
然而，传统方式最大的问题是数据分片的迁移策略是手工预设的，不同层级的资源池分级架构是静态的，无法对不同层级资源池的动态扩/缩容进行实时感知并同步调整，无法对上层应用存储访问模型中热度阀值的范围变化无法做到动态适应，从而会产生高层(高性能)存储介质无法充分利用的浪费现象或频繁的内容升降迁移造成对实际业务的访问性能降低。
针对相关技术中迁移策略由人工进行静态设置导致的业务的访问性能降低的问题，目前尚未提出有效的解决方案。
发明内容
本发明实施例提供了一种容量变更建议方法及装置，以至少解决迁移策略由人工进行静态设置导致的业务的访问性能降低的问题。
根据本发明的另一个实施例，提供了一种容量变更建议方法，包括：统计在预设时间段内在数据存储服务器集群上第一存储资源池和第二存储资源池之间双向迁移的次数超过预设迁移数的第三数据分片；根据所述第三数据分片的数据量，上报对所述第一存储资源池的容量变更建议。
优选地，在统计在所述预设时间段内在所述数据存储服务器集群上所述第一存储资源池和所述第二存储资源池之间双向迁移的次数超过所述预设迁移数的所述第三数据分片之后，所述方法还包括：上报针对所述第三数据分片的告警信息，其中，所述告警信息包括：用于指示当前迁移策略性能收益低的告警，和/或用于指示所述第三数据分片频繁迁移的告警。
优选地，在上报对所述第一存储资源池的所述容量变更建议之后，所述方法还包括：启 动计时器；在所述计数器超时且未收到所述容量变更建议的响应消息的情况下，提升第一访问热度阈值，和/或，提升第二访问热度阈值，其中，所述第一访问热度阈值为需要由所述第二存储资源池迁移至所述第一存储资源池的第一数据分片的最小访问热度值，所述第二访问热度阈值为需要由所述第一存储资源池迁移至所述第二存储资源池的第二数据分片的最大访问热度值。
根据本发明的另一个实施例，还提供了一种容量变更建议装置，包括：第三数据分片统计模块，设置为统计在预设时间段内在数据存储服务器集群上第一存储资源池和第二存储资源池之间双向迁移的次数超过预设迁移数的第三数据分片；容量变更建议上报模块，设置为根据所述第三数据分片的数据量，上报对所述第一存储资源池的容量变更建议。
优选地，所述装置还包括：告警信息上报模块，设置为上报针对所述第三数据分片的告警信息，其中，所述告警信息包括：用于指示当前迁移策略性能收益低的告警，和/或用于指示所述第三数据分片频繁迁移的告警。
优选地，所述装置还包括：计时器启动模块，设置为启动计时器；第一访问热度阈值和/或第二访问热度阈值提升模块，设置为在所述计数器超时且未收到所述容量变更建议的响应消息的情况下，提升第一访问热度阈值，和/或，提升第二访问热度阈值，其中，所述第一访问热度阈值为需要由所述第二存储资源池迁移至所述第一存储资源池的第一数据分片的最小访问热度值，所述第二访问热度阈值为需要由所述第一存储资源池迁移至所述第二存储资源池的第二数据分片的最大访问热度值。
在本发明实施例中，还提供了一种计算机存储介质，该计算机存储介质可以存储有执行指令，该执行指令用于执行上述实施例中的容量变更建议方法的实现。
通过本发明实施例，采用统计在预设时间段内在数据存储服务器集群上第一存储资源池和第二存储资源池之间双向迁移的次数超过预设迁移数的第三数据分片；根据第三数据分片的数据量，上报对第一存储资源池的容量变更建议的方式，解决了迁移策略由人工进行静态设置导致的业务的访问性能降低的问题，提升业务的访问性能。
附图说明
此处所说明的附图用来提供对本发明的进一步理解，构成本申请的一部分，本发明的示意性实施例及其说明用于解释本发明，并不构成对本发明的不当限定。在附图中：
图1是根据本发明实施例的迁移策略调整方法的流程图；
图2是根据本发明实施例的容量变更建议方法的流程图；
图3是根据本发明实施例的迁移策略调整装置的结构示意图；
图4是根据本发明实施例的迁移策略调整装置的优选结构示意图一；
图5是根据本发明实施例的迁移策略调整装置的优选结构示意图二；
图6是根据本发明实施例的迁移策略调整装置的优选结构示意图三；
图7是根据本发明实施例的迁移策略调整装置的优选结构示意图四；
图8是根据本发明实施例的迁移策略调整装置的优选结构示意图五；
图9是根据本发明实施例的容量变更建议装置的结构示意图；
图10是根据本发明实施例的容量变更建议装置的优选结构示意图一；
图11是根据本发明实施例的容量变更建议装置的优选结构示意图二；
图12是根据本发明优选实施例的智能分级存储***的结构示意图；
图13是根据本发明优选实施例的智能分级存储方法的流程图。
具体实施方式
下文中将参考附图并结合实施例来详细说明本发明。需要说明的是，在不冲突的情况下，本申请中的实施例及实施例中的特征可以相互组合。
需要说明的是，本发明的说明书和权利要求书及上述附图中的术语“第一”、“第二”等是用于区别类似的对象，而不必用于描述特定的顺序或先后次序。
在本实施例中提供了一种迁移策略调整方法，图1是根据本发明实施例的迁移策略调整方法的流程图，如图1所示，该流程包括如下步骤：
步骤S102，统计在预设时间段内数据存储服务器集群上的第一存储资源池的容量信息，其中，第一存储资源池用于存储访问热度值超过预设访问热度阈值的数据分片；
步骤S104，根据容量信息，判断第一存储资源池的存储容量是否发生变化；
步骤S106，在判断到存储容量发生变化的情况下，调整数据存储服务器集群的数据分片的迁移策略。
通过上述步骤，根据第一存储资源池的存储容量变化情况，对数据分片的迁移策略进行调整；例如，在第一存储资源池为高性能存储资源池的情况下，通过上述步骤可以动态根据高性能存储资源池的存储容量的变化对迁移策略进行调整，相对于相关技术中手工静态设置迁移策略的方式，解决了迁移策略由人工进行静态设置导致的高性能存储介质浪费和业务的访问性能降低的问题，可以提高高性能存储介质的利用率，提升业务的访问性能。
优选地，上述方法还包括：统计在预设时间段内元数据服务器集群上的数据分片的访问信息；根据访问信息，统计预设时间段内的数据分片的访问热度值；根据访问热度值确定数据分片中需进行迁移的数据分片，迁移需进行迁移的数据分片。上述的访问热度值是指数据分片被访问的频度，被访问频度越高，则访问热度值越高。具体的访问热度值统计方式可以根据实际需要进行设计，在本发明实施例中并不限制其统计方式。数据分片一般存储在多个 存储资源池中，例如，按照存储性能分类，可以分为高性能存储资源池(例如SSD介质的存储资源池)和低性能存储资源池(例如，串行高级技术附件接口(Serial Advanced Technology Attachment，简称为SATA))。高性能存储资源池中存储的数据分片是访问热度值较大的数据分片，由于这些数据分片被访问的频度高，因此对于读性能要求高；反之，低性能存储资源池中存储的数据分片是访问热度值较低的数据分片。
优选地，在根据访问热度值确定数据分片中需进行迁移的数据分片，迁移需进行迁移的数据分片时，可以根据第一存储资源池(例如高性能存储资源池)的剩余存储空间和待迁移至第一存储资源池的数据量进行处理。
例如，在第一存储资源池被占用比例未达到预设满阈值的情况下，根据访问热度值确定需要由第二存储资源池(例如低性能存储资源池)迁移至第一存储资源池的第一数据分片的第一数据量；判断第一存储资源池的剩余存储空间是否满足第一数据分片的存储需求；在判断到第一存储资源池的剩余存储空间满足第一数据分片的存储需求的情况下，将第一数据分片迁移至第一存储资源池。
优选地，在判断到第一存储资源池的剩余存储空间不能满足第一数据分片的存储需求的情况下，根据第一存储资源池的剩余存储空间和第一数据量，确定需要由第一存储资源池迁移至第二存储资源池的第二数据分片的第二数据量，其中，第二数据量大于或等于第一数据量与剩余存储空间的差值；将第二数据量的第二数据分片迁移至第二存储资源池；将第一数据量的第一数据分片迁移至第一存储资源池。通过上述方式，在第二数据量等于第一数据量与剩余存储空间的差值的情况下，实现了在高性能存储资源池和低性能存储资源池之间的最小数据量迁移，从而可以避免数据分片的大量迁移；同时，由于将尽可能多的数据分片存储在高性能存储资源池中，因此，也提高了高性能存储资源池的利用率，提升了数据分片的访问性能。
优选地，在步骤S106中，调整迁移策略时，可以统计在预设时间段内数据分片的迁移信息；根据迁移信息，确定在预设时间段内在第一存储资源池和第二存储资源池之间双向迁移的次数超过预设迁移数的第三数据分片；在判断到存储容量增加的情况下，在当前访问热度值统计周期之后的预设多个访问热度值统计周期内将第三数据分片驻留在第一存储资源池；或者，确定在预设时间段内在第一存储资源池和第二存储资源池之间双向迁移的次数低于预设迁移数的第四数据分片；在判断到第一存储资源池容量减少的情况下，在当前访问热度值统计周期之后的预设多个访问热度值统计周期内将第四数据分片驻留在第二存储资源池。通过上述方式，可以统计出频繁迁移的数据分片，并且将频繁迁移的数据分片在一定周期内保持驻留在低层级存储资源池或者高层级存储资源池，避免了数据分片的频繁迁移，从而可以提升***稳定性，降低数据分片迁移占用的资源。
优选地，在判断到第一存储资源池的容量发生变化的情况下，在步骤S106中，调整数据存储服务器集群的数据分片的迁移策略还可以包括以下至少之一：在判断到第一存储资源池容量增加的情况下，降低第一访问热度阈值；在判断到第一存储资源池容量增加的情况下，降低第二访问热度阈值；在判断到第一存储资源池容量减少的情况下，提升第一访问热度阈 值；在判断到第一存储资源池容量减少的情况下，提升第二访问热度阈值；其中，第一访问热度阈值为需要由第二存储资源池迁移至第一存储资源池的第一数据分片的最小访问热度值，第二访问热度阈值为需要由第一存储资源池迁移至第二存储资源池的第二数据分片的最大访问热度值。通过上述方式，可以根据高性能存储资源池的容量变化，对上迁/下迁的访问热度阈值进行调整，从而可以使得热点数据分片的数据量根据高性能存储资源池的容量变化而扩大或者收缩。
优选地，在统计数据存储服务集群上的第一存储资源池的容量信息之后，可以根据容量信息判断在连续的多个访问热度值统计周期内第一存储资源池的利用率是否低于预设利用率；在判断到在连续的多个访问热度值统计周期内第一存储资源池的利用率低于预设利用率的情况下，增大访问热度值统计周期，和/或，降低第二访问热度阈值，其中，第二访问热度阈值为需要由第一存储资源池迁移至第二存储资源池的第二数据分片的最大访问热度值。通过上述方式，可以根据高性能存储资源池的利用率对上迁/下迁的访问热度阈值进行调整，从而提升了高性能存储资源池的利用率。
优选地，上述方法还包括：统计在预设时间段内在第一存储资源池和第二存储资源池之间双向迁移的次数超过预设迁移数的第三数据分片；上报针对第三数据分片的告警信息，其中，告警信息包括：用于指示当前迁移策略性能收益低的告警，和/或用于指示第三数据分片频繁迁移的告警。
优选地，在上报告警信息时，还可以根据第三数据分片的数据量，上报对第一存储资源池的容量变更建议。采用上述方式，在将容量变更建议上报之后，高层后续可以根据容量变更建议对高性能存储资源池的容量进行调整。
优选地，在上报对第一存储资源池的容量变更建议之后，还可以启动计时器；在计数器超时且未收到容量变更建议的响应消息的情况下，提升第一访问热度阈值，和/或，提升第二访问热度阈值，其中，第一访问热度阈值为需要由第二存储资源池迁移至第一存储资源池的第一数据分片的最小访问热度值，第二访问热度阈值为需要由第一存储资源池迁移至第二存储资源池的第二数据分片的最大访问热度值。通过上述方式，在高层在预定时间内没有指示对高性能存储资源池的存储容量进行变更的情况下，可以通过对访问热度阈值的调整，避免第三数据分片的频繁迁移。
需要说明的是，在本发明实施例中所指的“第一数据分片”、“第二数据分片”、“第三数据分片”或者“第四数据分片”的数量可以是一个或者多个，并且，在通常情况下，由于存储***中数据量大，则迁移的数据一般也是由多个数据分片构成的数据分片集。
在本实施例中还提供了一种容量变更建议方法，图2是根据本发明实施例的容量变更建议方法的流程图，如图2所示，该流程包括如下步骤：
步骤S202，统计在预设时间段内在数据存储服务器集群上第一存储资源池和第二存储资源池之间双向迁移的次数超过预设迁移数的第三数据分片；
步骤S204，根据第三数据分片的数据量，上报对第一存储资源池的容量变更建议。
通过上述步骤，可以根据数据分片的迁移情况，上报对第一存储资源池(例如高性能存储资源池)的容量变更建议，从而使得高层可以自动根据容量变更建议对第一存储资源池的容量进行调整，解决了迁移策略由人工进行静态设置导致的高性能存储介质浪费或者业务的访问性能降低的问题，可以提高高性能存储介质的利用率，提升业务的访问性能。
优选地，在统计在预设时间段内在数据存储服务器集群上第一存储资源池和第二存储资源池之间双向迁移的次数超过预设迁移数的第三数据分片之后，还可以上报针对第三数据分片的告警信息，其中，告警信息包括：用于指示当前迁移策略性能收益低的告警，和/或用于指示第三数据分片频繁迁移的告警。
优选地，在上报对第一存储资源池的容量变更建议之后，可以启动计时器；在计数器超时且未收到容量变更建议的响应消息的情况下，提升第一访问热度阈值，和/或，提升第二访问热度阈值，其中，第一访问热度阈值为需要由第二存储资源池迁移至第一存储资源池的第一数据分片的最小访问热度值，第二访问热度阈值为需要由第一存储资源池迁移至第二存储资源池的第二数据分片的最大访问热度值。通过上述方式，在高层在预定时间内没有指示对高性能存储资源池的存储容量进行变更的情况下，可以通过对访问热度阈值的调整，避免第三数据分片的频繁迁移。
需要说明的是，在本发明实施例中所指的“第一数据分片”、“第二数据分片”、“第三数据分片”或者“第四数据分片”的数量可以是一个或者多个，并且，在通常情况下，由于存储***中数据量大，则迁移的数据一般也是由多个数据分片构成的数据分片集。
通过以上的实施方式的描述，本领域的技术人员可以清楚地了解到根据上述实施例的方法可借助软件加必需的通用硬件平台的方式来实现，当然也可以通过硬件，但很多情况下前者是更佳的实施方式。基于这样的理解，本发明的技术方案本质上或者说对现有技术做出贡献的部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质(如ROM/RAM、磁碟、光盘)中，包括若干指令用以使得一台终端设备(可以是手机，计算机，服务器，或者网络设备等)执行本发明各个实施例的方法。
在本实施例中还提供了一种迁移策略调整装置，用于实现上述实施例及优选实施方式，已经进行过说明的不再赘述。如以下所使用的，术语“模块”可以实现预定功能的软件和/或硬件的组合。尽管以下实施例所描述的装置较佳地以软件来实现，但是硬件，或者软件和硬件的组合的实现也是可能并被构想的。
图3是根据本发明实施例的迁移策略调整装置的结构示意图，如图3所示，该装置包括：容量信息统计模块32、容量信息判断模块34和迁移策略调整模块36，其中，容量信息统计模块32，设置为统计在预设时间段内数据存储服务器集群上的第一存储资源池的容量信息，其中，第一存储资源池用于存储访问热度值超过预设访问热度阈值的数据分片；容量信息判断模块34，设置为根据容量信息，判断第一存储资源池的存储容量是否发生变化；迁移策略调整模块36，设置为在判断到存储容量发生变化的情况下，调整数据存储服务器集群的数据 分片的迁移策略。
图4是根据本发明实施例的迁移策略调整装置的优选结构示意图一，如图4所示，优选地，容量信息统计模块32还设置为：统计在预设时间段内元数据服务器集群上的数据分片的访问信息；装置还包括：访问热度值统计模块42，耦合至容量信息统计模块32，设置为根据访问信息，统计预设时间段内的数据分片的访问热度值；数据分片迁移模块44，耦合至访问热度值统计模块42，设置为根据访问热度值确定数据分片中需进行迁移的数据分片，迁移需进行迁移的数据分片。
优选地，数据分片迁移模块44包括：第一数据量确定单元442，设置为在第一存储资源池被占用比例未达到预设满阈值的情况下，根据访问热度值确定需要由第二存储资源池迁移至第一存储资源池的第一数据分片的第一数据量；存储需求判断单元444，耦合至第一数据量确定单元442，设置为判断第一存储资源池的剩余存储空间是否满足第一数据分片的存储需求；第一数据分片迁移单元446，耦合至存储需求判断单元444，设置为在判断到第一存储资源池的剩余存储空间满足第一数据分片的存储需求的情况下，将第一数据分片迁移至第一存储资源池。
优选地，数据分片迁移模块44还包括：第二数据量确定单元448，耦合至存储需求判断单元444，设置为在判断到第一存储资源池的剩余存储空间不能满足第一数据分片的存储需求的情况下，根据第一存储资源池的剩余存储空间和第一数据量，确定需要由第一存储资源池迁移至第二存储资源池的第二数据分片的第二数据量，其中，第二数据量大于或等于第一数据量与剩余存储空间的差值；第二数据分片迁移单元449，耦合至第二数据量确定单元448，设置为将第二数据量的第二数据分片迁移至第二存储资源池；其中，第一数据分片迁移单元446，还设置为在将第二数据量的第二数据分片迁移至第二存储资源池之后，将第一数据量的第一数据分片迁移至第一存储资源池。
优选地，迁移策略调整模块36包括：迁移信息统计单元362，设置为统计在预设时间段内数据分片的迁移信息；第三数据分片确定单元364，耦合至迁移信息统计单元362，设置为根据迁移信息，确定在预设时间段内在第一存储资源池和第二存储资源池之间双向迁移的次数超过预设迁移数的第三数据分片；和/或，第四数据分片确定单元366，耦合至迁移信息统计单元362，设置为根据迁移信息，确定在预设时间段内在第一存储资源池和第二存储资源池之间双向迁移的次数低于预设迁移数的第四数据分片；第三数据分片驻留单元365，耦合至第三数据分片确定单元364，设置为在判断到存储容量增加的情况下，在当前访问热度值统计周期之后的预设多个访问热度值统计周期内将第三数据分片驻留在第一存储资源池；和/或第四数据分片驻留单元367，耦合至第四数据分片确定单元366，设置为在判断到第一存储资源池容量减少的情况下，在当前访问热度值统计周期之后的预设多个访问热度值统计周期内将第四数据分片驻留在第二存储资源池。
优选地，迁移策略调整模块36还包括以下至少之一：第一访问热度阈值降低单元，设置为在判断到第一存储资源池容量增加的情况下，降低第一访问热度阈值；第二访问热度阈值降低单元，设置为在判断到第一存储资源池容量增加的情况下，降低第二访问热度阈值；第 一访问热度阈值提升单元，设置为在判断到第一存储资源池容量减少的情况下，提升第一访问热度阈值；第二访问热度阈值提升单元，设置为在判断到第一存储资源池容量减少的情况下，提升第二访问热度阈值；其中，第一访问热度阈值为需要由第二存储资源池迁移至第一存储资源池的第一数据分片的最小访问热度值，第二访问热度阈值为需要由第一存储资源池迁移至第二存储资源池的第二数据分片的最大访问热度值。
图5是根据本发明实施例的迁移策略调整装置的优选结构示意图二，如图5所示，优选地，装置还包括：利用率判断模块52，耦合至容量信息统计模块32，设置为判断在连续的多个访问热度值统计周期内第一存储资源池的利用率是否低于预设利用率；访问热度值统计周期和/或第二访问热度阈值调整模块54，耦合至利用率判断模块52，设置为在判断到在连续的多个访问热度值统计周期内第一存储资源池的利用率低于预设利用率的情况下，增大访问热度值统计周期，和/或，降低第二访问热度阈值，其中，第二访问热度阈值为需要由第一存储资源池迁移至第二存储资源池的第二数据分片的最大访问热度值。
图6是根据本发明实施例的迁移策略调整装置的优选结构示意图三，如图6所示，优选地，容量信息统计模块32还设置为：统计在预设时间段内在第一存储资源池和第二存储资源池之间双向迁移的次数超过预设迁移数的第三数据分片；装置还包括：告警信息上报模块62，耦合至容量信息统计模块32，设置为上报针对第三数据分片的告警信息，其中，告警信息包括：用于指示当前迁移策略性能收益低的告警，和/或用于指示第三数据分片频繁迁移的告警。
图7是根据本发明实施例的迁移策略调整装置的优选结构示意图四，如图7所示，优选地，装置还包括：容量变更建议上报模块72，耦合至容量信息统计模块32，设置为根据第三数据分片的数据量，上报对第一存储资源池的容量变更建议。
图8是根据本发明实施例的迁移策略调整装置的优选结构示意图五，如图8所示，优选地，装置还包括：计时器启动模块82，耦合至容量变更建议上报模块72，设置为启动计时器；第一访问热度阈值和/或第二访问热度阈值提升模块84，耦合至计时器启动模块82，设置为在计数器超时且未收到容量变更建议的响应消息的情况下，提升第一访问热度阈值，和/或，提升第二访问热度阈值，其中，第一访问热度阈值为需要由第二存储资源池迁移至第一存储资源池的第一数据分片的最小访问热度值，第二访问热度阈值为需要由第一存储资源池迁移至第二存储资源池的第二数据分片的最大访问热度值。
本实施例还提供了一种容量变更建议装置，用于实现上述实施例及优选实施方式，已经进行过说明的不再赘述。
图9是根据本发明实施例的容量变更建议装置的结构示意图，如图9所示，该装置包括：第三数据分片统计模块92和容量变更建议上报模块94，其中，第三数据分片统计模块92，设置为统计在预设时间段内在数据存储服务器集群上第一存储资源池和第二存储资源池之间双向迁移的次数超过预设迁移数的第三数据分片；容量变更建议上报模块94，耦合至第三数据分片统计模块92，设置为根据第三数据分片的数据量，上报对第一存储资源池的容量变更建议。
图10是根据本发明实施例的容量变更建议装置的优选结构示意图一，如图10所示，优选地，装置还包括：告警信息上报模块102，耦合至第三数据分片统计模块92，设置为上报针对第三数据分片的告警信息，其中，告警信息包括：用于指示当前迁移策略性能收益低的告警，和/或用于指示第三数据分片频繁迁移的告警。
图11是根据本发明实施例的容量变更建议装置的优选结构示意图二，如图11所示，优选地，装置还包括：计时器启动模块112，耦合至容量变更建议上报模块94，设置为启动计时器；第一访问热度阈值和/或第二访问热度阈值提升模块114，耦合至计时器启动模块112，设置为在计数器超时且未收到容量变更建议的响应消息的情况下，提升第一访问热度阈值，和/或，提升第二访问热度阈值，其中，第一访问热度阈值为需要由第二存储资源池迁移至第一存储资源池的第一数据分片的最小访问热度值，第二访问热度阈值为需要由第一存储资源池迁移至第二存储资源池的第二数据分片的最大访问热度值。
需要说明的是，上述各个模块是可以通过软件或硬件来实现的，对于后者，可以通过以下方式实现，但不限于此：上述模块均位于同一处理器中；或者，上述模块分别位于多个处理器中。
本发明的实施例还提供了一种软件，该软件用于执行上述实施例及优选实施方式中描述的技术方案。
本发明的实施例还提供了一种存储介质。在本实施例中，上述存储介质可以被设置为存储用于执行以下步骤的程序代码：
步骤S102，统计在预设时间段内数据存储服务器集群上的第一存储资源池的容量信息，其中，第一存储资源池用于存储访问热度值超过预设访问热度阈值的数据分片；
步骤S104，根据容量信息，判断第一存储资源池的存储容量是否发生变化；
步骤S106，在判断到存储容量发生变化的情况下，调整数据存储服务器集群的数据分片的迁移策略。
本发明的实施例还提供了一种存储介质。在本实施例中，上述存储介质可以被设置为存储用于执行以下步骤的程序代码：
步骤S202，统计在预设时间段内在数据存储服务器集群上第一存储资源池和第二存储资源池之间双向迁移的次数超过预设迁移数的第三数据分片；
步骤S204，根据第三数据分片的数据量，上报对第一存储资源池的容量变更建议。
可选地，在本实施例中，上述存储介质可以包括但不限于：U盘、只读存储器(Read-Only Memory，简称为ROM)、随机存取存储器(Random Access Memory，简称为RAM)、移动硬盘、磁碟或者光盘等各种可以存储程序代码的介质。
可选地，本实施例中的具体示例可以参考上述实施例及可选实施方式中所描述的示例，本实施例在此不再赘述。
为了使本发明实施例的描述更加清楚，下面结合优选实施例进行描述和说明。
本发明优选实施例提供了一种基于分布式架构的智能分级存储***，以至少实现下列三个功能：
1、实时感知资源池硬件变化/调整，根据资源池的扩/缩容进行智能动态分级策略调整，减少/消除内部迁移抖动，优化***访问性能；
2、实时感知业务热点模型变化，包括热点内容范围的扩展和收缩；
3、根据实时业务热点模型变化给出智能资源池硬件调整建议，优化硬件配置适配业务应用性能需要；或进行智能动态分级策略调整，减少/消除内部迁移抖动，优化***访问性能。
本发明优选实施例将以云存储领域中基于分布式文件***且具有明显读访问热点一次写多次读(Write Once&Read Mostly，简称为WORM)模型的海量文件存储应用场景为例进行描述和说明。在本发明优选实施例中，高层级存储资源池(或者高性能存储资源池)相当于上述的第一存储资源池；低层级存储资源池(或者低性能存储资源池)相当于上述的第二存储资源池。
在本发明优选实施例中，可以通过内嵌于分布式文件存储***中的智能分级引擎实现，智能分级引擎可实时感知当前业务热点模型的动态变化(热点内容范围出现扩散和收缩)以及不同层级资源池扩/缩容动态变化，实时统计分析当前一段时间(包含多个周期内所有数据分片热度统计周期和迁移周期)的数据迁移方向和数据量，业务的访问分布和分级性能提升收益统计与记录，而实现分级存储中内容上升/下降热度阀值以及对应的迁移内容等分级策略的智能自适应调整，以及针对当前业务热点访问模型对***管理员提供智能硬件配置调整建议；以获得存储***对当前业务模型性能匹配的最优化和存储***性能利用的最优化。
为了实现上述功能，本发明优选实施例的方案包括如下三个部分：
第一部分，***当前分级配置下数据迁移代价和性能提升收益实时计算：
实时记录***中数据迁移(双向)量，数据迁移以在存储服务器集群中的单节点间不同层级存储介质组为优先，记录可以包括：迁移方向，迁移开始时间，迁移结束时间，迁移占用的不同层级存储介质间的通道(存储控制器)带宽，数据迁移前的访问量和访问频率，数据迁移后的访问量和访问频率，迁移前后各层级存储介质的空间占用情况。
第二部分，业务访问热点模型感知与分级策略智能自适应调整：
在元数据服务器集群中，对存储服务器集群中以数据分片为单位，按数据分片归属元数据服务器上实时记录存储数据分片的访问类型、访问量、访问频率，计算并记录当前***中所有数据分片访问的热度值，并以此为基础进行定时汇总和排序。
结合不同层级存储资源池的空间占用情况，对于高层级存储资源池整体占用比例未达到满阀值(假设满阀值对应的容量为T1)的情形，对低层级存储资源池中的访问热度高于等于高层级存储资源池中的数据分片内容进行迁移量计算(假设需要上升迁移的总量为A1)，当 上升迁移量加上当前无需迁移的数据容量(假设为C1)超过高层级存储池满阀值时(即：C1+A1>T1)，首先对高层级存储资源池中的访问热度低于等于低层级存储资源池中的数据分片内容按超出量(假设为D1，则D1＝C1+A1-T1)进行下降迁移，下降迁移完成后，对上升迁移总量(A1)进行上升迁移操作。当上升迁移量加上高层级存储当前无需迁移的数据容量(已占容量假设为C1)不超过高层级存储池满阀值时(即：C1+A1<＝T1)，无需先对高层级内的任何内容先进行降级而直接对低存储层级中需要上升迁移的总量数据分片(A1)进行上升迁移操作。
对于每次迁移完成后的连续多个数据分片热度统计周期内，同时增加对不同层级存储资源池(特别是高层级存储资源池)在每个周期内的空间占用情况统计，当出现热点范围收缩而导致在连续一段时间(例如，多个热度统计周期)内高层级存储资源池利用率低于满阀值的设定最大值，则：1)自动调整热度统计周期，例如，视热点内容收缩程度可将统计周期调整为原来的1.5或2倍以上，降低不同层级存储资源池内不必要的迁移消耗；2)在不超过高层级存储资源池满阀值的前提下，自动下调数据分片上升迁移的热度阀值，保证高层级存储资源池的适度充满和高利用率；
对于不同层级存储资源池，特别是高层级存储资源池动态扩/缩容做出适时反应，反应分为两个优先级，1)降低迁移特别是相同数据分片内容的双向迁移。使那些在连续几个周期内被频繁双向迁移的数据分片内容在一定时间内能一直驻留在高层级(在高层级存储资源池扩容的情况下)/低层级(在高层级存储资源池缩容的情况下)存储资源池中。2)降低低层级存储资源池中新热点数据上迁的热度阀值或降低高层级存储资源池中已有热点数据下迁的热度阀值(在高层级存储资源池扩容的情况下)，或者，提升低层级存储资源池中新热点数据上迁的热度阀值或提升高层级存储资源池中已有热点数据下迁的热度阀值(在高层级存储资源池缩容的情况下)；使高层级存储资源池内容随扩容而动态扩展。
第三部分，业务访问热点模型下的智能硬件配置调整建议与分级策略自适应调整：
当应用***的存储内容访问出现热点范围扩散(或高层级存储资源池被缩容)时，高层级存储资源池在容量占用达到满阀值(假设满阀值对应的容量为T1)时也无法完全以最小模式承载***中所有热点分片数据时，高层级中的前一个统计周期中热度排在后面的热点数据会不断被低层级中新统计周期中热度排在前面的热点数据置换，从而发生两个层级存储资源池之间双向的频繁的数据迁移。该迁移会导致***对外的服务能力降低。
其中，最小模式承载是指在***当前的存储冗余模式，例如多副本或者纠删码(Erasure Coding，简称为EC)模式，存储服务集群对满足上升迁移的数据量(A1)在高层级存储资源池驻留的最小规模。如n(n>＝1)完全副本模式下只有一个热点副本驻留在高层级存储资源池中，其余n-1份副本依然驻留在低层级存储资源池中；EC模式下(假设冗余比为n：m)只有n份副本数据驻留在高层级存储资源池中，其余m份副本数据驻留在低层级存储资源池中。
在该情况下，智能分级引擎根据记录的连续多个热度统计和数据迁移周期内数据分片迁移信息、访问热度统计信息，对频繁双向迁移的数据分片给出性能收益低告警和频繁迁移告 警。此外，还可以按频繁迁移的数据分片数，给出高层级存储资源池扩容建议，例如，最小(最小承载模式下)容量和最佳容量两个数量值建议。当***给出的高层级存储资源池扩容建议在一定时长内未被正常响应时，则自动提升低层级存储资源池数据分片内容上升迁移热度阀值，或者提升高层级存储资源池数据分片内容下降迁移热度阀值。
优选地，低层级存储资源池数据分片内容上升迁移热度阀值与高层级存储资源池数据分片内容下降迁移热度阀值可以为相同的值。
通过上述方案，可以实现存储介质资源变化实时感知与自动调整；***访问热点模型变化实时感知与自适应调整；以及基于***业务和环境变化感知的智能决策建议；使传统的分级存储的具有自适应性和智能性，为最大化发挥***资源的利用率，提升存储***的在WORM应用模型下对热点模型扩散和收缩、存储硬件介质变化的动态适应性，最大化分布式存储***对应用的性能响应。
下面将结合附图对本发明优选实施例进行描述和说明。
下面分别以高层级存储资源池扩容和应用热点模型变化(热点范围扩展)为例说明智能分级存储的内部自适应与调整过程。
图12是根据本发明优选实施例的智能分级存储***的结构示意图，如图12所示，该***包括元数据服务器集群、文件访问接入服务器集群、数据存储服务器集群以及智能分级存储引擎。其中，本发明优选实施例的方案通过智能分级存储引擎与其他服务集群的协作共同实现。
图13是根据本发明优选实施例的智能分级存储方法的流程图，下面结合图13对高层级存储资源池扩容的智能调整、高层级存储资源池缩容的智能调整、***热点范围扩展的智能硬件调整建议/自适应调整进行说明。
针对高层级存储资源池扩容的智能调整包括下列步骤：
(1)数据存储服务器集群中的部分/全部服务器新增/***固态盘(Solid State Drives，简称为SSD)介质；
(2)新增/***SSD的存储服务器识别并将新SSD盘正常加入***的高层级资源池中；
(3)部署于存储服务器上的存储资源池信息监控和上报模块将正常加入的SSD设备后的资源池信息上报给智能分级存储引擎，其中包括不同层级存储资源池的设备新组成信息，包括高层级存储资源池当前使用容量、新总容量等；
(4)智能分级存储引擎在新的数据分片访问统计周期内，收集由元数据服务器集群上的实时数据分片访问收集与上报模块上报的数据分片访问信息，其中包含数据分片所在资源池的位置，被访问次数等；
(5)智能分级存储引擎在新的数据分片访问统计周期内，收集并记录由数据存储服务器集群上的数据分片迁移与上报模块上报的基于数据分片详细迁移信息，包括迁移方向；
(6)根据步骤(5)方式记录的所有的前面连续2个统计周期内存在双向迁移的所有数据分片信息列表LT，包括当前位置信息和访问热度信息；
(7)智能分级存储引擎在新的数据分片访问统计周期内，收集并统计***内所有的数据分片访问热度信息，对比当前迁移热度阀值，形成新的待上升迁移数据分片列表LA和新的待下降迁移数据分片列表LD；
(8)对步骤(7)中的LD，与步骤(6)中的当前位置在低层级存储资源池LT数据分片列表进行选择性合并，合并后按热度统计信息形成新的LD1，计算高层级存储资源池下一周期空间使用比率：r＝LD1+LA+C1/C，其中C1表示高层级存储资源池中当前不需迁移数据分片；
(9)当r小于等于高层级存储资源池满阀值时，直接置空本次统计所得的LD列表。当r大于高层级存储资源池满阀值时，修正LD列表，修正后的LD列表中数据分片列表为：LD1+LA+C1-r·C；
通过上述步骤，扩容后的高层级资源池被充分利用，特别是对未扩容前频繁被双向迁移的数据分片，其抖动式迁移得以有效遏制，降低数据在不同层级资源池之间的内部迁移消耗，提升性能收益值，同时，高层级存储资源池包含的热点数据分片数量自动扩展为更多、更广。***对上层应用整体访问性能大幅提升。
针对高层级存储资源池缩容的智能调整包括下列步骤：
(1)数据存储服务器集群中的部分/全部服务器由于硬件设备异常/损坏导致部分SSD介质不可用，或由于人为原因拔出部分SSD介质；
(2)***存储服务器集群内由于正常可用的SSD盘减少导致***的高层级资源池容量变小；
(3)部署于存储服务器上的存储资源池信息监控和上报模块将SSD设备异常或SSD设备被拔出后的新资源池信息上报给智能分级存储引擎，其中包括不同层级存储资源池的设备新组成信息，包括高层级存储资源池当前使用容量、新总容量等。同时通过自身的操作维护管理(Operation&Maintenance Management，简称为OMM)模块对发生异常的SSD盘/被拔出的SSD盘对外(***管理员)给出相应的***告警机制；
(4)智能分级存储引擎在新的数据分片访问统计周期内，收集并记录由数据存储服务器集群上的数据分片迁移与上报模块上报的基于数据分片详细迁移信息，包括迁移方向；
(5)通常情况下，SSD盘发生异常(或SSD盘被拔出)时***的热点内容数并未改变，但由于SSD盘的异常(或SSD盘被拔出)导致高层级存储资源池总容量变小，导致在改变发生后的每个周期内高层级存储资源池空间使用比率r上升；
(6)统计高层级存储存储资源池使用比率上升后的至少两个统计周期内的平均新空间使 用率r1、数据分片访问热度记录、数据分片迁移记录；
(7)如果r1小于等于高层级存储资源池满阀值，则***可以什么也不做，表示当前的SSD盘异常(或SSD盘拔出)未对当前存储于高层存储资源池的热点内容访问造成影响(下降迁移)，亦即当前热点内容性能收益值保持不变；
(8)如果r1大于***当前高层级资源池满阀值，为保证高层级资源池的正常运行，***必须根据新的至少两个统计周期内数据分片访问热度记录，将高层级资源池中占用超出资源池满阀值的空间的热度最低的数据分片内容强行下降迁移到低层级存储资源池中，导致***性能提升收益下降；
(9)性能提升收益下降，由于热点内容范围未出现变动，进一步将导致高层级存储资源池和低层级存储资源池中热度接近临界值的数据分片内容出现双向的迁移抖动；
(10)步骤(9)中双向迁移抖动的内容将被智能分级存储引擎中的数据分片访问热度统计记录模块、数据分片迁移统计记录模块准确识别；
(11)***识别出双向迁移抖动之后，连续统计两个或以上周期内被双向迁移的数据分片的在每个周期内的被访问热度平均值Ha和热度最大值Max_a，同时统计连续两个或以上周期内驻留在高层级存储中无需下降迁移的数据分片被访问平均热度值Hb和热度最小值Min_b；
(12)对于步骤(11)中的Ha和Hb，Hb可能不存在，此时表明***每个周期内需要上升迁移的数据分片容量超过整个高层级资源池的最大容量r·C，则此时，将***数据分片上升迁移的阀值调整为：Ha*LA/(r·C)，智能完成热度阀值调整，动态消除/降低了高层级资源池容量缩减造成的***双向迁移抖动。
针对***热点范围扩展的智能硬件调整建议/自适应调整包括如下步骤：
(1)通常情况下，应用于WORM模型中热点明显的业务场景，智能分级存储开启后，高层级存储资源池的利用率都是接近或达到满阀值的；
(2)某一时间点开始，***内热点范围出现扩展，包含该时间点内的热度统计周期以及后续连续的几个热度统计周期发现热点内容出现扩展，随之由于高层级存储资源池有限而导致出现大量访问热度接近临界值的数据分片发生迁移抖动，同时伴随每个统计周期内性能提升收益下降；
(3)智能分级存储引擎根据连续两个或以上周期的数据分片双向迁移的统计与记录、以及相比热点未扩展前下降的性能提升收益，确定连续两个或以上周期内被频繁换出和换入的数据分片集，对该分片集所需的容量按分布式存储***当前存储冗余模式设置，结合满阀值计算出高层级存储资源池需要扩展的最小容量和最佳容量；
(4)智能分级存储引擎给出的高层级存储资源池需要扩展的最小容量和最佳容量通过分布式存储***已有的OMM模块告警渠道反馈给***管理员，其中还包含***具体的最近两 个或以上周期内的分级性能收益下降和数据分片迁移统计记录信息；
(5)智能分级引擎在反馈由于热点范围扩展而引起的高层级存储资源池扩展建议的同时，设定一个超时定时器，在超时定时器未到之前，***管理员按建议进行了扩容，则***自动按前面所述“高层级存储资源池扩容的智能调整”描述步骤进行；
(6)在超时定时器已到但***的高层级存储资源池仍未被有效扩容，则开始统计连续两个或以上周期内被双向迁移的数据分片的在每个周期内的被访问热度平均值Ha和热度最大值Max_a，同时统计连续两个或以上周期内驻留在高层级存储中无需下降迁移的数据分片被访问平均热度值Hb和热度最小值Min_b；
(7)对于步骤(6)中的Ha和Hb，Hb可能不存在，此时表明***每个周期内需要上升迁移的数据分片容量超过整个高层级资源池的最大容量r·C，则此时，将***数据分片上升迁移的阀值调整为：Ha*LA/(r·C)；
(8)对于步骤(6)中的Ha和Hb，如果Hb存在，则一定有(Hb>Ha)∩(Max_a≤Min_b)，此时将***数据上升迁移的阀值直接调整为Min_b；
(9)通过步骤(7)和步骤(8)的智能自动阀值调整，降低热点内容范围扩张导致的数据分片双向抖动迁移。
显然，本领域的技术人员应该明白，上述的本发明的各模块或各步骤可以用通用的计算装置来实现，它们可以集中在单个的计算装置上，或者分布在多个计算装置所组成的网络上，可选地，它们可以用计算装置可执行的程序代码来实现，从而，可以将它们存储在存储装置中由计算装置来执行，并且在某些情况下，可以以不同于此处的顺序执行所示出或描述的步骤，或者将它们分别制作成各个集成电路模块，或者将它们中的多个模块或步骤制作成单个集成电路模块来实现。这样，本发明不限制于任何特定的硬件和软件结合。
以上所述仅为本发明的优选实施例而已，并不用于限制本发明，对于本领域的技术人员来说，本发明可以有各种更改和变化。凡在本发明的精神和原则之内，所作的任何修改、等同替换、改进等，均应包含在本发明的保护范围之内。
工业实用性
本发明实施例提供的上述技术方案，可以应用于容量变更建立过程中，采用统计在预设时间段内在数据存储服务器集群上第一存储资源池和第二存储资源池之间双向迁移的次数超过预设迁移数的第三数据分片；根据第三数据分片的数据量，上报对第一存储资源池的容量变更建议的方式，解决了迁移策略由人工进行静态设置导致的业务的访问性能降低的问题，提升业务的访问性能。

Claims (6)

  1. 一种容量变更建议方法，包括：
    统计在预设时间段内在数据存储服务器集群上第一存储资源池和第二存储资源池之间双向迁移的次数超过预设迁移数的第三数据分片；
    根据所述第三数据分片的数据量，上报对所述第一存储资源池的容量变更建议。
  2. 根据权利要求1所述的方法，其中，在统计在所述预设时间段内在所述数据存储服务器集群上所述第一存储资源池和所述第二存储资源池之间双向迁移的次数超过所述预设迁移数的所述第三数据分片之后，所述方法还包括：
    上报针对所述第三数据分片的告警信息，其中，所述告警信息包括：用于指示当前迁移策略性能收益低的告警，和/或用于指示所述第三数据分片频繁迁移的告警。
  3. 根据权利要求1或2所述的方法，其中，在上报对所述第一存储资源池的所述容量变更建议之后，所述方法还包括：
    启动计时器；
    在所述计数器超时且未收到所述容量变更建议的响应消息的情况下，提升第一访问热度阈值，和/或，提升第二访问热度阈值，其中，所述第一访问热度阈值为需要由所述第二存储资源池迁移至所述第一存储资源池的第一数据分片的最小访问热度值，所述第二访问热度阈值为需要由所述第一存储资源池迁移至所述第二存储资源池的第二数据分片的最大访问热度值。
  4. 一种容量变更建议装置，包括：
    第三数据分片统计模块，设置为统计在预设时间段内在数据存储服务器集群上第一存储资源池和第二存储资源池之间双向迁移的次数超过预设迁移数的第三数据分片；
    容量变更建议上报模块，设置为根据所述第三数据分片的数据量，上报对所述第一存储资源池的容量变更建议。
  5. 根据权利要求4所述的装置，其中，所述装置还包括：
    告警信息上报模块，设置为上报针对所述第三数据分片的告警信息，其中，所述告警信息包括：用于指示当前迁移策略性能收益低的告警，和/或用于指示所述第三数据分片频繁迁移的告警。
  6. 根据权利要求4或5所述的装置，其中，所述装置还包括：
    计时器启动模块，设置为启动计时器；
    第一访问热度阈值和/或第二访问热度阈值提升模块，设置为在所述计数器超时且未收到所述容量变更建议的响应消息的情况下，提升第一访问热度阈值，和/或，提升第二访问热度阈值，其中，所述第一访问热度阈值为需要由所述第二存储资源池迁移至所述 第一存储资源池的第一数据分片的最小访问热度值，所述第二访问热度阈值为需要由所述第一存储资源池迁移至所述第二存储资源池的第二数据分片的最大访问热度值。
PCT/CN2016/104851 2015-09-06 2016-11-07 容量变更建议方法及装置 WO2017036428A2 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201510698063.3 2015-09-06
CN201510698063.3A CN106502578B (zh) 2015-09-06 2015-09-06 容量变更建议方法及装置

Publications (2)

Publication Number Publication Date
WO2017036428A2 true WO2017036428A2 (zh) 2017-03-09
WO2017036428A3 WO2017036428A3 (zh) 2017-04-13

Family

ID=58186719

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2016/104851 WO2017036428A2 (zh) 2015-09-06 2016-11-07 容量变更建议方法及装置

Country Status (2)

Country Link
CN (1) CN106502578B (zh)
WO (1) WO2017036428A2 (zh)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110008199A (zh) * 2019-03-25 2019-07-12 华南理工大学 一种基于访问热度的数据迁移部署方法
CN110825908A (zh) * 2019-11-04 2020-02-21 安超云软件有限公司 一种对象的迁移方法、装置、电子设备及存储介质
CN111124692A (zh) * 2020-01-02 2020-05-08 神州数码融信软件有限公司 一种业务请求处理***

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106502576B (zh) * 2015-09-06 2020-06-23 中兴通讯股份有限公司 迁移策略调整方法及装置
CN112948398B (zh) * 2021-04-29 2023-02-24 电子科技大学 一种面向冷热数据的分级存储***及方法
CN113590586B (zh) * 2021-07-29 2022-03-22 东方微银科技股份有限公司 分布式图数据库***节点间迁移分片数据的方法、装置
CN114936003B (zh) * 2022-05-06 2023-03-21 北京新科安云信息技术有限公司 资源池的数据分层迁移方法、装置、设备及可读存储介质
CN115373594A (zh) * 2022-07-12 2022-11-22 浙江大学 一种双向动态切换的存储***及动态切换管理方法

Family Cites Families (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102143206A (zh) * 2010-12-17 2011-08-03 浪潮(北京)电子信息产业有限公司 集群存储***中存储池的调整方法、装置及***
CN102043732A (zh) * 2010-12-30 2011-05-04 成都市华为赛门铁克科技有限公司 一种缓存分配方法及装置
CN103365781B (zh) * 2012-03-29 2016-05-04 国际商业机器公司 用于动态地重新配置存储***的方法和设备
CN103150263B (zh) * 2012-12-13 2016-01-20 深圳先进技术研究院 分级存储方法
CN103078933B (zh) * 2012-12-29 2015-12-02 深圳先进技术研究院 一种确定数据迁移时机的方法和装置
CN104657215A (zh) * 2013-11-19 2015-05-27 南京鼎盟科技有限公司 云计算中虚拟化节能***
CN104660578B (zh) * 2014-04-22 2017-12-19 董唯元 一种实现数据安全存储及数据访问控制的***及其方法
CN104536994A (zh) * 2014-12-11 2015-04-22 北京京东尚科信息技术有限公司 通用的数据迁移方法和装置

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110008199A (zh) * 2019-03-25 2019-07-12 华南理工大学 一种基于访问热度的数据迁移部署方法
CN110825908A (zh) * 2019-11-04 2020-02-21 安超云软件有限公司 一种对象的迁移方法、装置、电子设备及存储介质
CN111124692A (zh) * 2020-01-02 2020-05-08 神州数码融信软件有限公司 一种业务请求处理***
CN111124692B (zh) * 2020-01-02 2023-05-12 神州数码融信软件有限公司 一种业务请求处理***

Also Published As

Publication number Publication date
WO2017036428A3 (zh) 2017-04-13
CN106502578A (zh) 2017-03-15
CN106502578B (zh) 2019-06-11

Similar Documents

Publication Publication Date Title
WO2016165441A1 (zh) 迁移策略调整方法、容量变更建议方法及装置
WO2017036428A2 (zh) 容量变更建议方法及装置
CN100518088C (zh) 一种流媒体数据内容管理方法
US20230004436A1 (en) Container scheduling method and apparatus, and non-volatile computer-readable storage medium
US9727487B2 (en) Cache management method and apparatus for non-volatile storage device
US20180167461A1 (en) Method and apparatus for load balancing
US20120221730A1 (en) Resource control system and resource control method
US20150199281A1 (en) Apparatus and method for meeting performance metrics for users in file systems
CN103078933B (zh) 一种确定数据迁移时机的方法和装置
CN106156283B (zh) 基于数据温度和节点性能的异构Hadoop存储方法
CN109085995B (zh) 数据动态分片的存储方法、装置和***
US11157313B2 (en) Method, apparatus and computer storage medium for controlling a storage system
WO2015154352A1 (zh) 分布式文件***的数据迁移方法、装置及元数据服务器
CN104092620A (zh) 一种实现网络带宽调整的方法及装置
CN103631894A (zh) 一种基于hdfs的动态副本管理方法
CN111414070A (zh) 一种机箱功耗管理方法、***及电子设备和存储介质
WO2019011262A1 (zh) 分配资源的方法和装置
CN111143279A (zh) 一种数据迁移方法、装置、设备及可读存储介质
WO2022017290A1 (zh) 终端网速控制方法、装置、终端及存储介质
CN111506425B (zh) 服务质量数据处理方法和装置
EP3076295A1 (en) System resource balance adjustment method and device
CN104239230B (zh) 一种数据块迁移方法及装置
US11232228B2 (en) Method and device for improving data storage security
US10078642B1 (en) Dynamic memory shrinker for metadata optimization
CN103152377B (zh) 一种面向ftp服务的数据访问方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16840857

Country of ref document: EP

Kind code of ref document: A2

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 16840857

Country of ref document: EP

Kind code of ref document: A2