CN113448970B - Graph data storage method and system - Google Patents
Graph data storage method and system Download PDFInfo
- Publication number
- CN113448970B CN113448970B CN202111006840.5A CN202111006840A CN113448970B CN 113448970 B CN113448970 B CN 113448970B CN 202111006840 A CN202111006840 A CN 202111006840A CN 113448970 B CN113448970 B CN 113448970B
- Authority
- CN
- China
- Prior art keywords
- storage unit
- data
- migrated
- central
- memory usage
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/214—Database migration support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention is suitable for the field of computers, and provides a graph data storage method and a system, wherein the obtained data is stored in a corresponding storage unit, and the memory usage proportion of the storage unit is calculated periodically; and continuously monitoring and judging whether the memory usage proportion of the central storage unit is greater than the overflow proportion, namely regularly monitoring whether the memory of the central storage unit is about to run out, and automatically transferring the data in the central storage unit to a primary storage unit connected with the central storage unit when the memory usage proportion of the central storage unit is greater than the overflow proportion. And the data in the central storage unit is migrated to the primary storage unit, so that the memory of the central storage unit is released. The problem of among the prior art fixed memory cell, data memory space appears unbalance when the storage data, leads to some memory cell to load too big, the running speed slows down is solved.
Description
Technical Field
The invention belongs to the field of computers, and particularly relates to a graph data storage method and a graph data storage system.
Background
The data processing mode of the graph database is similar to OLTP (online processing) processing in the field of traditional databases, and related query, data updating and graph query language processing of a system with high efficiency and low delay are provided as main tasks. Aiming at the problem that the processing efficiency of a relational database on the association query is not ideal, the graph database particularly models graph data, supports efficient association query storage and specific graph index orientation optimization, and the method comprises the steps of avoiding accessing node, edge and attribute data which are irrelevant to the query under the condition of giving a graph query to narrow the search range and reduce the computation overhead of the query result.
Graph databases do have a wide range of applicability because connections exist in all corners of nature and society. Every thing is not isolated but is tied to other things, either tightly or loosely. With the progress of human society, the processing of various relationships becomes more and more important, and not only people but also connection relationships between things are more and more emphasized. For example, social interaction is a person-to-person connection, and graph databases that are intrinsic to graph data models are inherently suited to obvious contact-centric areas. The use of graph databases in social networks can facilitate the identification of direct or indirect connections between people/groups and things they communicate with, enabling users to efficiently score, comment on, discover things that exist in relation to each other and common. It is more intuitive to know how people interact, associate, do things or choose in groups among people in a social network. Social networks are the most basic graph models, and more contents such as personal preferences, purchased articles, daily life style and the like can be overlaid on the graph models, so that a more advanced graph database application mode is evolved.
In the relational graph data model, data such as nodes, edges, attributes and the like form a framework of the graph data model, one node is selected, the node is associated with a plurality of other surrounding nodes, and the nodes are associated with other nodes, so that a relational network which looks like taking one node as a center and continuously radiating to the surrounding is presented. One node is regarded as a small data storage unit, and the node is regarded as the corresponding relation among a plurality of data storage units, the existing graph data is stored fixedly on the node, a computer program can specify that the data is stored in the pre-distributed fixed storage units, so that more data are stored in some storage units in the storage process, less data are stored in some storage units, and the distribution of the whole data stored in each storage unit is unbalanced, so that the load of some storage units is too large, the running speed is slow, and the node is used as one node in the data query and storage process, and the running performance of the whole database is reduced.
Disclosure of Invention
The embodiment of the invention provides a graph data storage method and a graph data storage system, and aims to solve the problems that some storage units are overloaded and the running speed is slowed down due to data storage amount imbalance when the storage units with the fixed memory sizes are used for storing data in advance.
The embodiment of the invention is realized in such a way that, on one hand, a graph data storage method comprises the following steps:
acquiring data to be saved;
storing the data in a corresponding storage unit, and periodically calculating the memory usage proportion of the storage unit; defining a storage unit as a central storage unit, defining the storage unit directly connected with the central storage unit as a primary storage unit, defining the storage unit directly connected with the primary storage unit as a secondary storage unit except the central storage unit, and defining a multilayer hierarchical relationship by outwards radiating a plurality of layers of storage units in the same way;
judging whether the memory usage proportion of the central memory unit is greater than the overflow proportion; the overflow occupation ratio is an early warning value that the memory usage of the central storage unit is about to reach the total memory amount;
and when the memory usage ratio of the central storage unit is greater than the overflow ratio, automatically transferring the data in the central storage unit to the primary storage unit connected with the central storage unit.
As a modified scheme of the invention: the storing the data in the corresponding storage unit and periodically calculating the memory usage ratio of the storage unit specifically include:
storing the data in the corresponding storage unit;
carrying out timing monitoring on the storage unit, and triggering timing grabbing when the timing countdown time is zero;
and capturing the memory usage amount and the total memory amount of the storage unit to obtain the memory usage amount ratio.
As a further improvement of the invention: when the memory usage proportion of the central storage unit is greater than the overflow proportion, the automatically transferring the data in the central storage unit to the first-level storage unit connected with the central storage unit specifically includes:
when the memory usage ratio of the central storage unit is greater than the overflow ratio, acquiring the number of the primary storage units connected with the central storage unit at the moment;
determining data to be migrated in the central storage unit to form a data set to be migrated;
and distributing the data sets to be migrated in the central storage unit according to the number of the primary storage units, and migrating the data sets to the primary storage units.
As another improvement of the invention: the determining data to be migrated from the central storage unit and forming a data set to be migrated specifically includes:
when the memory usage ratio of the central storage unit is larger than the overflow ratio, forward calculation is carried out according to the time point of the last storage of different data as a reference, and data with a fixed magnitude is divided to form a data set to be migrated;
and identifying the corresponding relation between the data set to be migrated and different primary storage units.
As a further scheme of the invention: the specific method for segmenting the data with the fixed magnitude to form the data set to be migrated comprises the following steps: forward reckoning according to the time point of the last storage of different data as a reference, and dividing data in a fixed time to form a data set to be migrated; or forward reckoning according to the time point of the last storage of different data as a reference, and dividing the data with a fixed size to form a data set to be migrated.
As a still further scheme of the invention: the determining data to be migrated from the central storage unit and forming a data set to be migrated specifically includes:
capturing the memory usage amount and the total memory amount of each primary storage unit;
calculating the residual memory amount of each primary storage unit according to the memory usage amount and the total memory amount;
obtaining the ratio of the residual memory among the primary storage units according to the residual memory amount of each primary storage unit;
determining the total amount of data to be migrated in the central storage unit, distributing the total amount of data according to the ratio of the remaining memories, and determining the size of a data set to be migrated corresponding to each primary storage unit;
and according to the size of the data set to be migrated corresponding to each primary storage unit, forward calculation is carried out by taking the time point of the last storage of different data as a reference, and the data set to be migrated with the corresponding size is divided.
As an optimization scheme of the invention: the allocating the data sets to be migrated in the central storage unit according to the number of the primary storage units, and the migrating to the primary storage units specifically includes:
and respectively migrating the data sets to be migrated to the primary storage units according to the corresponding relation between the segmented data sets to be migrated and different primary storage units.
As another scheme of the invention: when the memory usage proportion of the central storage unit is greater than the overflow proportion, the method further comprises the following steps after the data in the central storage unit is automatically transferred to the first-level storage unit connected with the central storage unit:
after the data in the central storage unit is transferred to the lower-level storage unit, judging whether the memory usage proportion of the lower-level storage unit is greater than the overflow proportion;
when the memory usage ratio of the lower-level storage unit is greater than the overflow ratio, automatically transferring the data in the lower-level storage unit to a lower-level storage unit connected with the lower-level storage unit;
and stopping the storage unit receiving the migration data to continue data migration until the storage unit receiving the migration data finally judges that the memory usage ratio is not more than the overflow ratio.
As another optimization scheme of the invention: when the memory usage occupation ratio of the lower-level storage unit is greater than the overflow occupation ratio, after the data in the lower-level storage unit is automatically migrated to the next-level storage unit connected with the lower-level storage unit, the method further includes:
when data are migrated to the last-stage storage unit, and after judgment, the memory usage ratio of the last-stage storage unit is still larger than the overflow-full ratio, the last-stage storage unit is taken as a reference to be pushed upwards, and memory expansion is carried out on all storage units on the data migration path where the last-stage storage unit is located.
In another aspect, a graph data storage system includes:
the data acquisition module is used for acquiring data to be stored;
the memory usage ratio monitoring module is used for storing the data in the corresponding storage unit and periodically calculating the memory usage ratio of the storage unit; defining a storage unit as a central storage unit, defining the storage unit directly connected with the central storage unit as a primary storage unit, defining the storage unit directly connected with the primary storage unit as a secondary storage unit except the central storage unit, and defining a multilayer hierarchical relationship by radiating outwards by the multilayer storage units in the same way;
the judging module is used for judging whether the memory usage proportion of the central storage unit is greater than the overflow proportion; the overflow occupation ratio is an early warning value that the memory usage of the central storage unit is about to reach the total memory amount;
and the data migration module is used for automatically migrating the data in the central storage unit to the first-level storage unit connected with the central storage unit when the memory usage proportion of the central storage unit is greater than the overflow proportion.
The invention has the beneficial effects that: storing the acquired data in corresponding storage units, and periodically calculating the memory usage ratio of the storage units; and continuously monitoring and judging whether the memory usage proportion of the central storage unit is greater than the overflow proportion, namely regularly monitoring whether the memory of the central storage unit is about to run out, and automatically transferring the data in the central storage unit to a primary storage unit connected with the central storage unit when the memory usage proportion of the central storage unit is greater than the overflow proportion. The data in the central storage unit is migrated to the primary storage units, so that the memory of the central storage unit is released, the data of the central storage unit is dispersedly migrated to the plurality of primary storage units connected with the central storage unit, the data size is not large, the data can be received by the primary storage units, and even if the primary storage units cannot receive the data, the data in the primary storage units can be continuously migrated to the lower-layer storage units, so that the data are gradually differentiated. The problem of among the prior art fixed memory cell, data memory space appears unbalance when the storage data, leads to some memory cell to load too big, the running speed slows down is solved.
Drawings
FIG. 1 is a schematic diagram illustrating the architecture of an environment in which a data storage method operates;
FIG. 2 is a main flow diagram of a graph data storage method;
FIG. 3 is a flow chart of data set partitioning to be migrated in a graph data storage method;
FIG. 4 is a flow chart of data migration of multi-level memory cells in a graph data storage method;
fig. 5 is a schematic diagram of the internal structure of a graph data storage system.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
The method comprises the steps of storing acquired data in corresponding storage units, and periodically calculating the memory usage proportion of the storage units; and continuously monitoring and judging whether the memory usage proportion of the central storage unit is greater than the overflow proportion, namely regularly monitoring whether the memory of the central storage unit is about to run out, and automatically transferring the data in the central storage unit to a primary storage unit connected with the central storage unit when the memory usage proportion of the central storage unit is greater than the overflow proportion. The data in the central storage unit is migrated to the primary storage units, so that the memory of the central storage unit is released, the data of the central storage unit is dispersedly migrated to the plurality of primary storage units connected with the central storage unit, the data size is not large, the data can be received by the primary storage units, and even if the primary storage units cannot receive the data, the data in the primary storage units can be continuously migrated to the lower-layer storage units, so that the data are gradually differentiated. The problem of among the prior art fixed memory cell, data memory space appears unbalance when the storage data, leads to some memory cell to load too big, the running speed slows down is solved.
Fig. 1 shows a schematic structural diagram of an operating environment of a graph data storage method according to an embodiment of the present invention, where a data generation device 1 may be a mobile device, or may be various operation terminals, or a data processing module, and different data generation devices 1 upload data to a graph data storage system, and after verification and allocation of the system, the data are stored in corresponding storage units 2, and the storage units 2 are mutually communicated, and when one storage unit 2 is about to overflow, the data in the storage unit may migrate to flow to other storage units 2.
Fig. 2 shows a main flow chart of a graph data storage method according to an embodiment of the present invention, where the graph data storage method includes:
step S10: data to be saved is acquired. The data generating device 1 continuously uploads the data to the graph data storage system, different types of data are correspondingly stored in different storage units 2, and the storage units 2 are communicated with each other because the data are correlated with each other. These memory units 2 may be physically independent, being small memories one by one; or a large data storage server is divided on a virtual level, the memory of the data storage server is divided into small storage units 2, and the storage units 2 are independent and communicated with each other. The latter method is preferable in practical operation for convenience of use and operation.
Step S11: and storing the data in the corresponding storage unit 2, and periodically calculating the memory usage ratio of the storage unit 2. Defining a storage unit as a central storage unit, defining a storage unit directly connected with the central storage unit as a primary storage unit, defining a storage unit directly connected with the primary storage unit as a secondary storage unit except the central storage unit, and defining a multilayer hierarchical relationship by radiating outwards by the multilayer storage units in the same way. The period can be set to be a fixed time period such as every hour, every day, every week and the like, on one hand, unnecessary running programs brought to the storage unit by real-time monitoring are avoided, processing resources of the storage unit are occupied, and on the other hand, the storage unit can be monitored.
Step S12: judging whether the memory usage ratio of the central memory unit is greater than the overflow ratio; the overflow occupation ratio is an early warning value that the memory usage of the central storage unit is about to reach the total memory amount, which indicates that the memory of the central storage unit is about to be used up, and at this time, data migration or capacity expansion needs to be performed on the central storage unit.
Step S13: and when the memory usage ratio of the central storage unit is greater than the overflow ratio, automatically transferring the data in the central storage unit to the primary storage unit connected with the central storage unit. Assuming A, B two data storage units, the data between them are related to each other, in order to establish A, B relationship, there are some related data, which may be stored in both a and B, belonging to unfixed-attribute data, and are randomly stored in A, B when they are stored, so that unfixed-attribute data can be migrated from an overfull storage unit to a surplus-memory storage unit when one of a or B is about to be overfull.
In one aspect of this embodiment, the storing the data in the corresponding storage unit, and periodically calculating the memory usage ratio of the storage unit specifically includes:
step S110: and storing the data in the corresponding storage unit.
Step S111: and carrying out timing monitoring on the storage unit, and triggering timing grabbing when the timing countdown time is zero.
Step S112: and capturing the memory usage amount and the total memory amount of the storage unit to obtain the memory usage amount ratio. The usage ratio can be obtained by dividing the total memory usage.
In one aspect of this embodiment, when the memory usage ratio of the central storage unit is greater than the overflow ratio, the automatically migrating the data in the central storage unit to the first-stage storage unit connected to the central storage unit specifically includes:
step S130: and when the memory usage ratio of the central storage unit is greater than the overflow ratio, acquiring the number of the first-level storage units connected with the central storage unit at the moment. These primary storage units are located to facilitate subsequent data migration and to determine the data sets to be migrated.
Step S131: and determining data to be migrated from the central storage unit to form a data set to be migrated.
Step S132: and distributing the data sets to be migrated in the central storage unit according to the number of the primary storage units, and migrating the data sets to the primary storage units. The distribution according to the number of the primary storage units may refer to average distribution according to the number, or non-uniform distribution according to the number, and non-uniform distribution is performed according to the memory usage of each primary storage unit.
In one case of this embodiment, the determining data to be migrated from the central storage unit and forming the data set to be migrated specifically includes:
step S20: when the memory usage ratio of the central storage unit is larger than the overflow ratio, forward calculation is carried out according to the time point of the last storage of different data as a reference, and data with fixed magnitude is divided to form a data set to be migrated. The latest batch of data is migrated, because the latest batch of data has larger variability and is only related to the previous data, and because the latest batch of data has no data in the following process, the relevance of the following data is not influenced, and the structure of the previous data is not influenced. The specific method for segmenting the data with the fixed magnitude to form the data set to be migrated comprises the following steps: forward reckoning according to the time point of the last storage of different data as a reference, and dividing data in a fixed time to form a data set to be migrated; or forward reckoning according to the time point of the last storage of different data as a reference, and dividing the data with a fixed size to form a data set to be migrated.
Step S21: and identifying the corresponding relation between the data set to be migrated and different primary storage units.
Fig. 3 shows a flow chart of dividing a data set to be migrated in a graph data storage method according to an embodiment of the present invention, and in another method, the determining data to be migrated in a central storage unit, and forming the data set to be migrated specifically includes:
step S30: and capturing the memory usage amount and the total memory amount of each primary storage unit.
Step S31: and calculating the residual memory amount of each primary storage unit according to the memory usage amount and the total memory amount.
Step S32: and obtaining the residual memory ratio among the plurality of primary storage units according to the residual memory amount of each primary storage unit.
Step S33: determining the total amount of data to be migrated in the central storage unit, distributing the total amount of data according to the ratio of the remaining memories, and determining the size of the data set to be migrated corresponding to each primary storage unit.
Step S34: and according to the size of the data set to be migrated corresponding to each primary storage unit, forward calculation is carried out by taking the time point of the last storage of different data as a reference, and the data set to be migrated with the corresponding size is divided. The migration data is distributed according to the memory surplus condition of each primary storage unit, so that the data distributed to each primary storage unit is the data amount which can be borne by the primary storage unit, and the condition that a certain primary storage unit receives data which exceeds the storage capacity of the certain primary storage unit is avoided.
In a case of this embodiment, the allocating the data sets to be migrated in the central storage unit according to the number of the primary storage units, and migrating to the primary storage unit specifically includes:
step S40: and respectively migrating the data sets to be migrated to the primary storage units according to the corresponding relation between the segmented data sets to be migrated and different primary storage units.
Fig. 4 shows a data migration flow chart of a multi-level storage unit in a graph data storage method according to an embodiment of the present invention, where after the data in the central storage unit is automatically migrated to the first-level storage unit connected to the central storage unit when the memory usage ratio of the central storage unit is greater than the overflow ratio, the method further includes:
step S130: and after the data in the central storage unit is transferred to the lower-level storage unit, judging whether the memory usage proportion of the lower-level storage unit is greater than the overflow proportion.
Step S131: and when the memory usage ratio of the lower-level storage unit is greater than the overflow ratio, automatically migrating the data in the lower-level storage unit to a lower-level storage unit connected with the lower-level storage unit.
Step S132: and stopping the storage unit receiving the migration data to continue data migration until the storage unit receiving the migration data finally judges that the memory usage ratio is not more than the overflow ratio. Data continuously flows outwards in the layer migration mode, so that the memories of the data storage units are mutually adjusted to achieve balance, and the memory of each data storage unit can be fully utilized.
In one aspect of this embodiment, after automatically migrating data in a lower-level storage unit to a lower-level storage unit connected to the lower-level storage unit when the memory usage percentage of the lower-level storage unit is greater than the overflow percentage, the method further includes:
step S50: when data are migrated to the last-stage storage unit, and after judgment, the memory usage ratio of the last-stage storage unit is still larger than the overflow-full ratio, the last-stage storage unit is taken as a reference to be pushed upwards, and memory expansion is carried out on all storage units on the data migration path where the last-stage storage unit is located. When the data is migrated to the last-stage storage unit, and the data in the storage unit is still about to overflow, it indicates that the remaining memories of all the storage units on the data chain where the storage unit is located are few or the migrated data cannot be redistributed in the next data migration, and at this time, the storage units may be queried reversely to expand the memories of the storage units.
Fig. 5 is a schematic diagram illustrating an internal structure of a graph data storage system according to an embodiment of the present invention, where the graph data storage system includes:
a data obtaining module 100, configured to obtain data to be saved.
The memory usage ratio monitoring module 200 is configured to store data in the corresponding storage unit 2, and periodically calculate a memory usage ratio of the storage unit 2; defining a storage unit as a central storage unit, defining a storage unit directly connected with the central storage unit as a primary storage unit, defining a storage unit directly connected with the primary storage unit as a secondary storage unit except the central storage unit, and defining a multilayer hierarchical relationship by radiating outward a multilayer storage unit in the same way.
A judging module 300, configured to judge whether the memory usage proportion of the central storage unit is greater than the overflow proportion; the overflow occupation ratio is an early warning value that the memory usage of the central storage unit is about to reach the total memory amount.
And the data migration module 400 is configured to automatically migrate data in the central storage unit to the first-level storage unit connected to the central storage unit when the memory usage occupancy of the central storage unit is greater than the overflow occupancy.
In order to load the above method and system to operate successfully, the system may include more or less components than those described above, or combine some components, or different components, in addition to the various modules described above, for example, input/output devices, network access devices, buses, processors, memories, and the like.
It should be understood that, although the steps in the flowcharts of the embodiments of the present invention are shown in sequence as indicated by the arrows, the steps are not necessarily executed in sequence as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least a portion of the steps in various embodiments may include multiple sub-steps or multiple stages that are not necessarily performed at the same time, but may be performed at different times, and the order of performance of the sub-steps or stages is not necessarily sequential, but may be performed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.
The technical features of the embodiments described above may be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the embodiments described above are not described, but should be considered as being within the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above-mentioned embodiments only represent some preferred embodiments of the present invention, and the description thereof is more specific and detailed, but not to be construed as limiting the scope of the present invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the inventive concept, which falls within the scope of the present invention. Therefore, the protection scope of the present patent shall be subject to the appended claims.
Claims (2)
1. A graph data storage method, the method comprising:
acquiring data to be saved;
storing the data in a corresponding storage unit, and periodically calculating the memory usage ratio of the storage unit; defining a storage unit as a central storage unit, defining the storage unit directly connected with the central storage unit as a primary storage unit, defining the storage unit directly connected with the primary storage unit as a secondary storage unit except the central storage unit, and defining a multilayer hierarchical relationship by radiating outwards by the multilayer storage units in the same way;
judging whether the memory usage ratio of the central memory unit is greater than the overflow ratio; the overflow occupation ratio is an early warning value when the memory usage amount of the central storage unit reaches the total memory amount;
when the memory usage ratio of the central storage unit is greater than the overflow ratio, automatically transferring the data in the central storage unit to a primary storage unit connected with the central storage unit;
after the data in the central storage unit is transferred to the lower-level storage unit, judging whether the memory usage proportion of the lower-level storage unit is greater than the overflow proportion;
when the memory usage ratio of the lower-level storage unit is greater than the overflow ratio, automatically transferring the data in the lower-level storage unit to a lower-level storage unit connected with the lower-level storage unit;
stopping the storage unit receiving the migration data from continuing data migration until the storage unit receiving the migration data finally judges that the memory usage proportion is not more than the overflow proportion;
when the memory usage proportion of the central storage unit is greater than the overflow proportion, the automatically transferring the data in the central storage unit to the first-level storage unit connected with the central storage unit specifically comprises:
when the memory usage ratio of the central storage unit is greater than the overflow ratio, acquiring the number of primary storage units connected with the central storage unit;
determining data to be migrated in the central storage unit to form a data set to be migrated;
distributing the data sets to be migrated in the central storage unit according to the number of the primary storage units, and migrating the data sets to the primary storage units;
when the memory usage proportion of the central storage unit is larger than the overflow proportion, forward calculation is carried out according to the time point of the last storage of different data as a reference, and data of a fixed magnitude is divided to form a data set to be migrated;
identifying the corresponding relation between the data set to be migrated and different primary storage units;
the storing the data in the corresponding storage unit and periodically calculating the memory usage ratio of the storage unit specifically include:
storing the data in the corresponding storage unit;
carrying out timing monitoring on the storage unit, and triggering timing grabbing when the timing countdown time is zero;
capturing the memory usage amount and the total memory amount of the storage unit to obtain the memory usage amount ratio;
the specific method for segmenting the data with the fixed magnitude to form the data set to be migrated comprises the following steps: forward reckoning according to the time point of the last storage of different data as a reference, and dividing data in a fixed time to form a data set to be migrated; or forward reckoning according to the time point of the last storage of different data as a reference, and dividing the data with fixed size to form a data set to be migrated;
the determining data to be migrated from the central storage unit and forming a data set to be migrated specifically includes:
capturing the memory usage amount and the total memory amount of each primary storage unit;
calculating the residual memory amount of each primary storage unit according to the memory usage amount and the total memory amount;
obtaining the ratio of the residual memory among the primary storage units according to the residual memory amount of each primary storage unit;
determining the total amount of data to be migrated in the central storage unit, distributing the total amount of data according to the ratio of the remaining memories, and determining the size of a data set to be migrated corresponding to each primary storage unit;
according to the size of the data set to be migrated corresponding to each primary storage unit, forward calculation is carried out by taking the time point of the last storage of different data as a reference, and the data set to be migrated with the corresponding size is divided;
when the memory usage occupation ratio of the lower-level storage unit is greater than the overflow occupation ratio, after the data in the lower-level storage unit is automatically migrated to the next-level storage unit connected with the lower-level storage unit, the method further includes:
when data are migrated to the last-stage storage unit, and after judgment, the memory usage ratio of the last-stage storage unit is still larger than the overflow-full ratio, the last-stage storage unit is taken as a reference to be pushed upwards, and memory expansion is carried out on all storage units on the data migration path where the last-stage storage unit is located.
2. The graph data storage method according to claim 1, wherein the allocating the data sets to be migrated in the central storage unit according to the number of the primary storage units, and the migrating to the primary storage units specifically includes:
and respectively migrating the data sets to be migrated to the primary storage units according to the corresponding relation between the segmented data sets to be migrated and different primary storage units.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111006840.5A CN113448970B (en) | 2021-08-31 | 2021-08-31 | Graph data storage method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111006840.5A CN113448970B (en) | 2021-08-31 | 2021-08-31 | Graph data storage method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113448970A CN113448970A (en) | 2021-09-28 |
| CN113448970B true CN113448970B (en) | 2022-07-12 |
Family
ID=77819041
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111006840.5A Active CN113448970B (en) | 2021-08-31 | 2021-08-31 | Graph data storage method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113448970B (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102341779A (en) * | 2009-03-02 | 2012-02-01 | 国际商业机器公司 | Method, system and computer program product for managing the placement of storage data in a multi tier virtualized storage infrastructure |
| CN103177006A (en) * | 2011-12-21 | 2013-06-26 | 北京昆仑万维科技股份有限公司 | Data storage system and method for updating storage capacity thereof |
| CN103870206A (en) * | 2012-12-13 | 2014-06-18 | 华为技术有限公司 | Caching data receiving and reading method and device and router cache device |
| CN109783472A (en) * | 2018-12-14 | 2019-05-21 | 深圳壹账通智能科技有限公司 | Moving method, device, computer equipment and the storage medium of table data |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP5070315B2 (en) * | 2010-04-28 | 2012-11-14 | 株式会社日立製作所 | Storage device and data hierarchy management method in storage device |
| US9201751B1 (en) * | 2011-04-18 | 2015-12-01 | American Megatrends, Inc. | Data migration between multiple tiers in a storage system using policy based ILM for QOS |
| CN102364474B (en) * | 2011-11-17 | 2014-08-20 | 中国科学院计算技术研究所 | Metadata storage system for cluster file system and metadata management method |
| CN102662855B (en) * | 2012-04-17 | 2015-02-25 | 华为技术有限公司 | Storage method and system of binary tree |
| CN107870916A (en) * | 2016-09-23 | 2018-04-03 | 伊姆西Ip控股有限责任公司 | Memory management method and equipment |
| CN110858124B (en) * | 2018-08-24 | 2021-06-01 | 华为技术有限公司 | Data migration method and device |
-
2021
- 2021-08-31 CN CN202111006840.5A patent/CN113448970B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102341779A (en) * | 2009-03-02 | 2012-02-01 | 国际商业机器公司 | Method, system and computer program product for managing the placement of storage data in a multi tier virtualized storage infrastructure |
| CN103177006A (en) * | 2011-12-21 | 2013-06-26 | 北京昆仑万维科技股份有限公司 | Data storage system and method for updating storage capacity thereof |
| CN103870206A (en) * | 2012-12-13 | 2014-06-18 | 华为技术有限公司 | Caching data receiving and reading method and device and router cache device |
| CN109783472A (en) * | 2018-12-14 | 2019-05-21 | 深圳壹账通智能科技有限公司 | Moving method, device, computer equipment and the storage medium of table data |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113448970A (en) | 2021-09-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110166282B (en) | Resource allocation method, device, computer equipment and storage medium | |
| Tang et al. | An intermediate data placement algorithm for load balancing in spark computing environment | |
| CN108920153B (en) | Docker container dynamic scheduling method based on load prediction | |
| CN106534318B (en) | A kind of OpenStack cloud platform resource dynamic scheduling system and method based on flow compatibility | |
| CN110221915B (en) | Node scheduling method and device | |
| CN106354729B (en) | Graph data processing method, device and system | |
| US20080163237A1 (en) | Method of changing system configuration in shared-nothing database management system | |
| CN110147407B (en) | Data processing method and device and database management server | |
| CN107436813A (en) | A kind of method and system of meta data server dynamic load leveling | |
| CN105975345B (en) | A kind of video requency frame data dynamic equalization memory management method based on distributed memory | |
| CN103714098A (en) | Method and system used for sectioning data base | |
| CN107450855A (en) | A kind of model for distributed storage variable data distribution method and system | |
| CN107844402A (en) | A kind of resource monitoring method, device and terminal based on super fusion storage system | |
| CN112637263B (en) | Multi-data center resource optimization promotion method and system and storage medium | |
| CN104158902B (en) | A kind of Hbase data blocks distribution method and device based on number of request | |
| CN113448970B (en) | Graph data storage method and system | |
| CN106648829A (en) | Virtual machine transferring method for efficient utilization of cloud resource | |
| CN105635285A (en) | State-sensing-based VM migration scheduling method | |
| CN108932258A (en) | Data directory processing method and processing device | |
| CN112948113A (en) | Cluster resource management scheduling method, device, equipment and readable storage medium | |
| Guo et al. | Handling data skew at reduce stage in Spark by ReducePartition | |
| Irandoost et al. | Learning automata-based algorithms for MapReduce data skewness handling | |
| CN108958967A (en) | A kind of method and server of data processing | |
| Inostrosa-Psijas et al. | Load balance strategies for DEVS approximated parallel and distributed discrete-event simulations | |
| CN112395269A (en) | Method and device for building MySQL high-availability group |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |