CN117931083A - Data distributed storage local data access method and system - Google Patents
Data distributed storage local data access method and system Download PDFInfo
- Publication number
- CN117931083A CN117931083A CN202410131365.1A CN202410131365A CN117931083A CN 117931083 A CN117931083 A CN 117931083A CN 202410131365 A CN202410131365 A CN 202410131365A CN 117931083 A CN117931083 A CN 117931083A
- Authority
- CN
- China
- Prior art keywords
- data
- access request
- access
- distributed storage
- node
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 29
- 239000012634 fragment Substances 0.000 claims abstract description 55
- 238000013507 mapping Methods 0.000 claims abstract description 29
- 230000010354 integration Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000008450 motivation Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0604—Improving or facilitating administration, e.g. storage management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0629—Configuration or reconfiguration of storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/60—Scheduling or organising the servicing of application requests, e.g. requests for application data transmissions using the analysis and optimisation of the required network resources
- H04L67/63—Routing a service request depending on the request content or context
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application relates to a data distributed storage local data access method and a system, comprising the following steps: analyzing and processing the historical local data access request in a big data row to obtain a plurality of different access request items, and establishing a mapping association relation between the access request items and a plurality of data fragments; constructing access request nodes corresponding to the access request items one by one, and constructing the connection between the access request nodes and the corresponding storage nodes; by adopting the method, the large data is utilized to analyze and process to generate a plurality of access request items, a new access request node is constructed, and the connection between the access request node and the corresponding storage node is constructed according to the mapping association relation between the access request items and the data fragments, so that when the local data access is carried out, the request can be directly identified and transferred to the corresponding access request node for data processing, various local access requests can be well considered, and meanwhile, the conflict with the normal distributed storage service does not exist.
Description
Technical Field
The invention relates to the technical field of distributed storage, in particular to a data distributed storage local data access method and system.
Background
The distributed network storage system adopts an expandable system structure, utilizes a plurality of storage nodes to share the storage load, and utilizes the position server to position the storage information, so that the reliability, the availability and the access efficiency of the system are improved, and the system is easy to expand;
For local data access requirements, the existing data slicing mode is generally difficult to better cope with, because the motivation of the access requirements is often diversified, the slicing strategy is too single, the diversified local data access requirements are difficult to meet, and a data distributed storage local data access method and system capable of improving the situation are needed.
Disclosure of Invention
The technical problem to be solved by the invention is to provide a data distributed storage local data access method and a data distributed storage local data access system aiming at the defects in the prior art.
The technical scheme adopted for solving the technical problems is as follows:
a data distributed storage local data access method is constructed, which comprises the following steps:
The first step: the distributed storage server performs data slicing processing on the data to be stored, and performs content identification on the data slicing;
And a second step of: analyzing and processing the historical local data access request in a big data row to obtain a plurality of different access request items;
and a third step of: establishing a mapping association relation between the access request item and a plurality of data fragments according to the access request item;
Fourth step: the data fragments are stored in a distributed mode to different storage nodes according to a distributed storage strategy;
Fifth step: constructing access request nodes corresponding to the access request items one by one, and constructing the connection between the access request nodes and the corresponding storage nodes according to the mapping association relation between the access request items and the data fragments;
Sixth step: a connection is established between the access requesting node and the distributed storage server.
The invention relates to a data distributed storage local data access method, wherein the method further comprises the following steps:
When local data access is performed, sending a local data access request to a distributed storage server;
The distributed storage server analyzes and identifies the request and matches the request in a plurality of access request nodes;
the access request node successfully matched reads the stored data fragment content in the corresponding storage node and combines the data fragment content into target data;
and sending the target data to a distributed storage server.
The invention relates to a data distributed storage local data access method, wherein when a data update request is received by a distributed storage server, data update contents are identified, and the identified update contents are corresponding to data fragments;
Judging whether the mapping association relation between the data fragments and the access request item needs to be adjusted according to the updated data fragment content, if so, changing, otherwise, not acting;
and correspondingly changing the relation between the access request node and the corresponding storage node for the change of the mapping incidence relation.
The invention relates to a data distributed storage local data access method, wherein an access path between an access request node and a distributed storage server and an access path between the distributed storage server and a storage node are mutually independent.
A data distributed storage local data access system comprises a distributed storage server, a plurality of storage nodes, a plurality of access request nodes and a local data access processing unit;
the distributed storage server performs data slicing processing on the data to be stored and performs content identification on the data slicing; the data fragments are stored in a distributed mode to different storage nodes according to a distributed storage strategy;
The local data access processing unit analyzes and processes the historical local data access request in a big data row to obtain a plurality of different access request items; establishing a mapping association relation between the access request item and a plurality of data fragments according to the access request item; constructing access request nodes corresponding to the access request items one by one, and constructing the connection between the access request nodes and the corresponding storage nodes according to the mapping association relation between the access request items and the data fragments; a connection is established between the access requesting node and the distributed storage server.
The distributed storage server is also used for receiving local data access requests, carrying out request analysis and identification, and carrying out matching in a plurality of access request nodes;
And the access request node which is successfully matched reads the stored data fragment content in the corresponding storage node, combines the data fragment content into target data, and sends the target data to the distributed storage server.
The invention relates to a data distributed storage local data access system, wherein when a data update request is received, a distributed storage server identifies data update content and corresponds the identified update content to data fragments; judging whether the mapping association relation between the data fragments and the access request item needs to be adjusted according to the updated data fragment content, if so, changing, otherwise, not acting; and correspondingly changing the relation between the access request node and the corresponding storage node for the change of the mapping incidence relation.
The data distributed storage local data access system provided by the invention is characterized in that the access path between the access request node and the distributed storage server is mutually independent from the access path between the distributed storage server and the storage node.
The application has the beneficial effects that: by adopting the method, a plurality of access request items are generated by analyzing and processing big data, a new access request node is constructed, and the connection between the access request node and the corresponding storage node is constructed according to the mapping association relation between the access request item and the data fragments, so that when partial data access is performed, the request can be directly identified and transferred to the corresponding access request node for data processing, various partial access requests can be better considered by analyzing the big data, no conflict exists between the access request item and normal distributed storage service, and the consumption of computational resources and network resources of distributed storage is low.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the present invention will be further described with reference to the accompanying drawings and embodiments, in which the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained by those skilled in the art without inventive effort:
FIG. 1 is a flow chart of a method for accessing local data in a distributed data storage according to a preferred embodiment of the present invention;
FIG. 2 is a schematic block diagram of a data distributed storage local data access system in accordance with a preferred embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the following description will be made in detail with reference to the technical solutions in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by a person skilled in the art without any inventive effort, are intended to be within the scope of the present invention, based on the embodiments of the present invention.
The data distributed storage local data access method of the preferred embodiment of the invention, as shown in fig. 1, comprises the following steps:
s01: the distributed storage server performs data slicing processing on the data to be stored, and performs content identification on the data slicing;
s02: analyzing and processing the historical local data access request in a big data row to obtain a plurality of different access request items;
s03: establishing a mapping association relation between the access request item and a plurality of data fragments according to the access request item;
S04: the data fragments are stored in a distributed mode to different storage nodes according to a distributed storage strategy;
S05: constructing access request nodes corresponding to the access request items one by one, and constructing the connection between the access request nodes and the corresponding storage nodes according to the mapping association relation between the access request items and the data fragments;
s06: constructing a connection between the access request node and the distributed storage server;
By adopting the method, a plurality of access request items are generated by analyzing and processing big data, a new access request node is constructed, and the connection between the access request node and the corresponding storage node is constructed according to the mapping association relation between the access request item and the data fragments, so that when partial data access is performed, the request can be directly identified and transferred to the corresponding access request node for data processing, various partial access requests can be better considered by analyzing the big data, and meanwhile, no conflict exists between the normal distributed storage service.
When the local data access is performed, the method can be adopted:
Sending a local data access request to a distributed storage server;
The distributed storage server analyzes and identifies the request and matches the request in a plurality of access request nodes;
the access request node successfully matched reads the stored data fragment content in the corresponding storage node and combines the data fragment content into target data;
transmitting the target data to a distributed storage server;
the distributed storage server only needs to identify and match tasks, no matter the specific data integration and processing (a plurality of data fragments may have combined requirements according to different access requirements), and the consumption of computational resources of the distributed storage server is also less.
Preferably, the access path between the access request node and the distributed storage server is independent of the access path between the distributed storage server and the storage node, i.e. the local access and the normal access adopt two completely different lines, so that the two can be processed in parallel without collision.
Preferably, when updating data, the corresponding access request node needs to be updated, and the method can adopt:
when the distributed storage server receives the data updating request, the data updating content is identified, and the identified updating content is corresponding to the data fragments;
Judging whether the mapping association relation between the data fragments and the access request item needs to be adjusted according to the updated data fragment content, if so, changing, otherwise, not acting;
Correspondingly changing the relation between the access request node and the corresponding storage node for the change of the mapping incidence relation;
the updating process is convenient, and the updating speed is also faster.
A data distributed storage local data access system, as shown in fig. 2, comprises a distributed storage server 1, a plurality of storage nodes 2, a plurality of access request nodes 3 and a local data access processing unit 4;
The distributed storage server 1 performs data slicing processing on data to be stored and performs content identification on the data slicing; the data fragments are stored in a distributed mode to different storage nodes 2 according to a distributed storage strategy;
The local data access processing unit 4 analyzes and processes the historical local data access request in a big data row to obtain a plurality of different access request items; establishing a mapping association relation between the access request item and a plurality of data fragments according to the access request item; constructing access request nodes corresponding to the access request items one by one, and constructing the connection between the access request nodes 3 and the corresponding storage nodes 2 according to the mapping association relation between the access request items and the data fragments; constructing a connection between the access request node and the distributed storage server;
The system of the application utilizes big data to analyze and process to generate a plurality of access request items, constructs new access request nodes, constructs the connection between the access request nodes and corresponding storage nodes according to the mapping association relation between the access request items and the data fragments, can directly identify requests and flow to the corresponding access request nodes for data processing when local data access is carried out, and can better consider various local access requests by virtue of big data analysis, and meanwhile, has no conflict with normal distributed storage service.
Preferably, the distributed storage server 1 is further configured to receive a local data access request, analyze and identify the request, and match the request in a plurality of access request nodes;
the access request node successfully matched reads the stored data fragment content in the corresponding storage node and combines the data fragment content into target data, and the target data is sent to the distributed storage server;
the distributed storage server only needs to identify and match tasks, no matter the specific data integration and processing (a plurality of data fragments may have combined requirements according to different access requirements), and the consumption of computational resources of the distributed storage server is also less.
Preferably, the distributed storage server 1 identifies the data update content when receiving the data update request, and corresponds the identified update content to the data fragment; judging whether the mapping association relation between the data fragments and the access request item needs to be adjusted according to the updated data fragment content, if so, changing, otherwise, not acting; correspondingly changing the relation between the access request node and the corresponding storage node for the change of the mapping incidence relation; the updating process is convenient, and the updating speed is also faster.
Preferably, the access path between the access request node and the distributed storage server is independent of the access path between the distributed storage server and the storage node, i.e. the local access and the normal access adopt two completely different lines, so that the two can be processed in parallel without collision.
It will be understood that modifications and variations will be apparent to those skilled in the art from the foregoing description, and it is intended that all such modifications and variations be included within the scope of the following claims.
Claims (8)
1.A method for accessing local data in a data distributed storage, comprising the steps of:
The first step: the distributed storage server performs data slicing processing on the data to be stored, and performs content identification on the data slicing;
And a second step of: analyzing and processing the historical local data access request in a big data row to obtain a plurality of different access request items;
and a third step of: establishing a mapping association relation between the access request item and a plurality of data fragments according to the access request item;
Fourth step: the data fragments are stored in a distributed mode to different storage nodes according to a distributed storage strategy;
Fifth step: constructing access request nodes corresponding to the access request items one by one, and constructing the connection between the access request nodes and the corresponding storage nodes according to the mapping association relation between the access request items and the data fragments;
Sixth step: a connection is established between the access requesting node and the distributed storage server.
2. The method for accessing local data for data distributed storage according to claim 1, further comprising the steps of:
When local data access is performed, sending a local data access request to a distributed storage server;
The distributed storage server analyzes and identifies the request and matches the request in a plurality of access request nodes;
the access request node successfully matched reads the stored data fragment content in the corresponding storage node and combines the data fragment content into target data;
and sending the target data to a distributed storage server.
3. The data distributed storage local data access method according to claim 1, wherein when the distributed storage server receives a data update request, the data update content is identified, and the identified update content is corresponding to the data fragment;
Judging whether the mapping association relation between the data fragments and the access request item needs to be adjusted according to the updated data fragment content, if so, changing, otherwise, not acting;
and correspondingly changing the relation between the access request node and the corresponding storage node for the change of the mapping incidence relation.
4. The method of claim 1, wherein the access path from the access requesting node to the distributed storage server and the access path from the distributed storage server to the storage node are independent of each other.
5. The data distributed storage local data access system is characterized by comprising a distributed storage server, a plurality of storage nodes, a plurality of access request nodes and a local data access processing unit;
the distributed storage server performs data slicing processing on the data to be stored and performs content identification on the data slicing; the data fragments are stored in a distributed mode to different storage nodes according to a distributed storage strategy;
The local data access processing unit analyzes and processes the historical local data access request in a big data row to obtain a plurality of different access request items; establishing a mapping association relation between the access request item and a plurality of data fragments according to the access request item; constructing access request nodes corresponding to the access request items one by one, and constructing the connection between the access request nodes and the corresponding storage nodes according to the mapping association relation between the access request items and the data fragments; a connection is established between the access requesting node and the distributed storage server.
6. The data distributed storage local data access system according to claim 5, wherein the distributed storage server is further configured to receive local data access requests, perform request parsing and identification, and perform matching in a plurality of access request nodes;
And the access request node which is successfully matched reads the stored data fragment content in the corresponding storage node, combines the data fragment content into target data, and sends the target data to the distributed storage server.
7. The data distributed storage local data access system according to claim 5, wherein the distributed storage server, when receiving the data update request, identifies the data update content, and corresponds the identified update content to the data fragment; judging whether the mapping association relation between the data fragments and the access request item needs to be adjusted according to the updated data fragment content, if so, changing, otherwise, not acting; and correspondingly changing the relation between the access request node and the corresponding storage node for the change of the mapping incidence relation.
8. The data distributed storage local data access system of claim 5, wherein the access path between the access requesting node and the distributed storage server is independent of the access path between the distributed storage server and the storage node.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202410131365.1A CN117931083A (en) | 2024-01-31 | 2024-01-31 | Data distributed storage local data access method and system |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202410131365.1A CN117931083A (en) | 2024-01-31 | 2024-01-31 | Data distributed storage local data access method and system |
Publications (1)
Publication Number | Publication Date |
---|---|
CN117931083A true CN117931083A (en) | 2024-04-26 |
Family
ID=90766567
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202410131365.1A Pending CN117931083A (en) | 2024-01-31 | 2024-01-31 | Data distributed storage local data access method and system |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN117931083A (en) |
-
2024
- 2024-01-31 CN CN202410131365.1A patent/CN117931083A/en active Pending
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN110119292B (en) | System operation parameter query method, matching method, device and node equipment | |
CN111800458A (en) | Dynamic load balancing method and system for Kubernetes container cloud platform | |
EP4095768A1 (en) | Application recognition model updating method and apparatus, and storage medium | |
CN106991008B (en) | Resource lock management method, related equipment and system | |
CN113656176B (en) | Cloud equipment distribution method, device and system, electronic equipment, medium and product | |
US11922059B2 (en) | Method and device for distributed data storage | |
CN101673217B (en) | Method for realizing remote program call and system thereof | |
CN105681404A (en) | Metadata node management method and device of distributed cache system | |
CN112579319A (en) | Service calling method and device based on LRU Cache optimization | |
US20130318244A1 (en) | System and method for assigning server to terminal and efficiently delivering messages to the terminal | |
US7783810B2 (en) | Apparatus and method of processing information | |
CN114490100B (en) | Message queue telemetry transmission load balancing method, device and server | |
CN111581226B (en) | Data sharing method and device based on big data platform and administrative enterprise cloud platform | |
CN111600929B (en) | Transmission line detection method, routing strategy generation method and proxy server | |
CN117931083A (en) | Data distributed storage local data access method and system | |
CN112055077A (en) | Block link access method, device and storage medium | |
CN112052104A (en) | Message queue management method based on multi-computer-room realization and electronic equipment | |
US8548994B2 (en) | Reducing overheads in application processing | |
CN115185658A (en) | Task unloading scheduling method based on time and communication reliability and related product | |
CN112799849A (en) | Data processing method, device, equipment and storage medium | |
CN113783826A (en) | Micro-service exception handling method and device | |
CN113285997B (en) | Data processing method, device, medium and product based on heterogeneous system | |
CN112187816B (en) | RPC request processing method, equipment and storage medium | |
CN111782938B (en) | Information query pushing method and device and computer equipment | |
JP2000040051A (en) | Method and device for transmitting message in client server system |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination |