CN117931083A - Data distributed storage local data access method and system - Google Patents

Data distributed storage local data access method and system Download PDF

Info

Publication number
CN117931083A
CN117931083A CN202410131365.1A CN202410131365A CN117931083A CN 117931083 A CN117931083 A CN 117931083A CN 202410131365 A CN202410131365 A CN 202410131365A CN 117931083 A CN117931083 A CN 117931083A
Authority
CN
China
Prior art keywords
data
access request
access
distributed storage
node
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202410131365.1A
Other languages
Chinese (zh)
Inventor
张立旭
黎明
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Youwang Cloud Computing Co ltd
Original Assignee
Youwang Cloud Computing Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Youwang Cloud Computing Co ltd filed Critical Youwang Cloud Computing Co ltd
Priority to CN202410131365.1A priority Critical patent/CN117931083A/en
Publication of CN117931083A publication Critical patent/CN117931083A/en
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/0604—Improving or facilitating administration, e.g. storage management
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0629—Configuration or reconfiguration of storage systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0638—Organizing or formatting or addressing of data
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/50—Network services
    • H04L67/60—Scheduling or organising the servicing of application requests, e.g. requests for application data transmissions using the analysis and optimisation of the required network resources
    • H04L67/63—Routing a service request depending on the request content or context

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The application relates to a data distributed storage local data access method and a system, comprising the following steps: analyzing and processing the historical local data access request in a big data row to obtain a plurality of different access request items, and establishing a mapping association relation between the access request items and a plurality of data fragments; constructing access request nodes corresponding to the access request items one by one, and constructing the connection between the access request nodes and the corresponding storage nodes; by adopting the method, the large data is utilized to analyze and process to generate a plurality of access request items, a new access request node is constructed, and the connection between the access request node and the corresponding storage node is constructed according to the mapping association relation between the access request items and the data fragments, so that when the local data access is carried out, the request can be directly identified and transferred to the corresponding access request node for data processing, various local access requests can be well considered, and meanwhile, the conflict with the normal distributed storage service does not exist.

Description

Data distributed storage local data access method and system
Technical Field
The invention relates to the technical field of distributed storage, in particular to a data distributed storage local data access method and system.
Background
The distributed network storage system adopts an expandable system structure, utilizes a plurality of storage nodes to share the storage load, and utilizes the position server to position the storage information, so that the reliability, the availability and the access efficiency of the system are improved, and the system is easy to expand;
For local data access requirements, the existing data slicing mode is generally difficult to better cope with, because the motivation of the access requirements is often diversified, the slicing strategy is too single, the diversified local data access requirements are difficult to meet, and a data distributed storage local data access method and system capable of improving the situation are needed.
Disclosure of Invention
The technical problem to be solved by the invention is to provide a data distributed storage local data access method and a data distributed storage local data access system aiming at the defects in the prior art.
The technical scheme adopted for solving the technical problems is as follows:
a data distributed storage local data access method is constructed, which comprises the following steps:
The first step: the distributed storage server performs data slicing processing on the data to be stored, and performs content identification on the data slicing;
And a second step of: analyzing and processing the historical local data access request in a big data row to obtain a plurality of different access request items;
and a third step of: establishing a mapping association relation between the access request item and a plurality of data fragments according to the access request item;
Fourth step: the data fragments are stored in a distributed mode to different storage nodes according to a distributed storage strategy;
Fifth step: constructing access request nodes corresponding to the access request items one by one, and constructing the connection between the access request nodes and the corresponding storage nodes according to the mapping association relation between the access request items and the data fragments;
Sixth step: a connection is established between the access requesting node and the distributed storage server.
The invention relates to a data distributed storage local data access method, wherein the method further comprises the following steps:
When local data access is performed, sending a local data access request to a distributed storage server;
The distributed storage server analyzes and identifies the request and matches the request in a plurality of access request nodes;
the access request node successfully matched reads the stored data fragment content in the corresponding storage node and combines the data fragment content into target data;
and sending the target data to a distributed storage server.
The invention relates to a data distributed storage local data access method, wherein when a data update request is received by a distributed storage server, data update contents are identified, and the identified update contents are corresponding to data fragments;
Judging whether the mapping association relation between the data fragments and the access request item needs to be adjusted according to the updated data fragment content, if so, changing, otherwise, not acting;
and correspondingly changing the relation between the access request node and the corresponding storage node for the change of the mapping incidence relation.
The invention relates to a data distributed storage local data access method, wherein an access path between an access request node and a distributed storage server and an access path between the distributed storage server and a storage node are mutually independent.
A data distributed storage local data access system comprises a distributed storage server, a plurality of storage nodes, a plurality of access request nodes and a local data access processing unit;
the distributed storage server performs data slicing processing on the data to be stored and performs content identification on the data slicing; the data fragments are stored in a distributed mode to different storage nodes according to a distributed storage strategy;
The local data access processing unit analyzes and processes the historical local data access request in a big data row to obtain a plurality of different access request items; establishing a mapping association relation between the access request item and a plurality of data fragments according to the access request item; constructing access request nodes corresponding to the access request items one by one, and constructing the connection between the access request nodes and the corresponding storage nodes according to the mapping association relation between the access request items and the data fragments; a connection is established between the access requesting node and the distributed storage server.
The distributed storage server is also used for receiving local data access requests, carrying out request analysis and identification, and carrying out matching in a plurality of access request nodes;
And the access request node which is successfully matched reads the stored data fragment content in the corresponding storage node, combines the data fragment content into target data, and sends the target data to the distributed storage server.
The invention relates to a data distributed storage local data access system, wherein when a data update request is received, a distributed storage server identifies data update content and corresponds the identified update content to data fragments; judging whether the mapping association relation between the data fragments and the access request item needs to be adjusted according to the updated data fragment content, if so, changing, otherwise, not acting; and correspondingly changing the relation between the access request node and the corresponding storage node for the change of the mapping incidence relation.
The data distributed storage local data access system provided by the invention is characterized in that the access path between the access request node and the distributed storage server is mutually independent from the access path between the distributed storage server and the storage node.
The application has the beneficial effects that: by adopting the method, a plurality of access request items are generated by analyzing and processing big data, a new access request node is constructed, and the connection between the access request node and the corresponding storage node is constructed according to the mapping association relation between the access request item and the data fragments, so that when partial data access is performed, the request can be directly identified and transferred to the corresponding access request node for data processing, various partial access requests can be better considered by analyzing the big data, no conflict exists between the access request item and normal distributed storage service, and the consumption of computational resources and network resources of distributed storage is low.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the present invention will be further described with reference to the accompanying drawings and embodiments, in which the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained by those skilled in the art without inventive effort:
FIG. 1 is a flow chart of a method for accessing local data in a distributed data storage according to a preferred embodiment of the present invention;
FIG. 2 is a schematic block diagram of a data distributed storage local data access system in accordance with a preferred embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the following description will be made in detail with reference to the technical solutions in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by a person skilled in the art without any inventive effort, are intended to be within the scope of the present invention, based on the embodiments of the present invention.
The data distributed storage local data access method of the preferred embodiment of the invention, as shown in fig. 1, comprises the following steps:
s01: the distributed storage server performs data slicing processing on the data to be stored, and performs content identification on the data slicing;
s02: analyzing and processing the historical local data access request in a big data row to obtain a plurality of different access request items;
s03: establishing a mapping association relation between the access request item and a plurality of data fragments according to the access request item;
S04: the data fragments are stored in a distributed mode to different storage nodes according to a distributed storage strategy;
S05: constructing access request nodes corresponding to the access request items one by one, and constructing the connection between the access request nodes and the corresponding storage nodes according to the mapping association relation between the access request items and the data fragments;
s06: constructing a connection between the access request node and the distributed storage server;
By adopting the method, a plurality of access request items are generated by analyzing and processing big data, a new access request node is constructed, and the connection between the access request node and the corresponding storage node is constructed according to the mapping association relation between the access request item and the data fragments, so that when partial data access is performed, the request can be directly identified and transferred to the corresponding access request node for data processing, various partial access requests can be better considered by analyzing the big data, and meanwhile, no conflict exists between the normal distributed storage service.
When the local data access is performed, the method can be adopted:
Sending a local data access request to a distributed storage server;
The distributed storage server analyzes and identifies the request and matches the request in a plurality of access request nodes;
the access request node successfully matched reads the stored data fragment content in the corresponding storage node and combines the data fragment content into target data;
transmitting the target data to a distributed storage server;
the distributed storage server only needs to identify and match tasks, no matter the specific data integration and processing (a plurality of data fragments may have combined requirements according to different access requirements), and the consumption of computational resources of the distributed storage server is also less.
Preferably, the access path between the access request node and the distributed storage server is independent of the access path between the distributed storage server and the storage node, i.e. the local access and the normal access adopt two completely different lines, so that the two can be processed in parallel without collision.
Preferably, when updating data, the corresponding access request node needs to be updated, and the method can adopt:
when the distributed storage server receives the data updating request, the data updating content is identified, and the identified updating content is corresponding to the data fragments;
Judging whether the mapping association relation between the data fragments and the access request item needs to be adjusted according to the updated data fragment content, if so, changing, otherwise, not acting;
Correspondingly changing the relation between the access request node and the corresponding storage node for the change of the mapping incidence relation;
the updating process is convenient, and the updating speed is also faster.
A data distributed storage local data access system, as shown in fig. 2, comprises a distributed storage server 1, a plurality of storage nodes 2, a plurality of access request nodes 3 and a local data access processing unit 4;
The distributed storage server 1 performs data slicing processing on data to be stored and performs content identification on the data slicing; the data fragments are stored in a distributed mode to different storage nodes 2 according to a distributed storage strategy;
The local data access processing unit 4 analyzes and processes the historical local data access request in a big data row to obtain a plurality of different access request items; establishing a mapping association relation between the access request item and a plurality of data fragments according to the access request item; constructing access request nodes corresponding to the access request items one by one, and constructing the connection between the access request nodes 3 and the corresponding storage nodes 2 according to the mapping association relation between the access request items and the data fragments; constructing a connection between the access request node and the distributed storage server;
The system of the application utilizes big data to analyze and process to generate a plurality of access request items, constructs new access request nodes, constructs the connection between the access request nodes and corresponding storage nodes according to the mapping association relation between the access request items and the data fragments, can directly identify requests and flow to the corresponding access request nodes for data processing when local data access is carried out, and can better consider various local access requests by virtue of big data analysis, and meanwhile, has no conflict with normal distributed storage service.
Preferably, the distributed storage server 1 is further configured to receive a local data access request, analyze and identify the request, and match the request in a plurality of access request nodes;
the access request node successfully matched reads the stored data fragment content in the corresponding storage node and combines the data fragment content into target data, and the target data is sent to the distributed storage server;
the distributed storage server only needs to identify and match tasks, no matter the specific data integration and processing (a plurality of data fragments may have combined requirements according to different access requirements), and the consumption of computational resources of the distributed storage server is also less.
Preferably, the distributed storage server 1 identifies the data update content when receiving the data update request, and corresponds the identified update content to the data fragment; judging whether the mapping association relation between the data fragments and the access request item needs to be adjusted according to the updated data fragment content, if so, changing, otherwise, not acting; correspondingly changing the relation between the access request node and the corresponding storage node for the change of the mapping incidence relation; the updating process is convenient, and the updating speed is also faster.
Preferably, the access path between the access request node and the distributed storage server is independent of the access path between the distributed storage server and the storage node, i.e. the local access and the normal access adopt two completely different lines, so that the two can be processed in parallel without collision.
It will be understood that modifications and variations will be apparent to those skilled in the art from the foregoing description, and it is intended that all such modifications and variations be included within the scope of the following claims.

Claims (8)

1.A method for accessing local data in a data distributed storage, comprising the steps of:
The first step: the distributed storage server performs data slicing processing on the data to be stored, and performs content identification on the data slicing;
And a second step of: analyzing and processing the historical local data access request in a big data row to obtain a plurality of different access request items;
and a third step of: establishing a mapping association relation between the access request item and a plurality of data fragments according to the access request item;
Fourth step: the data fragments are stored in a distributed mode to different storage nodes according to a distributed storage strategy;
Fifth step: constructing access request nodes corresponding to the access request items one by one, and constructing the connection between the access request nodes and the corresponding storage nodes according to the mapping association relation between the access request items and the data fragments;
Sixth step: a connection is established between the access requesting node and the distributed storage server.
2. The method for accessing local data for data distributed storage according to claim 1, further comprising the steps of:
When local data access is performed, sending a local data access request to a distributed storage server;
The distributed storage server analyzes and identifies the request and matches the request in a plurality of access request nodes;
the access request node successfully matched reads the stored data fragment content in the corresponding storage node and combines the data fragment content into target data;
and sending the target data to a distributed storage server.
3. The data distributed storage local data access method according to claim 1, wherein when the distributed storage server receives a data update request, the data update content is identified, and the identified update content is corresponding to the data fragment;
Judging whether the mapping association relation between the data fragments and the access request item needs to be adjusted according to the updated data fragment content, if so, changing, otherwise, not acting;
and correspondingly changing the relation between the access request node and the corresponding storage node for the change of the mapping incidence relation.
4. The method of claim 1, wherein the access path from the access requesting node to the distributed storage server and the access path from the distributed storage server to the storage node are independent of each other.
5. The data distributed storage local data access system is characterized by comprising a distributed storage server, a plurality of storage nodes, a plurality of access request nodes and a local data access processing unit;
the distributed storage server performs data slicing processing on the data to be stored and performs content identification on the data slicing; the data fragments are stored in a distributed mode to different storage nodes according to a distributed storage strategy;
The local data access processing unit analyzes and processes the historical local data access request in a big data row to obtain a plurality of different access request items; establishing a mapping association relation between the access request item and a plurality of data fragments according to the access request item; constructing access request nodes corresponding to the access request items one by one, and constructing the connection between the access request nodes and the corresponding storage nodes according to the mapping association relation between the access request items and the data fragments; a connection is established between the access requesting node and the distributed storage server.
6. The data distributed storage local data access system according to claim 5, wherein the distributed storage server is further configured to receive local data access requests, perform request parsing and identification, and perform matching in a plurality of access request nodes;
And the access request node which is successfully matched reads the stored data fragment content in the corresponding storage node, combines the data fragment content into target data, and sends the target data to the distributed storage server.
7. The data distributed storage local data access system according to claim 5, wherein the distributed storage server, when receiving the data update request, identifies the data update content, and corresponds the identified update content to the data fragment; judging whether the mapping association relation between the data fragments and the access request item needs to be adjusted according to the updated data fragment content, if so, changing, otherwise, not acting; and correspondingly changing the relation between the access request node and the corresponding storage node for the change of the mapping incidence relation.
8. The data distributed storage local data access system of claim 5, wherein the access path between the access requesting node and the distributed storage server is independent of the access path between the distributed storage server and the storage node.
CN202410131365.1A 2024-01-31 2024-01-31 Data distributed storage local data access method and system Pending CN117931083A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202410131365.1A CN117931083A (en) 2024-01-31 2024-01-31 Data distributed storage local data access method and system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202410131365.1A CN117931083A (en) 2024-01-31 2024-01-31 Data distributed storage local data access method and system

Publications (1)

Publication Number Publication Date
CN117931083A true CN117931083A (en) 2024-04-26

Family

ID=90766567

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202410131365.1A Pending CN117931083A (en) 2024-01-31 2024-01-31 Data distributed storage local data access method and system

Country Status (1)

Country Link
CN (1) CN117931083A (en)

Similar Documents

Publication Publication Date Title
CN110119292B (en) System operation parameter query method, matching method, device and node equipment
CN111800458A (en) Dynamic load balancing method and system for Kubernetes container cloud platform
EP4095768A1 (en) Application recognition model updating method and apparatus, and storage medium
CN106991008B (en) Resource lock management method, related equipment and system
CN113656176B (en) Cloud equipment distribution method, device and system, electronic equipment, medium and product
US11922059B2 (en) Method and device for distributed data storage
CN101673217B (en) Method for realizing remote program call and system thereof
CN105681404A (en) Metadata node management method and device of distributed cache system
CN112579319A (en) Service calling method and device based on LRU Cache optimization
US20130318244A1 (en) System and method for assigning server to terminal and efficiently delivering messages to the terminal
US7783810B2 (en) Apparatus and method of processing information
CN114490100B (en) Message queue telemetry transmission load balancing method, device and server
CN111581226B (en) Data sharing method and device based on big data platform and administrative enterprise cloud platform
CN111600929B (en) Transmission line detection method, routing strategy generation method and proxy server
CN117931083A (en) Data distributed storage local data access method and system
CN112055077A (en) Block link access method, device and storage medium
CN112052104A (en) Message queue management method based on multi-computer-room realization and electronic equipment
US8548994B2 (en) Reducing overheads in application processing
CN115185658A (en) Task unloading scheduling method based on time and communication reliability and related product
CN112799849A (en) Data processing method, device, equipment and storage medium
CN113783826A (en) Micro-service exception handling method and device
CN113285997B (en) Data processing method, device, medium and product based on heterogeneous system
CN112187816B (en) RPC request processing method, equipment and storage medium
CN111782938B (en) Information query pushing method and device and computer equipment
JP2000040051A (en) Method and device for transmitting message in client server system

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination