CN110430264B - Data reading method in cloud computing - Google Patents

Data reading method in cloud computing Download PDF

Info

Publication number
CN110430264B
CN110430264B CN201910717096.6A CN201910717096A CN110430264B CN 110430264 B CN110430264 B CN 110430264B CN 201910717096 A CN201910717096 A CN 201910717096A CN 110430264 B CN110430264 B CN 110430264B
Authority
CN
China
Prior art keywords
cloud computing
data
time
storage layer
computing server
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201910717096.6A
Other languages
Chinese (zh)
Other versions
CN110430264A (en
Inventor
黄颖
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Jiangsu puxu Technology Co.,Ltd.
Original Assignee
JIANGSU PUXU SOFTWARE INFORMATION TECHNOLOGY CO LTD
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by JIANGSU PUXU SOFTWARE INFORMATION TECHNOLOGY CO LTD filed Critical JIANGSU PUXU SOFTWARE INFORMATION TECHNOLOGY CO LTD
Priority to CN201910717096.6A priority Critical patent/CN110430264B/en
Publication of CN110430264A publication Critical patent/CN110430264A/en
Application granted granted Critical
Publication of CN110430264B publication Critical patent/CN110430264B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/50—Network services
    • H04L67/60—Scheduling or organising the servicing of application requests, e.g. requests for application data transmissions using the analysis and optimisation of the required network resources
    • H04L67/62—Establishing a time schedule for servicing the requests
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/50—Network services
    • H04L67/60—Scheduling or organising the servicing of application requests, e.g. requests for application data transmissions using the analysis and optimisation of the required network resources
    • H04L67/63—Routing a service request depending on the request content or context

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The invention relates to a data reading method in cloud computing, which processes a data reading request of a cloud computing task in an application layer through a logic storage layer in a cloud computing server, determines an actual storage position of a reading target according to the request, estimates data reading time according to the actual storage position, and controls sending time of the data reading request according to the data reading time. The method can improve the utilization rate of system resources.

Description

Data reading method in cloud computing
Technical Field
The invention belongs to the field of cloud computing, and particularly relates to a data reading method in cloud computing.
Background
At present, cloud computing is a widely developed computing mode in the fields of computers and the internet, and a large amount of data needs to be stored in cloud computing, and usually a large number of server clusters are provided on a network, and the data are stored in the clusters by adopting a grid technology, a distributed file system and the like. When a certain server executes a cloud computing task, it may read data local to the server, and may also need to read data of other servers in the cluster, in which case the processing manner in the prior art may cause waste of system resources. For example, in the prior art, when the server initiates a read request, if the request needs to read data of the local server and the other servers, the system may initiate the read request to the local server and the other servers at the same time, and then waits for the read data to arrive, in this case, the local data is usually read first, but the data of the other servers needs to wait for a period of time, during this period of time, the local data occupies a certain memory area but cannot be used, and the memory area is wasted for a period of time. Even if the local data is not read, the data returned by different servers will be in sequence, which may also result in the same waste of resources.
Disclosure of Invention
In order to solve the above problems in the prior art, the present invention provides a data reading method in cloud computing.
The technical scheme adopted by the invention is as follows:
a data reading method in cloud computing comprises the following steps:
step 1: a cloud computing task executed by a cloud computing server A sends a reading request to a logic storage layer of the cloud computing server A;
step 2: the logic storage layer analyzes the reading request and determines the actual storage position of the reading target; the actual storage positions comprise local and other n cloud computing servers;
and step 3: the logic storage layer estimates the time T of reading data from the local according to the reading request and the actual storage position of the reading target0And the time to read data from each of the other n cloud computing servers;
and 4, step 4: the logic storage layer is used for sequencing the data reading time of the n cloud computing servers from large to small; the n sorted data have the reading time T1，T2，……，Tn；TiThe corresponding cloud computing server is marked as Bi(1≤i≤n)；
And 5: the logic storage layer sequentially sends data reading requests to corresponding cloud computing servers according to the sequence;
step 6: for data needing local reading, the logic storage layer judges T0And T1If T is0≥T1Then the logical storage layer is facing to cloud computing server B1Sending a data reading request to a local physical storage layer at the same time of sending the data reading request; if T is0＜T1Then the logical storage layer is facing to cloud computing server B1After sending the data read request, wait for T1-T0Sending a data reading request to a local physical storage layer;
and 7: each cloud computing server BiReading corresponding data according to the received data reading request, and returning the corresponding data to the logic storage layer of the cloud computing server A; the physical storage layer of the cloud computing server A also reads corresponding data from the local according to the received data reading request and returns the data to the logic storage layer; the logic storage layer stores the received data in a designated cache region;
and 8: and the logic storage layer returns the pointer of the cache region to the cloud computing task.
Further, the logical storage layer estimates a time to read from local based on an amount of local data that needs to be read.
Further, the specific method for estimating the data reading time of the cloud computing server a for another cloud computing service B by the logic storage layer includes: counting the time of reading data from the cloud computing server B each time by the cloud computing server A in a past period of time, and estimating the time of reading data from the cloud computing server B this time based on the counting.
Further, the time when the cloud computing server a reads the data amount basically the same as that of the current reading from the cloud computing server B is counted, and the average value of the times is taken as the estimated value of the current reading time.
Further, the network transmission time of the data between the servers A and B is estimated through a transmission test, and after the network transmission time is deducted, the speed of reading the data locally by the server B is estimated every time, so that the time of reading the data locally by the server B is estimated according to the data volume read this time, and the estimated network transmission time is added to obtain the time of reading the data from the server B this time.
Further, the step 5 comprises:
the logical storage layer firstly sends the data to a cloud computing server B1Sending a data reading request to any other cloud computing server Bj(1 < j ≦ n), if Tj-1-TjIf the value is less than the predefined threshold value, the logic storage layer is facing the cloud computing server Bj-1After the data reading request is sent, the data is immediately sent to the cloud computing server BjSending a data reading request; if T isj-1-TjNot less than the predefined threshold, the logical storage layer is facing the cloud computing server Bj-1After sending the data reading request, waiting for Tj-1-TjAfter the time, the cloud computing server B is started againjA data read request is sent.
Further, after the cloud computing task acquires the data which needs to be read, the logic storage layer is informed to release the cache area.
The invention has the beneficial effects that: invalid occupation of the cache is reduced, and the utilization efficiency of system resources is improved.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, and are not to be considered limiting of the invention, in which:
FIG. 1 is a system architecture to which the method of the present invention is applicable.
Detailed Description
The present invention will now be described in detail with reference to the drawings and specific embodiments, wherein the exemplary embodiments and descriptions are provided only for the purpose of illustrating the present invention and are not to be construed as limiting the present invention.
Referring to fig. 1, a cloud computing system architecture to which the present invention relates is illustrated. The cloud computing system comprises a plurality of cloud computing servers which are connected together through a network, wherein the network can be a local area network or the internet, and each server can execute a specified cloud computing task.
In order to execute the cloud computing task, the storage system of each server is divided into an application layer, a logic storage layer and a physical storage layer. The application layer includes a server running each program, such as the cloud computing task. The logical storage layer is used for shielding the specific physical storage mode from the application layer and providing a corresponding storage interface for the application layer, and therefore the application layer does not need to care where the specific data is stored. The physical storage layer is used for managing a specific physical storage mode, receiving and executing a read-write command of the logic storage layer, and returning a corresponding result.
Based on the above system architecture, the data reading method of the present invention is described in detail below.
Step 1: the cloud computing task executed by the cloud computing server A sends a reading request to the logic storage layer of the cloud computing server A.
Specifically, the logic storage layer may provide a corresponding data reading interface, and the cloud computing task calls the data reading interface to make a reading request. The call parameters of the interface (also the content of the read request) include the target of the read (e.g. a file, or a logical object) and the amount of data read. Multiple read targets may also be included in a read request.
Step 2: the logic storage layer analyzes the read request and determines the actual storage position of the read target.
In the cloud computing system, data is stored in a distributed manner through a server cluster, and even if the same reading target is the same, the data can be stored in different cloud computing servers in a split manner. Therefore, the logical storage layer needs to determine the actual storage location of the read target according to the actual situation. The specific cloud computing storage organization method is already the content of the prior art, and is not described herein again.
Obviously, if the actual storage location is local, the logical storage layer reads data directly from the local physical storage layer. However, the method of the present invention is directed to the case where the actual storage location further includes other cloud computing servers.
Therefore, in particular, the logical storage layer determines that the actual storage location of the read target includes the local (i.e., the cloud computing server a itself), and the other n cloud computing servers.
And step 3: the logic storage layer estimates the time T of reading data from the local according to the reading request and the actual storage position of the reading target0And a time to read data from each of the other n cloud computing servers.
The time for reading the data from the local means the time for reading the local data to be read completely into the cache or the memory. In general, the time to read data locally depends on the amount of data read, so the logical storage layer can estimate the time to read locally (denoted as T) according to the amount of local data that needs to be read0The same applies below).
The time for reading data from the other servers depends on the one hand on the amount of data read and on the other hand on the time for transmitting data over the network, including the time for transmitting data requests from the cloud computing server a to the other servers and the time for transmitting read data from the other servers to the cloud computing server a. For example, if the logical storage layer is to estimate data reading time of the cloud computing server a for another cloud computing service B, the logical storage layer may count time of each time the cloud computing server a reads data from the cloud computing server B in a past period of time, and estimate time of this time of reading data from the cloud computing server B based on the statistics. An alternative mode is to count the time for reading the data volume (for example, the difference between the data volume and the data volume read this time is within 1 mbyte) from the cloud computing server B, and take the average value of the times as the estimated value of the time read this time. Another alternative is to estimate the network transmission time of the data between the servers a and B by a transmission test (e.g. ping), and after deducting the network transmission time, estimate the speed of reading the data locally by the server B each time, so as to estimate the time of reading the data locally by the server B according to the data amount read this time, and then, in combination with the estimated network transmission time, obtain the time of reading the data from the server B this time. Other estimation methods may be used by those skilled in the art, and the present invention is not limited in this regard.
And 4, step 4: and the logic storage layer is used for sequencing the data reading time of the n cloud computing servers from large to small.
Specifically, it is not necessary to set the n data read times after the sorting as T1，T2，……，Tn；TiThe corresponding cloud computing server is marked as Bi(i is more than or equal to 1 and less than or equal to n). Thus, it can be expected that from cloud computing server B1The time taken to read the data is the most, and the overall data reading time is actually determined by T1。
And 5: and the logic storage layer sequentially sends data reading requests to the corresponding cloud computing servers according to the sequence.
Specifically, the logical storage layer firstly sends the data to the cloud computing server B1Sending a data reading request and then sending the data reading request to a cloud computing server B2Sending a data reading request, and so on until the cloud computing server Bn。
For any one of the cloud computing servers Bj(1 < j ≦ n), if Tj-1-TjIf the value is less than the predefined threshold value, the logic storage layer is facing the cloud computing server Bj-1After the data reading request is sent, the data is immediately sent to the cloud computing server BjSending data readsFetching a request; that is, to Bj-1And BjThe transmission data read request is continuously transmitted. If T isj-1-TjNot less than the predefined threshold, the logical storage layer is facing the cloud computing server Bj-1After the data reading request is sent, waiting for a period of time, and sending the data to a cloud computing server BjSending a data read request, waiting for a time equal to Tj-1-Tj。
Based on this step 5, the logical storage layer allows each cloud computing server to return data in substantially the same time by controlling the sending time of the data read request.
Step 6: for data needing local reading, the logic storage layer judges T0And T1If T is0≥T1Then the logical storage layer is facing to cloud computing server B1Sending a data reading request to a local physical storage layer at the same time of sending the data reading request; if T is0＜T1Then the logical storage layer is facing to cloud computing server B1After sending the data read request, wait for T1-T0And sending a data reading request to the local physical storage layer.
In general, the time estimate for reading data locally is reliable because it is not affected by network conditions, whereas reading data from other servers may be affected by network conditions, and thus even at T0≥T1For server B1Should also not be later than the local data read request, in which case both are starting data read requests as early as possible at the same time.
And 7: each cloud computing server BiReading corresponding data according to the received data reading request, and returning the corresponding data to the logic storage layer of the cloud computing server A; and the physical storage layer of the cloud computing server A also reads corresponding data from local according to the received data reading request and returns the data to the logic storage layer.
After step 7 is completed, the logical storage layer of the cloud computing server a receives data to be read from the other n cloud computing servers and the local physical storage layer, and the received data are stored in the designated cache area.
And 8: and the logic storage layer returns a cache region pointer for storing the read data to the cloud computing task.
Therefore, the cloud computing task of the application layer obtains the data which needs to be read, and then the cloud computing task can inform the logic storage layer to release the cache region.
Based on the steps of the method, the time for reading data is controlled by the logic storage layer, the invalid occupation of the cache is reduced, and the utilization efficiency of system resources is improved.
The above description is only a preferred embodiment of the present invention, and all equivalent changes or modifications of the structure, characteristics and principles described in the present invention are included in the scope of the present invention.

Claims (6)

1. A data reading method in cloud computing is characterized by comprising the following steps:
step 1: a cloud computing task executed by a cloud computing server A sends a reading request to a logic storage layer of the cloud computing server A;
step 2: the logic storage layer analyzes the reading request and determines the actual storage position of the reading target; the actual storage positions comprise local and other n cloud computing servers;
and step 3: the logic storage layer estimates the time T of reading data from the local according to the reading request and the actual storage position of the reading target0And the time to read data from each of the other n cloud computing servers;
and 4, step 4: the logic storage layer is used for sequencing the data reading time of the n cloud computing servers from large to small; the n sorted data have the reading time T1，T2，……，Tn；TiThe corresponding cloud computing server is marked as Bi，1≤i≤n；
And 5: the logic storage layer sequentially sends data reading requests to corresponding cloud computing servers according to the sequence, and the specific process is as follows: the logical storage layer firstly sends the data to a cloud computing server B1Sending a data reading request to any other cloud computing server BjJ is more than 1 and less than or equal to n, if Tj-1-TjIf the value is less than the predefined threshold value, the logic storage layer is facing the cloud computing server Bj-1After the data reading request is sent, the data is immediately sent to the cloud computing server BjSending a data reading request; if T isj-1-TjNot less than the predefined threshold, the logical storage layer is facing the cloud computing server Bj-1After sending the data reading request, waiting for Tj-1-TjAfter the time, the cloud computing server B is started againjSending a data reading request;
step 6: for data needing local reading, the logic storage layer judges T0And T1If T is0≥T1Then the logical storage layer is facing to cloud computing server B1Sending a data reading request to a local physical storage layer at the same time of sending the data reading request; if T is0＜T1Then the logical storage layer is facing to cloud computing server B1After sending the data read request, wait for T1-T0Sending a data reading request to a local physical storage layer;
and 7: each cloud computing server BiReading corresponding data according to the received data reading request, and returning the corresponding data to the logic storage layer of the cloud computing server A; the physical storage layer of the cloud computing server A also reads corresponding data from the local according to the received data reading request and returns the data to the logic storage layer; the logic storage layer stores the received data in a designated cache region;
and 8: and the logic storage layer returns the pointer of the cache region to the cloud computing task.
2. The method of claim 1, wherein the logical storage layer estimates the time to read from local based on the amount of local data that needs to be read.
3. The method according to any one of claims 1 to 2, wherein the specific method for the logic storage layer to estimate the data reading time of the cloud computing server A for another cloud computing server B comprises the following steps: counting the time of reading data from the cloud computing server B each time by the cloud computing server A in a past period of time, and estimating the time of reading data from the cloud computing server B this time based on the counting.
4. The method according to any one of claims 1 to 2, characterized in that the time when the cloud computing server A reads substantially the same data amount from the cloud computing server B at this time is counted, and the average value of the times is taken as the estimated value of the time at this time.
5. The method according to any one of claims 1-2, wherein the network transmission time of the data between the cloud computing servers a and B is estimated through a transmission test, the speed of the data read by the cloud computing server B at the local thereof each time is estimated after the network transmission time is deducted, so that the time of the data read by the cloud computing server B at the local is estimated according to the data amount read at the current time, and the estimated network transmission time is added to obtain the time of the data read from the cloud computing server B at the current time.
6. The method according to claim 1, wherein the cloud computing task notifies a logical storage layer to release the cache area after acquiring the data that the cloud computing task needs to read.
CN201910717096.6A 2019-08-05 2019-08-05 Data reading method in cloud computing Active CN110430264B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201910717096.6A CN110430264B (en) 2019-08-05 2019-08-05 Data reading method in cloud computing

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910717096.6A CN110430264B (en) 2019-08-05 2019-08-05 Data reading method in cloud computing

Publications (2)

Publication Number Publication Date
CN110430264A CN110430264A (en) 2019-11-08
CN110430264B true CN110430264B (en) 2020-09-25

Family

ID=68414210

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910717096.6A Active CN110430264B (en) 2019-08-05 2019-08-05 Data reading method in cloud computing

Country Status (1)

Country Link
CN (1) CN110430264B (en)

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103810009A (en) * 2014-02-20 2014-05-21 北京奇虎科技有限公司 Method and device for accelerating starting of computer operating system
CN107888660A (en) * 2017-10-13 2018-04-06 杭州朗和科技有限公司 Cloud service resource allocation method, medium, device and computing device
CN109617710A (en) * 2018-11-09 2019-04-12 西北大学 The big data transmission bandwidth dispatching method for thering is deadline to constrain between data center

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106406821A (en) * 2016-08-15 2017-02-15 平安科技（深圳）有限公司 Data processing request sorting method and device
CN107920094B (en) * 2016-10-08 2021-07-16 阿里巴巴集团控股有限公司 Data acquisition method and device, server and network equipment
CN108255928A (en) * 2017-11-30 2018-07-06 北京元心科技有限公司 Distributed system method for reading data and device

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103810009A (en) * 2014-02-20 2014-05-21 北京奇虎科技有限公司 Method and device for accelerating starting of computer operating system
CN107888660A (en) * 2017-10-13 2018-04-06 杭州朗和科技有限公司 Cloud service resource allocation method, medium, device and computing device
CN109617710A (en) * 2018-11-09 2019-04-12 西北大学 The big data transmission bandwidth dispatching method for thering is deadline to constrain between data center

Also Published As

Publication number Publication date
CN110430264A (en) 2019-11-08

Similar Documents

Publication Publication Date Title
KR101242954B1 (en) Using priority to determine whether to queue an input/output(i/o) request directed to storage
JP4028674B2 (en) Method and apparatus for controlling the number of servers in a multi-system cluster
CN107992359B (en) Task scheduling method for cost perception in cloud environment
JP4421637B2 (en) Distributed scheduling of subtask processors
CN110837410A (en) Task scheduling method and device, electronic equipment and computer readable storage medium
US20200219028A1 (en) Systems, methods, and media for distributing database queries across a metered virtual network
CN110383764B (en) System and method for processing events using historical data in a serverless system
US20080086734A1 (en) Resource-based scheduler
US9875141B2 (en) Managing pools of dynamic resources
US20190332328A1 (en) Storage Controller and IO Request Processing Method
US20180165112A1 (en) Managing test services in a distributed production service environment
CN111857992B (en) Method and device for allocating linear resources in Radosgw module
CN108196939B (en) Intelligent virtual machine management method and device for cloud computing
CN115269108A (en) Data processing method, device and equipment
CN114006946B (en) Method, device, equipment and storage medium for processing homogeneous resource request
US9769022B2 (en) Timeout value adaptation
CN111135586A (en) Game matching method, game matching device, storage medium and electronic device
CN112860387A (en) Distributed task scheduling method and device, computer equipment and storage medium
CN114564281A (en) Container scheduling method, device, equipment and storage medium
CN110430264B (en) Data reading method in cloud computing
CN116089477B (en) Distributed training method and system
CN106897124A (en) A kind of multi-tenant request processing method, device and equipment
CN113127179A (en) Resource scheduling method and device, electronic equipment and computer readable medium
CN111831408A (en) Asynchronous task processing method and device, electronic equipment and medium
US10637919B2 (en) Autonomous resource governor in distributed systems for protecting shared resources

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
TA01 Transfer of patent application right

Effective date of registration: 20200827

Address after: Qinhuai District of Nanjing city in Jiangsu province is 210000 Road No. 1

Applicant after: JIANGSU PUXU SOFTWARE INFORMATION TECHNOLOGY Co.,Ltd.

Address before: 214300 NO.67 Jiaotong Road, Hede Town, Sheyang County, Yancheng City, Jiangsu Province

Applicant before: Huang Ying

TA01 Transfer of patent application right
GR01 Patent grant
GR01 Patent grant
CP01 Change in the name or title of a patent holder

Address after: 210000 Qinhuai District, Nanjing City, Jiangsu Province, No. 1

Patentee after: Jiangsu puxu Technology Co.,Ltd.

Address before: 210000 Qinhuai District, Nanjing City, Jiangsu Province, No. 1

Patentee before: JIANGSU PUXU SOFTWARE INFORMATION TECHNOLOGY Co.,Ltd.

CP01 Change in the name or title of a patent holder