CN117579619A - Data downloading method, device, equipment and storage medium - Google Patents

Data downloading method, device, equipment and storage medium Download PDF

Info

Publication number
CN117579619A
CN117579619A CN202311533479.0A CN202311533479A CN117579619A CN 117579619 A CN117579619 A CN 117579619A CN 202311533479 A CN202311533479 A CN 202311533479A CN 117579619 A CN117579619 A CN 117579619A
Authority
CN
China
Prior art keywords
data
downloading
point
download
blocks
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202311533479.0A
Other languages
Chinese (zh)
Inventor
张正平
姜宝琦
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Baidu Netcom Science and Technology Co Ltd
Original Assignee
Beijing Baidu Netcom Science and Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Baidu Netcom Science and Technology Co Ltd filed Critical Beijing Baidu Netcom Science and Technology Co Ltd
Priority to CN202311533479.0A priority Critical patent/CN117579619A/en
Publication of CN117579619A publication Critical patent/CN117579619A/en
Pending legal-status Critical Current

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/14—Session management
    • H04L67/141—Setup of application sessions
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/34—Network arrangements or protocols for supporting network services or applications involving the movement of software or configuration parameters 

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Information Transfer Between Computers (AREA)

Abstract

The disclosure provides a data downloading method, a device, equipment and a storage medium, relates to the field of data processing, and particularly relates to the technical fields of data transmission, internet of things and the like. The data downloading method comprises the following steps: acquiring a downloading task aiming at target data; dividing target data into a plurality of data blocks; initializing a plurality of network threads in a network layer aiming at a plurality of data blocks; and establishing a plurality of point-to-point links of the current device and the data source by using a plurality of network threads, and downloading a plurality of data blocks through the plurality of point-to-point links. Therefore, the parallel capacity of the threads is improved through the multi-network threads responsible for point-to-point link establishment and data downloading in the network layer, and the data downloading efficiency is further improved.

Description

Data downloading method, device, equipment and storage medium
Technical Field
The disclosure relates to the field of data processing, in particular to the technical fields of data transmission, internet of things and the like, and particularly relates to a data downloading method, device, equipment and storage medium.
Background
The network disk can provide online downloading and online storage services, and along with the construction of a network disk opening platform, the user equipment can be connected with a software development kit (software development kit, SDK) for realizing the online downloading service, so that the localization of the online downloading service on the user equipment is realized.
In an internet of things (internet of things, IOT) scenario, a data download scheme includes: a single data source and single thread serial download scheme, a single data source/multiple data sources and single network thread parallel download scheme, a single data source/multiple data sources and multiple threads parallel download scheme, wherein each thread needs to be frequently switched in a plurality of tasks such as data receiving, data writing and the like.
The data downloading efficiency of the scheme is low.
Disclosure of Invention
The present disclosure provides a data downloading method, apparatus, device and storage medium for improving data downloading efficiency.
According to a first aspect of the present disclosure, there is provided a data downloading method, including: acquiring a downloading task aiming at target data; dividing the target data into a plurality of data blocks; initializing a plurality of network threads in a network layer according to the plurality of data blocks; and establishing a plurality of point-to-point links of the current equipment and the data source by using the plurality of network threads, and downloading the plurality of data blocks through the plurality of point-to-point links.
According to a second aspect of the present disclosure, there is provided a data downloading apparatus comprising: the task acquisition unit is used for acquiring a downloading task aiming at the target data; a data dividing unit for dividing the target data into a plurality of data blocks; a thread initializing unit, configured to initialize a plurality of network threads in a network layer for the plurality of data blocks; and the data block downloading unit is used for establishing a plurality of point-to-point links of the current equipment and the data source by using the plurality of network threads, and downloading the plurality of data blocks through the plurality of point-to-point links.
According to a third aspect of the present disclosure, there is provided an electronic device comprising: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the data download method according to the first aspect.
According to a fourth aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the data downloading method of the first aspect.
According to a fifth aspect of the present disclosure, there is provided a computer program product comprising: a computer program stored in a readable storage medium, from which it can be read by at least one processor of an electronic device, the at least one processor executing the computer program causing the electronic device to perform the data download method according to the first aspect.
According to the technical scheme provided by the disclosure, the downloading task aiming at the target data is obtained, the target data is divided into a plurality of data blocks, and the downloading task of the target data is divided into the downloading tasks of the plurality of data blocks. For a plurality of data blocks, initializing a plurality of network threads in a network layer, establishing a plurality of point-to-point links between current equipment and a data source by using the plurality of network threads, and downloading the plurality of data blocks through the plurality of point-to-point links, thereby realizing parallel downloading of the plurality of network threads. The difference between the multi-thread parallel downloading scheme and the multi-thread parallel downloading scheme is that the multi-network thread in the network layer is only responsible for the establishment of point-to-point links and the data downloading based on the point-to-point links, and frequent switching among a plurality of tasks such as data receiving and transmitting, data writing and the like is not needed, so that the task quantity of the thread is reduced, the thread parallel capability is improved, and the data downloading efficiency is further improved.
It should be understood that the description in this section is not intended to identify key or critical features of the embodiments of the disclosure, nor is it intended to be used to limit the scope of the disclosure. Other features of the present disclosure will become apparent from the following specification.
Drawings
The drawings are for a better understanding of the present solution and are not to be construed as limiting the present disclosure. Wherein:
fig. 1 is a schematic diagram of an application scenario to which embodiments of the present disclosure are applicable;
FIG. 2 is a schematic diagram according to a first embodiment of the present disclosure;
FIG. 3 is a schematic diagram according to a second embodiment of the present disclosure;
FIG. 4 is an exemplary diagram of a data download process provided in accordance with the above-described embodiments of the present disclosure;
FIG. 5 is a schematic diagram according to a third embodiment of the present disclosure;
fig. 6 is a schematic block diagram of an example electronic device 600 that may be used to implement embodiments of the present disclosure.
Detailed Description
Exemplary embodiments of the present disclosure are described below in conjunction with the accompanying drawings, which include various details of the embodiments of the present disclosure to facilitate understanding, and should be considered as merely exemplary. Accordingly, one of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
In an internet of things (internet of things, IOT) scenario, a data download scheme may include: a serial download scheme of single data source and single thread, a parallel download scheme of single/multiple data source and single network thread, and a parallel download scheme of single/multiple data source and multiple thread. The data downloading task relates to a network module (which can be understood as a network layer), a memory module and a disk module, the thread refers to a thread penetrating through the network module, the memory module and the disk module, and the network thread refers to a thread in the network module.
In the single data source+single thread serial downloading scheme, the single data source has risks of network link blockage, single point failure and the like, and the serial downloading speed is far slower than the parallel downloading speed; in the parallel downloading scheme of single/multiple data sources and single network threads, as the performance of equipment in an IOT scene is lower, the single network thread receiving and transmitting threads easily become bottlenecks for improving the data downloading speed, and the single network threads are always in a full-load state, so that the data downloading speed is low; in the parallel downloading scheme of single/multiple data sources and multiple threads, each thread needs to complete all operations in the whole downloading flow, namely, the method is not limited to executing data receiving and transmitting tasks in a network layer, and needs to be responsible for multiple tasks of other layers, such as data writing of a magnetic disk, data reading of the magnetic disk, scheduling among multiple tasks and the like, so that the parallel capability of the multiple threads is weaker, the performance is poorer, and the data downloading speed is slow. Therefore, the above schemes have the disadvantage of slow data download speed.
In order to improve the data downloading speed and the data downloading efficiency, the disclosure provides a data downloading method which can be applied to the technical fields of data transmission, the Internet of things and the like in the data processing field. The method initializes a plurality of network threads for a data downloading task at an application network layer, establishes a plurality of point-to-point links between a current device and a data source by using the plurality of network threads, and downloads data through the plurality of point-to-point links. The network thread is only responsible for data transceiving of a network layer, comprises the establishment of a plurality of point-to-point links and the data downloading based on the plurality of point-to-point links, and does not need to be responsible for other tasks such as data reading and writing in a memory, data reading and writing in a disk and the like, so that the task amount of the network thread is reduced, the parallel capacity of the plurality of network threads is improved, and the data downloading speed and the data downloading efficiency are further improved through the plurality of network threads.
It should be noted that, the downloaded data in this embodiment is not user data for a specific user, and cannot reflect personal information of a specific user.
In the technical scheme of the disclosure, the related processes of collecting, storing, using, processing, transmitting, providing, disclosing and the like of the personal information of the user accord with the regulations of related laws and regulations, and the public order colloquial is not violated.
Fig. 1 is a schematic diagram of an application scenario to which the present disclosure is applicable. In this application scenario, a data source 101 and a data downloading end 102 are included, where one or more data sources 101 may be used, and fig. 1 illustrates a plurality of data sources 101. On the data download side 102, an online download service of a network disk, such as an SDK implementing the online download service, may be deployed to request download data from the data source 101. In the data downloading process, a plurality of network threads can be applied to the network layer for data downloading, so that the data downloading speed and the data downloading efficiency are improved through parallel downloading of the plurality of network threads.
It can be understood that the scheme provided by the disclosure can realize optimization of online download service, especially realize optimization of download behavior of online download service of network disk on low-performance devices such as IOT, and provide faster data download speed for users with lower system resource overhead.
It should be noted that the embodiments of the present disclosure may be implemented on a terminal and/or a server, and fig. 1 is an example of implementation of the embodiments of the present disclosure on a terminal. The terminal may be a personal digital assistant (personal digital assistant, PDA for short), a handheld device with a wireless communication function (e.g., a smart phone, a tablet computer), a computing device (e.g., a personal computer (personal computer, PC for short)), a wearable device (e.g., a smart watch, a smart bracelet), a smart home device (e.g., a smart speaker, a smart display device), etc. The servers can be independent servers or server clusters, and can be local servers or cloud servers.
The following describes the technical scheme of the present disclosure and how the technical scheme of the present disclosure solves the above technical problems in detail with specific embodiments. The following embodiments may be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments. Embodiments of the present disclosure will be described below with reference to the accompanying drawings.
Fig. 2 is a schematic diagram according to a first embodiment of the present disclosure. As shown in fig. 2, the data downloading method provided in the first embodiment of the present disclosure includes:
s201, acquiring a download task for target data.
The target data may be document, audio, video, etc.
In this embodiment, a download request for target data may be received, where the download request is used to request execution of a download task for target data, where the download request may be input by a user, or may be automatically triggered when the target data is downloaded according to a data downloading sequence; alternatively, the download task for the target data may be obtained from a task list.
S202, dividing target data into a plurality of data blocks.
Wherein the data sizes of the plurality of data blocks may be the same or different.
In this embodiment, in the downloading task of the target data, the data amount of the target data may be included, and the target data may be divided into a plurality of data blocks according to the data amount of the target data, so as to convert the downloading task of the target data into the downloading task of the plurality of data blocks. The process may refer to data slicing downloading, and the data slicing process is not described in detail. Therefore, by dividing the data into a plurality of data blocks, each data block can be independently downloaded in parallel, and the data downloading efficiency is improved.
S203, initializing a plurality of network threads in a network layer for the plurality of data blocks.
Wherein the plurality of network threads are parallel.
In this embodiment, the allocation and initialization of the download resources are performed for a plurality of data blocks, and the initialization of the download resources includes performing initialization of a plurality of network threads in the network layer so as to use the plurality of network threads for downloading the plurality of data blocks. In addition to initializing the network thread, other threads (such as initializing the threads in a task layer, initializing the threads in a disk storage, etc.) and allocating software and hardware resources (such as resources of a processor, a memory, a disk, etc.) may be performed.
S204, a plurality of point-to-point links of the current device and the data source are established by using a plurality of network threads, and a plurality of data blocks are downloaded through the plurality of point-to-point links.
Wherein one or more point-to-point links may be established by an application of a network thread. The data sources may be one or more, a point-to-point link is used to establish a connection between the current device and a data source, and one or more point-to-point links may be established between the current device and a data source.
The network thread is used for executing data receiving and transmitting operation in the network layer, and the completion of the data receiving and transmitting operation firstly establishes a point-to-point link and then carries out data transmission based on the point-to-point link.
In this embodiment, the task information of the download task of the target data may include an address of a data source of the target data, or may be determined by a thread or a functional module for implementing a data source determining function based on the task information of the download task of the target data, or may be determined by other means, which is not specifically limited herein. After determining the address of the data source, a plurality of point-to-point links of the current device and the data source can be established by using a plurality of network threads, and after the plurality of point-to-point links are established, the plurality of network threads can download a plurality of data blocks through the plurality of point-to-point links, and finally the target data is obtained.
In the embodiment of the disclosure, each network thread in the network layer is only responsible for data transceiving of the network layer, including being responsible for establishing a point-to-point link and downloading data based on the point-to-point link, and is not required to be responsible for other tasks such as data reading and writing in a memory, data reading and writing in a disk, and the like, so that the task amount of the network thread is reduced, the parallel capacity and downloading performance of a plurality of network threads are improved, and further, the data downloading speed and the data downloading efficiency are improved through the plurality of network threads.
In one possible implementation, there is no lock between the plurality of network threads, and each network thread performs data download based on a respective established point-to-point link among the plurality of network threads and the plurality of point-to-point links. Therefore, no extra lock expenditure exists among a plurality of network threads, the system resource expenditure is reduced, and the faster data downloading speed is provided for the user with lower system resource expenditure.
Each network thread downloads data based on the point-to-point links established by the network thread, so that the network threads cannot compete for the same point-to-point link, and the condition that one point-to-point link is read and written by the network threads simultaneously does not occur, so that the network threads do not need to be locked. For example, there are 16 point-to-point links and 3 network threads, where the first network thread establishes the 1 st to 4 th point-to-point links, the second network thread establishes the 5 th to 10 th point-to-point links, and the third network thread establishes the 11 th to 16 th point-to-point links, and then the 3 network threads only manage the point-to-point links established by themselves. Locking of network threads refers to defining the order of access of network threads, such as defining multiple network threads to alternately read data on a point-to-point link.
In one possible implementation, the number of data sources is multiple, and in the process of establishing multiple point-to-point links between the current device and the data sources by using multiple network threads, for each data source, a point-to-point link of a data source corresponding type may be established between the current device and the data source. Thus, in the face of different data sources, a point-to-point link of a proper type is established to ensure the stability and accuracy of data downloading.
Wherein the plurality of data sources may include one or more of: internet data center (internet data center, IDC), content delivery network (content delivery network, CDN), PCDN (point-to-point technology combined with CDN), ant-CDN (self-built CDN).
Fig. 3 is a schematic diagram according to a second embodiment of the present disclosure. As shown in fig. 3, the data downloading method provided in the second embodiment of the present disclosure includes:
s301, acquiring a downloading task aiming at target data.
S302, dividing target data into a plurality of data blocks.
S303, initializing a plurality of network threads in a network layer for a plurality of data blocks.
S304, a plurality of point-to-point links of the current device and the data source are established by using a plurality of network threads.
The implementation principles and technical effects of S301 to S304 may refer to the foregoing embodiments, and are not repeated.
S305, sending a notification message to the task layer, wherein the notification message indicates that the plurality of point-to-point links are data downloading links of the plurality of data blocks.
In this embodiment, a notification message may be sent to the task layer through the network layer to notify the task layer that the multiple point-to-point links are successfully established and may be used for downloading multiple data blocks.
In one possible implementation, in the network layer, the notification message may be sent to the task layer in a callback manner. In the resource allocation and initialization process, the network layer receives a callback function provided by the task layer; after a plurality of point-to-point links of the current device and the data source are established by using a plurality of network threads, a notification message can be sent to the task layer in the network layer by calling a callback function. Therefore, the notification message is timely sent to the task layer in a callback mode.
In one possible implementation, to reduce the task amount of the network thread, a notification message may be sent to the task layer by other threads in the network layer.
S306, receiving download requests respectively corresponding to the plurality of point-to-point links, wherein the download requests respectively corresponding to the plurality of point-to-point links are distributed to the plurality of point-to-point links by the task layer, and the download requests corresponding to the point-to-point links indicate that the point-to-point links are responsible for requesting the downloaded data in the plurality of data blocks.
In this embodiment, in the task layer, after receiving the notification message, a corresponding download request may be allocated to the plurality of point-to-point links by a thread in the task layer, so as to allocate the download of the plurality of data blocks to the plurality of point-to-point links, and implement parallel download of the data in the plurality of data blocks by the plurality of point-to-point links. After the corresponding download requests are allocated to the plurality of point-to-point links, the download requests respectively corresponding to the plurality of point-to-point links can be sent to the network layer through the task layer. In the network layer, a plurality of network threads can be applied to receive downloading requests respectively corresponding to a plurality of point-to-point links.
In one possible implementation, a process for allocating corresponding download requests to a plurality of point-to-point links in a task layer includes: for a target link to be allocated with a request in a plurality of point-to-point links, determining an idle data block in the data blocks which are not downloaded and are contained in a plurality of data blocks, wherein the idle data block is a data block containing idle data pieces, and the idle data pieces are the data pieces in an unrendered state; and distributing the downloading request of the idle data pieces in the idle data blocks to the target links according to the data downloading capability of the target links. Thereby, the rationality of assigning download request assignments for a plurality of point-to-point links is improved.
In which a plurality of pieces of data may be contained in a data block, for example, one piece of data has a data size of 2 Megabytes (MB) and one piece of data has a data size of 16 kilobytes (Kbytes, KB), then one piece of data may contain 128 pieces of data. In the download distribution process, the data download capacity of different point-to-point links is different, and the size of the distributed downloaded data is different. In assigning download requests to multiple point-to-point links, several situations may occur: the entire data block has been allocated for point-to-point link download, a portion of the data pieces in the data block have been allocated for point-to-point link download, and none of the data pieces in the data block have been allocated for point-to-point link download.
Wherein if a piece of data is not assigned to a point-to-point link for download, the piece of data may be referred to as a free piece of data, and the piece of data containing the free piece of data is a free piece of data.
The data downloading capability of the point-to-point link may be preset according to factors such as downloading cost and downloading speed of the point-to-point link, and the data downloading capability of the point-to-point link may be represented as a downloading data amount supported by the point-to-point link, such as a maximum downloading data amount supported by the point-to-point link.
In this implementation, in the task layer, the downloading requests may be sequentially allocated to the plurality of point-to-point links according to the data blocks that are not downloaded and are included in the plurality of data blocks. In the process of sequentially distributing downloading requests, aiming at a target link to be distributed with the requests in a plurality of point-to-point links, determining an idle data block in unrendered data blocks contained in a plurality of data blocks; the number of pieces of data that match the data download capability of the target link may be determined and download requests for the number of pieces of free data in the free data block may be assigned to the target link. After the target link is allocated, a download request may be allocated for the next point-to-point link of the target link in the same manner until a download request is allocated for each point-to-point link or no data blocks exist that may be allocated among the plurality of data blocks. Therefore, the idle condition of the data and the link downloading capability are comprehensively considered, the downloading request is distributed, and the accuracy and the rationality of the downloading request distribution are improved.
By way of example, in assigning a download request, point-to-point link a assigns a download request that requests the first 64 pieces of data in the first data block to be downloaded, point-to-point link B assigns a download request that requests the remaining pieces of data (i.e., the remaining 64 pieces of data) in the first data block to be downloaded, and point-to-point link C assigns a download request that requests the second data block to be downloaded (i.e., the amount of download data assigned to point-to-point link C is the amount of data for the entire second data block).
In one possible implementation, if there is no free data block in the data block that is not downloaded, determining that preemption of the downloaded data piece is allowed in the data block in the downloaded state; a download request that allows preempting a downloaded piece of data is assigned to the target link. Therefore, the data downloading speed is improved through preemptive downloading of the data sheet, and the data downloading efficiency is further improved.
Whether the data sheet allows preemptive downloading or not can be preconfigured, and whether the data sheet allows preemptive downloading or not can be determined according to the actual downloading condition. For example, if the download request allocated to the point-to-point link a is a request for downloading the first 64 pieces of data in the first data block, the download request allocated to the point-to-point link B is a request for downloading the remaining pieces of data in the first data block, and the download request of the point-to-point link C is a request for downloading the second data block, if the point-to-point link B and the point-to-point link C complete the downloading of the corresponding data and the point-to-point link a does not complete the downloading of the corresponding data yet, the first 64 pieces of data in the first data block may be allocated to the point-to-point link B and/or the point-to-point link C to perform the synchronous downloading, so that the first 64 pieces of data in the first data block are obtained by the point-to-point link B and/or the point-to-point link C with a higher downloading capability as soon as possible, thereby improving the data downloading efficiency.
In one possible implementation, if there is no free data block in the data blocks that are not downloaded and there is no data piece that allows preemption of downloading in the data blocks that are in the downloading state, the target link may be switched to a dormant state or a suspended state, waiting to wake up when the downloading request is allocated next time.
S307, the corresponding downloading request is sent to the corresponding data source through a plurality of point-to-point links, and the downloading data returned by the corresponding data source is received.
In this embodiment, a plurality of network threads may be applied, and the download requests corresponding to the point-to-point links are sent to the data sources corresponding to the point-to-point links through the point-to-point links, that is, the data sources linked by the point-to-point links, so as to request the data sources to download corresponding data; after the download request is sent, the download data returned by the corresponding data source is received through a plurality of point-to-point links.
In the embodiment of the disclosure, each network thread in the network layer is only responsible for data transceiving of the network layer, including being responsible for establishment of point-to-point links and data downloading based on the point-to-point links, so that the task amount of the network threads is reduced, the parallel capacity and downloading performance of a plurality of network threads are improved, and the data downloading speed and the data downloading efficiency are improved. Meanwhile, in the process of downloading data by applying a plurality of network threads, the downloading request can be reasonably distributed for the point-to-point links established by the plurality of network threads, and the data downloading speed and the data downloading efficiency are further improved from the distribution of the downloading request.
In the case of multiple data sources, the transmission protocols adopted by different data sources may be different, so that the management and organization of the data packets at the application layer are difficult to unify, and the data unpacking and decryption operations may need to be performed in the middle. In the related art, after unpacking and decrypting to obtain pure real data, the data is organized into data pieces, and finally the data pieces are spliced into complete data blocks by corresponding modules, so that multiple memory copies are needed in the process, and the data processing performance is affected. To solve this problem, the present disclosure provides the following implementation manner based on any one of the above embodiments:
in one possible implementation manner, after downloading a plurality of data blocks through a plurality of point-to-point links, the downloaded data received by the plurality of point-to-point links may be obtained, a target data block to which the downloaded data belongs is determined, and the downloaded data is stored in a memory space corresponding to the target data block. The memory space corresponding to the target data block is applied before receiving the download data belonging to the target data block. Therefore, each data block is managed through independent memory space, and data from different data sources are managed through independent memory space in a unified way for the situation that the data in the data block are downloaded from different data sources, so that the data management effect in the data downloading process is improved.
Wherein the target data block is one or more data blocks in a download state.
In the implementation manner, in an application layer, whether the target data block has downloaded data or not can be judged, if not, a corresponding memory space can be applied for the target data block, so that an independent memory space can be applied for the target data block before the downloaded data belonging to the target data block is received; if the target data block has downloaded data, the downloaded data can be stored in the memory space corresponding to the target data block. Therefore, no matter from which data source the downloaded data in the data block comes, the downloaded data in the data block is uniformly managed through the independent memory space corresponding to the data block.
In one possible implementation manner, after the downloaded data is stored in the memory space corresponding to the target data block, under the condition that the downloaded data needs to be unpacked, unpacking operation is performed on the downloaded data in the memory space to obtain package receiving data corresponding to the downloaded data; and storing the packet data into the memory space. Therefore, the writing and unpacking of the downloaded data and the writing of the unpacked data are uniformly carried out in the memory space, multiple memory copies are not needed, the data processing efficiency is improved, and the data downloading efficiency is further improved.
In this implementation manner, after the downloaded data is stored in the memory space corresponding to the target data block, whether the downloaded data needs to be unpacked or not may be determined according to a transmission protocol corresponding to the downloaded data (i.e., a transmission protocol corresponding to a data source that sends the downloaded data). If the transmission protocol corresponding to the downloaded data is a private protocol, the downloaded data can be determined to need to be unpacked, the downloaded data is unpacked according to the transmission protocol corresponding to the downloaded data, and the unpacked data is stored in the memory space. Therefore, the in-situ unpacking of the downloaded data is carried out in the memory space, so that multiple memory copies in the unpacking process are avoided.
In one possible implementation, after the target data block is downloaded, if the target data block is encrypted data, the target data block is decrypted. Therefore, the data decryption operation is decoupled from the data downloading and data unpacking operation, the data decryption operation is postponed, and the data block is decrypted after the data block is downloaded, namely, the data block is decrypted after waiting for a complete data block to be cached, so that the decryption efficiency of the data block is improved, and the data downloading efficiency is further improved.
Further, whether the target data block is downloaded or not can be determined in the memory space corresponding to the target data block, if the target data block is downloaded, whether the target data block is encrypted data is judged, and if the target data block is encrypted data, the target data block is decrypted in the memory space. Therefore, the unified management of writing, unpacking and decrypting operation of the data blocks is realized through the memory space, multiple memory copies in the processes are saved, the data processing efficiency is effectively improved, and the data downloading efficiency is improved.
After the target data is decrypted, pure data of the target data can be obtained, and the pure data of the target data is delivered to a file layer for asynchronous disc landing.
In the case that the data size of the target data is large, multiple network threads may need to be applied to perform multiple downloads, and the following implementation manner of the loop judgment process of data downloading is provided:
in one possible implementation, it may be determined whether the downloading of the plurality of data blocks is complete; if the downloading of the plurality of data blocks is not completed, continuing to download the data blocks which are not completed in the plurality of data blocks through a plurality of network threads; and if the downloading of the plurality of data blocks is completed, releasing the plurality of network threads. Therefore, the complete downloading of the target data is ensured, and the network thread is released in time after the complete downloading of the target data, so that the resources are saved.
In this implementation manner, after receiving the download data returned by the corresponding data source, or before distributing the corresponding download requests for the plurality of point-to-point links, determining whether the downloading of the plurality of data blocks is completed in the task layer, if the downloading of the plurality of data blocks is not completed, distributing the corresponding download requests for the plurality of point-to-point links according to the data blocks which are not completed in the plurality of data blocks, otherwise, determining that the downloading of the target data is completed, releasing the plurality of network threads, and stopping downloading. Therefore, the continuous distribution of the downloading request under the condition that the downloading of a plurality of data blocks is completed is avoided, and the resource waste is reduced.
In addition to releasing multiple network threads, other resources occupied by the targeted data download may also be released.
In one possible implementation, determining whether the downloading of the plurality of data blocks is complete includes: acquiring a data bitmap corresponding to target data; based on the plurality of binary values, it is determined whether the downloading of the plurality of data blocks is complete. Therefore, whether the downloading of each data block is completed or not is recorded based on the data bitmap, and the accuracy of judging whether the downloading of a plurality of data blocks is completed or not is improved.
The binary values in the data bitmap correspond to the data blocks one by one, and the binary values corresponding to the data blocks indicate that the downloading of the data blocks is completed or not. For example, when the binary value corresponding to the data block is 0, indicating that the data block is not downloaded; and when the binary value corresponding to the data block is 1, indicating that the downloading of the data block is completed.
In this implementation manner, after the target data is divided into a plurality of data blocks, a data bitmap corresponding to the target data may be initialized according to the number of the plurality of data blocks, where a plurality of binary values in the initialized data bitmap are in one-to-one correspondence with the plurality of data blocks, and the binary value corresponding to the data block is a value indicating that the data block is not downloaded. In downloading a plurality of data blocks via a plurality of point-to-point links, each time a data block is downloaded, a binary value in a data bitmap corresponding to the data block is modified to a value indicating that the data block is downloaded. In the process of determining whether the downloading of the plurality of data blocks is completed, a data bitmap corresponding to the target data can be obtained, if the plurality of binary values in the data bitmap are all numerical values indicating that the downloading of the data blocks is completed, the downloading of the plurality of data blocks is determined to be completed, otherwise, the downloading of the plurality of data blocks is determined to not be completed.
By way of example, fig. 4 is an exemplary diagram of a data download process provided in accordance with the above-described embodiments of the present disclosure. As shown in fig. 4, the data download process includes the following steps S401 to S420:
s401, downloading is started, namely downloading of target data is started.
S402, initializing resources.
After the downloading is started, the resource initialization is carried out for the downloading of the target data, wherein the initialization and the allocation of various software and hardware resources such as threads, memories and magnetic discs can be included.
S403, judging whether the downloading is finished according to the data bitmap.
Wherein S403 may be performed after the resource initialization of S402; alternatively, S403 may be placed after S418 instead of S419; alternatively, S403 may be performed after S419. Fig. 4 is an example of S403 after S402. The data bitmap may refer to the foregoing embodiments, and will not be described in detail.
In the task layer, whether the downloading of the target data is completed may be determined according to whether the plurality of binary values in the data bitmap are 1, and if the plurality of binary values in the data bitmap are 1, the downloading of the target data is determined to be completed, and S419 is performed. If the binary value is 0 in the data bitmap, it is determined that the downloading of the target data is not completed, and S404 is performed.
S404, determining available data sources.
For downloading of the target data, one or more available data sources are determined. The determination process of the data source may refer to the foregoing embodiment, and will not be described in detail.
S405, constructing a plurality of point-to-point links.
Multiple network threads may be applied in the network layer to build multiple point-to-point links between the current device and the data source to enable multi-data source multi-network thread downloading of the target data. When the number of the data sources is multiple, one or more point-to-point links can be constructed between the current device and one data source; when the types of data sources are different, the plurality of point-to-point links may include different types of point-to-point links.
S406, notifying the task layer that the available point-to-point links are available.
After a plurality of point-to-point links are constructed by using a plurality of network threads, the network layer can inform the task layer that the point-to-point links are available. The network layer may notify the task layer that there is an available point-to-point link by calling a callback function, which may refer to the foregoing embodiment and will not be described in detail.
S407, the task layer distributes a downloading request for the point-to-point link.
After informing the task layer that there is a point-to-point link available, the task layer distributes the target data to the point-to-point link for downloading, i.e. distributes a downloading request for the point-to-point link.
Specifically, the task layer may download requests to a plurality of point-to-point links in turn, and the downloading request process of each point-to-point link may include S4071 to S4074:
s4071, whether there are spare data blocks to allocate.
The spare data block is the spare data block in the previous embodiment, if there is a spare data block in the target data that can be allocated, S4072 is executed, otherwise S4073 is executed. The foregoing embodiments may be referred to for dividing the target data into data blocks and determining whether there are any idle data blocks, which will not be described in detail.
S4072, distributing the data in the spare data blocks to the point-to-point links for downloading according to the factors such as the downloading cost, speed and the like of the point-to-point links.
If there are spare data blocks in the plurality of data blocks of the target data, the data downloading capability of the point-to-point link may be determined according to the downloading cost, speed and other factors of the point-to-point link (the data downloading capability of the point-to-point link may also be determined in advance according to the downloading cost, speed and other factors of the point-to-point link), and the data in the spare data blocks may be allocated to the point-to-point link for downloading based on the data downloading capability of the point-to-point link.
S4073, whether there is a preemptible data slice.
Wherein the preemptible data slice is the data slice that was allowed to preempt in the previous embodiment.
If there are no spare data blocks in the plurality of data blocks of the target data, it can be determined whether there are preemptible data pieces in the data blocks in the download state. If there is a preemptible data slice, S4074 is performed, otherwise S4075 is performed.
S4074, distributing the preemptively downloaded data pieces to the point-to-point link for downloading according to the factors such as the downloading cost, speed and the like of the point-to-point link.
If preemptive data pieces exist in the data block in the downloading state, determining the data downloading capability of the point-to-point link according to the factors such as the downloading cost and the speed of the point-to-point link (the data downloading capability of the point-to-point link can also be determined in advance according to the factors such as the downloading cost and the speed of the point-to-point link), and distributing the preemptive downloaded data pieces to the point-to-point link for downloading based on the data downloading capability of the point-to-point link.
S4075, wait for the next wakeup.
If no preemptive data piece exists in the data block in the downloading state, the downloading request can be respectively carried out without the point-to-point link, and the point-to-point link waits for the next awakening.
Thus, by executing S4071 to S4074 a plurality of times, download requests are allocated to a plurality of point-to-point links.
S408, sending a download request to the data source through the point-to-point link.
The network layer may receive download requests allocated by the task layer for the plurality of point-to-point links, and then may send the download requests to the corresponding data sources via the plurality of point-to-point links, respectively, to request for downloading data.
S409, whether the data block in the downloading has data to be returned.
After the download request is sent, it may be determined whether there is data return for the data block under download, if not, S410 is performed, otherwise S411 is performed.
S410, applying for the corresponding memory space for the data block.
And applying for the corresponding memory space for the data block under the condition that the data block in the downloading process has no data return. After S410, S411 may be performed to store data to the memory space after receiving the data of the data block
S411, the process returns a response.
In the case where there is a data return for the data block under download, a response message returned by the data source may be received via the point-to-point link, the response message including the data of the data block under download.
And S412, storing the returned data into the corresponding memory space. That is, the data returned for the data block under download is stored in the memory space corresponding to the data block.
S413, whether it belongs to a proprietary protocol.
And judging whether the transmission protocol corresponding to the returned data belongs to a private protocol, if so, unpacking the returned data, executing S414, otherwise executing S415.
S414, unpacking.
And unpacking the returned data.
S415, obtaining the downloaded data sheet.
Under the condition that the returned data does not need to be unpacked, the downloaded data pieces in the data blocks can be obtained from the returned data, and under the condition that the returned data needs to be unpacked, the downloaded data pieces in the data blocks can also be obtained from the unpacked data.
S416, whether the data collection is completed.
Whether the data source returns the data to the point-to-point link is completed is judged, if not, the process jumps to the execution S411, otherwise, the execution S417 is executed.
S417, whether the data block is downloaded.
Whether the downloading of the whole data block in the downloading is completed is judged, if the downloading is completed, S418 is executed, otherwise, the operation jumps to S407, and the task layer distributes the downloading request for the point-to-point link.
S418, decrypting the data and landing the data.
And decrypting the downloaded data block in the memory space corresponding to the downloaded data block to obtain decrypted data, and landing the decrypted data, namely storing the decrypted data in a magnetic disk. Therefore, the data decryption is postpositioned, and the decryption is carried out together after the whole data block is downloaded, so that the decryption efficiency is improved.
S419, checking whether the downloading is completed at the file layer.
After the data is stored in the disk, whether the downloading of the target data is completed can be checked in a file layer (such as a file system). In the file layer, a data bitmap corresponding to the target data may be established, and whether the downloading of the target data is completed is checked according to the data bitmap. If it is checked in the file layer that the downloading of the target data is completed, S420 is executed, otherwise, the process goes to S403.
S420, releasing the resources and stopping downloading.
And after the completion of the downloading of the target data is determined, releasing the software and hardware resources allocated for the downloading of the target data, and ending the downloading of the target data.
Fig. 5 is a schematic diagram of a third embodiment of the present disclosure. As shown in fig. 5, a data downloading apparatus 500 provided in a third embodiment of the present disclosure includes:
a task acquisition unit 501 configured to acquire a download task for target data;
a data dividing unit 502 for dividing the target data into a plurality of data blocks;
a thread initializing unit 503, configured to initialize a plurality of network threads in a network layer for a plurality of data blocks;
the data block downloading unit 504 is configured to establish a plurality of point-to-point links between the current device and the data source by using a plurality of network threads, and download a plurality of data blocks through the plurality of point-to-point links.
In one possible implementation, there is no lock between the plurality of network threads, and each network thread performs data download based on a respective established point-to-point link among the plurality of network threads and the plurality of point-to-point links.
In one possible implementation, the data block downloading unit 504 includes: a message sending module (not shown in the figure) for sending a notification message to the task layer, wherein the notification message indicates that the plurality of point-to-point links are data download links of the plurality of data blocks; the request receiving module is used for receiving the download requests which are distributed by the task layer for a plurality of point-to-point links, wherein the download requests which are respectively corresponding to the point-to-point links are distributed by the task layer for the point-to-point links, and the download requests which are corresponding to the point-to-point links indicate that the point-to-point links are responsible for requesting the downloaded data in a plurality of data blocks; and the request sending module is used for sending the corresponding downloading request to the corresponding data source through a plurality of point-to-point links and receiving the downloading data returned by the corresponding data source.
In a possible implementation manner, the data downloading device further includes a request allocation unit (not shown in the figure), where the request allocation unit is configured to allocate corresponding download requests for a plurality of point-to-point links in a task layer, and the request allocation unit includes: an idle data block determining module (not shown in the figure) configured to determine, for a target link to be allocated with a request among a plurality of point-to-point links, an idle data block among data blocks that are not downloaded and are included in a plurality of data blocks, the idle data block being a data block including an idle data slice, the idle data slice being a data slice that is in an unrendered state; a first download request distribution module (not shown in the figure) for distributing the download request of the idle data pieces in the idle data blocks to the target links according to the data download capability of the target links.
In one possible implementation, the request allocation unit further includes: a preemptive data slice determining module (not shown in the figure) for determining that preemptive downloaded data slices are allowed in the data blocks in the downloading state if no idle data blocks exist in the data blocks which are not downloaded; a second download request distribution module (not shown in the figure) for distributing the download request of the data piece which allows preemptive download to the target link.
In one possible implementation, the data downloading device further includes: a download data storage unit (not shown in the figure) for acquiring the download data received by the plurality of point-to-point links, determining a target data block to which the download data belongs, and storing the download data into a memory space corresponding to the target data block; the memory space is applied before the downloaded data is received.
In one possible implementation, the data downloading device further includes: a data unpacking unit (not shown in the figure) for unpacking the downloaded data in the memory space to obtain the package receiving data corresponding to the downloaded data when the downloaded data needs unpacking; and the unpacking data storage unit is used for storing the unpacking data into the memory space.
In one possible implementation, the data downloading device further includes: a data decryption unit (not shown) for decrypting the target data block if the target data block is encrypted data after the target data block is downloaded.
In one possible implementation, the data downloading device further includes: a download completion judging unit (not shown in the figure) for determining whether or not the downloading of the plurality of data blocks is completed; a download circulation unit (not shown) for continuing to download the data blocks which are not downloaded from the plurality of data blocks through the plurality of network threads if the plurality of data blocks are not downloaded; a resource releasing unit (not shown in the figure) for releasing the plurality of network threads if the downloading of the plurality of data blocks is completed.
In one possible implementation manner, the download completion judging unit includes: a data bitmap acquisition module (not shown in the figure) for acquiring a data bitmap corresponding to the target data, wherein a plurality of binary values in the data bitmap are in one-to-one correspondence with a plurality of data blocks, and the binary values corresponding to the data blocks indicate that the downloading of the data blocks is completed or not completed; a download completion judging module (not shown) for determining whether the downloading of the plurality of data blocks is completed based on the plurality of binary values.
The data downloading device provided in fig. 5 may perform the steps related to the terminal in the above corresponding method embodiments, and the implementation principle and technical effects are similar, which are not described herein again.
According to an embodiment of the present disclosure, the present disclosure further provides an electronic device, including: at least one processor; and a memory communicatively coupled to the at least one processor; the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the aspects provided in any one of the embodiments described above.
According to an embodiment of the present disclosure, there is also provided a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the solution provided by any one of the above embodiments.
According to an embodiment of the present disclosure, the present disclosure also provides a computer program product comprising: a computer program stored in a readable storage medium, from which at least one processor of an electronic device can read, the at least one processor executing the computer program causing the electronic device to perform the solution provided by any one of the embodiments described above.
Fig. 6 is a schematic block diagram of an example electronic device 600 that may be used to implement embodiments of the present disclosure. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile apparatuses, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing apparatuses. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the disclosure described and/or claimed herein.
As shown in fig. 6, the electronic device 600 includes a computing unit 601 that can perform various appropriate actions and processes according to a computer program stored in a Read Only Memory (ROM) (fig. 6 is exemplified by a ROM 602) or a computer program loaded from a storage unit 608 into a random access Memory (Random Access Memory, RAM) (fig. 6 is exemplified by a RAM 603). In the RAM 603, various programs and data required for the operation of the electronic device 600 can also be stored. The computing unit 601, ROM 602, and RAM 603 are connected to each other by a bus 604. An input/output (I/O) interface (I/O interface 605 is also connected to bus 604, as shown in FIG. 6.
A number of components in the electronic device 600 are connected to the I/O interface 605, including: an input unit 606 such as a keyboard, mouse, etc.; an output unit 607 such as various types of displays, speakers, and the like; a storage unit 608, such as a magnetic disk, optical disk, or the like; and a communication unit 609 such as a network card, modem, wireless communication transceiver, etc. The communication unit 609 allows the electronic device 600 to exchange information/data with other devices through a computer network, such as the internet, and/or various telecommunication networks.
The computing unit 601 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of computing unit 601 include, but are not limited to, a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphic Processing Unit, GPU), various dedicated artificial intelligence (Artificial Intelligence, AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (Digital Signal Process, DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the respective methods and processes described above, such as a data download method. For example, in some embodiments, the data download method may be implemented as a computer software program tangibly embodied on a machine-readable medium, such as the storage unit 608. In some embodiments, part or all of the computer program may be loaded and/or installed onto the electronic device 600 via the ROM 602 and/or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the data downloading method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the data download method in any other suitable way (e.g., by means of firmware).
Various implementations of the systems and techniques described here above can be implemented in digital electronic circuitry, integrated circuit systems, field programmable gate arrays (Field Program Gate Array, FPGAs), application specific integrated circuits (Application Specific Integrated Circuit, ASICs), application specific standard products (Application Specific Standard Parts, ASSPs), systems On a Chip (SOC), complex programmable logic devices (Complex Programming Logic Device, CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs, the one or more computer programs may be executed and/or interpreted on a programmable system including at least one programmable processor, which may be a special purpose or general-purpose programmable processor, that may receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.
Program code for carrying out methods of the present disclosure may be written in any combination of one or more programming languages. These program code may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus such that the program code, when executed by the processor or controller, causes the functions/operations specified in the flowchart and/or block diagram to be implemented. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.
In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-Only Memory (ROM), an erasable programmable read-Only Memory (Erasable Programmable Read Only Memory, EPROM or flash Memory), an optical fiber, a portable compact disc read-Only Memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and pointing device (e.g., a mouse or trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic input, speech input, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a background component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such background, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local area network (Local Area Network, LAN), wide area network (Wide Area Network, WAN) and the internet.
The computer system may include a client and a server. The client and server are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, so that the defects of high management difficulty and weak service expansibility in the traditional physical hosts and VPS service ("Virtual Private Server" or simply "VPS") are overcome. The server may also be a server of a distributed system or a server that incorporates a blockchain.
It should be appreciated that various forms of the flows shown above may be used to reorder, add, or delete steps. For example, the steps recited in the present disclosure may be performed in parallel, sequentially, or in a different order, provided that the desired results of the disclosed aspects are achieved, and are not limited herein.
The above detailed description should not be taken as limiting the scope of the present disclosure. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives are possible, depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present disclosure are intended to be included within the scope of the present disclosure.

Claims (23)

1. A data downloading method, comprising:
acquiring a downloading task aiming at target data;
dividing the target data into a plurality of data blocks;
initializing a plurality of network threads in a network layer according to the plurality of data blocks;
and establishing a plurality of point-to-point links of the current equipment and the data source by using the plurality of network threads, and downloading the plurality of data blocks through the plurality of point-to-point links.
2. The data download method according to claim 1, wherein the plurality of network threads are lock-free therebetween, each network thread performing data download based on a respective established point-to-point link among the plurality of network threads and the plurality of point-to-point links.
3. The data downloading method as claimed in claim 1, wherein the downloading of the plurality of data blocks through the plurality of point-to-point links comprises:
sending a notification message to a task layer, wherein the notification message indicates that the plurality of point-to-point links are data downloading links of the plurality of data blocks;
receiving downloading requests respectively corresponding to the plurality of point-to-point links, wherein the downloading requests respectively corresponding to the plurality of point-to-point links are distributed to the plurality of point-to-point links by the task layer, and the downloading requests corresponding to the point-to-point links indicate that the point-to-point links are responsible for requesting the downloaded data in the plurality of data blocks;
and sending the corresponding downloading request to the corresponding data source through the plurality of point-to-point links, and receiving the downloading data returned by the corresponding data source.
4. A data download method according to claim 3, wherein the process of assigning corresponding download requests to the plurality of point-to-point links in the task layer comprises:
Determining an idle data block in the data blocks which are not downloaded and are contained in the data blocks aiming at target links to be allocated with requests in the point-to-point links, wherein the idle data block is a data block containing idle data pieces, and the idle data pieces are the data pieces in an unrendered state;
and distributing the downloading request of the idle data piece in the idle data block to the target link according to the data downloading capability of the target link.
5. The data download method according to claim 4, further comprising:
if the idle data block does not exist in the data block which is not downloaded, determining that the data block in the downloading state allows preempting the downloaded data piece;
and distributing the downloading request of the data sheet allowing preemptive downloading to the target link.
6. The data downloading method according to any one of claims 1 to 5, further comprising, after the downloading of the plurality of data blocks via the plurality of point-to-point links:
acquiring the download data received by the plurality of point-to-point links;
determining a target data block to which the downloaded data belong;
storing the downloaded data into a memory space corresponding to the target data block;
The memory space is applied before the download data is received.
7. The data downloading method according to claim 6, wherein after storing the downloaded data in the memory space corresponding to the target data block, the method comprises:
under the condition that the downloaded data needs to be unpacked, unpacking the downloaded data in the memory space to obtain package receiving data corresponding to the downloaded data;
and storing the packet data into the memory space.
8. The data download method according to claim 6, further comprising:
after the downloading of the target data block is completed, if the target data block is encrypted data, decrypting the target data block.
9. The data downloading method according to any one of claims 1 to 5, further comprising:
determining whether the downloading of the plurality of data blocks is completed;
if the downloading of the plurality of data blocks is not completed, continuing to download the data blocks which are not completed in the plurality of data blocks through the plurality of network threads;
and if the downloading of the plurality of data blocks is completed, releasing the plurality of network threads.
10. The data downloading method as claimed in claim 9, wherein the determining whether the downloading of the plurality of data blocks is completed comprises:
Acquiring a data bitmap corresponding to the target data, wherein a plurality of binary values in the data bitmap are in one-to-one correspondence with the plurality of data blocks, and the binary values corresponding to the data blocks indicate that the downloading of the data blocks is completed or not completed;
and determining whether the downloading of the plurality of data blocks is finished according to the plurality of binary values.
11. A data downloading apparatus comprising:
the task acquisition unit is used for acquiring a downloading task aiming at the target data;
a data dividing unit for dividing the target data into a plurality of data blocks;
a thread initializing unit, configured to initialize a plurality of network threads in a network layer for the plurality of data blocks;
and the data block downloading unit is used for establishing a plurality of point-to-point links of the current equipment and the data source by using the plurality of network threads, and downloading the plurality of data blocks through the plurality of point-to-point links.
12. The data downloading device of claim 11, wherein the plurality of network threads are lock-free therebetween, each network thread performing data downloading based on a respective established point-to-point link among the plurality of network threads and the plurality of point-to-point links.
13. The data downloading apparatus as claimed in claim 11, wherein the data block downloading unit comprises:
the message sending module is used for sending a notification message to the task layer, wherein the notification message indicates that the plurality of point-to-point links are data downloading links of the plurality of data blocks;
the request receiving module is used for receiving the download requests distributed by the task layer for the plurality of point-to-point links, wherein the download requests respectively corresponding to the plurality of point-to-point links are distributed by the task layer for the plurality of point-to-point links, and the download requests corresponding to the point-to-point links indicate the point-to-point links to be responsible for requesting the downloaded data in the plurality of data blocks;
and the request sending module is used for sending the corresponding downloading request to the corresponding data source through the plurality of point-to-point links and receiving the downloading data returned by the corresponding data source.
14. The data downloading apparatus as claimed in claim 13, further comprising a request allocation unit for allocating corresponding download requests for the plurality of point-to-point links in the task layer, the request allocation unit comprising:
the idle data block determining module is used for determining idle data blocks aiming at target links to be allocated with requests in the plurality of point-to-point links, wherein the idle data blocks are data blocks containing idle data sheets, and the idle data sheets are data sheets in an un-downloaded state;
And the first download request distribution module is used for distributing the download request of the idle data piece in the idle data block to the target link according to the data download capability of the target link.
15. The data downloading device of claim 14, the request allocation unit further comprising:
the preemption data sheet determining module is used for determining that preemption of the downloaded data sheets is allowed in the data blocks in the downloading state if the idle data blocks do not exist in the data blocks which are not downloaded;
and the second download request distribution module is used for distributing the download request of the data sheet which allows preemptive download to the target link.
16. The data downloading device according to any one of claims 11 to 15, further comprising:
the download data storage unit is used for acquiring the download data received by the plurality of point-to-point links, determining a target data block to which the download data belongs, and storing the download data into a memory space corresponding to the target data block;
the memory space is applied before the download data is received.
17. The data downloading device of claim 16, further comprising:
the data unpacking unit is used for unpacking the downloaded data in the memory space under the condition that the downloaded data needs unpacking to obtain package receiving data corresponding to the downloaded data;
And the unpacking data storage unit is used for storing the packed data into the memory space.
18. The data downloading device of claim 16, further comprising:
and the data decryption unit is used for decrypting the target data block if the target data block is encrypted data after the target data block is downloaded.
19. The data downloading device according to any one of claims 11 to 15, further comprising:
a download completion judging unit configured to determine whether the downloading of the plurality of data blocks is completed;
a downloading circulation unit, configured to continuously download, if the downloading of the plurality of data blocks is not completed, the data blocks, which are not downloaded, in the plurality of data blocks through the plurality of network threads;
and the resource release unit is used for releasing the plurality of network threads if the downloading of the plurality of data blocks is completed.
20. The data downloading apparatus as claimed in claim 19, wherein the download completion judging unit comprises:
the data bitmap acquisition module is used for acquiring a data bitmap corresponding to the target data, wherein a plurality of binary values in the data bitmap are in one-to-one correspondence with the plurality of data blocks, and the binary values corresponding to the data blocks indicate that the downloading of the data blocks is completed or not completed;
And the downloading completion judging module is used for determining whether the downloading of the plurality of data blocks is completed according to the plurality of binary values.
21. An electronic device, comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the data download method according to any one of claims 1 to 10.
22. A non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the data download method according to any one of claims 1 to 10.
23. A computer program product comprising a computer program which, when executed by a processor, implements the steps of the data downloading method according to any one of claims 1 to 10.
CN202311533479.0A 2023-11-16 2023-11-16 Data downloading method, device, equipment and storage medium Pending CN117579619A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202311533479.0A CN117579619A (en) 2023-11-16 2023-11-16 Data downloading method, device, equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202311533479.0A CN117579619A (en) 2023-11-16 2023-11-16 Data downloading method, device, equipment and storage medium

Publications (1)

Publication Number Publication Date
CN117579619A true CN117579619A (en) 2024-02-20

Family

ID=89892924

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202311533479.0A Pending CN117579619A (en) 2023-11-16 2023-11-16 Data downloading method, device, equipment and storage medium

Country Status (1)

Country Link
CN (1) CN117579619A (en)

Similar Documents

Publication Publication Date Title
US10467725B2 (en) Managing access to a resource pool of graphics processing units under fine grain control
US10572290B2 (en) Method and apparatus for allocating a physical resource to a virtual machine
CN107241281B (en) Data processing method and device
CN113641457B (en) Container creation method, device, apparatus, medium, and program product
EP3716577A1 (en) Cloud service migration method and apparatus, and electronic device
CN107832143B (en) Method and device for processing physical machine resources
US11201836B2 (en) Method and device for managing stateful application on server
US10341264B2 (en) Technologies for scalable packet reception and transmission
CN109726005B (en) Method, server system and computer readable medium for managing resources
CN110166507B (en) Multi-resource scheduling method and device
JP2022105146A (en) Acceleration system, acceleration method, and computer program
EP3644182A1 (en) Container isolation method and device for netlink resource
CN112905342A (en) Resource scheduling method, device, equipment and computer readable storage medium
CN115237589A (en) SR-IOV-based virtualization method, device and equipment
US10673983B2 (en) Processing a unit of work
CN111290842A (en) Task execution method and device
CN109639599B (en) Network resource scheduling method and system, storage medium and scheduling device
CN113157611B (en) Data transmission control method, device, equipment and readable storage medium
CN117579619A (en) Data downloading method, device, equipment and storage medium
CN114640681A (en) Data processing method and system
CN114237902A (en) Service deployment method and device, electronic equipment and computer readable medium
US11954534B2 (en) Scheduling in a container orchestration system utilizing hardware topology hints
US9330036B2 (en) Interrupt reduction by dynamic application buffering
CN113760798A (en) RDMA device allocation method, computing device and storage medium
CN117891589A (en) Distribution method, device and equipment of heterogeneous hard disk and computer readable storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination