WO2021164178A1 - Cloud technology-based file fragment uploading method and apparatus, and device and storage medium - Google Patents

Cloud technology-based file fragment uploading method and apparatus, and device and storage medium Download PDF

Info

Publication number
WO2021164178A1
WO2021164178A1 PCT/CN2020/099534 CN2020099534W WO2021164178A1 WO 2021164178 A1 WO2021164178 A1 WO 2021164178A1 CN 2020099534 W CN2020099534 W CN 2020099534W WO 2021164178 A1 WO2021164178 A1 WO 2021164178A1
Authority
WO
WIPO (PCT)
Prior art keywords
file
uploaded
type
upload
fragments
Prior art date
Application number
PCT/CN2020/099534
Other languages
French (fr)
Chinese (zh)
Inventor
李清鹏
Original Assignee
平安科技(深圳)有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 平安科技(深圳)有限公司 filed Critical 平安科技(深圳)有限公司
Publication of WO2021164178A1 publication Critical patent/WO2021164178A1/en

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/13File access structures, e.g. distributed indices
    • G06F16/137Hash-based
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/18File system types
    • G06F16/182Distributed file systems

Definitions

  • This application relates to the field of big data technology, in particular to a method, device, device, and storage medium for uploading file fragments based on cloud technology.
  • Cloud object storage provides users with safe and reliable customized storage solutions, and its essence is to provide file storage services for registered users.
  • the cloud object storage service in the cloud platform is only responsible for providing two basic upload interfaces, one is the basic shard interface, and the other is the final slice interface. Among them, the final fragment interface is used to upload the last file fragment in the fragmented file, and the basic fragment interface is used to upload other file fragments in the fragmented file except the last file fragment.
  • the technical problem to be solved by the embodiments of this application is to provide a method, device, equipment, and storage medium for uploading files of different sizes based on cloud technology, which can be adapted to upload files of different sizes on different terminal devices, and can conveniently Record the upload status of the file and verify the integrity of the file.
  • an embodiment of the present application provides a method for uploading file fragments based on cloud technology, which adopts the following technical solutions:
  • a file segment upload method based on cloud technology including:
  • an embodiment of the present application also provides a file segment uploading device based on cloud technology, which adopts the following technical solutions:
  • a file segment upload device based on cloud technology including:
  • An instruction receiving module configured to receive an upload instruction configured by a target user, where the upload instruction specifies a file object to be uploaded;
  • the file type division module is used to read the specified file to be uploaded according to the file object to be uploaded in the upload instruction, and obtain the file size and file identifier of each file to be uploaded, and compare the file size of each file to be uploaded With the size of the preset fragmentation threshold, the file to be uploaded whose file size is less than or equal to the fragmentation threshold is recorded as the first type of file, and the file to be uploaded with the file size greater than the fragmentation threshold is recorded as the second type. document;
  • a file fragmentation module configured to perform fragmentation processing on the second-type file based on the fragmentation threshold, so as to split all the second-type files into several file fragments carrying a file identifier and a fragmentation identifier;
  • a hash processing module which is used to sequentially read several file fragments of the first type file and the second type file into the memory, and perform hash processing to obtain the first hash of each file to be uploaded value;
  • the file upload module is used to call the basic fragmentation interface and the final fragment interface, directly upload the first type file in the memory through the final fragment interface, and upload the basic fragment interface and the final fragment interface.
  • the file verification module is used to mark the upload status of the file to be uploaded based on the file identifier and/or the segment identifier.
  • the file segment corresponding to the second type of file is uploaded, the file segments with the same file identifier are merged together, In order to restore to the second type file, and perform hash processing on each of the first type file and the second type file that have been uploaded to obtain the corresponding second hash value, through the first hash value and all the files The second hash value is checked to confirm the integrity of the uploaded file to be uploaded.
  • the embodiments of the present application also provide a computer device, which adopts the following technical solutions:
  • a computer device includes a memory, a processor, and computer-readable instructions stored in the memory and capable of running on the processor.
  • the processor executes the computer-readable instructions, the following is achieved based on cloud technology
  • the embodiments of the present application also provide one or more computer-readable storage media storing computer-readable instructions, and the following technical solutions are adopted:
  • One or more computer-readable storage media storing computer-readable instructions, where the computer-readable instructions implement the following steps of the cloud-based file segment upload method when the computer-readable instructions are executed by a processor:
  • the embodiment of the application discloses a method, device, device, and storage medium for uploading file fragments based on cloud technology.
  • the method for uploading file fragments based on cloud technology after receiving the upload instruction; Upload files, record the files to be uploaded whose file size is less than or equal to the fragmentation threshold as the first type of files, and record the files to be uploaded with the file size greater than the fragmentation threshold as the second type of files; perform fragment processing on the second type of files , Split the second type file into several file fragments; and perform hash processing to obtain the first hash value of each file to be uploaded; after calling the basic fragment interface and the final fragment interface, pass the final fragment interface Upload the first type of file directly, upload several file segments of the second type of file through the basic segmentation interface and the final segment interface; finally confirm the integrity of the uploaded file to be uploaded through the hash value.
  • the method can avoid upload restrictions of terminal devices belonging to different systems, and enable different terminal devices to realize the synchronous or asynchronous upload of multiple file fragments of different sizes, as well as the function of resuming file transfers, and can record more conveniently The upload status of the file and the integrity of the verification file.
  • FIG. 1 is an exemplary system architecture diagram to which the embodiments of this application can be applied;
  • FIG. 2 is a flowchart of an embodiment of the method for uploading file fragments based on cloud technology in an embodiment of the application
  • FIG. 3 is a schematic structural diagram of an embodiment of the device for uploading file fragments based on cloud technology in an embodiment of the application;
  • Fig. 4 is a schematic structural diagram of an embodiment of a computer device in an embodiment of the application.
  • the system architecture 100 may include terminal devices 101, 102, 103, a network 104, and a server 105.
  • the network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105.
  • the network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, and so on.
  • the user can use the terminal devices 101, 102, and 103 to interact with the server 105 through the network 104 to receive or send messages and so on.
  • Various communication client applications such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, and social platform software, can be installed on the terminal devices 101, 102, and 103.
  • the terminal devices 101, 102, 103 may be various electronic devices with display screens and support for web browsing, including but not limited to smart phones, tablets, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III, dynamic images) Experts compress standard audio layer 3), MP4 (Moving Picture Experts Group Audio Layer IV, dynamic image experts compress standard audio layer 4) players, laptop portable computers and desktop computers, etc.
  • MP3 players Motion Picture Experts Group Audio Layer III, dynamic images
  • MP4 Moving Picture Experts Group Audio Layer IV, dynamic image experts compress standard audio layer 4
  • laptop portable computers and desktop computers etc.
  • the server 105 may be a server that provides various services, for example, a background server that provides support for pages displayed on the terminal devices 101, 102, and 103.
  • the method for uploading file fragments based on cloud technology provided by the embodiments of the present application is generally executed by a server/terminal device. Accordingly, the device for uploading file fragments based on cloud technology is generally set in the server/terminal device.
  • terminal devices, networks, and servers in FIG. 1 are merely illustrative. There can be any number of terminal devices, networks, and servers according to implementation needs.
  • FIG. 2 a flowchart of an embodiment of the method for uploading file fragments based on cloud technology in an embodiment of the present application is shown.
  • the method for uploading file fragments based on cloud technology includes the following steps:
  • Step 201 Receive an upload instruction configured by a target user, where the upload instruction specifies a file object to be uploaded.
  • the file object to be uploaded described in the upload instruction refers to the file that the target user expects to upload during this upload and the path corresponding to the file.
  • the number of files to be uploaded can be single or multiple. When the number is multiple, the target user can configure the file list of the files to be uploaded and the corresponding path of each file in the upload instruction.
  • the target user configures the upload instruction
  • he can also customize some of his upload requirements, such as specifying that he wants to upload in a synchronous or asynchronous manner.
  • the electronic device for example, the server/terminal device shown in FIG. 1 on which the cloud-based file segment upload method runs can receive the target user's sending through a wired connection or a wireless connection. Upload instructions.
  • the above-mentioned wireless connection methods may include, but are not limited to, 3G/4G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultrawideband) connection, and other currently known or future wireless connection methods.
  • Step 202 Read the specified file to be uploaded according to the file object to be uploaded in the upload instruction, and obtain the file size and file identifier of each file to be uploaded, and compare the file size of each file to be uploaded with the preset file size. For the size of the fragmentation threshold, a file to be uploaded with a file size smaller than or equal to the fragmentation threshold is recorded as a first type of file, and a file to be uploaded with a file size greater than the fragmentation threshold is recorded as a file of the second type.
  • the server responds to the upload instruction to access according to the file path described in the upload instruction, and obtains the corresponding file to be uploaded from the accessed file path, and the file size and file identifier of each file to be uploaded can be obtained from this.
  • uploading the file to be uploaded in the subsequent steps it needs to be read into the memory before uploading, and the file to be uploaded generally needs to be divided into pieces before uploading.
  • the fragmentation threshold is a criterion for judging whether to perform fragmentation processing on a file, and it is also a fragmentation basis when performing fragmentation processing on a file.
  • the file to be uploaded for fragmentation processing is a file whose size is greater than the fragmentation threshold.
  • files to be uploaded with a file size greater than the fragmentation threshold are recorded as the first type of file, and files to be uploaded with the file size greater than the fragmentation threshold are recorded as the second type of file.
  • the size of the fragmentation threshold can be preset, such as 100K, 500K, or 1M according to the needs of different scenarios.
  • the cloud technology-based file fragment upload method before the step of comparing the file size of each file to be uploaded with the size of the preset fragmentation threshold in step 202, the cloud technology-based file fragment upload method also include:
  • the server will pre-configure a mapping table for different network connection status scenarios.
  • the mapping table divides different network connection statuses into different network status levels.
  • the network status level corresponds to a fragmentation threshold.
  • the corresponding network status level is searched in the mapping table according to the network connection status, and the mapping table is further matched to the network status level
  • the corresponding fragmentation threshold is regarded as the fragmentation threshold used in the steps of the current cloud-based file fragmentation upload method.
  • Step 203 Perform fragmentation processing on the second-type file based on the fragmentation threshold, so as to split all the second-type files into a plurality of file fragments carrying a file identifier and a fragment identifier.
  • Fragmentation processing of the file will split the file into several file fragments, and the sum of the size of the several fragments is equal to the size of the original file.
  • these file fragments except for the last file fragment, other file fragments have the same size and the same size as the fragment threshold, and the size of the last file fragment is less than or equal to the fragment threshold.
  • the steps of fragmentation processing Assuming that the size of the original file is 1200K and the set fragmentation threshold is 500K, after fragmentation processing is performed on the file, the file will be split into three file fragments, the first two The size of each is 500K, and the data of 0-500K and the data of 500K-1000K of the original file are respectively recorded. The size of the third and last file segment is 200K, and the remaining 200K data in the original file is recorded.
  • the second type of file when the second type of file is fragmented, a number of file fragments are split, and each file fragment will carry its corresponding second type.
  • the file size of each file to be uploaded is compared with the preset file size in step 202.
  • the step of recording a file to be uploaded with a file size less than or equal to the fragmentation threshold as a first type of file, and recording a file to be uploaded with a file size greater than the fragmentation threshold as a file of the second type include:
  • the file size of each file to be uploaded is compared with the size of the corresponding fragmentation threshold.
  • the file to be uploaded whose file size is less than or equal to the corresponding fragmentation threshold is recorded as the first type of file, and the file size is greater than the corresponding fragmentation threshold.
  • the file to be uploaded with the fragmentation threshold of is recorded as the second type of file;
  • the step of performing fragmentation processing on the second-type file based on the fragmentation threshold in step 203 includes: based on the plurality of fragmentation thresholds specified in the upload instruction, each of the second-type files is corresponding to the fragmentation threshold. Perform fragmentation processing.
  • the target user can specify different fragmentation thresholds for different files to be uploaded in the upload instruction.
  • processing is performed based on the fragmentation threshold assigned to each file to be uploaded in the upload instruction.
  • Step 204 Read several file fragments of the first type file and the second type file into the memory in sequence, and perform hash processing to obtain the first hash value of each file to be uploaded.
  • the file to be uploaded will be read into the memory of the server before uploading for hash processing and upload.
  • MD5 Message-Digest Algorithm 5
  • MD5 is used to hash the file to be uploaded.
  • MD5 is a password hash function widely used at present. It is often used to verify the integrity of network file transmission and prevent files from being tampered with. Compared with other hash algorithms, MD5 has a stronger anti-cracking ability.
  • the step 204 includes:
  • the file of the first type is directly read into the memory to perform hash processing to obtain the first hash value of each file of the first type;
  • the fragment hash value, and whenever the fragment hash value corresponding to the last file fragment of a second type file is obtained, the fragment hash value corresponding to all file fragments of the second type file is performed Combine to obtain the first hash value of each file of the second type.
  • the type of the file to be uploaded it can be identified based on the file suffix or the content of the hidden parameter that can more accurately represent the file type. Due to the limitation of the memory size, when reading the second type file, it will read the file fragments after its split in turn, and after calculating the hash value of the fragments, the several files of the second type file A number of fragmented hash values of the fragments are combined through accumulation to obtain the first hash value of the entire second-type file.
  • MD5 supports streaming hash value accumulation calculations, such as a second type file with multiple file fragments, each time a file fragment is read, the file fragment is hashed to obtain the fragment Hash value to obtain a 128-bit string. After reading all file fragments of the second type of file, the corresponding fragment hash value corresponding to the number of file fragments can be obtained, and all the fragments are finally hashed The column value is accumulated, and the hash value corresponding to the entire second-type file can be obtained.
  • Step 205 Invoke the basic slice interface and the final slice interface, directly upload the first type file in the memory through the final slice interface, and upload the file in the memory through the basic slice interface and the final slice interface. Several file fragments of the second type file.
  • the two upload interfaces provided by the cloud object storage service in the Ping An cloud platform are the basic slice interface and the final slice interface.
  • the final film interface is used to upload the last part of the file data of a file to be uploaded, which can facilitate the system to record the status and progress of the file upload.
  • For the first type of file there is no need to split into multiple file fragments through fragmentation, so it can be regarded as the first and last file fragment, and uploading needs to be done through the final fragment interface when uploading.
  • the target user when uploading a file to be uploaded, the target user can configure upload requirements in the upload instruction in advance, hoping to upload the file in a synchronous or asynchronous manner.
  • the server uploads the file to be uploaded, there are no relevant restrictions if the hardware supports it.
  • select the method of synchronous or asynchronous upload Based on the upload requirements configured in the upload instruction, select the method of synchronous or asynchronous upload. Synchronous upload means that the next file segment of the first type file or the second type file must wait for the file segment of the previous first type file or the second type file to upload successfully before uploading; asynchronous upload means that there is no need to queue up and Wait, the file fragments of the first type of file or the second type of file can be uploaded after being read into the memory in turn.
  • asynchronous upload should be understood as uploading at the same time, but the time node of the upload will be affected by the read into the memory. There are minor differences in time sequence.
  • the step of uploading several file fragments of the second type file in the memory through the basic fragment interface and the final fragment interface in step 205 includes:
  • the second type of file to which it belongs is first identified based on the file identifier it carries, and then it is further identified based on the segment identifier it carries whether it is the first type of file. The last file segment of the second type file.
  • Step 206 Mark the upload status of the file to be uploaded based on the file identifier and/or the segment identifier. After the file segment corresponding to the second type of file is uploaded, the file segments with the same file identifier are combined to restore the first file segment. Type two files, and perform hash processing on each file of the first type and the file of the second type that have been uploaded to obtain the corresponding second hash value, and pass the first hash value and the second hash value. The column value is checked to confirm the integrity of the uploaded file that has been uploaded.
  • the upload status of the file to be uploaded may include data such as upload progress, upload node, and upload time.
  • data such as upload progress, upload node, and upload time.
  • the completeness can be understood as whether the file fragments of the second type of file have been uploaded, whether the first type of file has been uploaded, and whether there is a difference in the content comparison before and after the file is uploaded. If there is a difference, it means that there was an error during the upload of the file and it is complete. Sexual issues.
  • the content of the merged file will be hashed to obtain the second hash value, and the correctness and completeness of the file will be verified when uploading, and the first hash value will be verified Just compare with the second hash value. If they are consistent, it means that the content of the file to be uploaded has not been changed during the transmission process, and the entire file has been uploaded successfully; otherwise, it means that the file to be uploaded has been tampered with, and the uploaded file needs to be discarded.
  • the step of combining the file fragments with the same file identifier to restore to the second type of file includes :
  • the call to the basic fragment interface and the final fragment interface is stopped, and an alarm message is generated and sent based on the upload status of each file fragment. To the target user.
  • the last file segment under the same second type file may be smaller than other file segments, even if the last file segment is uploaded later on the time node during asynchronous upload, when the network condition is not good , It may be the first to complete the upload. At this time, it is necessary to check whether other file fragments under the second type of file have been uploaded. If there are still file fragments that have not been uploaded within the preset response time, it means that the current network environment is poor or the upload service is abnormal. , The upload status during the network connection is not good, and normal file upload cannot be performed. The upload operation should be interrupted, and the corresponding alarm message should be generated and sent to the target user as a reminder. After the network connection problem is found, the upload can be continued.
  • the cloud technology-based file segment upload method further includes:
  • the file node can be a first One file segment of a file of one type or a file of the second type may also be a plurality of file segments of files of the first type and/or files of the second type. From the file node, uploading the file fragments of the first type file and/or the second type file that have not been uploaded successfully and that have not been uploaded is restarted, so that the file can be resumably uploaded.
  • the cloud technology-based file fragment upload method described in the embodiment of the present application can avoid upload restrictions of terminal devices belonging to different systems, and enable different terminal devices to realize synchronous or asynchronous upload of multiple file fragments of different sizes, and The function of resuming the file transfer, and can more conveniently record the upload status of the file and verify the integrity of the file.
  • the aforementioned storage medium may be a non-volatile storage medium such as a magnetic disk, an optical disc, a read-only memory (Read-Only Memory, ROM), or a random access memory (Random Access Memory, RAM), etc.
  • Fig. 3 shows a schematic structural diagram of an embodiment of the device for uploading file fragments based on cloud technology in an embodiment of the present application.
  • this application provides an embodiment of a device for uploading file fragments based on cloud technology.
  • the device embodiment corresponds to the method embodiment shown in FIG. It can be applied to various electronic devices.
  • the device for uploading file fragments based on cloud technology in this embodiment includes:
  • Instruction receiving module 301 used to receive an upload instruction configured by a target user, and the upload instruction specifies a file object to be uploaded.
  • File type division module 302 used to read the specified file to be uploaded according to the file object to be uploaded in the upload instruction, and obtain the file size and file identifier of each file to be uploaded, and compare the files of each file to be uploaded
  • the file size is equal to the preset fragmentation threshold, the file to be uploaded whose file size is less than or equal to the fragmentation threshold is recorded as the first type of file, and the file to be uploaded with the file size greater than the fragmentation threshold is recorded as the second type.
  • Type file used to read the specified file to be uploaded according to the file object to be uploaded in the upload instruction, and obtain the file size and file identifier of each file to be uploaded, and compare the files of each file to be uploaded
  • the file size is equal to the preset fragmentation threshold, the file to be uploaded whose file size is less than or equal to the fragmentation threshold is recorded as the first type of file, and the file to be uploaded with the file size greater than the fragmentation threshold is recorded as the second type.
  • Type file used to read the specified file to be uploaded according
  • File fragmentation module 303 used to perform fragmentation processing on the second-type file based on the fragmentation threshold, so as to split all the second-type files into several file fragments that carry a file identifier and a fragmentation identifier. .
  • Hash processing module 304 used to read several file fragments of the first type file and the second type file into memory in turn, and perform hash processing to obtain the first hash of each file to be uploaded Column value.
  • File upload module 305 used to call the basic slice interface and the final slice interface, directly upload the first type file in the memory through the final slice interface, and upload all the files through the basic slice interface and the final slice interface Several file fragments of the second type file in the memory.
  • File verification module 306 used to mark the upload status of the file to be uploaded based on the file identifier and/or the segment identifier.
  • the file segment corresponding to the second type of file is uploaded, the file segments with the same file identifier are merged , To restore to the second type file, and perform hash processing on each of the first type file and the second type file that have been uploaded to obtain the corresponding second hash value, through the first hash value and The second hash value is checked to confirm the integrity of the file to be uploaded that has been uploaded.
  • the file type dividing module 302 further includes: a threshold setting submodule. Before the file type dividing module 302 compares the file size of each file to be uploaded with the size of a preset fragmentation threshold, the threshold setting submodule is used to detect the network connection status of the current network, and obtain the network of the current network. Status level; matching a preset fragmentation threshold corresponding to the network status level, and set as a fragmentation threshold for comparison with the file size of the file to be uploaded.
  • a threshold setting submodule Before the file type dividing module 302 compares the file size of each file to be uploaded with the size of a preset fragmentation threshold, the threshold setting submodule is used to detect the network connection status of the current network, and obtain the network of the current network. Status level; matching a preset fragmentation threshold corresponding to the network status level, and set as a fragmentation threshold for comparison with the file size of the file to be uploaded.
  • the file type division module 302 is also used to identify the file identifier of the file to be uploaded, from In the upload instruction, several fragmentation thresholds corresponding to different file identifiers are acquired; the file size of each file to be uploaded is compared with the size of the corresponding fragmentation threshold, and the file size is less than or equal to the corresponding fragmentation threshold.
  • a file to be uploaded with a fragment threshold is recorded as a first type of file, and a file to be uploaded with a file size greater than the corresponding fragment threshold is recorded as a file of the second type;
  • the file fragmentation module 303 is further configured to perform fragmentation processing for the corresponding second-type file based on several fragmentation thresholds specified in the upload instruction.
  • the hash processing module 304 is used to: identify the type of the file to be uploaded; if it is the first type file, directly read the first type file into the memory and perform the hash processing to obtain The first hash value of each file of the first type; if it is a file of the second type, several file segments of the file of the second type are read in sequence, and the file is segmented every time a file segment is read in Perform hash processing to obtain the fragment hash value corresponding to the file fragment, and whenever the fragment hash value corresponding to the last file fragment of a second type file is obtained, the second type file The fragment hash values corresponding to all the file fragments of are combined to obtain the first hash value of each file of the second type.
  • the file upload module 305 is configured to identify whether the file segment currently to be uploaded is the last file segment of the second type file to which it belongs based on the file identifier and the segment identifier; if If no, upload the file fragment currently to be uploaded through the basic fragment interface; if yes, upload the file fragment currently to be uploaded through the final fragment interface.
  • the file verification module 306 is configured to: whenever the upload of a file segment is completed through the final segment interface, it will cyclically detect the file segment within a preset response time. The upload status of other file segments with the same file ID, until it is detected that all other file segments are uploaded, all uploaded file segments with the same file ID as the file segment are combined and restored; and If it is not detected that all other file fragments are uploaded after the response time, stop calling the basic fragment interface and the final fragment interface, and generate an alarm message based on the upload status of each file fragment and send it to The target user.
  • the file verification module 306 further includes: a resuming sub-module. After the file verification module 306 generates an alarm message based on the upload status of each file segment and sends it to the target user, the resuming sub-module is used to receive the resuming instruction sent by the target user; read the last file upload time The upload status of each file to be uploaded is used to mark the file node for uploading in response to the resuming instruction; in response to the resuming instruction, the first file node that has not been uploaded successfully and has not been uploaded is checked. File fragments of type files and/or second type files are re-uploaded.
  • the cloud-based file segment upload device described in the embodiment of the present application can avoid upload restrictions of terminal devices belonging to different systems, and enable different terminal devices to realize synchronous or asynchronous upload of multiple file segments of different sizes, and The function of resuming the file transfer, and can more conveniently record the upload status of the file and verify the integrity of the file.
  • FIG. 4 is a block diagram of the basic structure of the computer device in this embodiment.
  • the computer device 6 includes a memory 61, a processor 62, and a network interface 63 that communicate with each other through a system bus. It should be pointed out that only the computer device 6 with components 61-63 is shown in the figure, but it should be understood that it is not required to implement all the illustrated components, and more or fewer components may be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculation and/or information processing in accordance with pre-set or stored instructions.
  • Its hardware includes, but is not limited to, a microprocessor, a dedicated Integrated Circuit (Application Specific Integrated Circuit, ASIC), Programmable Gate Array (Field-Programmable Gate Array, FPGA), Digital Processor (Digital Signal Processor, DSP), embedded equipment, etc.
  • ASIC Application Specific Integrated Circuit
  • ASIC Application Specific Integrated Circuit
  • FPGA Field-Programmable Gate Array
  • DSP Digital Processor
  • the computer device may be a computing device such as a desktop computer, a notebook, a palmtop computer, and a cloud server.
  • the computer device can interact with the user through a keyboard, a mouse, a remote control, a touch panel, or a voice control device.
  • the memory 61 includes at least one type of readable storage medium, the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static memory Random access memory (SRAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), programmable read only memory (PROM), magnetic memory, magnetic disk, optical disk, etc.
  • the computer-readable storage medium may be non-volatile or volatile.
  • the memory 61 may be an internal storage unit of the computer device 6, such as a hard disk or a memory of the computer device 6.
  • the memory 61 may also be an external storage device of the computer device 6, such as a plug-in hard disk equipped on the computer device 6, a smart media card (SMC), a secure digital (Secure Digital, SD) card, Flash Card, etc.
  • the memory 61 may also include both the internal storage unit of the computer device 6 and its external storage device.
  • the memory 61 is generally used to store an operating system and various application software installed on the computer device 6, such as computer-readable instructions of a method for uploading file fragments based on cloud technology.
  • the memory 61 can also be used to temporarily store various types of data that have been output or will be output.
  • the processor 62 may be a central processing unit (Central Processing Unit, CPU), controller, microcontroller, microprocessor, or other data processing chip in some embodiments.
  • the processor 62 is generally used to control the overall operation of the computer device 6.
  • the processor 62 is configured to run computer-readable instructions or processed data stored in the memory 61, for example, run the computer-readable instructions of the cloud-based file segment upload method.
  • the network interface 63 may include a wireless network interface or a wired network interface, and the network interface 63 is generally used to establish a communication connection between the computer device 6 and other electronic devices.
  • This application also provides another implementation manner, that is, to provide a computer-readable storage medium that stores a cloud-based file segment upload process, and the cloud-based file segment upload process
  • the process may be executed by at least one processor, so that the at least one processor executes the steps of the method for uploading file fragments based on cloud technology as described above.
  • the technical solution of this application essentially or the part that contributes to the existing technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium (such as ROM/RAM, magnetic disk, The optical disc) includes several instructions to make a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) execute the methods described in the various embodiments of the present application.
  • a terminal device which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.
  • the disclosed device and method may be implemented in other ways.
  • the device embodiments described above are merely illustrative.
  • the division of the modules is only a logical function division, and there may be other divisions in actual implementation, for example, multiple modules or components may be combined. Or it can be integrated into another system, or some features can be ignored or not implemented.
  • the modules or components may or may not be physically separated, and the components displayed as modules or components may or may not be physical modules, and may be located in one place or distributed on multiple network units. Some or all of the modules or components may be selected according to actual needs to achieve the objectives of the solutions of the embodiments.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Information Transfer Between Computers (AREA)

Abstract

A cloud technology-based file fragment uploading method and apparatus, and a device and a storage medium, relating to the technical field of big data. The method comprises: receiving an upload instruction; reading files to be uploaded, and dividing said files into a first type of files and a second type of files; splitting the second type of files into several file fragments; obtaining the first hash value of each file to be uploaded; uploading the first type of files by means of a final fragment interface, and uploading several file fragments of the second type of files by means of a basic fragment interface and the final fragment interface; performing hash processing on the uploaded files to obtain a second hash value, and perform verification by means of the first hash value and the second hash value to confirm the integrity of the files. The method can avoid uploading limits of different terminal devices, achieves the synchronous or asynchronous uploading of multiple file fragments of different sizes and the function of file breakpoint continuation, and can more conveniently record the upload status of files and verify the integrity of the files.

Description

基于云技术的文件分片上传方法、装置、设备及存储介质File segment upload method, device, equipment and storage medium based on cloud technology
本申请要求于2020年2月21日提交中国专利局、申请号为2020101062941,名称为“基于云技术的文件分片上传方法、装置、设备及存储介质”的中国专利申请的优先权,其全部内容通过引用结合在本申请中。This application claims the priority of a Chinese patent application filed with the Chinese Patent Office on February 21, 2020, with the application number 2020101062941 and the title "Cloud-based file upload method, device, equipment and storage medium", all of which The content is incorporated in this application by reference.
技术领域Technical field
本申请涉及大数据技术领域,特别是一种基于云技术的文件分片上传方法、装置、设备及存储介质。This application relates to the field of big data technology, in particular to a method, device, device, and storage medium for uploading file fragments based on cloud technology.
背景技术Background technique
云对象存储为用户提供安全可靠的定制化存储解决方案,其本质是为注册用户提供文件存储服务。对于文件上传,云平台中的云对象存储服务只负责提供两个基础的上传接口,一个是基础分片接口,一个是终片接口。其中,终片接口用于上传分片后的文件中的最后一个文件分片,基础分片接口用于上传分片后的文件中除最后一个文件分片外的其他文件分片。Cloud object storage provides users with safe and reliable customized storage solutions, and its essence is to provide file storage services for registered users. For file upload, the cloud object storage service in the cloud platform is only responsible for providing two basic upload interfaces, one is the basic shard interface, and the other is the final slice interface. Among them, the final fragment interface is used to upload the last file fragment in the fragmented file, and the basic fragment interface is used to upload other file fragments in the fragmented file except the last file fragment.
发明人发现,对于不同的终端设备来说,利用云平台中的云对象存储服务时,基于云平台提供的两个最基础的上传接口,目前缺乏一个通用、有效的、令人满意的综合文件上传服务解决方案,如期望能通过这两个上传接口进行上传时,同时实现文件的同步上传或是异步上传,以及文件的断点续传等功能服务。The inventor found that for different terminal devices, when using the cloud object storage service in the cloud platform, based on the two most basic upload interfaces provided by the cloud platform, there is currently a lack of a universal, effective and satisfactory comprehensive file. Upload service solutions, if you expect to be able to upload through these two upload interfaces, simultaneous file upload or asynchronous upload, as well as file breakpoint resumable upload and other functional services.
发明内容Summary of the invention
本申请实施例所要解决的技术问题是,提供一种基于云技术的文件分片上传方法、装置、设备及存储介质,能适应不同终端设备上对不同大小的文件进行文件上传,并能方便地记录文件的上传状态和验证文件的完整性。The technical problem to be solved by the embodiments of this application is to provide a method, device, equipment, and storage medium for uploading files of different sizes based on cloud technology, which can be adapted to upload files of different sizes on different terminal devices, and can conveniently Record the upload status of the file and verify the integrity of the file.
为了解决上述技术问题,本申请实施例提供一种基于云技术的文件分片上传方法,采用了如下所述的技术方案:In order to solve the above technical problems, an embodiment of the present application provides a method for uploading file fragments based on cloud technology, which adopts the following technical solutions:
一种基于云技术的文件分片上传方法,包括:A file segment upload method based on cloud technology, including:
接收目标用户配置的上传指令,所述上传指令中指明有待上传文件对象;Receiving an upload instruction configured by a target user, where the upload instruction specifies a file object to be uploaded;
根据所述上传指令中待上传文件对象读取指明的待上传文件,并获取每个待上传文件的文件大小及文件标识,比较每个待上传文件的所述文件大小与预设的分片阈值的大小,将文件大小小于或等于所述分片阈值的待上传文件记为第一类型文件,将文件大小大于所述分片阈值的待上传文件记为第二类型文件;Read the specified file to be uploaded according to the file object to be uploaded in the upload instruction, and obtain the file size and file identifier of each file to be uploaded, and compare the file size of each file to be uploaded with the preset fragmentation threshold The file to be uploaded whose file size is less than or equal to the fragmentation threshold is recorded as the first type of file, and the file to be uploaded with the file size greater than the fragmentation threshold is recorded as the second type of file;
基于所述分片阈值对所述第二类型文件进行分片处理,以将所有第二类型文件分别拆分为携带文件标识和分片标识的若干个文件分片;Performing fragmentation processing on the second-type file based on the fragmentation threshold, so as to split all the second-type files into a plurality of file fragments carrying a file identifier and a fragment identifier;
将所述第一类型文件和所述第二类型文件的若干个文件分片依次读入内存,并执行哈希处理,以获取每个待上传文件的第一散列值;Reading several file fragments of the first type file and the second type file into memory in sequence, and performing hash processing to obtain the first hash value of each file to be uploaded;
调用基础分片接口和终片接口,通过所述终片接口直接上传所述内存中的第一类型文件,通过所述基础分片接口和所述终片接口上传所述内存中的第二类型文件的若干个文件分片;Call the basic fragment interface and the final fragment interface, directly upload the first type file in the memory through the final fragment interface, and upload the second type file in the memory through the basic fragment interface and the final fragment interface Several file fragments of the file;
基于文件标识和/或分片标识标记待上传文件的上传状态,当第二类型文件对应的文件分片完成上传后,将具有相同文件标识的文件分片进行拼合,以还原为第二类型文件,并对已完成上传的每个第一类型文件和第二类型文件分别进行哈希处理以获取对应的第二散列值,通过所述第一散列值和所述第二散列值进行校验,以确认已完成上传的待上传文件的完整性。Mark the upload status of the file to be uploaded based on the file identifier and/or segment identifier. After the file segment corresponding to the second type of file is uploaded, the file segments with the same file identifier are combined to restore the second type of file , And perform hash processing on each file of the first type and the file of the second type that have been uploaded to obtain the corresponding second hash value. Check to confirm the integrity of the uploaded files that have been uploaded.
为了解决上述技术问题,本申请实施例还提供一种基于云技术的文件分片上传装置,采 用了如下所述的技术方案:In order to solve the above technical problems, an embodiment of the present application also provides a file segment uploading device based on cloud technology, which adopts the following technical solutions:
一种基于云技术的文件分片上传装置,包括:A file segment upload device based on cloud technology, including:
指令接收模块,用于接收目标用户配置的上传指令,所述上传指令中指明有待上传文件对象;An instruction receiving module, configured to receive an upload instruction configured by a target user, where the upload instruction specifies a file object to be uploaded;
文件类型划分模块,用于根据所述上传指令中待上传文件对象读取指明的待上传文件,并获取每个待上传文件的文件大小及文件标识,比较每个待上传文件的所述文件大小与预设的分片阈值的大小,将文件大小小于或等于所述分片阈值的待上传文件记为第一类型文件,将文件大小大于所述分片阈值的待上传文件记为第二类型文件;The file type division module is used to read the specified file to be uploaded according to the file object to be uploaded in the upload instruction, and obtain the file size and file identifier of each file to be uploaded, and compare the file size of each file to be uploaded With the size of the preset fragmentation threshold, the file to be uploaded whose file size is less than or equal to the fragmentation threshold is recorded as the first type of file, and the file to be uploaded with the file size greater than the fragmentation threshold is recorded as the second type. document;
文件分片模块,用于基于所述分片阈值对所述第二类型文件进行分片处理,以将所有第二类型文件分别拆分为携带文件标识和分片标识的若干个文件分片;A file fragmentation module, configured to perform fragmentation processing on the second-type file based on the fragmentation threshold, so as to split all the second-type files into several file fragments carrying a file identifier and a fragmentation identifier;
哈希处理模块,用于将所述第一类型文件和所述第二类型文件的若干个文件分片依次读入内存,并执行哈希处理,以获取每个待上传文件的第一散列值;A hash processing module, which is used to sequentially read several file fragments of the first type file and the second type file into the memory, and perform hash processing to obtain the first hash of each file to be uploaded value;
文件上传模块,用于调用基础分片接口和终片接口,通过所述终片接口直接上传所述内存中的第一类型文件,通过所述基础分片接口和所述终片接口上传所述内存中的第二类型文件的若干个文件分片;The file upload module is used to call the basic fragmentation interface and the final fragment interface, directly upload the first type file in the memory through the final fragment interface, and upload the basic fragment interface and the final fragment interface. Several file fragments of the second type file in the memory;
文件校验模块,用于基于文件标识和/或分片标识标记待上传文件的上传状态,当第二类型文件对应的文件分片完成上传后,将具有相同文件标识的文件分片进行拼合,以还原为第二类型文件,并对已完成上传的每个第一类型文件和第二类型文件分别进行哈希处理以获取对应的第二散列值,通过所述第一散列值和所述第二散列值进行校验,以确认已完成上传的待上传文件的完整性。The file verification module is used to mark the upload status of the file to be uploaded based on the file identifier and/or the segment identifier. When the file segment corresponding to the second type of file is uploaded, the file segments with the same file identifier are merged together, In order to restore to the second type file, and perform hash processing on each of the first type file and the second type file that have been uploaded to obtain the corresponding second hash value, through the first hash value and all the files The second hash value is checked to confirm the integrity of the uploaded file to be uploaded.
为了解决上述技术问题,本申请实施例还提供一种计算机设备,采用了如下所述的技术方案:In order to solve the above technical problems, the embodiments of the present application also provide a computer device, which adopts the following technical solutions:
一种计算机设备,包括存储器、处理器,以及存储在所述存储器中并可在所述处理器上运行的计算机可读指令,所述处理器执行所述计算机可读指令时实现如下基于云技术的文件分片上传方法的步骤:A computer device includes a memory, a processor, and computer-readable instructions stored in the memory and capable of running on the processor. When the processor executes the computer-readable instructions, the following is achieved based on cloud technology The steps of the file upload method:
接收目标用户配置的上传指令,所述上传指令中指明有待上传文件对象;Receiving an upload instruction configured by a target user, where the upload instruction specifies a file object to be uploaded;
根据所述上传指令中待上传文件对象读取指明的待上传文件,并获取每个待上传文件的文件大小及文件标识,比较每个待上传文件的所述文件大小与预设的分片阈值的大小,将文件大小小于或等于所述分片阈值的待上传文件记为第一类型文件,将文件大小大于所述分片阈值的待上传文件记为第二类型文件;Read the specified file to be uploaded according to the file object to be uploaded in the upload instruction, and obtain the file size and file identifier of each file to be uploaded, and compare the file size of each file to be uploaded with the preset fragmentation threshold The file to be uploaded whose file size is less than or equal to the fragmentation threshold is recorded as the first type of file, and the file to be uploaded with the file size greater than the fragmentation threshold is recorded as the second type of file;
基于所述分片阈值对所述第二类型文件进行分片处理,以将所有第二类型文件分别拆分为携带文件标识和分片标识的若干个文件分片;Performing fragmentation processing on the second-type file based on the fragmentation threshold, so as to split all the second-type files into a plurality of file fragments carrying a file identifier and a fragment identifier;
将所述第一类型文件和所述第二类型文件的若干个文件分片依次读入内存,并执行哈希处理,以获取每个待上传文件的第一散列值;Reading several file fragments of the first type file and the second type file into memory in sequence, and performing hash processing to obtain the first hash value of each file to be uploaded;
调用基础分片接口和终片接口,通过所述终片接口直接上传所述内存中的第一类型文件,通过所述基础分片接口和所述终片接口上传所述内存中的第二类型文件的若干个文件分片;Call the basic fragment interface and the final fragment interface, directly upload the first type file in the memory through the final fragment interface, and upload the second type file in the memory through the basic fragment interface and the final fragment interface Several file fragments of the file;
基于文件标识和/或分片标识标记待上传文件的上传状态,当第二类型文件对应的文件分片完成上传后,将具有相同文件标识的文件分片进行拼合,以还原为第二类型文件,并对已完成上传的每个第一类型文件和第二类型文件分别进行哈希处理以获取对应的第二散列值,通过所述第一散列值和所述第二散列值进行校验,以确认已完成上传的待上传文件的完整性。Mark the upload status of the file to be uploaded based on the file identifier and/or segment identifier. After the file segment corresponding to the second type of file is uploaded, the file segments with the same file identifier are combined to restore the second type of file , And perform hash processing on each file of the first type and the file of the second type that have been uploaded to obtain the corresponding second hash value. Check to confirm the integrity of the uploaded files that have been uploaded.
为了解决上述技术问题,本申请实施例还提供一个或多个存储有计算机可读指令的计算机可读存储介质,采用了如下所述的技术方案:In order to solve the above technical problems, the embodiments of the present application also provide one or more computer-readable storage media storing computer-readable instructions, and the following technical solutions are adopted:
一个或多个存储有计算机可读指令的计算机可读存储介质,其中,所述计算机可读指令被处理器执行时实现如下基于云技术的文件分片上传方法的步骤:One or more computer-readable storage media storing computer-readable instructions, where the computer-readable instructions implement the following steps of the cloud-based file segment upload method when the computer-readable instructions are executed by a processor:
接收目标用户配置的上传指令,所述上传指令中指明有待上传文件对象;Receiving an upload instruction configured by a target user, where the upload instruction specifies a file object to be uploaded;
根据所述上传指令中待上传文件对象读取指明的待上传文件,并获取每个待上传文件的 文件大小及文件标识,比较每个待上传文件的所述文件大小与预设的分片阈值的大小,将文件大小小于或等于所述分片阈值的待上传文件记为第一类型文件,将文件大小大于所述分片阈值的待上传文件记为第二类型文件;Read the specified file to be uploaded according to the file object to be uploaded in the upload instruction, and obtain the file size and file identifier of each file to be uploaded, and compare the file size of each file to be uploaded with the preset fragmentation threshold The file to be uploaded whose file size is less than or equal to the fragmentation threshold is recorded as the first type of file, and the file to be uploaded with the file size greater than the fragmentation threshold is recorded as the second type of file;
基于所述分片阈值对所述第二类型文件进行分片处理,以将所有第二类型文件分别拆分为携带文件标识和分片标识的若干个文件分片;Performing fragmentation processing on the second-type file based on the fragmentation threshold, so as to split all the second-type files into a plurality of file fragments carrying a file identifier and a fragment identifier;
将所述第一类型文件和所述第二类型文件的若干个文件分片依次读入内存,并执行哈希处理,以获取每个待上传文件的第一散列值;Reading several file fragments of the first type file and the second type file into memory in sequence, and performing hash processing to obtain the first hash value of each file to be uploaded;
调用基础分片接口和终片接口,通过所述终片接口直接上传所述内存中的第一类型文件,通过所述基础分片接口和所述终片接口上传所述内存中的第二类型文件的若干个文件分片;Call the basic fragment interface and the final fragment interface, directly upload the first type file in the memory through the final fragment interface, and upload the second type file in the memory through the basic fragment interface and the final fragment interface Several file fragments of the file;
基于文件标识和/或分片标识标记待上传文件的上传状态,当第二类型文件对应的文件分片完成上传后,将具有相同文件标识的文件分片进行拼合,以还原为第二类型文件,并对已完成上传的每个第一类型文件和第二类型文件分别进行哈希处理以获取对应的第二散列值,通过所述第一散列值和所述第二散列值进行校验,以确认已完成上传的待上传文件的完整性。Mark the upload status of the file to be uploaded based on the file identifier and/or segment identifier. After the file segment corresponding to the second type of file is uploaded, the file segments with the same file identifier are combined to restore the second type of file , And perform hash processing on each file of the first type and the file of the second type that have been uploaded to obtain the corresponding second hash value. Check to confirm the integrity of the uploaded files that have been uploaded.
与现有技术相比,本申请实施例主要有以下有益效果:Compared with the prior art, the embodiments of the present application mainly have the following beneficial effects:
本申请实施例公开了一种基于云技术的文件分片上传方法、装置、设备及存储介质,本申请实施例所述的基于云技术的文件分片上传方法,接收上传指令后;读取待上传文件,将文件大小小于或等于分片阈值的待上传文件记为第一类型文件,将文件大小大于分片阈值的待上传文件记为第二类型文件;对第二类型文件进行分片处理,将第二类型文件拆分为若干个文件分片;并执行哈希处理,以获取每个待上传文件的第一散列值;调用基础分片接口和终片接口后,通过终片接口直接上传第一类型文件,通过基础分片接口和终片接口上传第二类型文件的若干个文件分片;最终通过散列值确认已完成上传的待上传文件的完整性。所述方法能避免属于不同***的终端设备的上传限制,使不同的终端设备实现对不同大小的多文件分片的同步或异步上传,以及文件断点续传的功能,并能较方便地记录文件的上传状态和验证文件的完整。The embodiment of the application discloses a method, device, device, and storage medium for uploading file fragments based on cloud technology. The method for uploading file fragments based on cloud technology according to the embodiment of the application, after receiving the upload instruction; Upload files, record the files to be uploaded whose file size is less than or equal to the fragmentation threshold as the first type of files, and record the files to be uploaded with the file size greater than the fragmentation threshold as the second type of files; perform fragment processing on the second type of files , Split the second type file into several file fragments; and perform hash processing to obtain the first hash value of each file to be uploaded; after calling the basic fragment interface and the final fragment interface, pass the final fragment interface Upload the first type of file directly, upload several file segments of the second type of file through the basic segmentation interface and the final segment interface; finally confirm the integrity of the uploaded file to be uploaded through the hash value. The method can avoid upload restrictions of terminal devices belonging to different systems, and enable different terminal devices to realize the synchronous or asynchronous upload of multiple file fragments of different sizes, as well as the function of resuming file transfers, and can record more conveniently The upload status of the file and the integrity of the verification file.
附图说明Description of the drawings
为了更清楚地说明本申请实施例中的技术方案,下面将对实施例中所需要使用的附图作简单地介绍,显而易见地,下面描述中的附图仅仅是本申请的一些实施例,对于本领域普通技术人员来讲,在不付出创造性劳动的前提下,还可以根据这些附图获得其他的附图。In order to more clearly describe the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative work, other drawings can be obtained from these drawings.
图1为本申请实施例可以应用于其中的示例性***架构图;FIG. 1 is an exemplary system architecture diagram to which the embodiments of this application can be applied;
图2为本申请实施例中所述基于云技术的文件分片上传方法的一个实施例的流程图;2 is a flowchart of an embodiment of the method for uploading file fragments based on cloud technology in an embodiment of the application;
图3为本申请实施例中所述基于云技术的文件分片上传装置的一个实施例的结构示意图;3 is a schematic structural diagram of an embodiment of the device for uploading file fragments based on cloud technology in an embodiment of the application;
图4为本申请实施例中计算机设备的一个实施例的结构示意图。Fig. 4 is a schematic structural diagram of an embodiment of a computer device in an embodiment of the application.
具体实施方式Detailed ways
除非另有定义,本文所使用的所有的技术和科学术语与属于本申请的技术领域的技术人员通常理解的含义相同。本文中在本申请的说明书中所使用的术语只是为了描述具体的实施例的目的,不是旨在于限制本申请。Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field of this application. The terms used in the specification of the application herein are only for the purpose of describing specific embodiments, and are not intended to limit the application.
需要说明的是,本申请的说明书和权利要求书及上述附图中的术语“包括”、“包含”和“具有”以及它们任何变形,意图在于覆盖不排他的包含。例如包含了一系列步骤或单元的过程、方法、***、产品或设备没有限定于已列出的步骤或单元,而是可选地还包括没有列出的步骤或单元,或可选地还包括对于这些过程、方法、产品或设备固有的其他步骤或单元。在本申请的权利要求书、说明书以及说明书附图中的术语,诸如“第一”和“第二”等之类的关系术语仅仅用来将一个实体/操作/对象与另一个实体/操作/对象区分开来,而不一定要求或者暗示这些实体/操作/对象之间存在任何这种实际的关系或者顺序。It should be noted that the terms "including", "including" and "having" in the specification and claims of the present application and the above-mentioned drawings and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes unlisted steps or units, or optionally also includes Other steps or units inherent to these processes, methods, products or equipment. The terms in the claims, specification and drawings of this application, such as "first" and "second" and other relational terms are only used to associate one entity/operation/object with another entity/operation/ The distinction between objects does not necessarily require or imply any such actual relationship or order between these entities/operations/objects.
在本文中提及“实施例”意味着,结合实施例描述的特定特征、结构或特性可以包含在 本申请的至少一个实施例中。在说明书中的各个位置出现该短语并不一定均是指相同的实施例,也不是与其他实施例互斥的独立的或备选的实施例。本领域技术人员显式地和隐式地理解的是,本文所描述的实施例可以与其他实施例相结合。Reference to "embodiments" herein means that a specific feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art clearly and implicitly understand that the embodiments described herein can be combined with other embodiments.
为了使本技术领域的人员更好地理解本申请的方案,下面将结合本申请实施例中的相关附图,对本申请实施例中的技术方案进行清楚、完整地描述。In order to enable those skilled in the art to better understand the solutions of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely in conjunction with the relevant drawings in the embodiments of the present application.
如图1所示,***架构100可以包括终端设备101、102、103,网络104和服务器105。网络104用以在终端设备101、102、103和服务器105之间提供通信链路的介质。网络104可以包括各种连接类型,例如有线、无线通信链路或者光纤电缆等等。As shown in FIG. 1, the system architecture 100 may include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, and so on.
用户可以使用终端设备101、102、103通过网络104与服务器105交互,以接收或发送消息等。终端设备101、102、103上可以安装有各种通讯客户端应用,例如网页浏览器应用、购物类应用、搜索类应用、即时通信工具、邮箱客户端、社交平台软件等。The user can use the terminal devices 101, 102, and 103 to interact with the server 105 through the network 104 to receive or send messages and so on. Various communication client applications, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, and social platform software, can be installed on the terminal devices 101, 102, and 103.
终端设备101、102、103可以是具有显示屏并且支持网页浏览的各种电子设备,包括但不限于智能手机、平板电脑、电子书阅读器、MP3播放器(Moving Picture Experts GroupAudio Layer III,动态影像专家压缩标准音频层面3)、MP4(Moving Picture Experts GroupAudio Layer IV,动态影像专家压缩标准音频层面4)播放器、膝上型便携计算机和台式计算机等等。The terminal devices 101, 102, 103 may be various electronic devices with display screens and support for web browsing, including but not limited to smart phones, tablets, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III, dynamic images) Experts compress standard audio layer 3), MP4 (Moving Picture Experts Group Audio Layer IV, dynamic image experts compress standard audio layer 4) players, laptop portable computers and desktop computers, etc.
服务器105可以是提供各种服务的服务器,例如对终端设备101、102、103上显示的页面提供支持的后台服务器。The server 105 may be a server that provides various services, for example, a background server that provides support for pages displayed on the terminal devices 101, 102, and 103.
需要说明的是,本申请实施例所提供的基于云技术的文件分片上传方法一般由服务器/终端设备执行,相应地,基于云技术的文件分片上传装置一般设置于服务器/终端设备中。It should be noted that the method for uploading file fragments based on cloud technology provided by the embodiments of the present application is generally executed by a server/terminal device. Accordingly, the device for uploading file fragments based on cloud technology is generally set in the server/terminal device.
应该理解,图1中的终端设备、网络和服务器的数目仅仅是示意性的。根据实现需要,可以具有任意数目的终端设备、网络和服务器。It should be understood that the numbers of terminal devices, networks, and servers in FIG. 1 are merely illustrative. There can be any number of terminal devices, networks, and servers according to implementation needs.
继续参考图2,示出了本申请实施例中所述基于云技术的文件分片上传方法的一个实施例的流程图。所述基于云技术的文件分片上传方法,包括以下步骤:Continuing to refer to FIG. 2, a flowchart of an embodiment of the method for uploading file fragments based on cloud technology in an embodiment of the present application is shown. The method for uploading file fragments based on cloud technology includes the following steps:
步骤201:接收目标用户配置的上传指令,所述上传指令中指明有待上传文件对象。Step 201: Receive an upload instruction configured by a target user, where the upload instruction specifies a file object to be uploaded.
其中,所述上传指令中描述的待上传文件对象指本次上传时目标用户所期望上传的文件和文件对应的路径。其中,待上传文件的数量可以为单个也可以为多个,数量为多个时,目标用户可以在上传指令中配置待上传文件的文件列表及每个文件相应的路径。Wherein, the file object to be uploaded described in the upload instruction refers to the file that the target user expects to upload during this upload and the path corresponding to the file. Among them, the number of files to be uploaded can be single or multiple. When the number is multiple, the target user can configure the file list of the files to be uploaded and the corresponding path of each file in the upload instruction.
进一步的,目标用户在配置上传指令时,还可以在其中定制其部分上传需求,如指定期望采用同步或是异步的方式进行上传等。Furthermore, when the target user configures the upload instruction, he can also customize some of his upload requirements, such as specifying that he wants to upload in a synchronous or asynchronous manner.
在本申请实施例中,所述基于云技术的文件分片上传方法运行于其上的电子设备(例如图1所示的服务器/终端设备)可以通过有线连接方式或者无线连接方式接收目标用户发出的上传指令。需要指出的是,上述无线连接方式可以包括但不限于3G/4G连接、WiFi连接、蓝牙连接、WiMAX连接、Zigbee连接、UWB(ultrawideband)连接、以及其他现在已知或将来开发的无线连接方式。In the embodiment of the present application, the electronic device (for example, the server/terminal device shown in FIG. 1) on which the cloud-based file segment upload method runs can receive the target user's sending through a wired connection or a wireless connection. Upload instructions. It should be pointed out that the above-mentioned wireless connection methods may include, but are not limited to, 3G/4G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultrawideband) connection, and other currently known or future wireless connection methods.
步骤202:根据所述上传指令中待上传文件对象读取指明的待上传文件,并获取每个待上传文件的文件大小及文件标识,比较每个待上传文件的所述文件大小与预设的分片阈值的大小,将文件大小小于或等于所述分片阈值的待上传文件记为第一类型文件,将文件大小大于所述分片阈值的待上传文件记为第二类型文件。Step 202: Read the specified file to be uploaded according to the file object to be uploaded in the upload instruction, and obtain the file size and file identifier of each file to be uploaded, and compare the file size of each file to be uploaded with the preset file size. For the size of the fragmentation threshold, a file to be uploaded with a file size smaller than or equal to the fragmentation threshold is recorded as a first type of file, and a file to be uploaded with a file size greater than the fragmentation threshold is recorded as a file of the second type.
服务端响应于上传指令根据上传指令中描述的文件路径进行访问,并从访问的文件路径中获取对应的待上传文件,同时每个待上传文件的文件大小和文件标识均能由此获知。后续步骤中对待上传文件进行上传时,需要将其读入内存后再进行上传,而上传之前一般还需对待上传文件进行分片处理。The server responds to the upload instruction to access according to the file path described in the upload instruction, and obtains the corresponding file to be uploaded from the accessed file path, and the file size and file identifier of each file to be uploaded can be obtained from this. When uploading the file to be uploaded in the subsequent steps, it needs to be read into the memory before uploading, and the file to be uploaded generally needs to be divided into pieces before uploading.
分片阈值是是否对文件进行分片处理的一个判断标准,同时其也是对文件进行分片处理时的分片依据。进行分片处理的待上传文件为文件大小大于分片阈值的文件。本申请实施例中,为了便于区分,将文件大小大于分片阈值的待上传文件记为第一类型文件,将文件大小大于分片阈值的待上传文件记为第二类型文件。而分片阈值的大小可以预先设定,如根据不 同场景的需要设为100K或500K或1M等大小。The fragmentation threshold is a criterion for judging whether to perform fragmentation processing on a file, and it is also a fragmentation basis when performing fragmentation processing on a file. The file to be uploaded for fragmentation processing is a file whose size is greater than the fragmentation threshold. In the embodiments of the present application, to facilitate the distinction, files to be uploaded with a file size greater than the fragmentation threshold are recorded as the first type of file, and files to be uploaded with the file size greater than the fragmentation threshold are recorded as the second type of file. The size of the fragmentation threshold can be preset, such as 100K, 500K, or 1M according to the needs of different scenarios.
在本申请的一些实施例中,步骤202中所述比较每个待上传文件的所述文件大小与预设的分片阈值的大小的步骤之前,所述基于云技术的文件分片上传方法还包括:In some embodiments of the present application, before the step of comparing the file size of each file to be uploaded with the size of the preset fragmentation threshold in step 202, the cloud technology-based file fragment upload method also include:
检测当前网络的网络连接状况,获取当前网络的网络状态等级;Detect the network connection status of the current network, and obtain the network status level of the current network;
匹配与所述网络状态等级对应预设的分片阈值,并设为用于与待上传文件的文件大小进行比较的分片阈值。Match a preset fragmentation threshold corresponding to the network state level, and set it as a fragmentation threshold for comparison with the file size of the file to be uploaded.
由于文件上传时网络连接状况会对上传的状态影响极大,因此对于不同的网络连接状况,应对待上传的文件采用不同的分片处理方式。应当理解的是,网络连接状况越好,可将对文件进行分片处理拆分出的文件分片的大小设置为越大。Since the network connection status during file upload will greatly affect the upload status, different network connection statuses should be treated with different fragmentation processing methods for the uploaded files. It should be understood that the better the network connection is, the larger the size of the file fragments that are split by the file fragmentation processing can be set to be larger.
在本申请实施例的一种具体的实施方式中,服务端会预先为不同的网络连接状况场景配置一张映射表,映射表中将不同的网络连接状况划分为不同的网络状态等级,每个网络状态等级对应一个分片阈值,检测到当前网络的网络连接状况后,便根据其网络连接状况在映射表中搜索到与其对应的网络状态等级,并在映射表中进一步匹配到与网络状态等级对应的分片阈值,将其视为用于在当前基于云技术的文件分片上传方法的步骤中应用的分片阈值。In a specific implementation manner of the embodiments of the present application, the server will pre-configure a mapping table for different network connection status scenarios. The mapping table divides different network connection statuses into different network status levels. The network status level corresponds to a fragmentation threshold. After the current network connection status is detected, the corresponding network status level is searched in the mapping table according to the network connection status, and the mapping table is further matched to the network status level The corresponding fragmentation threshold is regarded as the fragmentation threshold used in the steps of the current cloud-based file fragmentation upload method.
步骤203:基于所述分片阈值对所述第二类型文件进行分片处理,以将所有第二类型文件分别拆分为携带文件标识和分片标识的若干个文件分片。Step 203: Perform fragmentation processing on the second-type file based on the fragmentation threshold, so as to split all the second-type files into a plurality of file fragments carrying a file identifier and a fragment identifier.
对文件进行分片处理,会将文件拆分为若干个文件分片,若干个分片的大小之和与原文件的大小相等。而这些文件分片中,除最后一个文件分片外,其他文件分片的大小均相同,且与分片阈值的大小相等,最后一个文件分片的大小则小于或等于分片阈值的大小。Fragmentation processing of the file will split the file into several file fragments, and the sum of the size of the several fragments is equal to the size of the original file. Among these file fragments, except for the last file fragment, other file fragments have the same size and the same size as the fragment threshold, and the size of the last file fragment is less than or equal to the fragment threshold.
进一步理解分片处理的步骤:假设原始文件的大小为1200K,设定的分片阈值为500K,则对该文件进行分片处理后,会将该文件拆分为三个文件分片,前两个的大小均为500K,分别记录原始文件的0-500K的数据和500K-1000K的数据,第三片即最后一个文件分片的大小为200K,记录原始文件中剩下的200K数据。Further understand the steps of fragmentation processing: Assuming that the size of the original file is 1200K and the set fragmentation threshold is 500K, after fragmentation processing is performed on the file, the file will be split into three file fragments, the first two The size of each is 500K, and the data of 0-500K and the data of 500K-1000K of the original file are respectively recorded. The size of the third and last file segment is 200K, and the remaining 200K data in the original file is recorded.
本申请实施例中,为了方便记录文件分片的上传状态,对第二类型文件进行分片时,拆分出的若干个文件分片,每个文件分片均会携带与其对应的第二类型文件的文件标识和该文件分片的分片标识。其中,文件标识用于区分文件分片所属的待上传文件,分片标识用于区分不同的文件分片。In the embodiment of this application, in order to conveniently record the upload status of file fragments, when the second type of file is fragmented, a number of file fragments are split, and each file fragment will carry its corresponding second type. The file ID of the file and the fragment ID of the file fragment. Among them, the file identifier is used to distinguish the file to be uploaded to which the file fragment belongs, and the fragment identifier is used to distinguish different file fragments.
在本申请的一些实施例中,若所述上传指令中对不同的待上传文件指定有对应的不同的分片阈值,步骤202中所述比较每个待上传文件的所述文件大小与预设的分片阈值的大小,将文件大小小于或等于所述分片阈值的待上传文件记为第一类型文件,将文件大小大于所述分片阈值的待上传文件记为第二类型文件的步骤包括:In some embodiments of the present application, if the upload instruction specifies different fragmentation thresholds for different files to be uploaded, the file size of each file to be uploaded is compared with the preset file size in step 202. The step of recording a file to be uploaded with a file size less than or equal to the fragmentation threshold as a first type of file, and recording a file to be uploaded with a file size greater than the fragmentation threshold as a file of the second type include:
识别待上传文件的文件标识,从所述上传指令中获取与不同所属文件标识对应的若干个分片阈值;Identify the file identifier of the file to be uploaded, and obtain several fragmentation thresholds corresponding to different file identifiers from the upload instruction;
将每个待上传文件的文件大小与对应的分片阈值的大小分别进行比较,将文件大小小于或等于与其对应的分片阈值的待上传文件记为第一类型文件,将文件大小大于与其对应的分片阈值的待上传文件记为第二类型文件;The file size of each file to be uploaded is compared with the size of the corresponding fragmentation threshold. The file to be uploaded whose file size is less than or equal to the corresponding fragmentation threshold is recorded as the first type of file, and the file size is greater than the corresponding fragmentation threshold. The file to be uploaded with the fragmentation threshold of is recorded as the second type of file;
步骤203中所述基于所述分片阈值对所述第二类型文件进行分片处理的步骤包括:基于所述上传指令中指定的若干个分片阈值,分别为对应的所述第二类型文件进行分片处理。The step of performing fragmentation processing on the second-type file based on the fragmentation threshold in step 203 includes: based on the plurality of fragmentation thresholds specified in the upload instruction, each of the second-type files is corresponding to the fragmentation threshold. Perform fragmentation processing.
有时由于待上传文件之间的大小差别较大,为了更合理地进行上传,目标用户可以为不同的待上传文件在上传指令中指定不同的分片阈值。以在划分第一类型文件和第二类型文件,和对第二类型文件进行分片处理时,基于上传指令中为每个待上传文件分配的分片阈值进行处理。Sometimes due to the large size difference between the files to be uploaded, in order to upload more reasonably, the target user can specify different fragmentation thresholds for different files to be uploaded in the upload instruction. When dividing the first type file and the second type file, and performing fragmentation processing on the second type file, processing is performed based on the fragmentation threshold assigned to each file to be uploaded in the upload instruction.
步骤204:将所述第一类型文件和所述第二类型文件的若干个文件分片依次读入内存,并执行哈希处理,以获取每个待上传文件的第一散列值。Step 204: Read several file fragments of the first type file and the second type file into the memory in sequence, and perform hash processing to obtain the first hash value of each file to be uploaded.
待上传文件在上传前会读入服务端的内存,以进行哈希处理和上传。The file to be uploaded will be read into the memory of the server before uploading for hash processing and upload.
本申请中,采用MD5(Message-Digest Algorithm 5,消息-摘要算法)对待上传文件进行 哈希处理。MD5是目前广泛使用的一种密码散列函数,其常常被用来验证网络文件传输的完整性,防止文件被人篡改,相比其他哈希算法,MD5的反破解能力更强。In this application, MD5 (Message-Digest Algorithm 5, Message-Digest Algorithm) is used to hash the file to be uploaded. MD5 is a password hash function widely used at present. It is often used to verify the integrity of network file transmission and prevent files from being tampered with. Compared with other hash algorithms, MD5 has a stronger anti-cracking ability.
在本申请的一些实施例中,所述步骤204包括:In some embodiments of the present application, the step 204 includes:
识别待上传文件的类型;Identify the type of file to be uploaded;
若为第一类型文件,便将第一类型文件直接读入内存执行哈希处理,以获取每个第一类型文件的第一散列值;If it is a file of the first type, the file of the first type is directly read into the memory to perform hash processing to obtain the first hash value of each file of the first type;
若为第二类型文件,则依次读入第二类型文件的若干个文件分片,在每读入一个文件分片时便对该文件分片执行哈希处理,以获取该文件分片对应的分片散列值,并每当获取到一个第二类型文件的最后一个文件分片对应的分片散列值后,对该第二类型文件的所有文件分片对应的分片散列值进行拼合,以获取到每个第二类型文件的第一散列值。If it is a file of the second type, read in several file segments of the file of the second type in turn, and perform hash processing on the file segment every time a file segment is read in to obtain the corresponding file segment The fragment hash value, and whenever the fragment hash value corresponding to the last file fragment of a second type file is obtained, the fragment hash value corresponding to all file fragments of the second type file is performed Combine to obtain the first hash value of each file of the second type.
识别待上传文件的类型,可以根据其文件后缀或携带的隐性参数中能更确切地表示文件类型的内容进行识别。由于内存大小的限制,对于第二类型文件读入时,会按其拆分后的文件分片进行依次读取,并分别计算出分片散列值后,将第二类型文件的若干个文件分片的若干个分片散列值通过累加进行拼合,以得到整个第二类型文件的第一散列值。To identify the type of the file to be uploaded, it can be identified based on the file suffix or the content of the hidden parameter that can more accurately represent the file type. Due to the limitation of the memory size, when reading the second type file, it will read the file fragments after its split in turn, and after calculating the hash value of the fragments, the several files of the second type file A number of fragmented hash values of the fragments are combined through accumulation to obtain the first hash value of the entire second-type file.
进一步理解,MD5支持流式的散列值累加计算,如一个具有多个文件分片的第二类型文件,每读取其一个文件分片,便对该文件分片进行哈希处理获取分片散列值,得到一个128位长度的字符串,读取完该第二类型文件的所有文件分片后,相应能得到与文件分片数量对应的分片散列值,最终将所有分片散列值累加,便能得出整个第二类型文件对应的散列值。It is further understood that MD5 supports streaming hash value accumulation calculations, such as a second type file with multiple file fragments, each time a file fragment is read, the file fragment is hashed to obtain the fragment Hash value to obtain a 128-bit string. After reading all file fragments of the second type of file, the corresponding fragment hash value corresponding to the number of file fragments can be obtained, and all the fragments are finally hashed The column value is accumulated, and the hash value corresponding to the entire second-type file can be obtained.
步骤205:调用基础分片接口和终片接口,通过所述终片接口直接上传所述内存中的第一类型文件,通过所述基础分片接口和所述终片接口上传所述内存中的第二类型文件的若干个文件分片。Step 205: Invoke the basic slice interface and the final slice interface, directly upload the first type file in the memory through the final slice interface, and upload the file in the memory through the basic slice interface and the final slice interface. Several file fragments of the second type file.
平安云平台中的云对象存储服务提供的两种上传接口为基础分片接口和终片接口。其中,终片接口用于上传一个待上传文件的最后一部分文件数据,能方便***记录文件上传的状态和进度。对于第一类型文件,无需通过分片处理拆分为多个文件分片,因此其本身可被视为第一个同时也是最后一个文件分片,上传时需通过终片接口进行上传。The two upload interfaces provided by the cloud object storage service in the Ping An cloud platform are the basic slice interface and the final slice interface. Among them, the final film interface is used to upload the last part of the file data of a file to be uploaded, which can facilitate the system to record the status and progress of the file upload. For the first type of file, there is no need to split into multiple file fragments through fragmentation, so it can be regarded as the first and last file fragment, and uploading needs to be done through the final fragment interface when uploading.
本申请中,对待上传文件进行上传时,目标用户可以预先在上传指令中配置上传需求,是希望通过同步或是异步的方式进行上传,服务端对待上传文件进行上传时,若硬件支持无相关限制,便基于上传指令中配置的上传需求,选用同步或异步上传的方式。同步上传表示下一个第一类型文件或第二类型文件的文件分片必须等待上一个第一类型文件或第二类型文件的文件分片上传成功后才能进行上传;异步上传则表示无需进行排队和等待,第一类型文件或第二类型文件的文件分片依次读入内存后便能进行上传,此时的异步上传应理解为可同时进行上传,但其上传的时间节点会因读入内存的时间先后有较小的差异。In this application, when uploading a file to be uploaded, the target user can configure upload requirements in the upload instruction in advance, hoping to upload the file in a synchronous or asynchronous manner. When the server uploads the file to be uploaded, there are no relevant restrictions if the hardware supports it. , Based on the upload requirements configured in the upload instruction, select the method of synchronous or asynchronous upload. Synchronous upload means that the next file segment of the first type file or the second type file must wait for the file segment of the previous first type file or the second type file to upload successfully before uploading; asynchronous upload means that there is no need to queue up and Wait, the file fragments of the first type of file or the second type of file can be uploaded after being read into the memory in turn. At this time, asynchronous upload should be understood as uploading at the same time, but the time node of the upload will be affected by the read into the memory. There are minor differences in time sequence.
在本申请的一些实施方式中,步骤205中所述通过所述基础分片接口和所述终片接口上传所述内存中的第二类型文件的若干个文件分片的步骤包括:In some embodiments of the present application, the step of uploading several file fragments of the second type file in the memory through the basic fragment interface and the final fragment interface in step 205 includes:
基于文件标识和分片标识,识别当前待上传的文件分片是否为其所属的第二类型文件的最后一个文件分片;Based on the file identifier and the segment identifier, identify whether the file segment currently to be uploaded is the last file segment of the second type file to which it belongs;
若不是,便通过所述基础分片接口上传当前待上传的文件分片;If not, upload the file fragments currently to be uploaded through the basic fragmentation interface;
若是,则通过所述终片接口上传当前待上传的文件分片。If yes, upload the file fragments currently to be uploaded through the final fragment interface.
该实施方式中,对于当前读入内存的文件分片进行上传时,首先基于其携带的文件标识识别出其所属的第二类型文件,然后进一步基于其携带的分片标识识别其是否为该第二类型文件的最后一个文件分片。In this embodiment, when uploading a file segment currently read into the memory, the second type of file to which it belongs is first identified based on the file identifier it carries, and then it is further identified based on the segment identifier it carries whether it is the first type of file. The last file segment of the second type file.
对于识别结果表示不为最后一个文件分片的待上传的文件分片,便通过基础分片接口进行上传,对于识别结果表示为最后一个分片的待上传的文件分片,则通过终片接口进行上传。For the file segment to be uploaded that is not the last file segment as the recognition result, upload it through the basic segmentation interface, and for the file segment to be uploaded that is the last segment as the recognition result, use the final segment interface To upload.
步骤206:基于文件标识和/或分片标识标记待上传文件的上传状态,当第二类型文件对应的文件分片完成上传后,将具有相同文件标识的文件分片进行拼合,以还原为第二类型文件,并对已完成上传的每个第一类型文件和第二类型文件分别进行哈希处理以获取对应的第 二散列值,通过所述第一散列值和所述第二散列值进行校验,以确认已完成上传的待上传文件的完整性。Step 206: Mark the upload status of the file to be uploaded based on the file identifier and/or the segment identifier. After the file segment corresponding to the second type of file is uploaded, the file segments with the same file identifier are combined to restore the first file segment. Type two files, and perform hash processing on each file of the first type and the file of the second type that have been uploaded to obtain the corresponding second hash value, and pass the first hash value and the second hash value. The column value is checked to confirm the integrity of the uploaded file that has been uploaded.
本申请中,待上传文件的上传状态可以包括上传进度、上传节点、上传时间等数据。通过记录的上传状态,用户能较详细地了解文件上传时的各种状况,并方便对问题的定位。这里完整性可以理解为如第二类型文件的文件分片是否全部完成上传,第一类型文件是否全部完成上传,以及文件上传前后内容对比是否出现差异,若有差异表示文件上传时出错,具有完整性问题。In this application, the upload status of the file to be uploaded may include data such as upload progress, upload node, and upload time. Through the recorded upload status, the user can understand the various conditions of the file upload in more detail, and facilitate the location of the problem. The completeness here can be understood as whether the file fragments of the second type of file have been uploaded, whether the first type of file has been uploaded, and whether there is a difference in the content comparison before and after the file is uploaded. If there is a difference, it means that there was an error during the upload of the file and it is complete. Sexual issues.
云平台中在进行合并文件时,会将合并后的文件内容进行哈希处理,以得到第二散列值,进行文件上传时正确性和完整性的校验,便通过对第一散列值和第二散列值进行比较即可。若一致则表示待上传文件在传输过程中内容未被更改,整个文件上传成功;否则表示待上传文件已经被篡改,需对完成上传的文件进行丢弃。When merging files in the cloud platform, the content of the merged file will be hashed to obtain the second hash value, and the correctness and completeness of the file will be verified when uploading, and the first hash value will be verified Just compare with the second hash value. If they are consistent, it means that the content of the file to be uploaded has not been changed during the transmission process, and the entire file has been uploaded successfully; otherwise, it means that the file to be uploaded has been tampered with, and the uploaded file needs to be discarded.
在本申请的一些实施例中,步骤206中所述当第二类型文件对应的文件分片完成上传后,将具有相同文件标识的文件分片进行拼合,以还原为第二类型文件的步骤包括:In some embodiments of the present application, when the file fragments corresponding to the second type of file are uploaded in step 206, the step of combining the file fragments with the same file identifier to restore to the second type of file includes :
每当通过所述终片接口完成对一个文件分片的上传后,在预设的响应时间内循环检测与该文件分片的文件标识相同的其他文件分片的上传状态,直至检测到其他文件分片均完成上传时,对与该文件分片的文件标识相同的所有已完成上传的文件分片进行拼合还原;Whenever the upload of a file segment is completed through the final segment interface, the upload status of other file segments with the same file identifier as the file segment is cyclically detected within the preset response time, until other files are detected When the upload of the fragments is completed, all uploaded file fragments with the same file identifier as the file fragment are combined and restored;
而若超过所述响应时间仍未检测到其他文件分片均完成上传,则停止对所述基础分片接口和所述终片接口的调用,并基于各文件分片的上传状态生成告警消息发送至所述目标用户。If the response time is exceeded and it is not detected that all other file fragments are uploaded, the call to the basic fragment interface and the final fragment interface is stopped, and an alarm message is generated and sent based on the upload status of each file fragment. To the target user.
由于同一个第二类型文件下的最后一个文件分片可能相较其他文件分片更小,异步上传时即使最后一个文件分片在时间节点上属于更晚进行上传的,但网络状况不佳时,可能反而会率先完成上传。此时便需检测该第二类型文件下的其他文件分片是否已完成上传,若在预设的响应时间内仍有文件分片未完成上传,则表示当前网络环境较差或上传服务出现异常,网络连接时的上传状态不佳,不能进行较正常的文件上传,应中断上传操作,生成相应的告警消息发送给目标用户进行提醒,待排查出网络连接的问题后再继续进行上传。Since the last file segment under the same second type file may be smaller than other file segments, even if the last file segment is uploaded later on the time node during asynchronous upload, when the network condition is not good , It may be the first to complete the upload. At this time, it is necessary to check whether other file fragments under the second type of file have been uploaded. If there are still file fragments that have not been uploaded within the preset response time, it means that the current network environment is poor or the upload service is abnormal. , The upload status during the network connection is not good, and normal file upload cannot be performed. The upload operation should be interrupted, and the corresponding alarm message should be generated and sent to the target user as a reminder. After the network connection problem is found, the upload can be continued.
在进一步的一种具体实施方式中,所述基于各文件分片的上传状态生成告警消息发送至所述目标用户的步骤之后,所述基于云技术的文件分片上传方法还包括:In a further specific implementation, after the step of generating an alarm message based on the upload status of each file segment and sending it to the target user, the cloud technology-based file segment upload method further includes:
接收目标用户发送的续传指令;Receive the resuming instruction sent by the target user;
读取上次文件上传时各待上传文件的上传状态,以标记出用于响应于所述续传指令进行上传的文件节点;Reading the upload status of each file to be uploaded when the file was uploaded last time to mark the file node used for uploading in response to the resuming instruction;
响应于所述续传指令,从所述文件节点处对未上传成功和未上传的第一类型文件和/或第二类型文件的文件分片重新进行上传。In response to the resuming instruction, re-upload the file segments of the first type file and/or the second type file that have not been uploaded successfully and have not been uploaded from the file node.
当排除掉网络故障或服务器服务异常的问题后,目标用户希望继续对上次未完成上传的待上传文件进行上传,此时重新上传会浪费许多不必要的服务器资源和时间,因此需要采用断点续传的方式进行上传。When network failures or server service abnormalities are eliminated, the target user wants to continue uploading the files to be uploaded that were not uploaded last time. At this time, re-uploading will waste a lot of unnecessary server resources and time, so breakpoints are needed. Upload by way of resuming.
通过读取在上次上传文件中断后所记录的上传过程中各待上传文件的上传状态,以了解上传进度,方便找出本次进行续传时的文件节点,所述文件节点可以为一个第一类型文件或第二类型文件的一个文件分片,也可以为多个第一类型文件和/或第二类型文件的若干个文件分片。从所述文件节点处重新开始对未上传成功和未进行上传的第一类型文件和/或第二类型文件的文件分片进行上传,由此便能实现对文件的断点续传。By reading the upload status of each file to be uploaded during the upload process recorded after the last upload of the file was interrupted, to understand the upload progress, it is convenient to find the file node when the upload is resumed this time. The file node can be a first One file segment of a file of one type or a file of the second type may also be a plurality of file segments of files of the first type and/or files of the second type. From the file node, uploading the file fragments of the first type file and/or the second type file that have not been uploaded successfully and that have not been uploaded is restarted, so that the file can be resumably uploaded.
本申请实施例所述的基于云技术的文件分片上传方法,能避免属于不同***的终端设备的上传限制,使不同的终端设备实现对不同大小的多文件分片的同步或异步上传,以及文件断点续传的功能,并能较方便地记录文件的上传状态和验证文件的完整性。The cloud technology-based file fragment upload method described in the embodiment of the present application can avoid upload restrictions of terminal devices belonging to different systems, and enable different terminal devices to realize synchronous or asynchronous upload of multiple file fragments of different sizes, and The function of resuming the file transfer, and can more conveniently record the upload status of the file and verify the integrity of the file.
本领域普通技术人员可以理解实现上述实施例方法中的全部或部分流程,是可以通过计算机可读指令来指令相关的硬件来完成,该计算机可读指令可存储于一个或多个计算机可读存储介质中,该计算机可读指令在被执行时,可包括如上述各方法的实施例的流程。其中,前述的存储介质可为磁碟、光盘、只读存储记忆体(Read-Only Memory,ROM)等非易失性 存储介质,或随机存储记忆体(RandomAccess Memory,RAM)等。A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments and methods can be implemented by instructing relevant hardware through computer-readable instructions, which can be stored in one or more computer-readable storages. In the medium, when the computer-readable instructions are executed, they may include the processes of the above-mentioned method embodiments. Among them, the aforementioned storage medium may be a non-volatile storage medium such as a magnetic disk, an optical disc, a read-only memory (Read-Only Memory, ROM), or a random access memory (Random Access Memory, RAM), etc.
应该理解的是,虽然附图的流程图中的各个步骤按照箭头的指示依次显示,但是这些步骤并不是必然按照箭头指示的顺序依次执行。除非本文中有明确的说明,这些步骤的执行并没有严格的顺序限制,其可以以其他的顺序执行。而且,附图的流程图中的至少一部分步骤可以包括多个子步骤或者多个阶段,这些子步骤或者阶段并不必然是在同一时刻执行完成,而是可以在不同的时刻执行,其执行顺序也不必然是依次进行,而是可以与其他步骤或者其他步骤的子步骤或者阶段的至少一部分轮流或者交替地执行。It should be understood that although the various steps in the flowchart of the drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily performed in sequence in the order indicated by the arrows. Unless explicitly stated in this article, the execution of these steps is not strictly limited in order, and they can be executed in other orders. Moreover, at least part of the steps in the flowchart of the drawings may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the order of execution is also It is not necessarily performed sequentially, but may be performed alternately or alternately with at least a part of other steps or sub-steps or stages of other steps.
进一步参考图3,图3示出了为本申请实施例中所述基于云技术的文件分片上传装置的一个实施例的结构示意图。作为对上述图2所示方法的实现,本申请提供了一种基于云技术的文件分片上传装置的一个实施例,该装置实施例与图2所示的方法实施例相对应,该装置具体可以应用于各种电子设备中。With further reference to Fig. 3, Fig. 3 shows a schematic structural diagram of an embodiment of the device for uploading file fragments based on cloud technology in an embodiment of the present application. As an implementation of the method shown in FIG. 2, this application provides an embodiment of a device for uploading file fragments based on cloud technology. The device embodiment corresponds to the method embodiment shown in FIG. It can be applied to various electronic devices.
如图3所示,本实施例所述的基于云技术的文件分片上传装置包括:As shown in FIG. 3, the device for uploading file fragments based on cloud technology in this embodiment includes:
指令接收模块301;用于接收目标用户配置的上传指令,所述上传指令中指明有待上传文件对象。 Instruction receiving module 301; used to receive an upload instruction configured by a target user, and the upload instruction specifies a file object to be uploaded.
文件类型划分模块302;用于根据所述上传指令中待上传文件对象读取指明的待上传文件,并获取每个待上传文件的文件大小及文件标识,比较每个待上传文件的所述文件大小与预设的分片阈值的大小,将文件大小小于或等于所述分片阈值的待上传文件记为第一类型文件,将文件大小大于所述分片阈值的待上传文件记为第二类型文件。File type division module 302; used to read the specified file to be uploaded according to the file object to be uploaded in the upload instruction, and obtain the file size and file identifier of each file to be uploaded, and compare the files of each file to be uploaded The file size is equal to the preset fragmentation threshold, the file to be uploaded whose file size is less than or equal to the fragmentation threshold is recorded as the first type of file, and the file to be uploaded with the file size greater than the fragmentation threshold is recorded as the second type. Type file.
文件分片模块303;用于基于所述分片阈值对所述第二类型文件进行分片处理,以将所有第二类型文件分别拆分为携带文件标识和分片标识的若干个文件分片。 File fragmentation module 303; used to perform fragmentation processing on the second-type file based on the fragmentation threshold, so as to split all the second-type files into several file fragments that carry a file identifier and a fragmentation identifier. .
哈希处理模块304;用于将所述第一类型文件和所述第二类型文件的若干个文件分片依次读入内存,并执行哈希处理,以获取每个待上传文件的第一散列值。 Hash processing module 304; used to read several file fragments of the first type file and the second type file into memory in turn, and perform hash processing to obtain the first hash of each file to be uploaded Column value.
文件上传模块305;用于调用基础分片接口和终片接口,通过所述终片接口直接上传所述内存中的第一类型文件,通过所述基础分片接口和所述终片接口上传所述内存中的第二类型文件的若干个文件分片。File upload module 305; used to call the basic slice interface and the final slice interface, directly upload the first type file in the memory through the final slice interface, and upload all the files through the basic slice interface and the final slice interface Several file fragments of the second type file in the memory.
文件校验模块306;用于基于文件标识和/或分片标识标记待上传文件的上传状态,当第二类型文件对应的文件分片完成上传后,将具有相同文件标识的文件分片进行拼合,以还原为第二类型文件,并对已完成上传的每个第一类型文件和第二类型文件分别进行哈希处理以获取对应的第二散列值,通过所述第一散列值和所述第二散列值进行校验,以确认已完成上传的待上传文件的完整性。 File verification module 306; used to mark the upload status of the file to be uploaded based on the file identifier and/or the segment identifier. When the file segment corresponding to the second type of file is uploaded, the file segments with the same file identifier are merged , To restore to the second type file, and perform hash processing on each of the first type file and the second type file that have been uploaded to obtain the corresponding second hash value, through the first hash value and The second hash value is checked to confirm the integrity of the file to be uploaded that has been uploaded.
在本申请的一些实施例中,所述文件类型划分模块302还包括:阈值设置子模块。所述文件类型划分模块302比较每个待上传文件的所述文件大小与预设的分片阈值的大小之前,所述阈值设置子模块用于检测当前网络的网络连接状况,获取当前网络的网络状态等级;匹配与所述网络状态等级对应预设的分片阈值,并设为用于与待上传文件的文件大小进行比较的分片阈值。In some embodiments of the present application, the file type dividing module 302 further includes: a threshold setting submodule. Before the file type dividing module 302 compares the file size of each file to be uploaded with the size of a preset fragmentation threshold, the threshold setting submodule is used to detect the network connection status of the current network, and obtain the network of the current network. Status level; matching a preset fragmentation threshold corresponding to the network status level, and set as a fragmentation threshold for comparison with the file size of the file to be uploaded.
在本申请的一些实施例中,若所述上传指令中对不同的待上传文件指定有对应的不同的分片阈值,所述文件类型划分模块302还用于识别待上传文件的文件标识,从所述上传指令中获取与不同所属文件标识对应的若干个分片阈值;将每个待上传文件的文件大小与对应的分片阈值的大小分别进行比较,将文件大小小于或等于与其对应的分片阈值的待上传文件记为第一类型文件,将文件大小大于与其对应的分片阈值的待上传文件记为第二类型文件;In some embodiments of the present application, if different fragmentation thresholds are specified for different files to be uploaded in the upload instruction, the file type division module 302 is also used to identify the file identifier of the file to be uploaded, from In the upload instruction, several fragmentation thresholds corresponding to different file identifiers are acquired; the file size of each file to be uploaded is compared with the size of the corresponding fragmentation threshold, and the file size is less than or equal to the corresponding fragmentation threshold. A file to be uploaded with a fragment threshold is recorded as a first type of file, and a file to be uploaded with a file size greater than the corresponding fragment threshold is recorded as a file of the second type;
所述文件分片模块303还用于基于所述上传指令中指定的若干个分片阈值,分别为对应的所述第二类型文件进行分片处理。The file fragmentation module 303 is further configured to perform fragmentation processing for the corresponding second-type file based on several fragmentation thresholds specified in the upload instruction.
在本申请的一些实施例中,所述哈希处理模块304用于:识别待上传文件的类型;若为第一类型文件,便将第一类型文件直接读入内存执行哈希处理,以获取每个第一类型文件的第一散列值;若为第二类型文件,则依次读入第二类型文件的若干个文件分片,在每读入一 个文件分片时便对该文件分片执行哈希处理,以获取该文件分片对应的分片散列值,并每当获取到一个第二类型文件的最后一个文件分片对应的分片散列值后,对该第二类型文件的所有文件分片对应的分片散列值进行拼合,以获取到每个第二类型文件的第一散列值。In some embodiments of the present application, the hash processing module 304 is used to: identify the type of the file to be uploaded; if it is the first type file, directly read the first type file into the memory and perform the hash processing to obtain The first hash value of each file of the first type; if it is a file of the second type, several file segments of the file of the second type are read in sequence, and the file is segmented every time a file segment is read in Perform hash processing to obtain the fragment hash value corresponding to the file fragment, and whenever the fragment hash value corresponding to the last file fragment of a second type file is obtained, the second type file The fragment hash values corresponding to all the file fragments of are combined to obtain the first hash value of each file of the second type.
在本申请的一些实施方式中,所述文件上传模块305用于基于文件标识和分片标识,识别当前待上传的文件分片是否为其所属的第二类型文件的最后一个文件分片;若不是,便通过所述基础分片接口上传当前待上传的文件分片;若是,则通过所述终片接口上传当前待上传的文件分片。In some embodiments of the present application, the file upload module 305 is configured to identify whether the file segment currently to be uploaded is the last file segment of the second type file to which it belongs based on the file identifier and the segment identifier; if If no, upload the file fragment currently to be uploaded through the basic fragment interface; if yes, upload the file fragment currently to be uploaded through the final fragment interface.
在本申请的一些实施例中,所述文件校验模块306用于:每当通过所述终片接口完成对一个文件分片的上传后,在预设的响应时间内循环检测与该文件分片的文件标识相同的其他文件分片的上传状态,直至检测到其他文件分片均完成上传时,对与该文件分片的文件标识相同的所有已完成上传的文件分片进行拼合还原;而若超过所述响应时间仍未检测到其他文件分片均完成上传,则停止对所述基础分片接口和所述终片接口的调用,并基于各文件分片的上传状态生成告警消息发送至所述目标用户。In some embodiments of the present application, the file verification module 306 is configured to: whenever the upload of a file segment is completed through the final segment interface, it will cyclically detect the file segment within a preset response time. The upload status of other file segments with the same file ID, until it is detected that all other file segments are uploaded, all uploaded file segments with the same file ID as the file segment are combined and restored; and If it is not detected that all other file fragments are uploaded after the response time, stop calling the basic fragment interface and the final fragment interface, and generate an alarm message based on the upload status of each file fragment and send it to The target user.
在进一步的一种具体实施方式中,所述文件校验模块306还包括:续传子模块。所述文件校验模块306基于各文件分片的上传状态生成告警消息发送至所述目标用户之后,所述续传子模块用于接收目标用户发送的续传指令;读取上次文件上传时各待上传文件的上传状态,以标记出用于响应于所述续传指令进行上传的文件节点;响应于所述续传指令,从所述文件节点处对未上传成功和未上传的第一类型文件和/或第二类型文件的文件分片重新进行上传。In a further specific implementation, the file verification module 306 further includes: a resuming sub-module. After the file verification module 306 generates an alarm message based on the upload status of each file segment and sends it to the target user, the resuming sub-module is used to receive the resuming instruction sent by the target user; read the last file upload time The upload status of each file to be uploaded is used to mark the file node for uploading in response to the resuming instruction; in response to the resuming instruction, the first file node that has not been uploaded successfully and has not been uploaded is checked. File fragments of type files and/or second type files are re-uploaded.
本申请实施例所述的基于云技术的文件分片上传装置,能避免属于不同***的终端设备的上传限制,使不同的终端设备实现对不同大小的多文件分片的同步或异步上传,以及文件断点续传的功能,并能较方便地记录文件的上传状态和验证文件的完整性。The cloud-based file segment upload device described in the embodiment of the present application can avoid upload restrictions of terminal devices belonging to different systems, and enable different terminal devices to realize synchronous or asynchronous upload of multiple file segments of different sizes, and The function of resuming the file transfer, and can more conveniently record the upload status of the file and verify the integrity of the file.
为解决上述技术问题,本申请实施例还提供计算机设备。具体请参阅图4,图4为本实施例计算机设备基本结构框图。In order to solve the above technical problems, the embodiments of the present application also provide computer equipment. Please refer to FIG. 4 for details. FIG. 4 is a block diagram of the basic structure of the computer device in this embodiment.
所述计算机设备6包括通过***总线相互通信连接存储器61、处理器62、网络接口63。需要指出的是,图中仅示出了具有组件61-63的计算机设备6,但是应理解的是,并不要求实施所有示出的组件,可以替代的实施更多或者更少的组件。其中,本技术领域技术人员可以理解,这里的计算机设备是一种能够按照事先设定或存储的指令,自动进行数值计算和/或信息处理的设备,其硬件包括但不限于微处理器、专用集成电路(Application Specific Integrated Circuit,ASIC)、可编程门阵列(Field-Programmable Gate Array,FPGA)、数字处理器(Digital Signal Processor,DSP)、嵌入式设备等。The computer device 6 includes a memory 61, a processor 62, and a network interface 63 that communicate with each other through a system bus. It should be pointed out that only the computer device 6 with components 61-63 is shown in the figure, but it should be understood that it is not required to implement all the illustrated components, and more or fewer components may be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculation and/or information processing in accordance with pre-set or stored instructions. Its hardware includes, but is not limited to, a microprocessor, a dedicated Integrated Circuit (Application Specific Integrated Circuit, ASIC), Programmable Gate Array (Field-Programmable Gate Array, FPGA), Digital Processor (Digital Signal Processor, DSP), embedded equipment, etc.
所述计算机设备可以是桌上型计算机、笔记本、掌上电脑及云端服务器等计算设备。所述计算机设备可以与用户通过键盘、鼠标、遥控器、触摸板或声控设备等方式进行人机交互。The computer device may be a computing device such as a desktop computer, a notebook, a palmtop computer, and a cloud server. The computer device can interact with the user through a keyboard, a mouse, a remote control, a touch panel, or a voice control device.
所述存储器61至少包括一种类型的可读存储介质,所述可读存储介质包括闪存、硬盘、多媒体卡、卡型存储器(例如,SD或DX存储器等)、随机访问存储器(RAM)、静态随机访问存储器(SRAM)、只读存储器(ROM)、电可擦除可编程只读存储器(EEPROM)、可编程只读存储器(PROM)、磁性存储器、磁盘、光盘等。所述计算机可读存储介质可以是非易失性,也可以是易失性。在一些实施例中,所述存储器61可以是所述计算机设备6的内部存储单元,例如该计算机设备6的硬盘或内存。在另一些实施例中,所述存储器61也可以是所述计算机设备6的外部存储设备,例如该计算机设备6上配备的插接式硬盘,智能存储卡(Smart Media Card,SMC),安全数字(Secure Digital,SD)卡,闪存卡(Flash Card)等。当然,所述存储器61还可以既包括所述计算机设备6的内部存储单元也包括其外部存储设备。本实施例中,所述存储器61通常用于存储安装于所述计算机设备6的操作***和各类应用软件,例如基于云技术的文件分片上传方法的计算机可读指令等。此外,所述存储器61还可以用于暂时地存储已经输出或者将要输出的各类数据。The memory 61 includes at least one type of readable storage medium, the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static memory Random access memory (SRAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), programmable read only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. The computer-readable storage medium may be non-volatile or volatile. In some embodiments, the memory 61 may be an internal storage unit of the computer device 6, such as a hard disk or a memory of the computer device 6. In other embodiments, the memory 61 may also be an external storage device of the computer device 6, such as a plug-in hard disk equipped on the computer device 6, a smart media card (SMC), a secure digital (Secure Digital, SD) card, Flash Card, etc. Of course, the memory 61 may also include both the internal storage unit of the computer device 6 and its external storage device. In this embodiment, the memory 61 is generally used to store an operating system and various application software installed on the computer device 6, such as computer-readable instructions of a method for uploading file fragments based on cloud technology. In addition, the memory 61 can also be used to temporarily store various types of data that have been output or will be output.
所述处理器62在一些实施例中可以是中央处理器(Central Processing Unit,CPU)、控 制器、微控制器、微处理器、或其他数据处理芯片。该处理器62通常用于控制所述计算机设备6的总体操作。本实施例中,所述处理器62用于运行所述存储器61中存储的计算机可读指令或者处理数据,例如运行所述基于云技术的文件分片上传方法的计算机可读指令。The processor 62 may be a central processing unit (Central Processing Unit, CPU), controller, microcontroller, microprocessor, or other data processing chip in some embodiments. The processor 62 is generally used to control the overall operation of the computer device 6. In this embodiment, the processor 62 is configured to run computer-readable instructions or processed data stored in the memory 61, for example, run the computer-readable instructions of the cloud-based file segment upload method.
所述网络接口63可包括无线网络接口或有线网络接口,该网络接口63通常用于在所述计算机设备6与其他电子设备之间建立通信连接。The network interface 63 may include a wireless network interface or a wired network interface, and the network interface 63 is generally used to establish a communication connection between the computer device 6 and other electronic devices.
本申请还提供了另一种实施方式,即提供一种计算机可读存储介质,所述计算机可读存储介质存储有基于云技术的文件分片上传流程,所述基于云技术的文件分片上传流程可被至少一个处理器执行,以使所述至少一个处理器执行如上述的基于云技术的文件分片上传方法的步骤。This application also provides another implementation manner, that is, to provide a computer-readable storage medium that stores a cloud-based file segment upload process, and the cloud-based file segment upload process The process may be executed by at least one processor, so that the at least one processor executes the steps of the method for uploading file fragments based on cloud technology as described above.
通过以上的实施方式的描述,本领域的技术人员可以清楚地了解到上述实施例方法可借助软件加必需的通用硬件平台的方式来实现,当然也可以通过硬件,但很多情况下前者是更佳的实施方式。基于这样的理解,本申请的技术方案本质上或者说对现有技术做出贡献的部分可以以软件产品的形式体现出来,该计算机软件产品存储在一个存储介质(如ROM/RAM、磁碟、光盘)中,包括若干指令用以使得一台终端设备(可以是手机,计算机,服务器,空调器,或者网络设备等)执行本申请各个实施例所述的方法。Through the description of the above implementation manners, those skilled in the art can clearly understand that the above-mentioned embodiment method can be implemented by means of software plus the necessary general hardware platform, of course, it can also be implemented by hardware, but in many cases the former is better.的实施方式。 Based on this understanding, the technical solution of this application essentially or the part that contributes to the existing technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium (such as ROM/RAM, magnetic disk, The optical disc) includes several instructions to make a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) execute the methods described in the various embodiments of the present application.
在本申请所提供的上述实施例中,应该理解到,所揭露的装置和方法,可以通过其他的方式实现。例如,以上所描述的装置实施例仅仅是示意性的,例如,所述模块的划分,仅仅为一种逻辑功能划分,实际实现时可以有另外的划分方式,例如,多个模块或组件可以结合或者可以集成到另一个***,或一些特征可以忽略,或不执行。In the foregoing embodiments provided in this application, it should be understood that the disclosed device and method may be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the modules is only a logical function division, and there may be other divisions in actual implementation, for example, multiple modules or components may be combined. Or it can be integrated into another system, or some features can be ignored or not implemented.
所述模块或组件可以是或者也可以不是物理上分开的,作为模块或组件显示的部件可以是或者也可以不是物理模块,既可以位于一个地方,或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部模块或组件来实现本实施例方案的目的。The modules or components may or may not be physically separated, and the components displayed as modules or components may or may not be physical modules, and may be located in one place or distributed on multiple network units. Some or all of the modules or components may be selected according to actual needs to achieve the objectives of the solutions of the embodiments.
本申请不限于上述实施方式,以上所述是本申请的优选实施方式,该实施例仅用于说明本申请而不用于限制本申请的范围,应当指出,对于本技术领域的普通技术人员来说,在不脱离本申请原理的前提下,其依然可以对前述各具体实施方式所记载的技术方案进行若干改进和修饰,或者对其中部分技术特征进行等效替换。凡是利用本申请说明书及附图内容所做的等效结构,直接或间接运用在其他相关的技术领域,均同理应视为包括在本申请的保护范围之内。This application is not limited to the above-mentioned embodiments. The above are the preferred embodiments of the application. This embodiment is only used to illustrate the application and not to limit the scope of the application. It should be noted that for those of ordinary skill in the art Without departing from the principle of the present application, it is still possible to make several improvements and modifications to the technical solutions described in the foregoing specific embodiments, or to equivalently replace some of the technical features. Any equivalent structure made using the content of the description and drawings of this application, directly or indirectly used in other related technical fields, shall equally be deemed to be included in the scope of protection of this application.
显然,以上所描述的实施例仅仅是本申请一部分实施例,而不是全部的实施例,附图中给出了本申请的较佳实施例,但并不限制本申请的专利范围。本申请可以以许多不同的形式来实现,相反地,提供这些实施例的目的是使对本申请的公开内容的理解更加透彻全面。尽管参照前述实施例对本申请进行了详细的说明,对于本领域的技术人员来而言,其依然可以对前述各具体实施方式所记载的技术方案进行修改,或者对其中部分技术特征进行等效替换。基于本申请中的实施例,本领域普通技术人员在没有作出创造性劳动前提下所获得的所有其他实施例,以及凡是利用本申请说明书及附图内容所做的等效结构,直接或间接运用在其他相关的技术领域,均同理在本申请专利保护范围之内。Obviously, the above-described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The drawings show preferred embodiments of the present application, but do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these examples is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although this application has been described in detail with reference to the foregoing embodiments, for those skilled in the art, it is still possible for those skilled in the art to modify the technical solutions described in each of the foregoing specific embodiments, or equivalently replace some of the technical features. . Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative work, as well as any equivalent structures made using the contents of the description and drawings of this application, are directly or indirectly used in Other related technical fields are similarly within the scope of the patent protection of this application.

Claims (20)

  1. 一种基于云技术的文件分片上传方法,其中,包括:A method for uploading files in pieces based on cloud technology, which includes:
    接收目标用户配置的上传指令,所述上传指令中指明有待上传文件对象;Receiving an upload instruction configured by a target user, where the upload instruction specifies a file object to be uploaded;
    根据所述上传指令中待上传文件对象读取指明的待上传文件,并获取每个待上传文件的文件大小及文件标识,比较每个待上传文件的所述文件大小与预设的分片阈值的大小,将文件大小小于或等于所述分片阈值的待上传文件记为第一类型文件,将文件大小大于所述分片阈值的待上传文件记为第二类型文件;Read the specified file to be uploaded according to the file object to be uploaded in the upload instruction, and obtain the file size and file identifier of each file to be uploaded, and compare the file size of each file to be uploaded with the preset fragmentation threshold The file to be uploaded whose file size is less than or equal to the fragmentation threshold is recorded as the first type of file, and the file to be uploaded with the file size greater than the fragmentation threshold is recorded as the second type of file;
    基于所述分片阈值对所述第二类型文件进行分片处理,以将所有第二类型文件分别拆分为携带文件标识和分片标识的若干个文件分片;Performing fragmentation processing on the second-type file based on the fragmentation threshold, so as to split all the second-type files into a plurality of file fragments carrying a file identifier and a fragment identifier;
    将所述第一类型文件和所述第二类型文件的若干个文件分片依次读入内存,并执行哈希处理,以获取每个待上传文件的第一散列值;Reading several file fragments of the first type file and the second type file into memory in sequence, and performing hash processing to obtain the first hash value of each file to be uploaded;
    调用基础分片接口和终片接口,通过所述终片接口直接上传所述内存中的第一类型文件,通过所述基础分片接口和所述终片接口上传所述内存中的第二类型文件的若干个文件分片;Call the basic fragment interface and the final fragment interface, directly upload the first type file in the memory through the final fragment interface, and upload the second type file in the memory through the basic fragment interface and the final fragment interface Several file fragments of the file;
    基于文件标识和/或分片标识标记待上传文件的上传状态,当第二类型文件对应的文件分片完成上传后,将具有相同文件标识的文件分片进行拼合,以还原为第二类型文件,并对已完成上传的每个第一类型文件和第二类型文件分别进行哈希处理以获取对应的第二散列值,通过所述第一散列值和所述第二散列值进行校验,以确认已完成上传的待上传文件的完整性。Mark the upload status of the file to be uploaded based on the file identifier and/or segment identifier. After the file segment corresponding to the second type of file is uploaded, the file segments with the same file identifier are combined to restore the second type of file , And perform hash processing on each file of the first type and the file of the second type that have been uploaded to obtain the corresponding second hash value. Check to confirm the integrity of the uploaded files that have been uploaded.
  2. 根据权利要求1所述的基于云技术的文件分片上传方法,其中,所述比较每个待上传文件的所述文件大小与预设的分片阈值的大小的步骤之前,所述方法还包括:The method for uploading file fragments based on cloud technology according to claim 1, wherein before the step of comparing the file size of each file to be uploaded with the size of a preset fragmentation threshold, the method further comprises :
    检测当前网络的网络连接状况,获取当前网络的网络状态等级;Detect the network connection status of the current network, and obtain the network status level of the current network;
    匹配与所述网络状态等级对应预设的分片阈值,并设为用于与待上传文件的文件大小进行比较的分片阈值。Match a preset fragmentation threshold corresponding to the network state level, and set it as a fragmentation threshold for comparison with the file size of the file to be uploaded.
  3. 根据权利要求1所述的基于云技术的文件分片上传方法,其中,若所述上传指令中对不同的待上传文件指定有对应的不同的分片阈值,所述比较每个待上传文件的所述文件大小与预设的分片阈值的大小,将文件大小小于或等于所述分片阈值的待上传文件记为第一类型文件,将文件大小大于所述分片阈值的待上传文件记为第二类型文件的步骤包括:The method for uploading file fragments based on cloud technology according to claim 1, wherein, if the upload instruction specifies different fragmentation thresholds for different files to be uploaded, the comparison of each file to be uploaded The file size and the size of the preset fragmentation threshold, the file to be uploaded whose file size is less than or equal to the fragmentation threshold is recorded as the first type of file, and the file to be uploaded whose file size is greater than the fragmentation threshold is recorded The steps for the second type of file include:
    识别待上传文件的文件标识,从所述上传指令中获取与不同所属文件标识对应的若干个分片阈值;Identify the file identifier of the file to be uploaded, and obtain several fragmentation thresholds corresponding to different file identifiers from the upload instruction;
    将每个待上传文件的文件大小与对应的分片阈值的大小分别进行比较,将文件大小小于或等于与其对应的分片阈值的待上传文件记为第一类型文件,将文件大小大于与其对应的分片阈值的待上传文件记为第二类型文件;The file size of each file to be uploaded is compared with the size of the corresponding fragmentation threshold. The file to be uploaded whose file size is less than or equal to the corresponding fragmentation threshold is recorded as the first type of file, and the file size is greater than the corresponding fragmentation threshold. The file to be uploaded with the fragmentation threshold of is recorded as the second type of file;
    所述基于所述分片阈值对所述第二类型文件进行分片处理的步骤包括:基于所述上传指令中指定的若干个分片阈值,分别为对应的所述第二类型文件进行分片处理。The step of performing fragmentation processing on the second-type file based on the fragmentation threshold includes: fragmenting the corresponding second-type file based on a plurality of fragmentation thresholds specified in the upload instruction deal with.
  4. 根据权利要求1所述的基于云技术的文件分片上传方法,其中,所述将所述第一类型文件和所述第二类型文件的若干个文件分片依次读入内存,并执行哈希处理,以获取每个待上传文件的第一散列值的步骤包括:The method for uploading file fragments based on cloud technology according to claim 1, wherein said reading several file fragments of said first type file and said second type file into memory in sequence, and performing hashing The steps of processing to obtain the first hash value of each file to be uploaded include:
    识别待上传文件的类型;Identify the type of file to be uploaded;
    若为第一类型文件,便将第一类型文件直接读入内存执行哈希处理,以获取每个第一类型文件的第一散列值;If it is a file of the first type, the file of the first type is directly read into the memory to perform hash processing to obtain the first hash value of each file of the first type;
    若为第二类型文件,则依次读入第二类型文件的若干个文件分片,在每读入一个文件分片时便对该文件分片执行哈希处理,以获取该文件分片对应的分片散列值,并每当获取到一个第二类型文件的最后一个文件分片对应的分片散列值后,对该第二类型文件的所有文件分片对应的分片散列值进行拼合,以获取到每个第二类型文件的第一散列值。If it is a file of the second type, read in several file segments of the file of the second type in turn, and perform hash processing on the file segment every time a file segment is read in to obtain the corresponding file segment The fragment hash value, and whenever the fragment hash value corresponding to the last file fragment of a second type file is obtained, the fragment hash value corresponding to all file fragments of the second type file is performed Combine to obtain the first hash value of each file of the second type.
  5. 根据权利要求1所述的基于云技术的文件分片上传方法,其中,所述通过所述基础分片接口和所述终片接口上传所述内存中的第二类型文件的若干个文件分片的步骤包括:The method for uploading file fragments based on cloud technology according to claim 1, wherein the uploading of a plurality of file fragments of the second type file in the memory through the basic fragment interface and the final fragment interface The steps include:
    基于文件标识和分片标识,识别当前待上传的文件分片是否为其所属的第二类型文件的最后一个文件分片;Based on the file identifier and the segment identifier, identify whether the file segment currently to be uploaded is the last file segment of the second type file to which it belongs;
    若不是,便通过所述基础分片接口上传当前待上传的文件分片;If not, upload the file fragments currently to be uploaded through the basic fragmentation interface;
    若是,则通过所述终片接口上传当前待上传的文件分片。If yes, upload the file fragments currently to be uploaded through the final fragment interface.
  6. 根据权利要求1所述的基于云技术的文件分片上传方法,其中,所述当第二类型文件对应的文件分片完成上传后,将具有相同文件标识的文件分片进行拼合,以还原为第二类型文件的步骤包括:The method for uploading file fragments based on cloud technology according to claim 1, wherein, after the file fragments corresponding to the second type files are uploaded, the file fragments with the same file identifier are combined to restore to The steps for the second type of file include:
    每当通过所述终片接口完成对一个文件分片的上传后,在预设的响应时间内循环检测与该文件分片的文件标识相同的其他文件分片的上传状态,直至检测到其他文件分片均完成上传时,对与该文件分片的文件标识相同的所有已完成上传的文件分片进行拼合还原;Whenever the upload of a file segment is completed through the final segment interface, the upload status of other file segments with the same file identifier as the file segment is cyclically detected within the preset response time, until other files are detected When the upload of the fragments is completed, all uploaded file fragments with the same file identifier as the file fragment are combined and restored;
    而若超过所述响应时间仍未检测到其他文件分片均完成上传,则停止对所述基础分片接口和所述终片接口的调用,并基于各文件分片的上传状态生成告警消息发送至所述目标用户。If the response time is exceeded and it is not detected that all other file fragments are uploaded, the call to the basic fragment interface and the final fragment interface is stopped, and an alarm message is generated and sent based on the upload status of each file fragment. To the target user.
  7. 根据权利要求6所述的基于云技术的文件分片上传方法,其中,所述基于各文件分片的上传状态生成告警消息发送至所述目标用户的步骤之后,所述方法还包括:The method for uploading file fragments based on cloud technology according to claim 6, wherein after the step of generating an alarm message based on the upload status of each file fragment and sending it to the target user, the method further comprises:
    接收目标用户发送的续传指令;Receive the resuming instruction sent by the target user;
    读取上次文件上传时各待上传文件的上传状态,以标记出用于响应于所述续传指令进行上传的文件节点;Reading the upload status of each file to be uploaded when the file was uploaded last time to mark the file node used for uploading in response to the resuming instruction;
    响应于所述续传指令,从所述文件节点处对未上传成功和未上传的第一类型文件和/或第二类型文件的文件分片重新进行上传。In response to the resuming instruction, re-upload the file segments of the first type file and/or the second type file that have not been uploaded successfully and have not been uploaded from the file node.
  8. 一种基于云技术的文件分片上传装置,其中,包括:A device for uploading file fragments based on cloud technology, which includes:
    指令接收模块,用于接收目标用户配置的上传指令,所述上传指令中指明有待上传文件对象;An instruction receiving module, configured to receive an upload instruction configured by a target user, where the upload instruction specifies a file object to be uploaded;
    文件类型划分模块,用于根据所述上传指令中待上传文件对象读取指明的待上传文件,并获取每个待上传文件的文件大小及文件标识,比较每个待上传文件的所述文件大小与预设的分片阈值的大小,将文件大小小于或等于所述分片阈值的待上传文件记为第一类型文件,将文件大小大于所述分片阈值的待上传文件记为第二类型文件;The file type division module is used to read the specified file to be uploaded according to the file object to be uploaded in the upload instruction, and obtain the file size and file identifier of each file to be uploaded, and compare the file size of each file to be uploaded With the size of the preset fragmentation threshold, the file to be uploaded whose file size is less than or equal to the fragmentation threshold is recorded as the first type of file, and the file to be uploaded with the file size greater than the fragmentation threshold is recorded as the second type. document;
    文件分片模块,用于基于所述分片阈值对所述第二类型文件进行分片处理,以将所有第二类型文件分别拆分为携带文件标识和分片标识的若干个文件分片;A file fragmentation module, configured to perform fragmentation processing on the second-type file based on the fragmentation threshold, so as to split all the second-type files into several file fragments carrying a file identifier and a fragmentation identifier;
    哈希处理模块,用于将所述第一类型文件和所述第二类型文件的若干个文件分片依次读入内存,并执行哈希处理,以获取每个待上传文件的第一散列值;A hash processing module, which is used to sequentially read several file fragments of the first type file and the second type file into the memory, and perform hash processing to obtain the first hash of each file to be uploaded value;
    文件上传模块,用于调用基础分片接口和终片接口,通过所述终片接口直接上传所述内存中的第一类型文件,通过所述基础分片接口和所述终片接口上传所述内存中的第二类型文件的若干个文件分片;The file upload module is used to call the basic fragmentation interface and the final fragment interface, directly upload the first type file in the memory through the final fragment interface, and upload the basic fragment interface and the final fragment interface. Several file fragments of the second type file in the memory;
    文件校验模块,用于基于文件标识和/或分片标识标记待上传文件的上传状态,当第二类型文件对应的文件分片完成上传后,将具有相同文件标识的文件分片进行拼合,以还原为第二类型文件,并对已完成上传的每个第一类型文件和第二类型文件分别进行哈希处理以获取对应的第二散列值,通过所述第一散列值和所述第二散列值进行校验,以确认已完成上传的待上传文件的完整性。The file verification module is used to mark the upload status of the file to be uploaded based on the file identifier and/or the segment identifier. When the file segment corresponding to the second type of file is uploaded, the file segments with the same file identifier are merged together, In order to restore to the second type file, and perform hash processing on each of the first type file and the second type file that have been uploaded to obtain the corresponding second hash value, through the first hash value and all the files The second hash value is checked to confirm the integrity of the uploaded file to be uploaded.
  9. 一种计算机设备,包括存储器、处理器,以及存储在所述存储器中并可在所述处理器上运行的计算机可读指令,所述处理器执行所述计算机可读指令时实现如下基于云技术的文件分片上传方法的步骤:A computer device includes a memory, a processor, and computer-readable instructions stored in the memory and capable of running on the processor. When the processor executes the computer-readable instructions, the following is achieved based on cloud technology The steps of the file upload method:
    接收目标用户配置的上传指令,所述上传指令中指明有待上传文件对象;Receiving an upload instruction configured by a target user, where the upload instruction specifies a file object to be uploaded;
    根据所述上传指令中待上传文件对象读取指明的待上传文件,并获取每个待上传文件的文件大小及文件标识,比较每个待上传文件的所述文件大小与预设的分片阈值的大小,将文件大小小于或等于所述分片阈值的待上传文件记为第一类型文件,将文件大小大于所述分片阈值的待上传文件记为第二类型文件;Read the specified file to be uploaded according to the file object to be uploaded in the upload instruction, and obtain the file size and file identifier of each file to be uploaded, and compare the file size of each file to be uploaded with the preset fragmentation threshold The file to be uploaded whose file size is less than or equal to the fragmentation threshold is recorded as the first type of file, and the file to be uploaded with the file size greater than the fragmentation threshold is recorded as the second type of file;
    基于所述分片阈值对所述第二类型文件进行分片处理,以将所有第二类型文件分别拆分为携带文件标识和分片标识的若干个文件分片;Performing fragmentation processing on the second-type file based on the fragmentation threshold, so as to split all the second-type files into a plurality of file fragments carrying a file identifier and a fragment identifier;
    将所述第一类型文件和所述第二类型文件的若干个文件分片依次读入内存,并执行哈希处理,以获取每个待上传文件的第一散列值;Reading several file fragments of the first type file and the second type file into memory in sequence, and performing hash processing to obtain the first hash value of each file to be uploaded;
    调用基础分片接口和终片接口,通过所述终片接口直接上传所述内存中的第一类型文件,通过所述基础分片接口和所述终片接口上传所述内存中的第二类型文件的若干个文件分片;Call the basic fragment interface and the final fragment interface, directly upload the first type file in the memory through the final fragment interface, and upload the second type file in the memory through the basic fragment interface and the final fragment interface Several file fragments of the file;
    基于文件标识和/或分片标识标记待上传文件的上传状态,当第二类型文件对应的文件分片完成上传后,将具有相同文件标识的文件分片进行拼合,以还原为第二类型文件,并对已完成上传的每个第一类型文件和第二类型文件分别进行哈希处理以获取对应的第二散列值,通过所述第一散列值和所述第二散列值进行校验,以确认已完成上传的待上传文件的完整性。Mark the upload status of the file to be uploaded based on the file identifier and/or segment identifier. After the file segment corresponding to the second type of file is uploaded, the file segments with the same file identifier are combined to restore the second type of file , And perform hash processing on each file of the first type and the file of the second type that have been uploaded to obtain the corresponding second hash value. Check to confirm the integrity of the uploaded files that have been uploaded.
  10. 根据权利要求9所述的计算机设备,其中,所述比较每个待上传文件的所述文件大小与预设的分片阈值的大小的步骤之前,所述方法还包括:9. The computer device according to claim 9, wherein before the step of comparing the file size of each file to be uploaded with the size of a preset fragmentation threshold, the method further comprises:
    检测当前网络的网络连接状况,获取当前网络的网络状态等级;Detect the network connection status of the current network, and obtain the network status level of the current network;
    匹配与所述网络状态等级对应预设的分片阈值,并设为用于与待上传文件的文件大小进行比较的分片阈值。Match a preset fragmentation threshold corresponding to the network state level, and set it as a fragmentation threshold for comparison with the file size of the file to be uploaded.
  11. 根据权利要求9所述的计算机设备,其中,若所述上传指令中对不同的待上传文件指定有对应的不同的分片阈值,所述比较每个待上传文件的所述文件大小与预设的分片阈值的大小,将文件大小小于或等于所述分片阈值的待上传文件记为第一类型文件,将文件大小大于所述分片阈值的待上传文件记为第二类型文件的步骤包括:8. The computer device according to claim 9, wherein if the upload instruction specifies different fragmentation thresholds for different files to be uploaded, the file size of each file to be uploaded is compared with a preset The step of recording a file to be uploaded with a file size less than or equal to the fragmentation threshold as a first type of file, and recording a file to be uploaded with a file size greater than the fragmentation threshold as a file of the second type include:
    识别待上传文件的文件标识,从所述上传指令中获取与不同所属文件标识对应的若干个分片阈值;Identify the file identifier of the file to be uploaded, and obtain several fragmentation thresholds corresponding to different file identifiers from the upload instruction;
    将每个待上传文件的文件大小与对应的分片阈值的大小分别进行比较,将文件大小小于或等于与其对应的分片阈值的待上传文件记为第一类型文件,将文件大小大于与其对应的分片阈值的待上传文件记为第二类型文件;The file size of each file to be uploaded is compared with the size of the corresponding fragmentation threshold. The file to be uploaded whose file size is less than or equal to the corresponding fragmentation threshold is recorded as the first type of file, and the file size is greater than the corresponding fragmentation threshold. The file to be uploaded with the fragmentation threshold of is recorded as the second type of file;
    所述基于所述分片阈值对所述第二类型文件进行分片处理的步骤包括:基于所述上传指令中指定的若干个分片阈值,分别为对应的所述第二类型文件进行分片处理。The step of performing fragmentation processing on the second-type file based on the fragmentation threshold includes: fragmenting the corresponding second-type file based on a plurality of fragmentation thresholds specified in the upload instruction deal with.
  12. 根据权利要求9所述的计算机设备,其中,所述将所述第一类型文件和所述第二类型文件的若干个文件分片依次读入内存,并执行哈希处理,以获取每个待上传文件的第一散列值的步骤包括:The computer device according to claim 9, wherein the said first type of file and the plurality of file fragments of the second type of file are sequentially read into the memory, and hash processing is performed to obtain each file to be The steps of uploading the first hash value of the file include:
    识别待上传文件的类型;Identify the type of file to be uploaded;
    若为第一类型文件,便将第一类型文件直接读入内存执行哈希处理,以获取每个第一类型文件的第一散列值;If it is a file of the first type, the file of the first type is directly read into the memory to perform hash processing to obtain the first hash value of each file of the first type;
    若为第二类型文件,则依次读入第二类型文件的若干个文件分片,在每读入一个文件分片时便对该文件分片执行哈希处理,以获取该文件分片对应的分片散列值,并每当获取到一个第二类型文件的最后一个文件分片对应的分片散列值后,对该第二类型文件的所有文件分片对应的分片散列值进行拼合,以获取到每个第二类型文件的第一散列值。If it is a file of the second type, read in several file segments of the file of the second type in turn, and perform hash processing on the file segment every time a file segment is read in to obtain the corresponding file segment The fragment hash value, and whenever the fragment hash value corresponding to the last file fragment of a second type file is obtained, the fragment hash value corresponding to all file fragments of the second type file is performed Combine to obtain the first hash value of each file of the second type.
  13. 根据权利要求9所述的计算机设备,其中,所述通过所述基础分片接口和所述终片接口上传所述内存中的第二类型文件的若干个文件分片的步骤包括:The computer device according to claim 9, wherein the step of uploading several file fragments of the second type file in the memory through the basic fragment interface and the final fragment interface comprises:
    基于文件标识和分片标识,识别当前待上传的文件分片是否为其所属的第二类型文件的最后一个文件分片;Based on the file identifier and the segment identifier, identify whether the file segment currently to be uploaded is the last file segment of the second type file to which it belongs;
    若不是,便通过所述基础分片接口上传当前待上传的文件分片;If not, upload the file fragments currently to be uploaded through the basic fragmentation interface;
    若是,则通过所述终片接口上传当前待上传的文件分片。If yes, upload the file fragments currently to be uploaded through the final fragment interface.
  14. 根据权利要求9所述的计算机设备,其中,所述当第二类型文件对应的文件分片完成上传后,将具有相同文件标识的文件分片进行拼合,以还原为第二类型文件的步骤包括:9. The computer device according to claim 9, wherein the step of combining the file fragments with the same file identifier to restore the file of the second type after the file fragments corresponding to the second type of file are uploaded. :
    每当通过所述终片接口完成对一个文件分片的上传后,在预设的响应时间内循环检测与该文件分片的文件标识相同的其他文件分片的上传状态,直至检测到其他文件分片均完成上传时,对与该文件分片的文件标识相同的所有已完成上传的文件分片进行拼合还原;Whenever the upload of a file segment is completed through the final segment interface, the upload status of other file segments with the same file identifier as the file segment is cyclically detected within the preset response time, until other files are detected When the upload of the fragments is completed, all uploaded file fragments with the same file identifier as the file fragment are combined and restored;
    而若超过所述响应时间仍未检测到其他文件分片均完成上传,则停止对所述基础分片接 口和所述终片接口的调用,并基于各文件分片的上传状态生成告警消息发送至所述目标用户。If the response time is exceeded and it is not detected that all other file fragments are uploaded, the call to the basic fragment interface and the final fragment interface is stopped, and an alarm message is generated and sent based on the upload status of each file fragment. To the target user.
  15. 一个或多个存储有计算机可读指令的计算机可读存储介质,其中,所述计算机可读指令被处理器执行时实现如下基于云技术的文件分片上传方法的步骤:One or more computer-readable storage media storing computer-readable instructions, where the computer-readable instructions implement the following steps of the cloud-based file segment upload method when the computer-readable instructions are executed by a processor:
    接收目标用户配置的上传指令,所述上传指令中指明有待上传文件对象;Receiving an upload instruction configured by a target user, where the upload instruction specifies a file object to be uploaded;
    根据所述上传指令中待上传文件对象读取指明的待上传文件,并获取每个待上传文件的文件大小及文件标识,比较每个待上传文件的所述文件大小与预设的分片阈值的大小,将文件大小小于或等于所述分片阈值的待上传文件记为第一类型文件,将文件大小大于所述分片阈值的待上传文件记为第二类型文件;Read the specified file to be uploaded according to the file object to be uploaded in the upload instruction, and obtain the file size and file identifier of each file to be uploaded, and compare the file size of each file to be uploaded with the preset fragmentation threshold The file to be uploaded whose file size is less than or equal to the fragmentation threshold is recorded as the first type of file, and the file to be uploaded with the file size greater than the fragmentation threshold is recorded as the second type of file;
    基于所述分片阈值对所述第二类型文件进行分片处理,以将所有第二类型文件分别拆分为携带文件标识和分片标识的若干个文件分片;Performing fragmentation processing on the second-type file based on the fragmentation threshold, so as to split all the second-type files into a plurality of file fragments carrying a file identifier and a fragment identifier;
    将所述第一类型文件和所述第二类型文件的若干个文件分片依次读入内存,并执行哈希处理,以获取每个待上传文件的第一散列值;Reading several file fragments of the first type file and the second type file into memory in sequence, and performing hash processing to obtain the first hash value of each file to be uploaded;
    调用基础分片接口和终片接口,通过所述终片接口直接上传所述内存中的第一类型文件,通过所述基础分片接口和所述终片接口上传所述内存中的第二类型文件的若干个文件分片;Call the basic fragment interface and the final fragment interface, directly upload the first type file in the memory through the final fragment interface, and upload the second type file in the memory through the basic fragment interface and the final fragment interface Several file fragments of the file;
    基于文件标识和/或分片标识标记待上传文件的上传状态,当第二类型文件对应的文件分片完成上传后,将具有相同文件标识的文件分片进行拼合,以还原为第二类型文件,并对已完成上传的每个第一类型文件和第二类型文件分别进行哈希处理以获取对应的第二散列值,通过所述第一散列值和所述第二散列值进行校验,以确认已完成上传的待上传文件的完整性。Mark the upload status of the file to be uploaded based on the file identifier and/or segment identifier. After the file segment corresponding to the second type of file is uploaded, the file segments with the same file identifier are combined to restore the second type of file , And perform hash processing on each file of the first type and the file of the second type that have been uploaded to obtain the corresponding second hash value. Check to confirm the integrity of the uploaded files that have been uploaded.
  16. 根据权利要求15所述的计算机可读存储介质,其中,所述比较每个待上传文件的所述文件大小与预设的分片阈值的大小的步骤之前,所述方法还包括:15. The computer-readable storage medium according to claim 15, wherein before the step of comparing the file size of each file to be uploaded with the size of a preset fragmentation threshold, the method further comprises:
    检测当前网络的网络连接状况,获取当前网络的网络状态等级;Detect the network connection status of the current network, and obtain the network status level of the current network;
    匹配与所述网络状态等级对应预设的分片阈值,并设为用于与待上传文件的文件大小进行比较的分片阈值。Match a preset fragmentation threshold corresponding to the network state level, and set it as a fragmentation threshold for comparison with the file size of the file to be uploaded.
  17. 根据权利要求15所述的计算机可读存储介质,其中,若所述上传指令中对不同的待上传文件指定有对应的不同的分片阈值,所述比较每个待上传文件的所述文件大小与预设的分片阈值的大小,将文件大小小于或等于所述分片阈值的待上传文件记为第一类型文件,将文件大小大于所述分片阈值的待上传文件记为第二类型文件的步骤包括:The computer-readable storage medium according to claim 15, wherein if the upload instruction specifies different fragmentation thresholds for different files to be uploaded, the file size of each file to be uploaded is compared With the size of the preset fragmentation threshold, the file to be uploaded whose file size is less than or equal to the fragmentation threshold is recorded as the first type of file, and the file to be uploaded with the file size greater than the fragmentation threshold is recorded as the second type. The steps of the file include:
    识别待上传文件的文件标识,从所述上传指令中获取与不同所属文件标识对应的若干个分片阈值;Identify the file identifier of the file to be uploaded, and obtain several fragmentation thresholds corresponding to different file identifiers from the upload instruction;
    将每个待上传文件的文件大小与对应的分片阈值的大小分别进行比较,将文件大小小于或等于与其对应的分片阈值的待上传文件记为第一类型文件,将文件大小大于与其对应的分片阈值的待上传文件记为第二类型文件;The file size of each file to be uploaded is compared with the size of the corresponding fragmentation threshold. The file to be uploaded whose file size is less than or equal to the corresponding fragmentation threshold is recorded as the first type of file, and the file size is greater than the corresponding fragmentation threshold. The file to be uploaded with the fragmentation threshold of is recorded as the second type of file;
    所述基于所述分片阈值对所述第二类型文件进行分片处理的步骤包括:基于所述上传指令中指定的若干个分片阈值,分别为对应的所述第二类型文件进行分片处理。The step of performing fragmentation processing on the second-type file based on the fragmentation threshold includes: fragmenting the corresponding second-type file based on a plurality of fragmentation thresholds specified in the upload instruction deal with.
  18. 根据权利要求15所述的计算机可读存储介质,其中,所述将所述第一类型文件和所述第二类型文件的若干个文件分片依次读入内存,并执行哈希处理,以获取每个待上传文件的第一散列值的步骤包括:The computer-readable storage medium according to claim 15, wherein the several file fragments of the first type file and the second type file are sequentially read into the memory, and hash processing is performed to obtain The steps of the first hash value of each file to be uploaded include:
    识别待上传文件的类型;Identify the type of file to be uploaded;
    若为第一类型文件,便将第一类型文件直接读入内存执行哈希处理,以获取每个第一类型文件的第一散列值;If it is a file of the first type, the file of the first type is directly read into the memory to perform hash processing to obtain the first hash value of each file of the first type;
    若为第二类型文件,则依次读入第二类型文件的若干个文件分片,在每读入一个文件分片时便对该文件分片执行哈希处理,以获取该文件分片对应的分片散列值,并每当获取到一个第二类型文件的最后一个文件分片对应的分片散列值后,对该第二类型文件的所有文件分片对应的分片散列值进行拼合,以获取到每个第二类型文件的第一散列值。If it is a file of the second type, read in several file segments of the file of the second type in turn, and perform hash processing on the file segment every time a file segment is read in to obtain the corresponding file segment The fragment hash value, and whenever the fragment hash value corresponding to the last file fragment of a second type file is obtained, the fragment hash value corresponding to all file fragments of the second type file is performed Combine to obtain the first hash value of each file of the second type.
  19. 根据权利要求15所述的计算机可读存储介质,其中,所述通过所述基础分片接口和所述终片接口上传所述内存中的第二类型文件的若干个文件分片的步骤包括:15. The computer-readable storage medium according to claim 15, wherein the step of uploading several file fragments of the second type file in the memory through the basic fragment interface and the final fragment interface comprises:
    基于文件标识和分片标识,识别当前待上传的文件分片是否为其所属的第二类型文件的最后一个文件分片;Based on the file identifier and the segment identifier, identify whether the file segment currently to be uploaded is the last file segment of the second type file to which it belongs;
    若不是,便通过所述基础分片接口上传当前待上传的文件分片;If not, upload the file fragments currently to be uploaded through the basic fragmentation interface;
    若是,则通过所述终片接口上传当前待上传的文件分片。If yes, upload the file fragments currently to be uploaded through the final fragment interface.
  20. 根据权利要求15所述的计算机可读存储介质,其中,所述当第二类型文件对应的文件分片完成上传后,将具有相同文件标识的文件分片进行拼合,以还原为第二类型文件的步骤包括:15. The computer-readable storage medium according to claim 15, wherein, after the file fragments corresponding to the second type of file are uploaded, the file fragments with the same file identifier are combined to restore the second type of file The steps include:
    每当通过所述终片接口完成对一个文件分片的上传后,在预设的响应时间内循环检测与该文件分片的文件标识相同的其他文件分片的上传状态,直至检测到其他文件分片均完成上传时,对与该文件分片的文件标识相同的所有已完成上传的文件分片进行拼合还原;Whenever the upload of a file segment is completed through the final segment interface, the upload status of other file segments with the same file identifier as the file segment is cyclically detected within the preset response time, until other files are detected When the upload of the fragments is completed, all uploaded file fragments with the same file identifier as the file fragment are combined and restored;
    而若超过所述响应时间仍未检测到其他文件分片均完成上传,则停止对所述基础分片接口和所述终片接口的调用,并基于各文件分片的上传状态生成告警消息发送至所述目标用户。If the response time is exceeded and it is not detected that all other file fragments are uploaded, the call to the basic fragment interface and the final fragment interface is stopped, and an alarm message is generated and sent based on the upload status of each file fragment. To the target user.
PCT/CN2020/099534 2020-02-21 2020-06-30 Cloud technology-based file fragment uploading method and apparatus, and device and storage medium WO2021164178A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202010106294.1A CN111414334B (en) 2020-02-21 2020-02-21 File fragment uploading method, device, equipment and storage medium based on cloud technology
CN202010106294.1 2020-02-21

Publications (1)

Publication Number Publication Date
WO2021164178A1 true WO2021164178A1 (en) 2021-08-26

Family

ID=71494130

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/099534 WO2021164178A1 (en) 2020-02-21 2020-06-30 Cloud technology-based file fragment uploading method and apparatus, and device and storage medium

Country Status (2)

Country Link
CN (1) CN111414334B (en)
WO (1) WO2021164178A1 (en)

Cited By (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113852642A (en) * 2021-10-09 2021-12-28 珠海迈科智能科技股份有限公司 TS (transport stream) fragment uploading method and device based on DVB (digital video broadcasting) standard
CN113949897A (en) * 2021-11-30 2022-01-18 紫光云(南京)数字技术有限公司 Fragment cache acceleration method for MP4 video streaming media on demand
CN114301900A (en) * 2021-12-27 2022-04-08 中国电信股份有限公司 File transmission method and device and electronic equipment
CN114363321A (en) * 2021-12-30 2022-04-15 支付宝(杭州)信息技术有限公司 File transmission method, equipment and system
CN114448975A (en) * 2022-03-04 2022-05-06 中国建设银行股份有限公司 File transmission method, device, equipment and medium
CN114915622A (en) * 2022-05-31 2022-08-16 广东三维睿新科技有限公司 HTTP-based file transmission method for web side
CN115002100A (en) * 2022-05-27 2022-09-02 平安普惠企业管理有限公司 File transmission method and device, electronic equipment and storage medium
CN115174561A (en) * 2022-09-08 2022-10-11 北京亿赛通网络安全技术有限公司 File segmented transmission method and system
CN115421649A (en) * 2022-08-02 2022-12-02 佳源科技股份有限公司 Indexable and extensible parameter file fragment storage system and method
CN115618428A (en) * 2022-12-19 2023-01-17 江西曼荼罗软件有限公司 Data integrity checking method and system based on block chain technology

Families Citing this family (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111835867A (en) * 2020-07-28 2020-10-27 浪潮云信息技术股份公司 File uploading method and system
CN112398904A (en) * 2020-09-24 2021-02-23 中国电建集团海外投资有限公司 Data sending method based on public cloud
CN112134789A (en) * 2020-09-28 2020-12-25 中国银行股份有限公司 Method and device for sending and receiving mails
CN112532740B (en) * 2020-12-11 2022-11-15 平安科技(深圳)有限公司 File uploading method and device and file checking method and device
CN112559464A (en) * 2020-12-22 2021-03-26 鲁班(北京)电子商务科技有限公司 File uploading method and uploading device
CN112769906A (en) * 2020-12-25 2021-05-07 三盟科技股份有限公司 File breakpoint uploading method and system, computer equipment and storage medium
CN112948340A (en) * 2021-02-04 2021-06-11 北京金山云网络技术有限公司 Data synchronization method and device, electronic equipment and readable storage medium
CN113079219A (en) * 2021-04-12 2021-07-06 北京明略昭辉科技有限公司 Large file fragment uploading method and system
CN113726838B (en) * 2021-06-17 2023-09-19 武汉理工数字传播工程有限公司 File transmission method, device, equipment and storage medium
CN113411393A (en) * 2021-06-17 2021-09-17 中国工商银行股份有限公司 File pushing method and device
CN114338646A (en) * 2021-11-29 2022-04-12 王建冬 File interactive transmission method, device, equipment and storage medium
CN114553852A (en) * 2022-02-28 2022-05-27 傲普(上海)新能源有限公司 Rapid uploading method based on asynchronous industrial file server
CN114553885A (en) * 2022-03-02 2022-05-27 上海弘玑信息技术有限公司 DHT network-based storage method and device, electronic equipment and storage medium
CN114884937B (en) * 2022-05-27 2024-01-09 宁夏中宁县京能新能源有限公司 New energy centralized control system data breakpoint continuous transmission method
CN114866536A (en) * 2022-05-31 2022-08-05 北京奕斯伟计算技术有限公司 File uploading method and WEB front end
CN115037739B (en) * 2022-06-13 2024-02-23 深圳乐播科技有限公司 File transmission method and device, electronic equipment and storage medium
CN117979118A (en) * 2024-03-29 2024-05-03 杭州海康威视数字技术股份有限公司 Data stream recording method, device, recorder and recording system

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130091097A1 (en) * 2011-06-10 2013-04-11 Microsoft Corporation Identification of moved or renamed files in file synchronization
CN109739810A (en) * 2018-12-07 2019-05-10 中山市江波龙电子有限公司 File synchronisation method, server, client and the device with store function
CN109831506A (en) * 2019-01-31 2019-05-31 百度在线网络技术(北京)有限公司 File uploading method, device, terminal, server and readable storage medium storing program for executing
CN110401725A (en) * 2019-08-23 2019-11-01 深圳市鹰硕技术有限公司 File uploading method, device, terminal, server, system and storage medium

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103139241A (en) * 2011-11-24 2013-06-05 腾讯科技(深圳)有限公司 Off-line download system and method for network resource files
US20180219871A1 (en) * 2017-02-01 2018-08-02 Futurewei Technologies, Inc. Verification of fragmented information centric network chunks
CN109522270A (en) * 2018-10-19 2019-03-26 平安科技(深圳)有限公司 File storing and reading method, electronic device and readable storage medium storing program for executing based on block chain

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130091097A1 (en) * 2011-06-10 2013-04-11 Microsoft Corporation Identification of moved or renamed files in file synchronization
CN109739810A (en) * 2018-12-07 2019-05-10 中山市江波龙电子有限公司 File synchronisation method, server, client and the device with store function
CN109831506A (en) * 2019-01-31 2019-05-31 百度在线网络技术(北京)有限公司 File uploading method, device, terminal, server and readable storage medium storing program for executing
CN110401725A (en) * 2019-08-23 2019-11-01 深圳市鹰硕技术有限公司 File uploading method, device, terminal, server, system and storage medium

Cited By (16)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113852642B (en) * 2021-10-09 2023-05-09 珠海迈科智能科技股份有限公司 DVB standard-based TS (transport stream) stream fragment uploading method and device
CN113852642A (en) * 2021-10-09 2021-12-28 珠海迈科智能科技股份有限公司 TS (transport stream) fragment uploading method and device based on DVB (digital video broadcasting) standard
CN113949897A (en) * 2021-11-30 2022-01-18 紫光云(南京)数字技术有限公司 Fragment cache acceleration method for MP4 video streaming media on demand
CN113949897B (en) * 2021-11-30 2024-02-13 紫光云(南京)数字技术有限公司 Slicing cache acceleration method for MP4 video streaming media on demand
CN114301900A (en) * 2021-12-27 2022-04-08 中国电信股份有限公司 File transmission method and device and electronic equipment
CN114363321A (en) * 2021-12-30 2022-04-15 支付宝(杭州)信息技术有限公司 File transmission method, equipment and system
CN114448975A (en) * 2022-03-04 2022-05-06 中国建设银行股份有限公司 File transmission method, device, equipment and medium
CN115002100A (en) * 2022-05-27 2022-09-02 平安普惠企业管理有限公司 File transmission method and device, electronic equipment and storage medium
CN115002100B (en) * 2022-05-27 2024-03-08 河北燕鸣科技有限公司 File transmission method and device, electronic equipment and storage medium
CN114915622A (en) * 2022-05-31 2022-08-16 广东三维睿新科技有限公司 HTTP-based file transmission method for web side
CN114915622B (en) * 2022-05-31 2024-01-30 广东三维睿新科技有限公司 File transmission method based on http for web terminal
CN115421649B (en) * 2022-08-02 2023-10-20 佳源科技股份有限公司 Indexable and extensible parameter file slicing storage system and method
CN115421649A (en) * 2022-08-02 2022-12-02 佳源科技股份有限公司 Indexable and extensible parameter file fragment storage system and method
CN115174561B (en) * 2022-09-08 2022-11-22 北京亿赛通网络安全技术有限公司 File segmented transmission method and system
CN115174561A (en) * 2022-09-08 2022-10-11 北京亿赛通网络安全技术有限公司 File segmented transmission method and system
CN115618428A (en) * 2022-12-19 2023-01-17 江西曼荼罗软件有限公司 Data integrity checking method and system based on block chain technology

Also Published As

Publication number Publication date
CN111414334B (en) 2024-04-05
CN111414334A (en) 2020-07-14

Similar Documents

Publication Publication Date Title
WO2021164178A1 (en) Cloud technology-based file fragment uploading method and apparatus, and device and storage medium
WO2021036228A1 (en) File upload method and apparatus, terminal, server, system, and storage medium
US10560261B1 (en) Systems and techniques for capture of trusted media data
CN112527816B (en) Data blood relationship analysis method, system, computer equipment and storage medium
WO2019075978A1 (en) Data transmission method and apparatus, computer device, and storage medium
US11563560B2 (en) Blockchain-based data evidence storage method and apparatus
WO2017215646A1 (en) Data transmission method and apparatus
US20140250158A1 (en) Method and device for obtaining file
WO2020220536A1 (en) Data backup method and device, and computer readable storage medium
WO2017097262A1 (en) Data exchange method and apparatus, computer program, and computer-readable medium
CN112948340A (en) Data synchronization method and device, electronic equipment and readable storage medium
WO2021169275A1 (en) Sdn network device access method and apparatus, computer device, and storage medium
US10917484B2 (en) Identifying and managing redundant digital content transfers
CN111935227A (en) Method for uploading file through browser, browser and electronic equipment
CN112380227A (en) Data synchronization method, device and equipment based on message queue and storage medium
CN110633046A (en) Storage method and device of distributed system, storage equipment and storage medium
CN116049109A (en) File verification method, system, equipment and medium based on filter
WO2019062019A1 (en) Data list exporting method and terminal using same
WO2019001021A1 (en) Data processing method, apparatus and system, server, and computer storage medium
CN112436943B (en) Request deduplication method, device, equipment and storage medium based on big data
WO2016169212A1 (en) File management method and device
CN106682130B (en) Similar picture detection method and device
WO2021051932A1 (en) Data updating method and apparatus, and computer device and storage medium
EP3349416A1 (en) Relationship chain processing method and system, and storage medium
CN115242783A (en) Transmission method, transmission device, electronic apparatus, and medium

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20919618

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20919618

Country of ref document: EP

Kind code of ref document: A1