CN113590994A - Data processing method, data processing device, computer equipment and storage medium - Google Patents
Data processing method, data processing device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN113590994A CN113590994A CN202110881360.7A CN202110881360A CN113590994A CN 113590994 A CN113590994 A CN 113590994A CN 202110881360 A CN202110881360 A CN 202110881360A CN 113590994 A CN113590994 A CN 113590994A
- Authority
- CN
- China
- Prior art keywords
- data
- target
- file
- target file
- edge node
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 19
- 238000000605 extraction Methods 0.000 claims abstract description 39
- 238000000034 method Methods 0.000 claims abstract description 38
- 230000002159 abnormal effect Effects 0.000 claims abstract description 17
- 238000012795 verification Methods 0.000 claims description 82
- 238000004590 computer program Methods 0.000 claims description 15
- 238000013075 data extraction Methods 0.000 claims description 5
- 230000004044 response Effects 0.000 abstract description 7
- 238000010586 diagram Methods 0.000 description 8
- 230000008569 process Effects 0.000 description 8
- 238000006243 chemical reaction Methods 0.000 description 6
- 238000013524 data verification Methods 0.000 description 5
- 230000008859 change Effects 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 230000001360 synchronised effect Effects 0.000 description 2
- 101000695861 Arabidopsis thaliana Brefeldin A-inhibited guanine nucleotide-exchange protein 5 Proteins 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/957—Browsing optimisation, e.g. caching or content distillation
- G06F16/9574—Browsing optimisation, e.g. caching or content distillation of access to content, e.g. by caching
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/11—File system administration, e.g. details of archiving or snapshots
- G06F16/116—Details of conversion of file system types or formats
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application relates to a data processing method, a data processing device, computer equipment and a storage medium. The method comprises the steps of carrying out self-checking on files stored in edge nodes, using the files to be checked in the edge nodes as target files, enabling each file stored in the edge nodes to be possibly a target file, searching first check data of the target file in the target edge nodes, wherein the first check data are partial data acquired from the target file according to a preset extraction scheme when the edge nodes receive the target file, used for checking the target file, extracting second check data from the target file according to the same preset extraction scheme, checking the target file by comparing the first check data with the second check data, determining that the target file is normal and correct under the condition of successful checking, determining that the target file is abnormal under the condition of failed checking, and avoiding sending the file with an abnormal state as a response to a client.
Description
Technical Field
The present application relates to the field of cloud computing technologies, and in particular, to a data processing method and apparatus, a computer device, and a storage medium.
Background
With the requirement of a user on the speed of requesting response becoming higher and higher, most data requests acquire corresponding response data from the edge node, so that more and more data are stored in the edge node, and if the data in the edge node is abnormal, but the edge node cannot detect the data state stored by the edge node, the abnormal data is easily returned to the user, so that the user cannot browse normally according to the abnormal data, the user needs to send a request again to acquire the required data, the time for the user to wait for the correct response data is increased, and the browsing experience of the user is easily affected.
Disclosure of Invention
In order to solve the technical problem, the application provides a data processing method, a data processing device, a computer device and a storage medium.
In a first aspect, the present application provides a data processing method, which is applied to an edge node, and the method includes:
determining a target file to be verified, wherein the target file is a file stored in a target edge node of a target edge node;
acquiring first check data of the target file from the target edge node, wherein the target edge node stores the file and the check data of the file, and the check data of the file is partial data acquired from the file according to a preset extraction scheme when the target edge node receives the file;
extracting second check-up data from the target file according to the preset extraction scheme;
verifying the target file by comparing the first verification data and the second verification data.
Optionally, the searching for the first check data of the target file from the target edge node includes:
reading third check data of the target file from the target edge node, wherein the coding format of the third check data is a first coding format;
and converting the third check data in the first coding format into the first check data in a second coding format, wherein the second coding format is a coding format adopted by data in the target file when the target edge node receives the target file, and the first coding format is different from the second coding format.
Optionally, before determining the target file to be verified, the method further includes:
receiving the target file, wherein the data in the target file is in the second encoding format;
extracting first check data from the target file according to the preset extraction scheme, wherein the first check data are partial data in the target file, and the coding format of the first check data is the second coding format;
converting the first parity data in the second encoding format into the third parity data in the first encoding format;
and storing the target file and the first check data corresponding to the target file into the target edge node.
Optionally, the extracting first verification data from the target file according to the preset extraction scheme includes:
splitting the target file into a plurality of split data according to a preset split scheme, wherein the split scheme is used for indicating the data number of the split data or indicating the data volume of each split data, and each split data comprises a plurality of subdata;
extracting subdata positioned at a specified position in each split data to obtain target data, wherein each split data corresponds to one target data;
and generating the first verification data according to all the extracted target data in the target file.
Optionally, the saving the target file and the first check data corresponding to the target file to the target edge node includes:
and storing each split data in the target file into one data block of the target edge node.
Optionally, the verifying the target file by comparing the first verification data and the second verification data includes:
determining that the target file passes the verification under the condition that the data at each position in the first verification data is the same as the data at the same position in the second verification data;
and determining that the target file is not verified under the condition that the data of at least one position in the first verification data is different from the data of the same position in the second verification data.
Optionally, the determining the target file to be verified includes:
taking a file corresponding to the downloading request in the target edge node as the target file under the condition of receiving the downloading request; or the like, or, alternatively,
and taking the file reaching the regular verification time in the target edge node as the target file.
Optionally, in a case where a download request is received, after the verifying the target file by comparing the first verification data and the second verification data, the method further includes:
under the condition that the target file passes the verification, encrypting the target file and then sending the encrypted target file to a terminal corresponding to the file downloading request;
and if the target file does not pass the verification, sending information indicating that the target file is abnormal to a dispatching system, and enabling the dispatching system to allocate connectable edge nodes for the downloading request again according to the information, wherein the connectable edge nodes do not contain the target edge nodes.
In a second aspect, the present application provides a data processing apparatus comprising:
the system comprises a target determining module, a verification module and a verification module, wherein the target determining module is used for determining a target file to be verified, and the target file is a file stored in a target edge node;
the file acquisition module is used for acquiring first check data of the target file from the target edge node, wherein the target edge node stores the file and the check data of the file, and the check data of the file is part of data acquired from the file according to a preset extraction scheme when the target edge node receives the file;
the data extraction module is used for extracting second check data from the target file according to the preset extraction scheme;
a data verification module for verifying the target file by comparing the first verification data and the second verification data.
A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the following steps when executing the computer program:
determining a target file to be verified, wherein the target file is a file stored in a target edge node of a target edge node;
acquiring first check data of the target file from the target edge node, wherein the target edge node stores the file and the check data of the file, and the check data of the file is partial data acquired from the file according to a preset extraction scheme when the target edge node receives the file;
extracting second check-up data from the target file according to the preset extraction scheme;
verifying the target file by comparing the first verification data and the second verification data.
A computer-readable storage medium, on which a computer program is stored which, when executed by a processor, carries out the steps of:
determining a target file to be verified, wherein the target file is a file stored in a target edge node of a target edge node;
acquiring first check data of the target file from the target edge node, wherein the target edge node stores the file and the check data of the file, and the check data of the file is partial data acquired from the file according to a preset extraction scheme when the target edge node receives the file;
extracting second check-up data from the target file according to the preset extraction scheme;
verifying the target file by comparing the first verification data and the second verification data.
Based on the data processing method, the embodiment of the application performs self-checking on files stored in edge nodes to ensure that the files in an abnormal state are sent to a client, the files to be checked in the edge nodes are used as target files, each file stored in the edge nodes can be a target file, first check data of the target file is searched in the target edge nodes, the first check data is part of data extracted from the target file according to a preset extraction scheme when the edge nodes receive the target file and is used for checking the target file, second check data is extracted from the target file according to the same preset extraction scheme, the target file is checked by comparing the first check data and the second check data, the target file is determined to be normal and correct under the condition that the checking is successful, the target file is determined to be abnormal under the condition that the checking is failed, the edge node can master the file state stored by the edge node, and the file with an abnormal state is prevented from being sent to the client as a response.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and together with the description, serve to explain the principles of the invention.
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without inventive exercise.
FIG. 1 is a diagram of an application environment of a data processing method in one embodiment;
FIG. 2 is a flow diagram illustrating a data processing method according to one embodiment;
FIG. 3 is a block diagram of a data processing apparatus according to an embodiment;
FIG. 4 is a diagram illustrating an internal structure of a computer device according to an embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
FIG. 1 is a diagram of an application environment of a data processing method in one embodiment. Referring to fig. 1, the data processing method is applied to a data processing system. The data processing system includes a terminal 110, an edge node 120, and a scheduling system 130. The terminal 110 and the edge node 120, the scheduling system 130 are connected through a network. The terminal 110 may specifically be a desktop terminal or a mobile terminal, and the mobile terminal may specifically be at least one of a mobile phone, a tablet computer, a notebook computer, and the like. The edge node 120 may be specifically a home node or a CDN node, where the home node is a router or a set-top box that provides a download service for a common user home by using an idle bandwidth. The CDN node is a node in a Content Delivery Network (CDN Network). The scheduling system 130 is configured to manage all nodes, each node reports data to the scheduling system at regular time, the scheduling system can know a storage state and an operating state of the node and an operator network corresponding to the node according to the reported data, and the scheduling system allocates a corresponding node to a data request of a terminal according to the reported data fed back by the node in real time, so as to provide a data service for the terminal.
In one embodiment, fig. 2 is a flow chart illustrating a data processing method in one embodiment, and referring to fig. 2, a data processing method is provided. The present embodiment is mainly exemplified by applying the method to the edge node 120 in fig. 1, and the data processing method specifically includes the following steps:
step S210, determining a target file to be verified, where the target file is a file stored in a target edge node.
In this embodiment, the target file to be verified refers to a file that meets a verification condition, where the verification condition may be specifically used to indicate that verification needs to be performed when the verification time within a verification period is reached, or when a download request for the file is received, the file needs to be verified before the file is sent. The files stored in the target edge node are used to provide data services for the client.
Step S220, obtaining first verification data of the target file from the target edge node, where the target edge node stores a file and verification data of the file, and the verification data of the file is a part of data obtained from the file according to a preset extraction scheme when the target edge node receives the file.
In this embodiment, each file stored in the target edge node has corresponding check data, the check data is composed of partial data in the file, the first check data is partial data extracted from the target file according to a preset extraction scheme when the target file is received, and in a subsequent checking process, the first check data is used as a reference standard for determining whether the target file changes. The preset extraction scheme refers to a rule for extracting data from a file, and specifically includes an extraction data amount, extraction positions, position intervals among the extraction positions, an extraction sequence and the like.
Step S230, extracting second check-up data from the target file according to the preset extraction scheme.
In this embodiment, the second check data is extracted from the target file in the same manner as the first check data is generated, so as to ensure consistency of data sampling before data checking. The second check-up data is part of data in the current target file.
Step S240, verifying the target file by comparing the first verification data and the second verification data.
In this embodiment, the first check data and the second check data which are generated in the same extraction manner and at different times are compared, the target file is checked by comparing the first check data and the second check data, the target file is determined to be normal and correct under the condition that the check is successful, and the target file is determined to be abnormal under the condition that the check is failed, so that the edge node grasps the file state stored by the edge node, and the file with the abnormal state is prevented from being sent to the client as a response.
In one embodiment, the searching the first check data of the target file from the target edge node includes:
reading third check data of the target file from the target edge node, wherein the coding format of the third check data is a first coding format; and converting the third check data in the first coding format into the first check data in a second coding format, wherein the second coding format is a coding format adopted by data in the target file when the target edge node receives the target file, and the first coding format is different from the second coding format.
Specifically, each file in the target edge node has corresponding check data, the check data is generated after format conversion is performed on the check data according to partial data extracted from the file when the file is received, the third check data is generated by format conversion according to the first check data when the target file is received, the encoding format of the third check data is the first encoding format, the third check data is subjected to format conversion to obtain the first check data in the second encoding format, the encoding format may be specifically but not limited to ASCII, GB2312, GBK, BIG5 and GB18030, and the first encoding format is different from the second encoding format.
In one embodiment, before the determining the target file to be verified, the method further includes:
receiving the target file, wherein the data in the target file is in the second encoding format; extracting first check data from the target file according to the preset extraction scheme, wherein the first check data are partial data in the target file, and the coding format of the first check data is the second coding format; converting the first parity data in the second encoding format into the third parity data in the first encoding format; and storing the target file and the first check data corresponding to the target file into the target edge node.
Specifically, when an object file in a second coding format is received, first check data in the same coding format as the second coding format is extracted from the object file according to a preset extraction scheme, format conversion is performed on the first check data to obtain third check data in the same coding format as the first coding format, the object file and the third check data corresponding to the object file are stored in an object edge node, the first check data are not directly stored in the object edge node, but stored in the form of the third check data, and therefore the situation that when the object file is changed, part of data in the object file is changed accordingly, the check data extracted from the object data can also be changed, and the object file can not be checked according to the changed check data serving as a reference standard is avoided. However, format conversion is performed on the first check data and the first check data is stored in the form of third check data, and when the target file is changed, the third check data does not change with data change in the target file, so that it can be ensured that the first check data obtained by performing code format conversion on the third check data according to the previous embodiment does not change with data change in the target file.
In an embodiment, the extracting the first verification data from the target file according to the preset extraction scheme includes:
splitting the target file into a plurality of split data according to a preset split scheme, wherein the split scheme is used for indicating the data number of the split data or indicating the data volume of each split data, and each split data comprises a plurality of subdata; extracting subdata positioned at a specified position in each split data to obtain target data, wherein each split data corresponds to one target data; and generating the first verification data according to all the extracted target data in the target file.
Specifically, the preset splitting scheme may split the target file according to the number of data, or split the target file according to the data amount specified by each split data, illustratively, split the target file according to the number of data, where the number of data is 4096, and 4096 split data is obtained. Each split data comprises a plurality of subdata, each subdata corresponds to one byte, namely each split data comprises a plurality of bytes, the plurality of bytes in the split data are arranged in sequence, and the subdata at a specified position in the split data is extracted, namely the byte at the specified position in the split data is extracted as target data, for example, each split data comprises 1000 bytes, if the specified position is a first position, the byte at the first position in the split data is extracted, and the extracted byte is used as the target data; if the designated position is the first position, the tenth position, the hundredth position and the first thousand position, the bytes positioned at the first position, the tenth position, the hundredth position and the first thousand position in the split data are respectively extracted, and the target data is generated according to the four extracted bytes. For example, the target data is generated by extracting bytes from only one split data, the bytes are extracted from the same designated position in different split data of the same file, and the target data corresponding to all the split data are spliced and combined into the first check data.
In one embodiment, the saving the target file and the first check data corresponding to the target file to the target edge node includes:
and storing each split data in the target file into one data block of the target edge node.
Specifically, the local disk of the target edge node may not support direct storage of a file with a large data size, for example, if the disk file management manner in the fat32 format does not support direct storage of a file with a large data size, the file with a large data size is split to obtain a plurality of split data, and the split data is stored by using the data blocks in the target edge node, so as to implement block storage of the file with a large data size.
In one embodiment, said checking said target file by comparing said first check data and said second check data comprises:
determining that the target file passes the verification under the condition that the data at each position in the first verification data is the same as the data at the same position in the second verification data; and determining that the target file is not verified under the condition that the data of at least one position in the first verification data is different from the data of the same position in the second verification data.
Specifically, the first check data is partial data extracted according to a preset extraction scheme when a target file is received, the second check data is partial data extracted from the target file according to the preset extraction scheme at present, the target file to be checked at present is split according to the preset splitting scheme to obtain a plurality of current split data, bytes at a specified position in each current split data are extracted to generate current target data, then the current target data corresponding to all the current split data are spliced into the second check data, each bit of data in the first check data is compared with data at the same position in the second check data, if the data at each position in the first check data is the same as the data at the same position in the second check data, it indicates that the target file to be checked at present is the same as the received target file, if the file is not changed, determining that the target file passes verification; and if the data of at least one position in the first check data is different from the data of the same position in the second check data, the currently checked target file is changed relative to the received target file, and the target file is determined not to be checked.
In one embodiment, the determining the target file to be verified includes:
taking a file corresponding to the downloading request in the target edge node as the target file under the condition of receiving the downloading request; or, the file reaching the regular check time in the target edge node is used as the target file.
Specifically, under the condition that the download request is received, the file requested to be downloaded by the download request is used as the target file, the target file is verified, and the influence on user experience caused by the fact that an abnormal file is returned to the client corresponding to the data request as a response is avoided. Alternatively, the files in the target edge node are checked regularly, the file reaching the regular check time is used as the target file, the same check time can be set for all the files, and different check times can be set for a plurality of files, so that the data processing amount during each check is reduced.
In one embodiment, in a case where a download request is received, after the verifying the target file by comparing the first verification data and the second verification data, the method further includes:
under the condition that the target file passes the verification, encrypting the target file and then sending the encrypted target file to a terminal corresponding to the file downloading request; and if the target file does not pass the verification, sending information indicating that the target file is abnormal to a dispatching system, and enabling the dispatching system to allocate connectable edge nodes for the downloading request again according to the information, wherein the connectable edge nodes do not contain the target edge nodes.
Specifically, after the target file passes verification, the target file is encrypted, and in order to protect the target file from being directly used after being stolen, the encryption method may specifically be a Base64 encryption method, an MD5 encryption method, a key string encryption method, or by using a symmetric encryption algorithm, and the like, and the encrypted file is returned to the terminal of the data request. And under the condition that the target file is not verified, sending information indicating that the target file is abnormal to the scheduling system, wherein the information indicates that the target edge node cannot provide data service related to the target file for the client, and the scheduling system can timely reallocate the edge node capable of providing the data service for the data request according to the feedback of the target edge node.
And the target edge node also deletes the target file which is not verified, so that the storage space of the node is prevented from being occupied by the file which cannot provide the data service, and self-checking is performed on all the files local to the disk, the target edge node sends a self-checking notification to the scheduling system, the self-checking notification also carries the information of deleting the target file which is not verified, so that the scheduling system does not schedule the target edge node in the self-checking process of the target edge node, namely the target edge node is prohibited from providing the data service in the self-checking process, so that the abnormal file which is not verified and found is also stored locally on the disk is avoided.
FIG. 2 is a flow diagram illustrating a data processing method according to an embodiment. It should be understood that, although the steps in the flowchart of fig. 2 are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least a portion of the steps in fig. 2 may include multiple sub-steps or multiple stages that are not necessarily performed at the same time, but may be performed at different times, and the order of performance of the sub-steps or stages is not necessarily sequential, but may be performed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.
In one embodiment, as shown in fig. 3, there is provided a data processing apparatus including:
a target determining module 310, configured to determine a target file to be verified, where the target file is a file stored in a target edge node of a target edge node;
the file obtaining module 320 is configured to obtain first check data of the target file from the target edge node, where the target edge node stores a file and check data of the file, and the check data of the file is partial data obtained from the file according to a preset extraction scheme when the target edge node receives the file;
the data extraction module 330 is configured to extract second check-up data from the target file according to the preset extraction scheme;
a data verification module 340, configured to verify the target file by comparing the first verification data and the second verification data.
In one embodiment, the file obtaining module 320 is further configured to:
reading third check data of the target file from the target edge node, wherein the coding format of the third check data is a first coding format;
and converting the third check data in the first coding format into the first check data in a second coding format, wherein the second coding format is a coding format adopted by data in the target file when the target edge node receives the target file, and the first coding format is different from the second coding format.
In one embodiment, the apparatus further comprises a processing module to:
receiving the target file, wherein the data in the target file is in the second encoding format;
extracting first check data from the target file according to the preset extraction scheme, wherein the first check data are partial data in the target file, and the coding format of the first check data is the second coding format;
converting the first parity data in the second encoding format into the third parity data in the first encoding format;
and storing the target file and the first check data corresponding to the target file into the target edge node.
In one embodiment, the processing module is to:
splitting the target file into a plurality of split data according to a preset split scheme, wherein the split scheme is used for indicating the data number of the split data or indicating the data volume of each split data, and each split data comprises a plurality of subdata;
extracting subdata positioned at a specified position in each split data to obtain target data, wherein each split data corresponds to one target data;
and generating the first verification data according to all the extracted target data in the target file.
In one embodiment, the processing module is to:
and storing each split data in the target file into one data block of the target edge node.
In one embodiment, the data verification module 340 is configured to:
determining that the target file passes the verification under the condition that the data at each position in the first verification data is the same as the data at the same position in the second verification data;
and determining that the target file is not verified under the condition that the data of at least one position in the first verification data is different from the data of the same position in the second verification data.
In one embodiment, the goal determination module 310 is configured to:
taking a file corresponding to the downloading request in the target edge node as the target file under the condition of receiving the downloading request; or the like, or, alternatively,
and taking the file reaching the regular verification time in the target edge node as the target file.
In one embodiment, the apparatus further comprises a data feedback module to:
under the condition that the target file passes the verification, encrypting the target file and then sending the encrypted target file to a terminal corresponding to the file downloading request;
and if the target file does not pass the verification, sending information indicating that the target file is abnormal to a dispatching system, and enabling the dispatching system to allocate connectable edge nodes for the downloading request again according to the information, wherein the connectable edge nodes do not contain the target edge nodes.
FIG. 4 is a diagram illustrating an internal structure of a computer device in one embodiment. The computer device may specifically be the terminal 110 (or the server 120) in fig. 1. As shown in fig. 4, the computer apparatus includes a processor, a memory, a network interface, an input device, and a display screen connected through a system bus. Wherein the memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium of the computer device stores an operating system and may also store a computer program which, when executed by the processor, causes the processor to implement the data processing method. The internal memory may also have stored therein a computer program that, when executed by the processor, causes the processor to perform a data processing method. The display screen of the computer equipment can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment can be a touch layer covered on the display screen, a key, a track ball or a touch pad arranged on the shell of the computer equipment, an external keyboard, a touch pad or a mouse and the like.
Those skilled in the art will appreciate that the architecture shown in fig. 4 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.
In one embodiment, the data processing apparatus provided herein may be implemented in the form of a computer program that is executable on a computer device such as that shown in fig. 4. The memory of the computer device may store various program modules constituting the data processing apparatus, such as the object determination module 310, the file acquisition module 320, the data extraction module 330, and the data verification module 340 shown in fig. 3. The computer program constituted by the respective program modules causes the processor to execute the steps in the data processing method of the respective embodiments of the present application described in the present specification.
The computer device shown in fig. 4 may perform the determination of the target file to be verified by the target determination module 310 in the data processing apparatus shown in fig. 3, wherein the target file is a file stored in a target edge node of the target edge nodes. The computer device may execute, by the file obtaining module 320, obtaining first verification data of the target file from the target edge node, where the target edge node stores a file and verification data of the file, and the verification data of the file is partial data obtained from the file according to a preset extraction scheme when the target edge node receives the file. The computer device may extract the second check-up data from the target file according to the preset extraction scheme through the data extraction module 330. The computer device may perform the verification of the target file by comparing the first verification data and the second verification data through the data verification module 340.
In one embodiment, a computer device is provided, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing the method of any of the above embodiments when executing the computer program.
In an embodiment, a computer-readable storage medium is provided, on which a computer program is stored, which computer program, when being executed by a processor, carries out the method of any of the above embodiments.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by instructing the relevant hardware through a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, target edge node, or other medium used in the embodiments provided herein can include non-volatile and/or volatile memory, among others. Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), double-rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), synchronous Link (Synchlink) DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and bus dynamic RAM (RDRAM).
It is noted that, in this document, relational terms such as "first" and "second," and the like, may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
The foregoing are merely exemplary embodiments of the present invention, which enable those skilled in the art to understand or practice the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (11)
1. A data processing method applied to an edge node, the method comprising:
determining a target file to be verified, wherein the target file is a file stored in a target edge node;
acquiring first check data of the target file from the target edge node, wherein the target edge node stores the file and the check data of the file, and the check data of the file is partial data acquired from the file according to a preset extraction scheme when the target edge node receives the file;
extracting second check-up data from the target file according to the preset extraction scheme;
verifying the target file by comparing the first verification data and the second verification data.
2. The method of claim 1, wherein the searching the first parity data of the target file from the target edge node comprises:
acquiring third check data of the target file from the target edge node, wherein the coding format of the third check data is a first coding format;
and converting the third check data in the first coding format into the first check data in a second coding format, wherein the second coding format is a coding format adopted by data in the target file when the target edge node receives the target file, and the first coding format is different from the second coding format.
3. The method of claim 2, wherein prior to determining the target document to be verified, the method further comprises:
receiving the target file, wherein the data in the target file is in the second encoding format;
extracting the first check data from the target file according to the preset extraction scheme, wherein the first check data is partial data in the target file, and the encoding format of the first check data is the second encoding format;
converting the first parity data in the second encoding format into the third parity data in the first encoding format;
and storing the target file and the third check data corresponding to the target file into the target edge node.
4. The method according to claim 3, wherein the extracting the first verification data from the target file according to the preset extraction scheme comprises:
splitting the target file into a plurality of split data according to a preset split scheme, wherein the split scheme is used for indicating the data number of the split data or indicating the data volume of each split data, and each split data comprises a plurality of subdata;
extracting subdata positioned at a specified position in each split data to obtain target data, wherein each split data corresponds to one target data;
and generating the first verification data according to all the extracted target data in the target file.
5. The method according to claim 4, wherein the saving the target file and the first parity data corresponding to the target file to the target edge node comprises:
and storing each split data in the target file into one data block of the target edge node.
6. The method of claim 1, wherein verifying the target file by comparing the first verification data and the second verification data comprises:
determining that the target file passes the verification under the condition that the data at each position in the first verification data is the same as the data at the same position in the second verification data;
and determining that the target file is not verified under the condition that the data of at least one position in the first verification data is different from the data of the same position in the second verification data.
7. The method of claim 1, wherein determining the target file to be verified comprises:
under the condition of receiving a downloading request, taking a file matched with the downloading request in the target edge node as the target file; or the like, or, alternatively,
and taking the file reaching the regular verification time in the target edge node as the target file.
8. The method of claim 7, wherein upon receiving a download request, after verifying the target file by comparing the first verification data and the second verification data, the method further comprises:
under the condition that the target file passes the verification, encrypting the target file and then sending the encrypted target file to a terminal corresponding to the file downloading request;
and if the target file does not pass the verification, sending information indicating that the target file is abnormal to a dispatching system, and enabling the dispatching system to allocate connectable edge nodes for the downloading request again according to the information, wherein the connectable edge nodes do not contain the target edge nodes.
9. A data processing apparatus, characterized in that the apparatus comprises:
the system comprises a target determining module, a verification module and a verification module, wherein the target determining module is used for determining a target file to be verified, and the target file is a file stored in a target edge node;
the file acquisition module is used for acquiring first check data of the target file from the target edge node, wherein the target edge node stores the file and the check data of the file, and the check data of the file is part of data acquired from the file according to a preset extraction scheme when the target edge node receives the file;
the data extraction module is used for extracting second check data from the target file according to the preset extraction scheme;
and the data comparison module is used for comparing the first check data with the second check data to check the target file.
10. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the steps of the method of any of claims 1 to 8 are implemented when the computer program is executed by the processor.
11. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110881360.7A CN113590994A (en) | 2021-08-02 | 2021-08-02 | Data processing method, data processing device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110881360.7A CN113590994A (en) | 2021-08-02 | 2021-08-02 | Data processing method, data processing device, computer equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113590994A true CN113590994A (en) | 2021-11-02 |
Family
ID=78253795
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110881360.7A Pending CN113590994A (en) | 2021-08-02 | 2021-08-02 | Data processing method, data processing device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113590994A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2023093079A1 (en) * | 2021-11-26 | 2023-06-01 | 浪潮通信信息***有限公司 | Consistency check method and apparatus for distributed edge cloud edge nodes |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101453478A (en) * | 2007-11-28 | 2009-06-10 | 新奥特（北京）视频技术有限公司 | Integrity checking method in document transmission |
| US20130226887A1 (en) * | 2012-02-24 | 2013-08-29 | Xyratex Technology Limited | Method of, and apparatus for, improved data integrity in a networked storage system |
| CN104978253A (en) * | 2015-06-24 | 2015-10-14 | 合肥格易集成电路有限公司 | Target file checking method and apparatus |
| CN109492436A (en) * | 2018-11-29 | 2019-03-19 | 深圳市网心科技有限公司 | A kind of file verification method and relevant apparatus |
| CN110889143A (en) * | 2018-09-07 | 2020-03-17 | 阿里巴巴集团控股有限公司 | File verification method and device |
| CN111385354A (en) * | 2020-03-02 | 2020-07-07 | 拉扎斯网络科技（上海）有限公司 | File transmission method and device, electronic equipment and nonvolatile storage medium |
| WO2020143317A1 (en) * | 2019-01-08 | 2020-07-16 | 平安科技（深圳）有限公司 | Fragmented file verification method and terminal device |
| CN112839003A (en) * | 2019-11-22 | 2021-05-25 | 上海哔哩哔哩科技有限公司 | Data verification method and system |
| CN112866310A (en) * | 2019-11-12 | 2021-05-28 | 北京金山云网络技术有限公司 | CDN back-to-source verification method and verification server, and CDN cluster |
-
2021
- 2021-08-02 CN CN202110881360.7A patent/CN113590994A/en active Pending
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101453478A (en) * | 2007-11-28 | 2009-06-10 | 新奥特（北京）视频技术有限公司 | Integrity checking method in document transmission |
| US20130226887A1 (en) * | 2012-02-24 | 2013-08-29 | Xyratex Technology Limited | Method of, and apparatus for, improved data integrity in a networked storage system |
| CN104978253A (en) * | 2015-06-24 | 2015-10-14 | 合肥格易集成电路有限公司 | Target file checking method and apparatus |
| CN110889143A (en) * | 2018-09-07 | 2020-03-17 | 阿里巴巴集团控股有限公司 | File verification method and device |
| CN109492436A (en) * | 2018-11-29 | 2019-03-19 | 深圳市网心科技有限公司 | A kind of file verification method and relevant apparatus |
| WO2020143317A1 (en) * | 2019-01-08 | 2020-07-16 | 平安科技（深圳）有限公司 | Fragmented file verification method and terminal device |
| CN112866310A (en) * | 2019-11-12 | 2021-05-28 | 北京金山云网络技术有限公司 | CDN back-to-source verification method and verification server, and CDN cluster |
| CN112839003A (en) * | 2019-11-22 | 2021-05-25 | 上海哔哩哔哩科技有限公司 | Data verification method and system |
| CN111385354A (en) * | 2020-03-02 | 2020-07-07 | 拉扎斯网络科技（上海）有限公司 | File transmission method and device, electronic equipment and nonvolatile storage medium |
Non-Patent Citations (3)
| Title |
|---|
| GAO, XL ET AL.: "Research on GML Data Validity Verification Based on Stack Mechanism", 《2013 SIXTH INTERNATIONAL SYMPOSIUM ON COMPUTATIONAL INTELLIGENCE AND DESIGN (ISCID)》, 31 December 2013 (2013-12-31) * |
| 刘琦;: "浅谈P2P网络文件传输", 科教文汇(下旬刊), no. 01, 30 January 2016 (2016-01-30) * |
| 贺鹏程;王劲林;邓浩江;孙鹏;: "P2P文件完整性校验延迟隐藏算法", 计算机工程, no. 15 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2023093079A1 (en) * | 2021-11-26 | 2023-06-01 | 浪潮通信信息***有限公司 | Consistency check method and apparatus for distributed edge cloud edge nodes |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109309725B (en) | Application downloading and updating method and device, computer equipment and storage medium | |
| CN112187804B (en) | Communication method and device of server, computer equipment and storage medium | |
| US9632765B1 (en) | Customized application package with context specific token | |
| CN110888838B (en) | Request processing method, device, equipment and storage medium based on object storage | |
| US20120260096A1 (en) | Method and system for monitoring a secure document | |
| CN111625782B (en) | Access authority control method and device for source code, computer equipment and storage medium | |
| CN108418797B (en) | Webpage access method and device, computer equipment and storage medium | |
| CN107395784B (en) | Method and device for importing internet protocol IP pools in batches | |
| US11063922B2 (en) | Virtual content repository | |
| CN111460394A (en) | Copyright file verification method and device and computer readable storage medium | |
| CN111475376A (en) | Method and device for processing test data, computer equipment and storage medium | |
| CN111177735A (en) | Identity authentication method, device, system and equipment and storage medium | |
| US9665732B2 (en) | Secure Download from internet marketplace | |
| CN113505318A (en) | Short link generation method, device, equipment and storage medium | |
| CN113590994A (en) | Data processing method, data processing device, computer equipment and storage medium | |
| CN110597820A (en) | Block chain based information processing method and device, storage medium and equipment | |
| US9397995B2 (en) | Information processing apparatus and user authentication method | |
| CN111158606B (en) | Storage method, storage device, computer equipment and storage medium | |
| CN112788084B (en) | Application program installation package downloading method, application program installation package pushing device and computer equipment | |
| CN109766084B (en) | Customized development method and device for payment application, computer equipment and storage medium | |
| CN113806815B (en) | File signing method and system | |
| CN115935414A (en) | Block chain based data verification method and device, electronic equipment and storage medium | |
| CN115185551A (en) | Application program installation method, device, system and storage medium | |
| CN114064100A (en) | Configuration information processing method, system and device | |
| CN111552551A (en) | User management method and device based on master-slave system, computer equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |