CN111586187A - Data transmission method, device, application server and data transmission system - Google Patents
Data transmission method, device, application server and data transmission system Download PDFInfo
- Publication number
- CN111586187A CN111586187A CN202010397620.9A CN202010397620A CN111586187A CN 111586187 A CN111586187 A CN 111586187A CN 202010397620 A CN202010397620 A CN 202010397620A CN 111586187 A CN111586187 A CN 111586187A
- Authority
- CN
- China
- Prior art keywords
- file
- transmission
- transmitted
- data
- server
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000005540 biological transmission Effects 0.000 title claims abstract description 336
- 238000000034 method Methods 0.000 title claims abstract description 45
- 238000012360 testing method Methods 0.000 claims abstract description 52
- 238000003860 storage Methods 0.000 claims description 50
- 238000012795 verification Methods 0.000 claims description 15
- 238000012545 processing Methods 0.000 abstract description 14
- 230000032683 aging Effects 0.000 abstract description 3
- 239000004065 semiconductor Substances 0.000 description 8
- 238000010586 diagram Methods 0.000 description 7
- 238000012546 transfer Methods 0.000 description 6
- 230000008569 process Effects 0.000 description 5
- 230000008859 change Effects 0.000 description 2
- 238000004140 cleaning Methods 0.000 description 2
- 230000001915 proofreading effect Effects 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 230000005856 abnormality Effects 0.000 description 1
- 238000013475 authorization Methods 0.000 description 1
- 230000000977 initiatory effect Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 238000013508 migration Methods 0.000 description 1
- 230000005012 migration Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention provides a data transmission method, a data transmission device, an application server and a data transmission system, and relates to the technical field of data processing. According to the data transmission method, the data transmission device, the application server and the data transmission system, after the configuration information of the test transmission site is obtained, the transmission list is generated based on the transmission rule included in the configuration information, the target file is obtained according to the file path included in the transmission list, the obtained target file is verified according to the file attribute information included in the transmission list, the file to be transmitted is obtained, and then the file to be transmitted is transmitted to the remote server. So, alright in order replacing the manual work, realize carrying out the automatic transmission of data, guarantee data transmission's ageing and accuracy, reduced manual handling data transmission's cost to a great extent.
Description
Technical Field
The invention relates to the technical field of data processing, in particular to a data transmission method, a data transmission device, an application server and a data transmission system.
Background
In recent years, with the expansion of the semiconductor industry, more and more data are generated in semiconductor testing, and more data are required to be transmitted. At present, data transmission is mostly carried out manually, so that data is manually selected and then transmitted. However, as the amount of data increases, the amount of manual data throwing increases, which means that the investment of manpower increases, but the processing efficiency is low and the error rate is high.
Disclosure of Invention
Based on the above research, the present invention provides a data transmission method, an apparatus, an application server, and a data transmission system, so as to improve the above problems.
Embodiments of the invention may be implemented as follows:
in a first aspect, an embodiment of the present invention provides a data transmission method, which is applied to an application server, where the method includes:
acquiring configuration information of a test transmission site;
generating a transmission list based on the transmission rule included in the configuration information; the transmission list comprises file attribute information and a file path;
acquiring a target file according to a file path included in the transmission list, and verifying the acquired target file according to file attribute information included in the transmission list to obtain a file to be transmitted;
and transmitting the file to be transmitted to a remote server.
In an optional embodiment, the configuration information further includes transmission address information, and before transmitting the file to be transmitted to a remote server, the method further includes:
judging whether the remote server is connected or not;
if not, connecting with the remote server for at most N times according to the transmission address information, and transmitting the file to be transmitted to the remote server after the connection is successful;
and if the connection times exceed N times, marking the file to be transmitted with a set identifier.
In an optional embodiment, the step of transmitting the file to be transmitted to a remote server includes:
judging whether the file to be transmitted is transmitted completely;
if not, acquiring the transmission times of the file to be transmitted, and judging whether the transmission times of the file to be transmitted is greater than a first preset threshold value or not;
if the transmission frequency of the file to be transmitted is not greater than the first preset threshold value, retransmitting the file to be transmitted, and updating the transmission frequency of the file to be transmitted;
and if the file size is larger than the first preset threshold value, marking the file to be transmitted with a set identifier.
In an optional embodiment, after the file to be transmitted is completely transmitted, the method further includes:
according to the file attribute information included in the transmission list, the file attribute information of the file to be transmitted to the remote server is corrected;
if the verification fails, judging whether the transmission times of the file to be transmitted are larger than a second preset threshold value or not;
if the transmission frequency of the file to be transmitted is not greater than the second preset threshold value, retransmitting the file to be transmitted, and updating the transmission frequency of the file to be transmitted;
if the file size is larger than the second preset threshold value, marking the file to be transmitted with a set identifier;
and if the verification is successful, transferring the file attribute information and the file path of the file to be transmitted in the transmission list to a historical data table.
In an optional implementation manner, the step of obtaining the target file according to the file path included in the transmission list includes:
if the file is stored in the application server, searching the application server to obtain the target file according to the file path included in the transmission list;
if the file is stored in a storage server, after the connection with the storage server is established, searching the target file in the storage server according to a file path included in the transmission list, and judging whether the target file is in a transmittable state;
if the target file is in a non-transferable state, judging whether the searching times of the target file is greater than a third preset threshold value;
if the target file is larger than the third preset threshold, marking a set downloading identifier of the target file; if the target file is not larger than the third preset threshold, searching the target file in the storage server again, and judging whether the target file is in a transmittable state or not;
and if the target file is in a transmittable state, downloading the target file to the application server.
In an optional embodiment, the step of verifying the obtained target file according to the file attribute information included in the transmission list includes:
judging whether the obtained file attribute information of the target file is consistent with the file attribute information included in the transmission list or not;
if the two are consistent, the verification is successful;
if not, setting the target file by a set identifier; the file attribute information includes a file name, a size, and a hash value of the file.
In an alternative embodiment, the transmission rule includes a path field and a data type field; the step of generating a transmission list based on the transmission rule included in the configuration information includes:
generating a data path according to the path field, and searching a folder under the data path according to the data path;
acquiring a folder matched with the data type field according to the data type field;
and obtaining the file path based on the file attribute information of the file in the folder and the data path, and generating the transmission list based on the file attribute information of the file and the file path.
In a second aspect, an embodiment of the present invention provides a data transmission device, which is applied to an application server, where the data transmission device includes an information acquisition module, a list generation module, a file verification module, and a file transmission module;
the information acquisition module is used for acquiring configuration information of the test transmission site;
the list generation module is used for generating a transmission list based on the transmission rule included in the configuration information; the transmission list comprises file attribute information and a file path;
the file checking module is used for acquiring a target file according to a file path included in the transmission list and checking the acquired target file according to file attribute information included in the transmission list to obtain a file to be transmitted;
and the file transmission module is used for transmitting the file to be transmitted to a remote application server.
In a third aspect, an embodiment of the present invention provides an application server, where the application server is configured to:
acquiring configuration information of a test transmission site;
generating a transmission list based on the transmission rule included in the configuration information; the transmission list comprises file attribute information and a file path;
acquiring a target file according to a file path included in the transmission list, and verifying the acquired target file according to file attribute information included in the transmission list to obtain a file to be transmitted;
and transmitting the file to be transmitted to a remote server.
In a fourth aspect, an embodiment of the present invention provides a data transmission system, including a database server, a remote server, and the application server in the foregoing embodiment;
the application server is used for acquiring configuration information of the test transmission site from the database server and generating a transmission list based on a transmission rule included in the configuration information; the transmission list comprises file attribute information and a file path;
the application server is used for acquiring a target file according to the file path included in the transmission list, verifying the acquired target file according to the file attribute information included in the transmission list to obtain a file to be transmitted, and transmitting the file to be transmitted to the remote server.
According to the data transmission method, the data transmission device, the application server and the data transmission system, after the configuration information of the test transmission site is obtained, the transmission list is generated based on the transmission rule included in the configuration information, the target file is obtained according to the file path included in the transmission list, the obtained target file is verified according to the file attribute information included in the transmission list, the file to be transmitted is obtained, and then the file to be transmitted is transmitted to the remote server. So, alright in order replacing the manual work, realize carrying out the automatic transmission of data, guarantee data transmission's ageing and accuracy, reduced manual handling data transmission's cost to a great extent.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the embodiments will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present invention and therefore should not be considered as limiting the scope, and for those skilled in the art, other related drawings can be obtained according to the drawings without inventive efforts.
Fig. 1 is a schematic structural diagram of a data transmission system according to an embodiment of the present invention.
Fig. 2 is a functional diagram of an application server according to an embodiment of the present invention.
Fig. 3 is a flowchart illustrating a data transmission method according to an embodiment of the present invention.
Fig. 4 is a flow chart illustrating sub-steps of a data transmission method according to an embodiment of the present invention.
Fig. 5 is a second flow chart illustrating sub-steps of the data transmission method according to the embodiment of the present invention.
Fig. 6 is a block diagram of a data transmission apparatus according to an embodiment of the present invention.
Icon: 100-a data transmission system; 10-an application server; 11-data transmission means; 111-an information acquisition module; 112-list generation module; 113-file check module; 114-file transfer module; 20-a database server; 30-a remote server; 40-a storage server; 50-MES server.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. The components of embodiments of the present invention generally described and illustrated in the figures herein may be arranged and designed in a wide variety of different configurations.
Thus, the following detailed description of the embodiments of the present invention, presented in the figures, is not intended to limit the scope of the invention, as claimed, but is merely representative of selected embodiments of the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined and explained in subsequent figures.
In the description of the present invention, it should be noted that if the terms "upper", "lower", "inside", "outside", etc. indicate an orientation or a positional relationship based on that shown in the drawings or that the product of the present invention is used as it is, this is only for convenience of description and simplification of the description, and it does not indicate or imply that the device or the element referred to must have a specific orientation, be constructed in a specific orientation, and be operated, and thus should not be construed as limiting the present invention.
Furthermore, the appearances of the terms "first," "second," and the like, if any, are used solely to distinguish one from another and are not to be construed as indicating or implying relative importance.
It should be noted that the features of the embodiments of the present invention may be combined with each other without conflict.
As described in the background, as the semiconductor industry expands, more and more data is generated by semiconductor testing, and more data needs to be transmitted. At present, software on the market often only supports a data transmission mode of a File Transfer Protocol (FTP) or a Secure File Transfer Protocol (SFTP), and mainly uses FTP transmission, so that the software rarely has SFTP data transmission capability and only has single SFTP transmission capability, but still needs to manually select data to be sent, and the data needs to be transmitted one by one, so that the efficiency is low, and the transmission error rate is high under the condition of large data volume. After the transfer is completed, the integrity of the file is often judged manually, for example, whether the file is stored on a client server (remote server), whether the size of the transferred file is consistent, and the like.
In addition, currently, the deployment platform of the mainstream data transmission software is limited to a specific operating environment, and based on this, technicians with relevant capabilities and relevant server equipment are required, which results in high equipment cost.
Based on the above research, the present embodiment provides a data transmission system to improve the above problems.
Referring to fig. 1, fig. 1 is a block diagram of a data transmission system 100 provided in the present embodiment, the data transmission system 100 provided in the present embodiment includes an application server 10, a database server 20, and a remote server 30, and in addition, the data transmission system 100 provided in the present embodiment may further include a storage server 40, a Manufacturing Execution System (MES) server 50, and the like.
As shown in fig. 1, the application server 10 provided in the present embodiment is communicatively connected to the database server 20, the remote server 30, and the storage server 40, and the database server 20 is communicatively connected to the MES server 50.
The MES server 50 is configured to record configuration information of the test transport sites of the semiconductor product in the test link, and the MES server 50 stores the configuration information of the test transport sites in the database server 20, so that the application server 10 can read the configuration information of the test transport sites from the database server 20. The configuration information of the semiconductor product at the test transmission site in the test link can be transmitted to the MES server 50 for recording by the operator through the client.
The storage server 40 is used for storing specific test data of the semiconductor product in a test link, such as test programs specified by MES, test time, materials used in the test, and the like. In a particular use scenario, these specific test data may be entered into the test equipment by a worker, and after the test is completed, automatically uploaded by the test equipment to the storage server 40 for storage, so that the application server 10 reads the data therefrom and transmits the data to the remote server 30.
In order to facilitate the application server 10 to read data from the storage server 40 and to facilitate uploading of data of the test equipment, in this embodiment, before each test equipment performs a test, each test equipment further records information such as a serial number, an ID, and a product lot. After a certain testing device is completed, the storage server 40 may generate a folder based on the information such as the number, ID, and product lot recorded on the testing device, and then store the file of the specific testing data measured by the testing device in the corresponding folder.
It should be noted that the information such as the code, the ID, and the product lot recorded in each test device may be different, and the generated folders are different according to the information such as the code, the ID, and the product lot.
The application server 10 is configured to obtain configuration information of the test transmission site from the database server 20, generate a transmission list based on a transmission rule included in the configuration information, obtain a target file from the storage server 40 or a local storage server according to a file path included in the transmission list, verify the obtained target file according to file attribute information included in the transmission list, obtain a file to be transmitted, and transmit the file to be transmitted to the remote server 30.
Optionally, in this embodiment, the storage server 40 and the application server 10 may be integrally configured, that is, may be deployed in the same device, or may be separately configured, so that the application server 10 provided in this embodiment may read data from the storage server 40 or locally.
In this embodiment, the application server 10 may be a single physical server, or may be a server group including a plurality of physical servers for executing different data processing functions. In some possible embodiments, such as where the application server 10 employs a single physical server, the physical server may be assigned different logical server components based on different service functions.
In order to implement the data transmission method described in this embodiment, the application server 10 provided in this embodiment is configured with multiple threads, as shown in fig. 2, the application server is configured with a HangFire main thread, and under the HangFire main thread, the application server 10 is further hung with an asynchronous log storage thread, a polling scan thread, a file check thread, a data processing thread, and a file transmission thread. The asynchronous log storage threads are used for storing logs of other threads, and log storage processes of the threads are not interfered with one another. Through the analysis and processing of the thread log, a user can quickly locate error information, trace and process the information, and the resource cost is saved.
And a polling scanning thread for acquiring the configuration information of the test transmission station from the database server 20 and generating a transmission list based on the configuration information. And the file checking thread is used for checking the local file, and checking the file transmitted to the remote server. And the data processing thread is used for processing the file, and comprises the processing of the number rule of file downloading and transmission, file cleaning, information migration in a transmission list, log cache cleaning and the like. The file transmission thread is used for file transmission of the remote server 30 and file downloading in the storage server 40, and is also used for recording the progress of file transmission in real time.
In addition, a control information thread is configured in the application server 10 for controlling the operations of other threads. If the user needs to change the control information parameters, the user can change the parameters in the database server 20, and then the control thread reads the control parameters from the database server 20 at regular time to perform data synchronization, thereby ensuring the continuity of real-time transmission.
The file transfer thread, the data processing thread, the file verification thread, and the polling scan thread configured in the application server 10 are respectively 4 independent threads. For a plurality of remote servers 30, the file transmission thread, the data processing thread, the file verification thread and the polling scanning thread can be synchronously performed, and the number of synchronous transmission and verification can be configured in a self-defined mode. For the same remote server 30, the file transmission to the remote server 30 and the file proofreading to the remote server 30 are performed at the same time, and only one operation is allowed, so that high delay and abnormality of the data network link can be avoided to a certain extent.
Based on the above configured threads, the application server 10 provided in the present embodiment can provide functions of control information configuration, presentation of real-time transmission information, presentation of history transmission information, log analysis, and manual analysis processing, as shown in fig. 2.
The application server 10 provided in this embodiment is not only applicable to a mainstream deployment manner, that is, may be deployed in the same device as the storage server 40, or may be deployed separately, but also applicable to a mainstream application system (for example, a cross-platform, a Liunx or Windows system, etc.). In this way, servers for a particular system may not need to be configured, while also reducing the need for human handling capacity, and therefore, reducing labor and equipment costs.
Referring to fig. 3, based on the schematic diagram of the data transmission system 100 shown in fig. 1 and the configuration diagram of the application server 10 shown in fig. 2, fig. 3 is a schematic flow chart of the data transmission method provided in the present embodiment. The data transmission method provided by the present embodiment is applied to the application server 10 in fig. 1, and the application server 10 executes the data transmission method provided by the present embodiment. The flow diagram shown in fig. 3 is explained in detail below.
Step S10: and acquiring configuration information of the test transmission station.
Step S20: a transmission list is generated based on the transmission rules included in the configuration information.
The generated transmission list comprises file attribute information and a file path.
Step S30: and acquiring a target file according to the file path included in the transmission list, and verifying the acquired target file according to the file attribute information included in the transmission list to obtain the file to be transmitted.
Step S40: and transmitting the file to be transmitted to a remote server.
In the data transmission method provided in this embodiment, after the configuration information of the test transmission site is obtained, a transmission list is generated based on the transmission rule included in the configuration information, a target file is obtained according to a file path included in the transmission list, the obtained target file is verified according to file attribute information included in the transmission list, a file to be transmitted is obtained, and then the file to be transmitted is transmitted to the remote server. Therefore, the method can replace manual work, realize automatic transmission of data, ensure the timeliness and accuracy of data transmission and reduce the cost of manual data transmission.
In a test link of a semiconductor product, there are many test items, such as an appearance test, a conductivity test, and the like, which are how many test items are configured for the test transmission site described in this embodiment, and how many test transmission sites are configured for each test transmission site, and the configuration information includes a transmission rule, and based on the transmission rule, a transmission list can be generated.
Optionally, in this embodiment, the transmission rule includes a path field and a data type field, and the step of generating the transmission list based on the transmission rule included in the configuration information may be:
firstly: and generating a data path according to the path field, and searching a folder under the data path according to the data path.
Secondly, the method comprises the following steps: and acquiring the folder matched with the data type field according to the data type field.
And finally: and obtaining a file path based on the file attribute information and the data path of the file in the folder, and generating a transmission list based on the file attribute information and the file path of the file.
The path field provided by this embodiment may include fields such as a number, an ID, and a product lot. And after the application server acquires the configuration information of the test transmission site from the database server, generating a data path according to a path field included in the transmission rule in the configuration information. For example, the data path generated based on fields such as number, ID, and product lot may be number \ ID \ product lot.
After the data path is generated, the folders under the data path can be searched in the storage server or locally according to the data path, and after the folders under the data path are found, the folders matched with the data type fields are selected based on the data type fields included in the transmission rule, for example, if the data type fields are x.
And after the matched folder is obtained, obtaining a file path based on the file name and the data path included in the file attribute information of the file in the folder. For example, the file name is A, the data path is number \ ID \ product batch, and the file path is number \ ID \ product batch \ A.
After the file path is obtained, a transmission list can be generated based on the file attribute information of the file and the file path, that is, the generated transmission list includes the file attribute information and the file path of the file.
As an optional implementation manner, the configuration information provided in this embodiment may further include configuration information such as a transmission time and a transmission identifier of the file. Therefore, the transmission list provided by this embodiment may further include transmission time and a transmission identifier of the file, and the data transmission method provided by this embodiment may further transmit the file to be transmitted to the remote server based on the transmission time and the transmission identifier.
Optionally, in this embodiment, the transmission time of the file may be calculated based on the current time, and a preset time period is added to the current time, so that the transmission time is obtained, that is, the transmission time is equal to the current time plus the preset time period, where the preset time period may be set arbitrarily. For example, after the file to be transmitted is obtained, if the current time is 8 points and the preset time period is 1 hour, the file to be transmitted needs to be transmitted until 9 points. Therefore, the automatic timing transmission of the files can be realized, the transmission time does not need to be monitored manually, and the cost of manual processing is greatly reduced.
Optionally, in this embodiment, the transmission identifier of the file included in the transmission list may be an identifier of whether the file is transmitted, and if the transmission identifier of the file to be transmitted is the identifier of the file to be transmitted, the file to be transmitted is transmitted after the file to be transmitted is obtained. If the file is the identifier which is not transmitted, the file is not transmitted for the moment, and the file to be transmitted is transmitted after the transmission identifier of the file is changed into the identifier which is transmitted.
After the transmission list is generated, the corresponding target file can be obtained based on the file path in the transmission list. In this embodiment, the storage server and the application server may be deployed in the same device, or may be separately deployed. Therefore, in consideration of different deployment manners, in this embodiment, the step of obtaining the target file according to the file path included in the transmission list may include:
if the file is stored in the application server, searching and obtaining a target file in the application server according to a file path included in the transmission list;
if the file is stored in the storage server, after the connection with the storage server is established, searching a target file in the storage server according to a file path included in the transmission list, and judging whether the target file is in a transmittable state;
if the target file is in a non-transferable state, judging whether the searching times of the target file is greater than a third preset threshold value;
if the download identifier is larger than the third preset threshold, marking the set download identifier of the target file; if the target file is not larger than the third preset threshold, searching the target file in the storage server again, and judging whether the target file is in a transmittable state or not;
and if the target file is in a transmittable state, downloading the target file to the application server.
When the storage server and the application server are integrally deployed, namely the file is stored in the application server, the application server directly performs local scanning and searching according to the file path included in the transmission list to obtain the target file.
As an optional implementation manner, when the application server provided in this embodiment is a server group formed by a plurality of physical servers, and the server group adopts a sharing manner, local scanning and searching may be directly performed according to a file path included in the transmission list, so as to obtain the target file.
When the storage server and the application server are separately deployed, namely the file is stored in the storage server, after the connection with the storage server is required to be established, the target file is searched in the storage server according to the file path included in the transmission list.
Optionally, in this embodiment, the configuration information may further include transmission address information of the storage server, and therefore, the application server may establish a connection with the storage server based on the transmission address information of the storage server.
In practical application, when the storage server and the application server are separately deployed, if a file on the storage server is in an untransmittable state, that is, the file is in an occupied state, for example, the file is opened, moved, and stored, the file cannot be downloaded by the application server on the storage server, and at this time, the file in the storage server needs to be waited to release the occupied state, and the file is downloaded manually, so that labor cost is high, and it is also impossible to know in time when the file is released, and time cost is high. Therefore, when the target file is obtained, it is necessary to determine whether the target file is in a transferable state.
If the target file is in the non-transferable state, judging whether the searching times of the target file is greater than a third preset threshold value, if the searching times of the target file is not greater than the third preset threshold value, scanning the storage database again to search the target file, updating the searching times of the target file, adding 1 to the original searching times, if the target file searched again is still in the non-transferable state, continuously judging whether the searching times of the target file searched again is greater than the third preset threshold value, and if the searching times of the target file searched again is greater than the third preset threshold value, marking the target file by setting a downloading identifier. And if the searched target file is in a transmittable state, downloading the searched target to the application server.
Optionally, in this embodiment, the set download identifier may be any distinguishing identifier, and may be marked with any character, for example, a, 1, a.
It should be noted that, when the application server marks a certain target file with a set download identifier, the target file needs to be marked with the set download identifier in the transmission list.
Optionally, in this embodiment, the third preset threshold may be set according to an actual requirement, for example, may be set for 3 times, 5 times, 10 times, and the like, and this embodiment is not limited specifically.
According to the data transmission method provided by the embodiment, when the file of the storage server is in the non-transmittable state, the transmission state and the transmission times of the file are searched and judged through multiple scanning, the file can be downloaded in time when the file in the storage server is unoccupied, the time cost can be effectively reduced, manual downloading can be replaced, and the labor cost is reduced.
In practical applications, when the file data volume is large, an error transmission may occur. In order to ensure the accuracy of file transmission and avoid file mistransmission, after the target file is obtained, the obtained target file also needs to be verified in the embodiment.
In an optional embodiment, the step of verifying the obtained target file according to the file attribute information included in the transmission list may include the following steps:
firstly: judging whether the obtained file attribute information of the target file is consistent with the file attribute information included in the transmission list or not;
if the two are consistent, the verification is successful; if not, setting the target file by a set identifier;
the file attribute information includes a file name, a size, and a hash value of the file. And judging whether the obtained file attribute information of the target is consistent with the file attribute information included in the transmission list, namely judging whether the file name, the size and the hash value of the obtained target file are consistent with the file name, the size and the hash value corresponding to the target file in the transmission list. If the file is consistent with the target file, the target file is successfully verified, the file is correctly downloaded, and the target file is the file to be transmitted. If the file downloading is inconsistent with the target file downloading, the file downloading is wrong, and the target file is set by the set identification.
Optionally, in this embodiment, the setting identifier may be any identifier, and may also be the same as the setting download identifier. When the target file is marked by the set mark, the application server does not process the target file and can transfer to manual processing. When the application server marks a certain target file with a set identifier, the target file needs to be marked with the set identifier in the transmission list.
As an optional implementation manner, in this embodiment, after the target file is successfully verified, the target file may be marked with an identifier that is successfully verified, and the target file that is successfully verified, that is, the file to be transmitted, is transmitted according to the identifier that is successfully verified. The mark which is successfully verified is different from the set downloading mark and the set mark.
According to the embodiment, the target file is verified, so that the condition of file mistransmission can be effectively avoided, and the accuracy of file transmission is ensured.
In order to ensure normal transmission of a file, before transmitting the file to be transmitted to a remote server, the data transmission method provided in this embodiment further includes the following steps:
and judging whether the remote server is connected or not.
And if the connection is not established, connecting the remote server for at most N times according to the transmission address information, and transmitting the file to be transmitted to the remote server after the connection is successful.
And if the connection times exceed N times, marking the file to be transmitted with a set identifier.
As an alternative implementation manner, in this embodiment, the application server may determine whether to connect with the remote server based on initiating a connection request conforming to a transmission protocol (e.g., FTP transmission protocol) to the remote server. Specifically, the application server initiates a connection request to the remote server, and if the connection request is received by the remote server, the remote server feeds back information to the application server, and after receiving a message fed back by the remote server, the application server determines to be connected with the remote server and transmits the file to be transmitted to the remote server. And if the application server does not receive the message fed back by the remote server, judging that the application server is not connected with the remote server.
And when the connection with the remote server is judged not to be connected, the application server establishes connection with the remote server according to the transmission address information for establishing connection with the remote server in the configuration information.
The transmission address information for establishing connection with the remote server includes information (such as an account number and a password) such as an address of the remote server, a port number, and authorization information of a protocol.
In this embodiment, the connection between the application server and the remote server follows a connection rule of at most N times, when the application server is connected to the remote server based on the address information, the application server may determine whether the number of times of connection with the remote server is less than or equal to N times, if the number of times of connection is less than or equal to N times, establish connection with the remote server, if the connection is successful, transmit the file, if the connection is failed, add 1 to the original number of times of connection, and determine whether the increased number of times of connection is less than or equal to N times, if the increased number of times of connection exceeds N times, mark the file to be transmitted with a set identifier.
Optionally, in this embodiment, N may be set according to an actual requirement, and this embodiment is not particularly limited.
According to the embodiment, the connection rule of the remote server and the application server is set, so that the remote server and the application server follow a connection mode for at most N times, on one hand, connection is not required to be established manually, the transmission operation of the file is set, on the other hand, the file to be transmitted can be transmitted to the remote server in time when the remote server is connected with the application server, and the real-time performance of data transmission is guaranteed.
In the actual file transmission process, the application server may be disconnected from the remote server due to network fluctuation, so that file transmission fails, manual retransmission is required, and it cannot be guaranteed that the file is transmitted in time when the application server is reconnected with the remote server. In order to ensure normal transmission of data and improve timeliness of data transmission, please refer to fig. 4 in combination, in this embodiment, the step of transmitting the file to be transmitted to the remote server includes:
step S41: and judging whether the file to be transmitted is transmitted completely.
If not, go to step S42; if the transmission is completed, step S50 is executed.
Step S42: acquiring the transmission times of the file to be transmitted, and judging whether the transmission times of the file to be transmitted is greater than a first preset threshold value.
If not, go to step S43. If the threshold value is greater than the first preset threshold value, step S44 is executed.
Step S43: and retransmitting the file to be transmitted, and updating the transmission times of the file to be transmitted.
Step S44: and marking the file to be transmitted with a set identifier.
After the transmission of the file to be transmitted is completed, a completion identifier is returned, so that the application server can judge whether the transmission of the file to be transmitted is completed according to the completion identifier. And when the application server is disconnected with the remote server and does not receive the completion identifier, the file to be transmitted is not completed.
When the file to be transmitted is not transmitted, acquiring the transmission times of the file to be transmitted, judging whether the transmission times of the file to be transmitted is greater than a first preset threshold value or not, if not, retransmitting the file to be transmitted, and updating the transmission times of the file to be transmitted. And if the number of the files to be transmitted is larger than the first preset threshold value, marking the files to be transmitted with the set marks.
Optionally, the first preset threshold may be set according to actual requirements, for example, 3 times, 5 times, 10 times, 100 times, and the like, and the embodiment is not particularly limited.
As an optional implementation manner, in the data transmission method provided in this embodiment, when the file to be transmitted is transmitted to the remote server, the transmission progress of the file to be transmitted may also be recorded, and the progress is updated once every time the file is transmitted to the preset data amount.
According to the embodiment, the first preset threshold value is set, when the transmission times is not greater than the first preset threshold value, if the transmission of the file to be transmitted fails, the file to be transmitted can be automatically transmitted again, so that manual transmission is not needed, the file to be transmitted can be transmitted in time when the application server and the remote server are reconnected, and the timeliness of data transmission is improved.
Because the application server is disconnected from the remote server due to network fluctuation in the actual file transmission process, the file transmission is also incomplete, and therefore, in this embodiment, after the file to be transmitted is completely transmitted, please refer to fig. 5 in combination, the method provided in this embodiment further includes the following steps:
step S50: and checking the file attribute information of the file to be transmitted to the remote server according to the file attribute information included in the transmission list.
If the verification fails, go to step S51; if the verification is successful, go to step S52.
Step S51: and judging whether the transmission times of the file to be transmitted is greater than a second preset threshold value.
If not, go to step S53, otherwise go to step S54.
Step S53: and retransmitting the file to be transmitted, and updating the transmission times of the file to be transmitted.
Step S54: and marking the file to be transmitted with a set identifier.
Step S52: and migrating the file attribute information and the file path of the file to be transmitted in the transmission list to a historical data table.
The file attribute information includes a file name, a size, and a hash value of the file. When the file attribute information of the file to be transmitted to the remote server is corrected according to the file attribute information included in the transmission list, the file attribute information of the file to be transmitted which is transmitted to the remote server is compared with the corresponding file attribute information in the transmission list, whether the file name, the size and the hash value of the file are consistent or not is judged, if so, the verification is successful, the file transmission is complete and successful, and the file attribute information and the file path of the file to be transmitted in the transmission list are transferred to the historical data table.
If the transmission times of the files to be transmitted are not greater than the second preset threshold, the files to be transmitted are retransmitted, the transmission times of the files to be transmitted are updated, and if the transmission times of the files to be transmitted are greater than the second preset threshold, the files to be transmitted are marked by the set marks for manual supplementary transmission.
It should be noted that the historical data table is a data table generated separately, and when the file transmission is completed and the remote file is successfully collated, all information related to the file, such as file attribute information, file path, transmission identifier, and the like of the file in the transmission list, is migrated to the historical data table, and all information related to the file in the transmission list is deleted.
As an optional implementation manner, when the application server provided in this embodiment is separately deployed from the storage server, in order to reduce the local storage space of the application server, after the file transmission is completed and the remote file proofreading is successful, the file downloaded from the storage server and stored in the local storage of the application server is deleted, so as to release the resource space of the application server.
Optionally, in this embodiment, the first preset threshold may be the same as or different from the second preset threshold, and the second preset threshold may also be set in a self-defined manner according to actual requirements, which is not limited in this embodiment.
According to the data transmission method provided by the embodiment, after transmission is completed, the file is checked, and transmission is performed again under the condition that the checking is unsuccessful, so that the integrity of file transmission can be ensured.
The data transmission method provided by the embodiment replaces manual work to realize automatic transmission of data, avoids the problems of low timeliness of data transmission, low accuracy of data transmission and high time cost of data transmission caused by manual transmission, and simultaneously corrects the integrity of a file after transmission is completed, so that the integrity and reliability of the transmitted data can be ensured.
Based on the same inventive concept, please refer to fig. 6 in this embodiment, the present embodiment further provides a data transmission apparatus 11, which is applied to the application server 10, and the data transmission apparatus 11 includes an information obtaining module 111, a list generating module 112, a file verifying module 113, and a file transmitting module 114.
The information obtaining module 111 is configured to obtain configuration information of the test transmission site;
the list generating module 112 is configured to generate a transmission list based on the transmission rule included in the configuration information; the transmission list includes file attribute information and a file path.
The file checking module 113 is configured to obtain a target file according to a file path included in the transmission list, and check the obtained target file according to file attribute information included in the transmission list to obtain a file to be transmitted.
The file transmission module 114 is used for transmitting the file to be transmitted to the remote server.
On the basis, the present embodiment further provides an application server, where the application server is configured to:
and acquiring configuration information of the test transmission station.
Generating a transmission list based on the transmission rule included in the configuration information; the transmission list includes file attribute information and a file path.
And acquiring a target file according to the file path included in the transmission list, and verifying the acquired target file according to the file attribute information included in the transmission list to obtain the file to be transmitted.
And transmitting the file to be transmitted to a remote server.
It can be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working processes of the data transmission device and the application server described above may refer to the corresponding processes in the foregoing method, and will not be described in detail herein.
In summary, according to the data transmission method, the data transmission apparatus, the application server, and the data transmission system provided in this embodiment, after the configuration information of the test transmission site is obtained, the transmission list is generated based on the transmission rule included in the configuration information, the target file is obtained according to the file path included in the transmission list, the obtained target file is verified according to the file attribute information included in the transmission list, the file to be transmitted is obtained, and then the file to be transmitted is transmitted to the remote server. So, alright in order replacing the manual work, realize carrying out the automatic transmission of data, guarantee data transmission's ageing and accuracy, reduced manual handling data transmission's cost to a great extent.
The above description is only for the specific embodiment of the present invention, but the scope of the present invention is not limited thereto, and any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope of the present invention are included in the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the appended claims.
Claims (10)
1. A data transmission method is applied to an application server, and the method comprises the following steps:
acquiring configuration information of a test transmission site;
generating a transmission list based on the transmission rule included in the configuration information; the transmission list comprises file attribute information and a file path;
acquiring a target file according to a file path included in the transmission list, and verifying the acquired target file according to file attribute information included in the transmission list to obtain a file to be transmitted;
and transmitting the file to be transmitted to a remote server.
2. The data transmission method according to claim 1, wherein the configuration information further includes transmission address information, and before transmitting the file to be transmitted to a remote server, the method further comprises:
judging whether the remote server is connected or not;
if not, connecting with the remote server for at most N times according to the transmission address information, and transmitting the file to be transmitted to the remote server after the connection is successful;
and if the connection times exceed N times, marking the file to be transmitted with a set identifier.
3. The data transmission method according to claim 1, wherein the step of transmitting the file to be transmitted to a remote server comprises:
judging whether the file to be transmitted is transmitted completely;
if not, acquiring the transmission times of the file to be transmitted, and judging whether the transmission times of the file to be transmitted is greater than a first preset threshold value or not;
if the transmission frequency of the file to be transmitted is not greater than the first preset threshold value, retransmitting the file to be transmitted, and updating the transmission frequency of the file to be transmitted;
and if the file size is larger than the first preset threshold value, marking the file to be transmitted with a set identifier.
4. The data transmission method according to claim 3, wherein after the file to be transmitted is completely transmitted, the method further comprises:
according to the file attribute information included in the transmission list, the file attribute information of the file to be transmitted to the remote server is corrected;
if the verification fails, judging whether the transmission times of the file to be transmitted are larger than a second preset threshold value or not;
if the transmission frequency of the file to be transmitted is not greater than the second preset threshold value, retransmitting the file to be transmitted, and updating the transmission frequency of the file to be transmitted;
if the file size is larger than the second preset threshold value, marking the file to be transmitted with a set identifier;
and if the verification is successful, transferring the file attribute information and the file path of the file to be transmitted in the transmission list to a historical data table.
5. The data transmission method according to claim 1, wherein the step of obtaining the target file according to the file path included in the transmission list includes:
if the file is stored in the application server, searching the application server to obtain the target file according to the file path included in the transmission list;
if the file is stored in a storage server, after the connection with the storage server is established, searching the target file in the storage server according to a file path included in the transmission list, and judging whether the target file is in a transmittable state;
if the target file is in a non-transferable state, judging whether the searching times of the target file is greater than a third preset threshold value;
if the target file is larger than the third preset threshold, marking a set downloading identifier of the target file; if the target file is not larger than the third preset threshold, searching the target file in the storage server again, and judging whether the target file is in a transmittable state or not;
and if the target file is in a transmittable state, downloading the target file to the application server.
6. The data transmission method according to claim 1, wherein the step of verifying the acquired target file according to the file attribute information included in the transmission list includes:
judging whether the obtained file attribute information of the target file is consistent with the file attribute information included in the transmission list or not;
if the two are consistent, the verification is successful;
if not, setting the target file by a set identifier; the file attribute information includes a file name, a size, and a hash value of the file.
7. The data transmission method according to claim 1, wherein the transmission rule includes a path field and a data type field; the step of generating a transmission list based on the transmission rule included in the configuration information includes:
generating a data path according to the path field, and searching a folder under the data path according to the data path;
acquiring a folder matched with the data type field according to the data type field;
and obtaining the file path based on the file attribute information of the file in the folder and the data path, and generating the transmission list based on the file attribute information of the file and the file path.
8. A data transmission device is characterized by being applied to an application server and comprising an information acquisition module, a list generation module, a file verification module and a file transmission module;
the information acquisition module is used for acquiring configuration information of the test transmission site;
the list generation module is used for generating a transmission list based on the transmission rule included in the configuration information; the transmission list comprises file attribute information and a file path;
the file checking module is used for acquiring a target file according to a file path included in the transmission list and checking the acquired target file according to file attribute information included in the transmission list to obtain a file to be transmitted;
and the file transmission module is used for transmitting the file to be transmitted to a remote application server.
9. An application server, wherein the application server is configured to:
acquiring configuration information of a test transmission site;
generating a transmission list based on the transmission rule included in the configuration information; the transmission list comprises file attribute information and a file path;
acquiring a target file according to a file path included in the transmission list, and verifying the acquired target file according to file attribute information included in the transmission list to obtain a file to be transmitted;
and transmitting the file to be transmitted to a remote server.
10. A data transmission system comprising a database server, a remote server and an application server as claimed in claim 9;
the application server is used for acquiring configuration information of the test transmission site from the database server and generating a transmission list based on a transmission rule included in the configuration information; the transmission list comprises file attribute information and a file path;
the application server is used for acquiring a target file according to the file path included in the transmission list, verifying the acquired target file according to the file attribute information included in the transmission list to obtain a file to be transmitted, and transmitting the file to be transmitted to the remote server.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010397620.9A CN111586187A (en) | 2020-05-12 | 2020-05-12 | Data transmission method, device, application server and data transmission system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010397620.9A CN111586187A (en) | 2020-05-12 | 2020-05-12 | Data transmission method, device, application server and data transmission system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111586187A true CN111586187A (en) | 2020-08-25 |
Family
ID=72126639
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010397620.9A Pending CN111586187A (en) | 2020-05-12 | 2020-05-12 | Data transmission method, device, application server and data transmission system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111586187A (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103793749A (en) * | 2014-02-27 | 2014-05-14 | 四川物联恒安安全技术研究有限公司 | Safety production comprehensive supervisory platform data transmission method |
| US20140208420A1 (en) * | 2013-01-23 | 2014-07-24 | Owl Computing Technologies, Inc. | System for remotely monitoring status information of devices connected to a network |
| CN106445951A (en) * | 2015-08-07 | 2017-02-22 | 中兴通讯股份有限公司 | File transmission method and apparatus |
| CN109040252A (en) * | 2018-08-07 | 2018-12-18 | 平安科技（深圳）有限公司 | Document transmission method, system, computer equipment and storage medium |
| CN109146272A (en) * | 2018-08-08 | 2019-01-04 | 甬矽电子(宁波)股份有限公司 | Product management method and management of product device |
| CN109474606A (en) * | 2018-12-04 | 2019-03-15 | 平安科技（深圳）有限公司 | Document transmission method, device, computer equipment and storage medium |
-
2020
- 2020-05-12 CN CN202010397620.9A patent/CN111586187A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140208420A1 (en) * | 2013-01-23 | 2014-07-24 | Owl Computing Technologies, Inc. | System for remotely monitoring status information of devices connected to a network |
| CN103793749A (en) * | 2014-02-27 | 2014-05-14 | 四川物联恒安安全技术研究有限公司 | Safety production comprehensive supervisory platform data transmission method |
| CN106445951A (en) * | 2015-08-07 | 2017-02-22 | 中兴通讯股份有限公司 | File transmission method and apparatus |
| CN109040252A (en) * | 2018-08-07 | 2018-12-18 | 平安科技（深圳）有限公司 | Document transmission method, system, computer equipment and storage medium |
| CN109146272A (en) * | 2018-08-08 | 2019-01-04 | 甬矽电子(宁波)股份有限公司 | Product management method and management of product device |
| CN109474606A (en) * | 2018-12-04 | 2019-03-15 | 平安科技（深圳）有限公司 | Document transmission method, device, computer equipment and storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 温少扬: ""基于STUN和云平台的点对点文件传输"", 《中国优秀硕士论文全文数据库》 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107688530B (en) | Software testing method and device | |
| US6941361B1 (en) | Method and apparatus for obtaining inventory and user information for a remote computer device | |
| CN104468843A (en) | File uploading method and device | |
| CN106681924A (en) | Software testing method and software testing system | |
| CN108052474B (en) | Data format conversion method, device, equipment and computer readable storage medium | |
| CN112272243B (en) | Address allocation method | |
| CN112737856A (en) | Link tracking method and device, storage medium and electronic device | |
| CN112600703B (en) | Network equipment remote access fault positioning method and device | |
| CN112015540B (en) | Program management method, device, system and readable storage medium | |
| CN111586187A (en) | Data transmission method, device, application server and data transmission system | |
| CN114490746B (en) | Equipment testing method and device, equipment to be tested and storage medium | |
| EP4177735A1 (en) | Novel portable program upgrading method | |
| CN105407003A (en) | Data loading and unloading method for 1553 bus | |
| CN114520749A (en) | Modbus data monitoring method and system based on cloud platform deployment object model | |
| CN111966394B (en) | ETL-based data analysis method, device, equipment and storage medium | |
| CN112380129B (en) | Debugging method, device, equipment and medium suitable for butt joint proxy system | |
| CN110012109B (en) | Method for establishing engineering information capable of realizing high accuracy | |
| JPH10133916A (en) | Problem analysis system of computer system and medium wherein problem analyzing tool of computer system is recorded | |
| CN110365627B (en) | Application program synchronization method and device, computing equipment and storage medium | |
| CN115372803B (en) | Motherboard test system, method, device and storage medium | |
| CN116886691B (en) | File management control method, device, system and storage medium | |
| CN117591159A (en) | 5G core network software upgrading method and device and computer storage medium | |
| CN114328260A (en) | Automatic testing method and system | |
| CN111162974B (en) | Configurable two-out-of-two hardware platform aging test system and test method | |
| CN116303065A (en) | Firewall upgrading automatic test system, device and method and computer storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200825 |