CN113163009A - Data transmission method, device, electronic equipment and storage medium - Google Patents
Data transmission method, device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN113163009A CN113163009A CN202110426597.6A CN202110426597A CN113163009A CN 113163009 A CN113163009 A CN 113163009A CN 202110426597 A CN202110426597 A CN 202110426597A CN 113163009 A CN113163009 A CN 113163009A
- Authority
- CN
- China
- Prior art keywords
- data
- file
- preset
- standard
- processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to a data processing technology, and discloses a data transmission method, which comprises the following steps: the method comprises the steps that a data set to be transmitted is subjected to batching processing, and the batched data are uploaded to a preset message middleware to be subjected to queue generation processing, so that a standard message queue is obtained; grouping and packaging the standard message queues, and sending the packaged standard message queues to a plurality of servers for idle instance operation to obtain a standard data set; writing the standard data set into a preset file, and uploading the file to a file storage server; and performing data verification processing on the file in the file storage server, and sending the data in the file to a target database when the data verification is passed. In addition, the invention also relates to a block chain technology, and the standard message queue can be stored in a node of the block chain. The invention also provides a data transmission device, electronic equipment and a computer readable storage medium. The invention can solve the problem of low stability of data transmission.
Description
Technical Field
The present invention relates to the field of data processing, and in particular, to a data transmission method, an apparatus, an electronic device, and a computer-readable storage medium.
Background
In program development, a large amount of data is often required to complete operation in a short time, and complete data is transmitted to another independent system library within a limited time for daily staff to use in an off-line system. For example, the interest rate data of the post-loan component users in the micro-server cluster credit project system is very large, up to millions, and is increasing day by day, so that the operation based on the interest rate data is very frequent, and a large amount of obtained calculation results need to be transmitted to the collection urging system.
In the existing industry, data are transmitted independently and operation is completed, time consumption is very long, the data are transmitted to a corresponding system through a timing task calling system independent interface after the operation is completed, the stability of the timing task calling system is relied on, and if the timing task calling system fails to process, the data need to be transmitted and operated again, so that the transmission of subsequent data is influenced.
Disclosure of Invention
The invention provides a data transmission method, a data transmission device, electronic equipment and a computer readable storage medium, and mainly aims to improve the stability of data transmission.
In order to achieve the above object, a data transmission method provided by the present invention includes:
the method comprises the steps that a data set to be transmitted is subjected to batching processing, and the batched data are uploaded to a preset message middleware to be subjected to queue generation processing, so that a standard message queue is obtained;
grouping and packaging the standard message queues to obtain multiple groups of packaged data, issuing the multiple groups of packaged data to multiple servers according to a preset data transmission channel to trigger idle instance events and perform idle instance operation to obtain a standard data set;
writing the standard data set into a preset file by using an input function, and uploading the file to a file storage server;
and performing data verification processing on the file in the file storage server, and sending the data in the file to a target database when the data verification is passed.
Optionally, the packing the standard message queue packet includes:
carrying out delay processing on the data in the standard message queue by using preset delay time;
and grouping and packaging the data after the delay processing according to a preset data batch size.
Optionally, the issuing the multiple groups of packed data to multiple servers according to a preset data transmission channel to trigger an idle instance event and perform idle instance operation includes:
acquiring a data channel;
and transmitting the standard message queue to a plurality of servers through the data channel according to a preset transmission protocol to trigger idle instance events and perform idle instance operation.
Optionally, the writing the standard data set into a preset file by using an input function includes:
dividing the standard data set into a plurality of units with preset sizes;
and adding a preset character string into the head of each unit, and writing the unit added with the character string into the preset file by using the input function.
Optionally, the performing data verification processing on the file in the file storage server includes:
reading data in a file in the file storage server;
judging whether the data in the file is consistent with the data in a data list in a target database;
if the data in the file is inconsistent with the data in the data list, performing batch and queue generation processing on the data in the file;
and if the data in the file is consistent with the data in the data list, sending the data in the file to the target database.
Optionally, the batching the data set to be transmitted includes:
extracting the ID corresponding to each piece of data to be transmitted in the data set to be transmitted;
and carrying out batch processing on the data to be transmitted according to the acquisition time of the ID.
Optionally, before the batch processing of the data set to be transmitted, the method further includes:
generating an acquisition probe based on a byte code enhancement technology, and burying points in a target server by using the acquisition probe;
and collecting data information in the target server by using the acquisition probe to obtain the data set to be transmitted.
In order to solve the above problem, the present invention also provides a data transfer apparatus, including:
the queue generating module is used for batching the data set to be transmitted and uploading the batched data to a preset message middleware for queue generating processing to obtain a standard message queue;
the queue issuing module is used for grouping and packaging the standard message queues to obtain a plurality of groups of packaged data, issuing the plurality of groups of packaged data to a plurality of servers according to a preset data transmission channel to trigger idle instance events and perform idle instance operation to obtain a standard data set;
the file uploading module is used for writing the standard data set into a preset file by using an input function and uploading the file to a file storage server;
and the data sending module is used for carrying out data verification processing on the file in the file storage server and sending the data in the file to a target database when the data verification is passed.
In order to solve the above problem, the present invention also provides an electronic device, including:
a memory storing at least one instruction; and
and the processor executes the instructions stored in the memory to realize the data transmission method.
In order to solve the above problem, the present invention further provides a computer-readable storage medium, which stores at least one instruction, and the at least one instruction is executed by a processor in an electronic device to implement the data transmission method.
According to the embodiment of the invention, the batch processing is carried out on the transmission data set, and the batched data is uploaded to the message middleware for queue generation processing, so that the batch processing ensures the stability of data uploading, the condition that the message middleware is unstable due to the fact that a large data set is uploaded at one time is avoided, the packed standard message queue is sent to a plurality of servers for idle instance operation, and the plurality of servers operate at the same time, so that the time is saved. Therefore, the data transmission method and the data transmission device can solve the problem of low stability of data transmission.
Drawings
Fig. 1 is a flowchart illustrating a data transmission method according to an embodiment of the present invention;
FIG. 2 is a flow chart illustrating a step in the data transmission method shown in FIG. 1;
FIG. 3 is a functional block diagram of a data transmission apparatus according to an embodiment of the present invention;
fig. 4 is a schematic structural diagram of an electronic device implementing the data transmission method according to an embodiment of the present invention.
The implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.
Detailed Description
It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
The embodiment of the application provides a data transmission method. The execution main body of the data transmission method includes, but is not limited to, at least one of electronic devices that can be configured to execute the method provided by the embodiments of the present application, such as a server, a terminal, and the like. In other words, the data transmission method may be performed by software or hardware installed in the terminal device or the server device, and the software may be a block chain platform. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, and the like.
Fig. 1 is a schematic flow chart of a data transmission method according to an embodiment of the present invention.
In this embodiment, the data transmission method includes:
and S1, batching the data set to be transmitted, uploading the batched data to a preset message middleware for queue generation processing, and obtaining a standard message queue.
In the embodiment of the invention, the data set to be transmitted refers to a data set which needs to be calculated in a short time and transmitted to another independent system library within a limited time.
Specifically, the batch processing of the data set to be transmitted includes:
extracting the ID corresponding to each piece of data to be transmitted in the data set to be transmitted;
and carrying out batch processing on the data to be transmitted according to the acquisition time of the ID.
In detail, each piece of data to be transmitted refers to data which needs to be operated and is appointed to be transmitted to an independent system, an ID corresponding to each piece of data to be transmitted in the data set to be transmitted is used for uniquely identifying each piece of transmitted data, the time for extracting the ID is different, the time for acquiring the ID is synchronously recorded, and therefore the data to be transmitted can be directly subjected to batch processing according to the time for acquiring the ID.
Further, the batch processing the data to be transmitted according to the acquisition time of the ID includes:
acquiring a preset first time threshold and a preset second time threshold;
judging the obtaining time of the ID and the size of the first time threshold and the second time threshold;
sorting the corresponding data to be transmitted which is less than or equal to the first time threshold into a first data set;
sorting the corresponding data to be transmitted which are greater than the first time threshold and less than or equal to the second time threshold into a second data set;
and sorting the corresponding data to be transmitted which is greater than the second time threshold into a third data set, wherein the first data set, the second data set and the third data set are batched data.
The first time threshold is smaller than the second time threshold, and both the first time threshold and the second time threshold refer to time points.
In detail, a preset first time threshold and a preset second time threshold are obtained, where the first time threshold and the second time threshold refer to two time points set in advance, the first time threshold is smaller than the second time threshold, when the obtaining time of the ID is smaller than or equal to the first time threshold, the data to be transmitted corresponding to the ID is named as a first data set, when the obtaining time of the ID is greater than the first time threshold and smaller than or equal to the second time threshold, the corresponding data to be transmitted is named as a second data set, when the obtaining time of the ID is greater than the second time threshold, the corresponding data to be transmitted is named as a third data set, and at this time, the first data set, the second data set, and the third data set are batched data.
Further, the uploading the batched data to a preset message middleware to obtain a standard message queue includes:
sequencing the batched data according to the ID acquisition time to obtain a task queue;
and executing message distribution, application decoupling, asynchronous message and flow peak clipping operation on the task queue by using a preset message middleware to obtain a standard message queue.
Specifically, the batched data are sequenced according to the time of the ID acquisition time to obtain a task queue, and the preset message middleware comprises ActiveMQ, RabbitMQ, RocktMQ and Kafka. In the embodiment of the present invention, Kafka middleware is adopted, and the Kafka is an open source stream processing platform, which is written by Java and Scala, and is a high-throughput distributed message system, and can process all information stream data in the task queue. The standard message queue refers to mq (message queue) information, which adopts a first-in first-out data structure, puts data to be transmitted (data to be processed) in a queue, and uses a queue mechanism to realize message transmission. The "first-in first-out" means that the information arranged in front of the queue information is processed in advance. In the embodiment of the invention, the application decoupling means that when one message is processed by a plurality of systems or modules correspondingly, the Kafka middleware can reduce the coupling among the systems or modules. The asynchronous message refers to the asynchronous processing of computer multithreading, and the thread of the current system or module is not blocked to wait for the completion of the processing, but the subsequent system or module is allowed to operate. The flow peak clipping means that under the condition of high concurrency, the Kafka middleware uses MQ information for buffering, disperses the information in the MQ information, and processes the information within a period of time to provide peak service processing capacity.
In another embodiment of the present invention, before the batching the data set to be transmitted, the method further includes:
generating an acquisition probe based on a byte code enhancement technology, and burying points in a target server by using the acquisition probe;
and collecting data information in the target server by using the acquisition probe to obtain the data set to be transmitted.
In detail, the acquisition probe is used for embedding points into the server through a byte code enhancement technology and collecting index information of the server. The method for embedding points by using the bytecode enhancement technology does not need to modify codes, different programming languages can be directly implanted into a language operating environment or a basic library through different technologies, and data acquisition can be quickly completed without auxiliary means such as logs and the like.
The target server is a server selected at will, the same application program on the target server is deployed on different servers, the target server is used for sending data to be transmitted, and other servers except the target server are used for receiving the data after subsequent processing.
And S2, grouping and packaging the standard message queues to obtain multiple groups of packaged data, issuing the multiple groups of packaged data to multiple servers according to a preset data transmission channel to trigger idle instance events and perform idle instance operation to obtain a standard data set.
In this embodiment of the present invention, referring to fig. 2, the packet packing of the standard message queue includes:
s201, delaying the data in the standard message queue by using preset delay time;
and S202, grouping and packaging the data after the delay processing according to a preset data batch size.
Specifically, the data in the standard message queue is delayed uniformly to a preset delay time, and the data after the delay processing is packed in packets, where the data amount after the delay processing may be large, so that the data needs to be packed in packets according to a preset data batch size.
In detail, since there is a lot of data in the standard message queue, and one-time transmission may cause pressure on the server, a standard data batch needs to be set, and the data after the delay processing is packed in packets according to the data batch.
In the embodiment of the present invention, the preset delay time may be 10s, because the data in the standard message queue comes from an MQ cluster, and the data passing through the MQ cluster is asynchronous, which may cause delay and result in data not arriving according to the production time sequence, the data with a disordered sequence is prevented from being acquired, and the data of the last 10 seconds is not read, which is equivalent to processing after delaying for 10 s.
Further, the issuing the multiple groups of packed data to multiple servers according to a preset data transmission channel to trigger idle instance events and perform idle instance operation includes:
acquiring a data channel;
and transmitting the standard message queue to a plurality of servers through the data channel according to a preset transmission protocol to trigger idle instance events and perform idle instance operation.
In detail, in the embodiment of the present invention, the data channel is a dedicated communication channel, and may also be provided by a data exchange network, a telephone exchange network, or another type of exchange network, and the preset transmission protocol may be a TCP protocol, where the TCP protocol is a connection-oriented, reliable, byte stream-based transport layer communication protocol, and transmits the standard message queue to multiple servers through the data channel according to the TCP protocol, and simultaneously triggers an idle instance event and performs an idle instance operation.
The preset data operation is an operation mode selected by combining a specific application scene, for example, because the post-loan component user interest data in the micro-uniform cluster credit project is very large, the data cleaning operation is performed on the daily cutting business calculation interest and the daily cutting business calculation interest is transmitted to the collection urging system.
And S3, writing the standard data set into a preset file by using an input function, and uploading the file to a file storage server.
In this embodiment of the present invention, the writing the standard data set into a preset file includes:
dividing the standard data set into a plurality of units with preset sizes;
and adding a preset character string into the head of each unit, and writing the unit added with the character string into the preset file by using the input function.
In detail, the standard data set is divided into a plurality of units with a preset size, the preset size can be set according to actual conditions and can be 16K or 128K, the character strings are similar to character arrays in storage, and single elements of each bit can be extracted, wherein the character strings are arranged at the front ends of the units and play a role in identifying each unit, so that in the subsequent data file reading process, even if a part of units in the data file are damaged, the corresponding units can be found in a mode of searching the character strings, the preset character strings are added to the head of each unit, and the units added with the character strings are written into the preset file by using the input function, wherein the input function can be a scan function or a get function.
And the preset file storage server is an FTP server.
And S4, performing data verification processing on the file in the file storage server, and sending the data in the file to a target database when the data verification is passed.
In this embodiment of the present invention, the performing data verification processing on the file in the file storage server includes:
reading data in a file in the file storage server;
judging whether the data in the file is consistent with the data in a data list in a target database;
if the data in the file is inconsistent with the data in the data list, performing batch and queue generation processing on the data in the file;
and if the data in the file is consistent with the data in the data list, sending the data in the file to the target database.
In detail, reading data in a file in the file storage server by using a read-write function, wherein a data list in a target database is preset, judging whether the data in the file is consistent with the data in the data list in the target database, if the data in the file is inconsistent with the data in the data list, the data in the file does not reach a sending standard, performing batch and queue generation processing on the data in the file, writing in a preset file again and uploading the file to a file storage server, and if the data in the file is consistent with the data in the data list, sending the data in the file to the target database.
According to the embodiment of the invention, the batch processing is carried out on the transmission data set, and the batched data is uploaded to the message middleware for queue generation processing, so that the batch processing ensures the stability of data uploading, the condition that the message middleware is unstable due to the fact that a large data set is uploaded at one time is avoided, the packed standard message queue is sent to a plurality of servers for idle instance operation, and the plurality of servers operate at the same time, so that the time is saved. Therefore, the data transmission method provided by the invention can solve the problem of low stability of data transmission.
Fig. 3 is a functional block diagram of a data transmission apparatus according to an embodiment of the present invention.
The data transmission device 100 of the present invention may be installed in an electronic device. According to the implemented functions, the data transmission apparatus 100 may include a queue generating module 101, a queue issuing module 102, a file uploading module 103, and a data sending module 104. The module of the present invention, which may also be referred to as a unit, refers to a series of computer program segments that can be executed by a processor of an electronic device and that can perform a fixed function, and that are stored in a memory of the electronic device.
In the present embodiment, the functions regarding the respective modules/units are as follows:
the queue generating module 101 is configured to batch process a data set to be transmitted, and upload the batched data to a preset message middleware to perform queue generating processing, so as to obtain a standard message queue;
the queue issuing module 102 is configured to group and package the standard message queues to obtain multiple groups of packaged data, issue the multiple groups of packaged data to multiple servers according to a preset data transmission channel to trigger idle instance events and perform idle instance operation to obtain a standard data set;
the file uploading module 103 is configured to write the standard data set into a preset file by using an input function, and upload the file to a file storage server;
the data sending module 104 is configured to perform data verification processing on the file in the file storage server, and send the data in the file to the target database when the data verification is passed.
In detail, the modules in the data transmission apparatus 100, when executed by a processor of an electronic device, may implement a data transmission method including the following steps:
the queue generating module 101 performs batch processing on a data set to be transmitted, and uploads the batch data to a preset message middleware for queue generating processing to obtain a standard message queue.
In the embodiment of the invention, the data set to be transmitted refers to a data set which needs to be calculated in a short time and transmitted to another independent system library within a limited time.
Specifically, the batch processing of the data set to be transmitted includes:
extracting the ID corresponding to each piece of data to be transmitted in the data set to be transmitted;
and carrying out batch processing on the data to be transmitted according to the acquisition time of the ID.
In detail, each piece of data to be transmitted refers to data which needs to be operated and is appointed to be transmitted to an independent system, an ID corresponding to each piece of data to be transmitted in the data set to be transmitted is used for uniquely identifying each piece of transmitted data, the time for extracting the ID is different, the time for acquiring the ID is synchronously recorded, and therefore the data to be transmitted can be directly subjected to batch processing according to the time for acquiring the ID.
Further, the batch processing the data to be transmitted according to the acquisition time of the ID includes:
acquiring a preset first time threshold and a preset second time threshold;
buzhou；
sorting the corresponding data to be transmitted which is less than or equal to the first time threshold into a first data set;
sorting the corresponding data to be transmitted which are greater than the first time threshold and less than or equal to the second time threshold into a second data set;
and sorting the corresponding data to be transmitted which is greater than the second time threshold into a third data set, wherein the first data set, the second data set and the third data set are batched data.
The first time threshold is smaller than the second time threshold, and both the first time threshold and the second time threshold refer to time points.
In detail, a preset first time threshold and a preset second time threshold are obtained, where the first time threshold and the second time threshold refer to two time points set in advance, the first time threshold is smaller than the second time threshold, when the obtaining time of the ID is smaller than or equal to the first time threshold, the data to be transmitted corresponding to the ID is named as a first data set, when the obtaining time of the ID is greater than the first time threshold and smaller than or equal to the second time threshold, the corresponding data to be transmitted is named as a second data set, when the obtaining time of the ID is greater than the second time threshold, the corresponding data to be transmitted is named as a third data set, and at this time, the first data set, the second data set, and the third data set are batched data.
Further, the uploading the batched data to a preset message middleware to obtain a standard message queue includes:
sequencing the batched data according to the ID acquisition time to obtain a task queue;
and executing message distribution, application decoupling, asynchronous message and flow peak clipping operation on the task queue by using a preset message middleware to obtain a standard message queue.
Specifically, the batched data are sequenced according to the time of the ID acquisition time to obtain a task queue, and the preset message middleware comprises ActiveMQ, RabbitMQ, RocktMQ and Kafka. In the embodiment of the present invention, Kafka middleware is adopted, and the Kafka is an open source stream processing platform, which is written by Java and Scala, and is a high-throughput distributed message system, and can process all information stream data in the task queue. The standard message queue refers to mq (message queue) information, which adopts a first-in first-out data structure, puts data to be transmitted (data to be processed) in a queue, and uses a queue mechanism to realize message transmission. The "first-in first-out" means that the information arranged in front of the queue information is processed in advance. In the embodiment of the invention, the application decoupling means that when one message is processed by a plurality of systems or modules correspondingly, the Kafka middleware can reduce the coupling among the systems or modules. The asynchronous message refers to the asynchronous processing of computer multithreading, and the thread of the current system or module is not blocked to wait for the completion of the processing, but the subsequent system or module is allowed to operate. The flow peak clipping means that under the condition of high concurrency, the Kafka middleware uses MQ information for buffering, disperses the information in the MQ information, and processes the information within a period of time to provide peak service processing capacity.
In another embodiment of the present invention, before the batching the data set to be transmitted, the method further includes:
generating an acquisition probe based on a byte code enhancement technology, and burying points in a target server by using the acquisition probe;
and collecting data information in the target server by using the acquisition probe to obtain the data set to be transmitted.
In detail, the acquisition probe is used for embedding points into the server through a byte code enhancement technology and collecting index information of the server. The method for embedding points by using the bytecode enhancement technology does not need to modify codes, different programming languages can be directly implanted into a language operating environment or a basic library through different technologies, and data acquisition can be quickly completed without auxiliary means such as logs and the like.
The target server is a server selected at will, the same application program on the target server is deployed on different servers, the target server is used for sending data to be transmitted, and other servers except the target server are used for receiving the data after subsequent processing.
And step two, the queue issuing module 102 groups and packages the standard message queues to obtain a plurality of groups of packaged data, issues the plurality of groups of packaged data to a plurality of servers according to a preset data transmission channel to trigger idle instance events and perform idle instance operation to obtain a standard data set.
In this embodiment of the present invention, the grouping and packaging the standard message queues includes:
carrying out delay processing on the data in the standard message queue by using preset delay time;
and grouping and packaging the data after the delay processing according to a preset data batch size.
Specifically, the data in the standard message queue is delayed uniformly to a preset delay time, and the data after the delay processing is packed in packets, where the data amount after the delay processing may be large, so that the data needs to be packed in packets according to a preset data batch size.
In detail, since there is a lot of data in the standard message queue, and one-time transmission may cause pressure on the server, a standard data batch needs to be set, and the data after the delay processing is packed in packets according to the data batch.
In the embodiment of the present invention, the preset delay time may be 10s, because the data in the standard message queue comes from an MQ cluster, and the data passing through the MQ cluster is asynchronous, which may cause delay and result in data not arriving according to the production time sequence, the data with a disordered sequence is prevented from being acquired, and the data of the last 10 seconds is not read, which is equivalent to processing after delaying for 10 s.
Further, the issuing the multiple groups of packed data to multiple servers according to a preset data transmission channel to trigger idle instance events and perform idle instance operation includes:
acquiring a data channel;
and transmitting the standard message queue to a plurality of servers through the data channel according to a preset transmission protocol to trigger idle instance events and perform idle instance operation.
In detail, in the embodiment of the present invention, the data channel is a dedicated communication channel, and may also be provided by a data exchange network, a telephone exchange network, or another type of exchange network, and the preset transmission protocol may be a TCP protocol, where the TCP protocol is a connection-oriented, reliable, byte stream-based transport layer communication protocol, and transmits the standard message queue to multiple servers through the data channel according to the TCP protocol, and simultaneously triggers an idle instance event and performs an idle instance operation.
The preset data operation is an operation mode selected by combining a specific application scene, for example, because the post-loan component user interest data in the micro-uniform cluster credit project is very large, the data cleaning operation is performed on the daily cutting business calculation interest and the daily cutting business calculation interest is transmitted to the collection urging system.
And thirdly, writing the standard data set into a preset file by using the file uploading module 103 with an input function, and uploading the file to a file storage server.
In this embodiment of the present invention, the writing the standard data set into a preset file includes:
dividing the standard data set into a plurality of units with preset sizes;
and adding a preset character string into the head of each unit, and writing the unit added with the character string into the preset file by using the input function.
In detail, the standard data set is divided into a plurality of units with a preset size, the preset size can be set according to actual conditions and can be 16K or 128K, the character strings are similar to character arrays in storage, and single elements of each bit can be extracted, wherein the character strings are arranged at the front ends of the units and play a role in identifying each unit, so that in the subsequent data file reading process, even if a part of units in the data file are damaged, the corresponding units can be found in a mode of searching the character strings, the preset character strings are added to the head of each unit, and the units added with the character strings are written into the preset file by using the input function, wherein the input function can be a scan function or a get function.
And the preset file storage server is an FTP server.
And fourthly, the data sending module 104 performs data verification processing on the file in the file storage server, and sends the data in the file to a target database when the data verification is passed.
In this embodiment of the present invention, the performing data verification processing on the file in the file storage server includes:
reading data in a file in the file storage server;
judging whether the data in the file is consistent with the data in a data list in a target database;
if the data in the file is inconsistent with the data in the data list, performing batch and queue generation processing on the data in the file;
and if the data in the file is consistent with the data in the data list, sending the data in the file to the target database.
In detail, reading data in a file in the file storage server by using a read-write function, wherein a data list in a target database is preset, judging whether the data in the file is consistent with the data in the data list in the target database, if the data in the file is inconsistent with the data in the data list, the data in the file does not reach a sending standard, performing batch and queue generation processing on the data in the file, writing in a preset file again and uploading the file to a file storage server, and if the data in the file is consistent with the data in the data list, sending the data in the file to the target database.
Fig. 4 is a schematic structural diagram of an electronic device implementing a data transmission method according to an embodiment of the present invention.
The electronic device 1 may comprise a processor 10, a memory 11 and a bus, and may further comprise a computer program, such as a data transfer program 12, stored in the memory 11 and executable on the processor 10.
The memory 11 includes at least one type of readable storage medium, which includes flash memory, removable hard disk, multimedia card, card-type memory (e.g., SD or DX memory, etc.), magnetic memory, magnetic disk, optical disk, etc. The memory 11 may in some embodiments be an internal storage unit of the electronic device 1, such as a removable hard disk of the electronic device 1. The memory 11 may also be an external storage device of the electronic device 1 in other embodiments, such as a plug-in mobile hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like, which are provided on the electronic device 1. Further, the memory 11 may also include both an internal storage unit and an external storage device of the electronic device 1. The memory 11 may be used not only to store application software installed in the electronic device 1 and various types of data, such as codes of the data transfer program 12, but also to temporarily store data that has been output or is to be output.
The processor 10 may be composed of an integrated circuit in some embodiments, for example, a single packaged integrated circuit, or may be composed of a plurality of integrated circuits packaged with the same or different functions, including one or more Central Processing Units (CPUs), microprocessors, digital Processing chips, graphics processors, and combinations of various control chips. The processor 10 is a Control Unit (Control Unit) of the electronic device, connects various components of the electronic device by using various interfaces and lines, and executes various functions and processes data of the electronic device 1 by running or executing programs or modules (e.g., data transfer programs, etc.) stored in the memory 11 and calling data stored in the memory 11.
The bus may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is arranged to enable connection communication between the memory 11 and at least one processor 10 or the like.
Fig. 4 only shows an electronic device with components, and it will be understood by those skilled in the art that the structure shown in fig. 4 does not constitute a limitation of the electronic device 1, and may comprise fewer or more components than those shown, or some components may be combined, or a different arrangement of components.
For example, although not shown, the electronic device 1 may further include a power supply (such as a battery) for supplying power to each component, and preferably, the power supply may be logically connected to the at least one processor 10 through a power management device, so as to implement functions of charge management, discharge management, power consumption management, and the like through the power management device. The power supply may also include any component of one or more dc or ac power sources, recharging devices, power failure detection circuitry, power converters or inverters, power status indicators, and the like. The electronic device 1 may further include various sensors, a bluetooth module, a Wi-Fi module, and the like, which are not described herein again.
Further, the electronic device 1 may further include a network interface, and optionally, the network interface may include a wired interface and/or a wireless interface (such as a WI-FI interface, a bluetooth interface, etc.), which are generally used for establishing a communication connection between the electronic device 1 and other electronic devices.
Optionally, the electronic device 1 may further comprise a user interface, which may be a Display (Display), an input unit (such as a Keyboard), and optionally a standard wired interface, a wireless interface. Alternatively, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, or the like. The display, which may also be referred to as a display screen or display unit, is suitable for displaying information processed in the electronic device 1 and for displaying a visualized user interface, among other things.
It is to be understood that the described embodiments are for purposes of illustration only and that the scope of the appended claims is not limited to such structures.
The data transfer program 12 stored in the memory 11 of the electronic device 1 is a combination of instructions that, when executed in the processor 10, may implement:
the method comprises the steps that a data set to be transmitted is subjected to batching processing, and the batched data are uploaded to a preset message middleware to be subjected to queue generation processing, so that a standard message queue is obtained;
grouping and packaging the standard message queues to obtain multiple groups of packaged data, issuing the multiple groups of packaged data to multiple servers according to a preset data transmission channel to trigger idle instance events and perform idle instance operation to obtain a standard data set;
writing the standard data set into a preset file by using an input function, and uploading the file to a file storage server;
and performing data verification processing on the file in the file storage server, and sending the data in the file to a target database when the data verification is passed.
Specifically, the specific implementation method of the processor 10 for the instruction may refer to the description of the relevant steps in the embodiment corresponding to fig. 1, which is not described herein again.
Further, the integrated modules/units of the electronic device 1, if implemented in the form of software functional units and sold or used as separate products, may be stored in a computer readable storage medium. The computer readable storage medium may be volatile or non-volatile. For example, the computer-readable medium may include: any entity or device capable of carrying said computer program code, recording medium, U-disk, removable hard disk, magnetic disk, optical disk, computer Memory, Read-Only Memory (ROM).
The present invention also provides a computer-readable storage medium, storing a computer program which, when executed by a processor of an electronic device, may implement:
the method comprises the steps that a data set to be transmitted is subjected to batching processing, and the batched data are uploaded to a preset message middleware to be subjected to queue generation processing, so that a standard message queue is obtained;
grouping and packaging the standard message queues to obtain multiple groups of packaged data, issuing the multiple groups of packaged data to multiple servers according to a preset data transmission channel to trigger idle instance events and perform idle instance operation to obtain a standard data set;
writing the standard data set into a preset file by using an input function, and uploading the file to a file storage server;
and performing data verification processing on the file in the file storage server, and sending the data in the file to a target database when the data verification is passed.
In the embodiments provided in the present invention, it should be understood that the disclosed apparatus, device and method can be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the modules is only one logical functional division, and other divisions may be realized in practice.
The modules described as separate parts may or may not be physically separate, and parts displayed as modules may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment.
In addition, functional modules in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, or in a form of hardware plus a software functional module.
It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential attributes thereof.
The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference signs in the claims shall not be construed as limiting the claim concerned.
The block chain is a novel application mode of computer technologies such as distributed data storage, point-to-point transmission, a consensus mechanism, an encryption algorithm and the like. A block chain (Blockchain), which is essentially a decentralized database, is a series of data blocks associated by using a cryptographic method, and each data block contains information of a batch of network transactions, so as to verify the validity (anti-counterfeiting) of the information and generate a next block. The blockchain may include a blockchain underlying platform, a platform product service layer, an application service layer, and the like.
Furthermore, it is obvious that the word "comprising" does not exclude other elements or steps, and the singular does not exclude the plural. A plurality of units or means recited in the system claims may also be implemented by one unit or means in software or hardware. The terms second, etc. are used to denote names, but not any particular order.
Finally, it should be noted that the above embodiments are only for illustrating the technical solutions of the present invention and not for limiting, and although the present invention is described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that modifications or equivalent substitutions may be made on the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.
Claims (10)
1. A method of data transfer, the method comprising:
the method comprises the steps that a data set to be transmitted is subjected to batching processing, and the batched data are uploaded to a preset message middleware to be subjected to queue generation processing, so that a standard message queue is obtained;
grouping and packaging the standard message queues to obtain multiple groups of packaged data, issuing the multiple groups of packaged data to multiple servers according to a preset data transmission channel to trigger idle instance events and perform idle instance operation to obtain a standard data set;
writing the standard data set into a preset file by using an input function, and uploading the file to a file storage server;
and performing data verification processing on the file in the file storage server, and sending the data in the file to a target database when the data verification is passed.
2. The data transfer method of claim 1, wherein said packing the standard message queue packets comprises:
carrying out delay processing on the data in the standard message queue by using preset delay time;
and grouping and packaging the data after the delay processing according to a preset data batch size.
3. The data transmission method of claim 1, wherein the sending the plurality of groups of packed data to a plurality of servers according to a preset data transmission channel to trigger idle instance events and perform idle instance operations comprises:
acquiring a data channel;
and transmitting the standard message queue to a plurality of servers through the data channel according to a preset transmission protocol to trigger idle instance events and perform idle instance operation.
4. The data transfer method of claim 1, wherein the writing the standard data set to a preset file using an input function comprises:
dividing the standard data set into a plurality of units with preset sizes;
and adding a preset character string into the head of each unit, and writing the unit added with the character string into the preset file by using the input function.
5. The data transfer method of claim 1, wherein the performing data verification processing on the file in the file storage server comprises:
reading data in a file in the file storage server;
judging whether the data in the file is consistent with the data in a data list in a target database;
if the data in the file is inconsistent with the data in the data list, performing batch and queue generation processing on the data in the file;
and if the data in the file is consistent with the data in the data list, sending the data in the file to the target database.
6. The data transfer method of claim 1, wherein batching the set of data to be transferred comprises:
extracting the ID corresponding to each piece of data to be transmitted in the data set to be transmitted;
and carrying out batch processing on the data to be transmitted according to the acquisition time of the ID.
7. The data transfer method of any one of claims 1 to 6, wherein prior to batching the data set to be transferred, the method further comprises:
generating an acquisition probe based on a byte code enhancement technology, and burying points in a target server by using the acquisition probe;
and collecting data information in the target server by using the acquisition probe to obtain the data set to be transmitted.
8. A data transfer apparatus, characterized in that the apparatus comprises:
the queue generating module is used for batching the data set to be transmitted and uploading the batched data to a preset message middleware for queue generating processing to obtain a standard message queue;
the queue issuing module is used for grouping and packaging the standard message queues to obtain a plurality of groups of packaged data, issuing the plurality of groups of packaged data to a plurality of servers according to a preset data transmission channel to trigger idle instance events and perform idle instance operation to obtain a standard data set;
the file uploading module is used for writing the standard data set into a preset file by using an input function and uploading the file to a file storage server;
and the data sending module is used for carrying out data verification processing on the file in the file storage server and sending the data in the file to a target database when the data verification is passed.
9. An electronic device, characterized in that the electronic device comprises:
at least one processor; and the number of the first and second groups,
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform a method of data transfer according to any one of claims 1 to 7.
10. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out a data transfer method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110426597.6A CN113163009A (en) | 2021-04-20 | 2021-04-20 | Data transmission method, device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110426597.6A CN113163009A (en) | 2021-04-20 | 2021-04-20 | Data transmission method, device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113163009A true CN113163009A (en) | 2021-07-23 |
Family
ID=76867613
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110426597.6A Pending CN113163009A (en) | 2021-04-20 | 2021-04-20 | Data transmission method, device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113163009A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114039981A (en) * | 2021-11-11 | 2022-02-11 | 中国建设银行股份有限公司 | Message processing method, device, server and storage medium |
| CN114296910A (en) * | 2021-12-02 | 2022-04-08 | 武汉物易云通网络科技有限公司 | Multi-partition queue sequential consumption method and device |
| CN115243318A (en) * | 2022-07-01 | 2022-10-25 | 华迪计算机集团有限公司 | Internet of things data transparent transmission method and system |
| CN116028247A (en) * | 2023-03-27 | 2023-04-28 | 中新宽维传媒科技有限公司 | File acquisition system based on message middleware |
| CN116302616A (en) * | 2023-03-28 | 2023-06-23 | 之江实验室 | Data processing method and device, storage medium and electronic equipment |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103605479A (en) * | 2013-10-16 | 2014-02-26 | 北京奇虎科技有限公司 | Data file writing method and system and data file reading method and system |
| CN109684099A (en) * | 2018-11-23 | 2019-04-26 | 新华三大数据技术有限公司 | Message treatment method and device |
| CN112000492A (en) * | 2020-08-24 | 2020-11-27 | 中国银行股份有限公司 | Management system and method for public number user label |
| CN112631806A (en) * | 2020-12-28 | 2021-04-09 | 平安银行股份有限公司 | Asynchronous message arranging and scheduling method and device, electronic equipment and storage medium |
-
2021
- 2021-04-20 CN CN202110426597.6A patent/CN113163009A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103605479A (en) * | 2013-10-16 | 2014-02-26 | 北京奇虎科技有限公司 | Data file writing method and system and data file reading method and system |
| CN109684099A (en) * | 2018-11-23 | 2019-04-26 | 新华三大数据技术有限公司 | Message treatment method and device |
| CN112000492A (en) * | 2020-08-24 | 2020-11-27 | 中国银行股份有限公司 | Management system and method for public number user label |
| CN112631806A (en) * | 2020-12-28 | 2021-04-09 | 平安银行股份有限公司 | Asynchronous message arranging and scheduling method and device, electronic equipment and storage medium |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114039981A (en) * | 2021-11-11 | 2022-02-11 | 中国建设银行股份有限公司 | Message processing method, device, server and storage medium |
| CN114039981B (en) * | 2021-11-11 | 2024-04-02 | 中国建设银行股份有限公司 | Message processing method, device, server and storage medium |
| CN114296910A (en) * | 2021-12-02 | 2022-04-08 | 武汉物易云通网络科技有限公司 | Multi-partition queue sequential consumption method and device |
| CN114296910B (en) * | 2021-12-02 | 2022-11-22 | 武汉物易云通网络科技有限公司 | Multi-partition queue sequential consumption method and device |
| CN115243318A (en) * | 2022-07-01 | 2022-10-25 | 华迪计算机集团有限公司 | Internet of things data transparent transmission method and system |
| CN116028247A (en) * | 2023-03-27 | 2023-04-28 | 中新宽维传媒科技有限公司 | File acquisition system based on message middleware |
| CN116028247B (en) * | 2023-03-27 | 2023-06-13 | 中新宽维传媒科技有限公司 | File acquisition system based on message middleware |
| CN116302616A (en) * | 2023-03-28 | 2023-06-23 | 之江实验室 | Data processing method and device, storage medium and electronic equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113163009A (en) | Data transmission method, device, electronic equipment and storage medium | |
| WO2021189826A1 (en) | Message generation method and apparatus, electronic device, and computer-readable storage medium | |
| CN112653760B (en) | Cross-server file transmission method and device, electronic equipment and storage medium | |
| CN112631806B (en) | Asynchronous message arrangement and scheduling method and device, electronic equipment and storage medium | |
| CN112115143A (en) | Automatic data updating and synchronizing method and device, electronic equipment and storage medium | |
| CN114185895A (en) | Data import and export method and device, electronic equipment and storage medium | |
| CN112702228A (en) | Service current limiting response method and device, electronic equipment and readable storage medium | |
| CN113890712A (en) | Data transmission method and device, electronic equipment and readable storage medium | |
| CN112732567A (en) | Mock data testing method and device based on ip, electronic equipment and storage medium | |
| CN113282854A (en) | Data request response method and device, electronic equipment and storage medium | |
| CN115129753A (en) | Data blood relationship analysis method and device, electronic equipment and storage medium | |
| CN116755637B (en) | Transaction data storage method, device, equipment and medium | |
| CN114185776A (en) | Big data point burying method, device, equipment and medium for application program | |
| CN113051224A (en) | File transmission method and device, electronic equipment and computer readable storage medium | |
| CN114697316B (en) | Batch downloading method, device and equipment of data and computer readable medium | |
| CN114598731B (en) | Cluster log acquisition method, device, equipment and storage medium | |
| CN114201466B (en) | Anti-cache breakdown method, device, equipment and readable storage medium | |
| CN113347451B (en) | Video uploading method and device, electronic equipment and computer readable storage medium | |
| CN115002011A (en) | Flow bidirectional test method and device, electronic equipment and storage medium | |
| CN114625712A (en) | Data synchronization method and device, electronic equipment and storage medium | |
| CN115145870A (en) | Method and device for positioning reason of failed task, electronic equipment and storage medium | |
| CN114185588A (en) | Incremental package generation method, device, equipment and storage medium | |
| CN112596717B (en) | Multi-data packing method, device, electronic equipment and computer readable storage medium | |
| CN115964307B (en) | Automatic test method, device, equipment and medium for transaction data | |
| CN113452785B (en) | Service access method and device based on offline resources, electronic equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210723 |