CN112270600A - Multi-source data processing method, system and related device - Google Patents
Multi-source data processing method, system and related device Download PDFInfo
- Publication number
- CN112270600A CN112270600A CN202011180781.9A CN202011180781A CN112270600A CN 112270600 A CN112270600 A CN 112270600A CN 202011180781 A CN202011180781 A CN 202011180781A CN 112270600 A CN112270600 A CN 112270600A
- Authority
- CN
- China
- Prior art keywords
- data
- transaction
- target
- queue
- service
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q40/00—Finance; Insurance; Tax strategies; Processing of corporate or income taxes
- G06Q40/04—Trading; Exchange, e.g. stocks, commodities, derivatives or currency exchange
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/217—Database tuning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2457—Query processing with adaptation to user needs
- G06F16/24578—Query processing with adaptation to user needs using ranking
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- Business, Economics & Management (AREA)
- Accounting & Taxation (AREA)
- Finance (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Development Economics (AREA)
- Economics (AREA)
- Marketing (AREA)
- Strategic Management (AREA)
- Technology Law (AREA)
- General Business, Economics & Management (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a method, a system and a related device for processing multi-source data, wherein the method comprises the steps of collecting a plurality of transaction data corresponding to a plurality of target business system ends; preprocessing the transaction data, and storing the preprocessed transaction data in a preset structured table data queue; and acquiring a service requirement, searching target transaction data matched with the service requirement from the preset structured table data queue, and sending the target transaction data to a corresponding target service system end for data processing. The invention aims to acquire the data of each transaction system to one queue in a unified and standard manner, acquire the data of the transaction systems from the same queue in real time aiming at diversified business requirements and expansion, isolate the respective transaction systems, and not increase the load pressure and the statistical pressure of the database of the transaction systems, thereby improving the data processing efficiency, reducing the development and maintenance difficulty and facilitating the users.
Description
Technical Field
The invention belongs to the technical field of data processing, and particularly relates to a method, a system and a related device for processing multi-source data.
Background
With the gradual migration of the traditional POS card swiping business to mobile payment, online payment or sign-up payment, some additional business functions along with payment orders are also brought forward while transaction payment is completed, for example, a docking merchant needs to acquire the order state in real time to check accounts, a cloud voice broadcasting device needs to broadcast payment information after payment, a cloud printer needs to print payment certificate information after payment, and the order data of a plurality of payment platforms need to be collected and analyzed.
In the traditional processing scheme, independent system development needs to be carried out on each service type and the service type is connected to a database corresponding to a payment platform, so that great bearing pressure and data operation pressure are caused on the payment system database. Moreover, as the business scenes in which payment order details need to be associated gradually extend, the uncontrollable load pressure on the database is gradually increased, so that the business scenes needed by real-time order data are more and more, but the storage of the order data is more dispersed, the business modes of the order are diversified, the data processing efficiency of the related technology is low, the maintenance and development difficulty is increased, and the cost is increased.
Therefore, how to improve the processing efficiency of multi-source data and further reduce the cost is a technical problem that needs to be solved by those skilled in the art.
Accordingly, the prior art is yet to be developed and improved.
Disclosure of Invention
In view of the foregoing, it is necessary to provide a method, a system and a related apparatus for processing multi-source data.
A multi-source data processing method comprises the following steps:
collecting a plurality of transaction data corresponding to a plurality of target service system ends;
preprocessing the transaction data, and storing the preprocessed transaction data in a preset structured table data queue;
and acquiring a service requirement, searching target transaction data matched with the service requirement from the preset structured table data queue, and sending the target transaction data to a corresponding target service system end for data processing.
The multi-source data processing method includes the following specific steps that the acquisition of a plurality of transaction data corresponding to a plurality of target business system ends includes:
determining a plurality of target transaction system ends corresponding to a plurality of target service system ends;
and inquiring databases of respective target transaction system ends, and extracting corresponding transaction data belonging to the target service system end from the databases of the respective target transaction system ends.
The processing method of the multi-source data further comprises the following steps:
monitoring whether data files stored in databases of respective target transaction system ends are changed;
if the situation that the data files stored in the database of at least one target transaction system end are changed is monitored, the changed data files are obtained, and the changed data files are preprocessed;
and storing the preprocessed changed data file in a preset structured data queue.
The multi-source data processing method includes the specific steps of monitoring whether the data files stored in the databases of the target transaction system ends change or not:
starting the canal service of each target transaction system end;
and monitoring Mysql binlog binary log files in the databases of the respective target transaction system ends to determine whether the data files stored in the respective databases are changed or not through the respective Mysql binlog binary log files.
The multi-source data processing method includes the steps of preprocessing the transaction data, and storing the preprocessed transaction data in a preset structured table data queue:
acquiring a preset structured table data queue; the structured table data queue is configured with a standard format;
respectively converting the transaction data into unified standard format data by adopting a parallel processing mode through canal service;
and storing the converted data in the standard format in a preset structured table data queue.
The multi-source data processing method is characterized in that the standard format is a json format.
The multi-source data processing method includes the steps of obtaining business requirements, searching target transaction data matched with the business requirements from the preset structured table data queue, and sending the target transaction data to a corresponding target business system end for data processing, wherein the step of obtaining business requirements specifically includes:
analyzing the acquired service requirement, and determining the transaction identification information of a target service system end corresponding to the service requirement;
determining whether target transaction data identical to the transaction identification information exists in the structured table data queue or not based on the transaction identification information;
if target transaction data identical to the transaction identification information exists, the target transaction data are sent to a target service system end corresponding to the transaction identification information;
and carrying out data processing on the target transaction data through the target service system end.
The application also provides a processing system of multisource data, the processing system of multisource data includes a plurality of business system ends, preliminary treatment server end and a plurality of transaction system ends, a plurality of business system ends with a plurality of transaction system ends all with the preliminary treatment server end is connected, the processing system of multisource data is used for realizing the step of the processing method of multisource data.
The present application further provides a server cluster, which includes:
a memory and a processor; the memory is used for storing a computer program, and the processor is used for realizing the steps of the multi-source data processing method when executing the computer program.
The application also provides a computer temporary storage medium, which stores one or more programs, and the programs are executed by a processor to realize the steps in the processing method of the multi-source data.
The embodiment of the invention has the following beneficial effects:
the invention discloses a method, a system and a related device for processing multi-source data, wherein the method for processing the multi-source data comprises the steps of collecting a plurality of transaction data corresponding to a plurality of target business system ends; preprocessing the transaction data, and storing the preprocessed transaction data in a preset structured table data queue; and acquiring a service requirement, searching target transaction data matched with the service requirement from the preset structured table data queue, and sending the target transaction data to a corresponding target service system end for data processing. The invention aims to acquire the data of each transaction system to one queue in a unified and standard manner, acquire the data of the transaction systems from the same queue in real time aiming at diversified business requirements and expansion, isolate the respective transaction systems, and not increase the load pressure and the statistical pressure of the database of the transaction systems, thereby improving the data processing efficiency, reducing the development and maintenance difficulty and facilitating the users.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
Wherein:
fig. 1 is a block diagram of a processing system for multi-source data according to the present invention.
Fig. 2 is a flowchart of an embodiment of a multi-source data processing method according to the present invention.
Fig. 3 is a block diagram of a server cluster according to the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Referring to fig. 1, fig. 1 illustrates a block diagram of a processing system for multi-source data according to the present invention, as shown in fig. 1, the processing system for multi-source data includes a plurality of business system terminals, a preprocessing server terminal, and a plurality of transaction system terminals, and the business system terminals and the transaction system terminals are connected to the preprocessing server terminal.
Specifically, the business system side refers to an object that needs to acquire transaction data, i.e., a target object. When a plurality of target objects exist, the user ID is used to distinguish different target objects. For example, the service system end may be an ordering service system end, a takeout service system end, and the like. The transaction system end refers to an object which completes transaction operation and stores transaction data, such as various banks, mobile terminals and the like, wherein the transaction data are order data, pipeline data and the like corresponding to various payment channels. The preprocessing server side acquires transaction data required by the business system side from the transaction system side, performs massive preprocessing, and stores the preprocessed data in the structured table data queue, so that each business system side directly reads the transaction data from the structured table data queue without extracting the transaction data from a database of the transaction system side. Therefore, the data of the transaction system ends are independent and not interfered with each other by taking the preprocessing end as a middleware from the sending end to the preprocessing end and then to the receiving end, and the data processing efficiency is accelerated.
Referring to fig. 2, fig. 2 is a flowchart illustrating an embodiment of a multi-source data processing method according to the present invention. The processing method of the multi-source data is applied to a multi-source data processing system shown in fig. 1, and in a popular way, the design deployment architecture of the method is a data acquisition end-a data preprocessing end-a data receiving end, and the data acquisition end is a large amount of data acquisition middleware which is installed at a business system end in a heartbeat mode; the data preprocessing end refers to a server cluster which is distributed and deployed and used for preprocessing the service data in a large batch; and the data receiving end is a server cluster which stores the preprocessed data and performs timing synchronization of the database and the search engine. The scene of data exchange is from the data acquisition end to the preprocessing end and then to the data receiving end, the type, structure, processing method and the like of the data to be processed can be judged in advance through the preprocessing link, and the data can be processed by referring to a knowledge base (past experience and history accumulation mode). In the embodiment of the application, the traditional data processing mode is changed from the sending end to the receiving end to the preprocessing mode and then to the receiving end, so that the data processing efficiency can be effectively accelerated, the beneficial effect of rapid processing on the terminal is realized, and the bottleneck problem of multi-source data processing can be effectively solved due to the adoption of a dynamic expansion mode in the processing process.
Specifically, as shown in fig. 2, the method for processing multi-source data includes:
and S10, collecting a plurality of transaction data corresponding to a plurality of target service system ends.
In this embodiment, the target service system end refers to an object that needs to acquire transaction data, that is, a target object. When a plurality of target objects exist, the user ID is used to distinguish different target objects. That is, the user ID uniquely identifies the target object. The user ID in this embodiment refers to a serial number automatically created by the system for each target object or a device identifier of the device itself. The identifier is not only used for determining the target object, but also used for determining the target object corresponding to each data stored in the structured table data queue through the incidence relation in the structured table data queue, and the user ID is similar to the index information and is used for quick query.
Different business system ends can obtain corresponding transaction data through different transaction system ends. The transaction system end can be a third-party transaction end or platform and can also be different payment channels. The payment channel can be a lottery, a WeChat, a Paibao, a rich payment, a Unionpay, a cloud flash payment and the like, and can be divided into payment types such as mobile payment, offline card swiping, Unionpay, cash payment and the like.
In order to avoid the situation that the load pressure of each database is increased and the data among the databases is interfered when a plurality of target service system ends perform data processing on the databases of different transaction system ends, the target transaction data corresponding to the target transaction system required by each target service system end are stored in a middleware, the required transaction data are acquired from the middleware and downloaded to the target service system end to perform corresponding data processing locally, so that the transaction data acquired by the databases are ensured to be isolated and not interfered, and the load pressure and the statistical pressure of the databases of the transaction system ends are reduced.
Illustratively, the collecting of the transaction data corresponding to the target service system terminals specifically includes:
s11, determining a plurality of target transaction system ends corresponding to the target service system ends;
and S12, querying the database of each target transaction system end, and extracting the transaction data corresponding to the target service system end from the database of each target transaction system end.
That is, each transaction system side corresponds to a database Mysql, which stores transaction data required by each transaction system side. Therefore, the preprocessing server collects transaction data from the respective databases in the transaction system terminals, then preprocesses the collected transaction data, and then sends the preprocessed transaction data to the target service system terminal.
Since the preprocessing server can correspond to a plurality of business system ends, a target business system end corresponding to the needed transaction data needs to be determined, and the transaction data needs to be obtained from which target transaction system ends by analyzing the business requirements and obtaining the identification information of the target object, such as the user ID or the equipment identification.
Once the target trading system end is determined, the preprocessing server collects trading data from the database of the target trading system end.
S20, preprocessing the transaction data, and storing the preprocessed transaction data in a preset structured table data queue.
Specifically, due to the diversification of format data of the acquired transaction data, such as format file streams of excel, txt, json, compressed packet, xtp, and the like, invalid data is reduced, a storage space is saved, and meanwhile, in order to improve processing efficiency, the acquired transaction data needs to be preprocessed. In this embodiment, the preprocessing refers to converting the acquired transaction data into unified standard format data through canal service, where the standard format data is json format data. JSON (JavaScript Object Notation) is a lightweight data exchange format, which stores and represents data in a text format completely independent of a programming language, and a simple and clear hierarchical structure makes JSON an ideal data exchange language, which is easy for human reading and writing, easy for machine parsing and generation, and effectively improves network transmission efficiency.
It should be noted that the standard format is not limited, and may also be set according to service requirements, such as one or more of file streams of excel, txt, compressed packet, xtp, and the like.
Of course, the preprocessing may be to determine the type, structure, processing method, etc. of the data to be processed in advance, and may refer to a knowledge base (past experience and history accumulated mode) for processing, or may be to customize the flow corresponding to the preprocessing flow. Therefore, the preprocessing is not limited and can be set according to business requirements.
A structured table data queue is pre-created in a preprocessing server. The structured table data queue is configured with the standard format. The structured table data queue is used for storing the preprocessed transaction data. And further, creating a working thread for monitoring the data queue of the structured table, wherein the working thread is used for monitoring the transaction of the data files in the data queue of the structured table.
Furthermore, when it is monitored that any stored data file is deleted, the working thread of the structured table data queue corresponding to the data file is deleted.
For example, the preprocessing the transaction data and storing the preprocessed transaction data in a preset structured table data queue specifically includes:
s21, acquiring a preset structured table data queue; the structured table data queue is configured with a standard format;
s22, converting the transaction data into unified standard format data respectively by adopting a parallel processing mode through canal service;
and S23, storing each converted standard format data in a preset structured table data queue.
Specifically, the database of each transaction system corresponds to a canal service, which is used for monitoring and transforming. After the acquired transaction data is subjected to the canal service, a parallel processing mode is adopted, namely, a plurality of transaction processes are selected in a multi-thread mode, and are analyzed and converted synchronously in parallel, so that standard format data corresponding to each transaction data is obtained.
Meanwhile, the canal service monitors in real time whether the data files stored in the database of the respective transaction system end are changed, and the change refers to whether the stored data files are modified, such as added, deleted, changed and the like. The method is particularly characterized in that Mysql binlog binary log files in the databases of the respective target transaction system sides are monitored, the Mysql binlog binary log files can record data update or potential update of Mysql (such as data which is deleted by a DELETE statement and does not meet conditions actually), and therefore the Mysql binlog binary log files are used for determining whether the data files stored in the respective databases are changed or not.
If it is monitored that the data file stored in the database of at least one target transaction system end is changed, the changed data file is obtained, the changed data file is preprocessed, the subsequent operation of the step S22 is executed, and the preprocessed changed data file is stored in a preset structured data queue.
S30, acquiring a service requirement, searching target transaction data matched with the service requirement from the preset structured table data queue, and sending the target transaction data to a corresponding target service system end for data processing.
Specifically, based on step S20, the preprocessing server may obtain multiple service demands, and therefore, when there are multiple service demands, it is necessary to determine a target service system terminal that sends the service demand and transaction data that matches each service demand.
All transaction data can be obtained from the structured table data queue, so that the target transaction data matched with the service requirement is searched from the structured table data queue through the incidence relation that the target service system end corresponding to the service requirement and the transaction system end have the same transaction identifier.
And then the preprocessing server forwards the target transaction data to a target service system end corresponding to the service requirement for data processing. Specifically, the target business system end downloads the target transaction data corresponding to the business requirement from the preprocessing server end to the local, and then processes the target transaction data.
Furthermore, the transaction data acquired by the target service system end is from the structured table data queue of the preprocessing server and does not directly act on the database of the source, namely the database corresponding to the target transaction system, so that the acquired transaction data can be subjected to secondary service processing, analysis or message pushing scenes, the data in the database corresponding to the original target transaction system is not influenced at all, and the transaction system is completely isolated, so that the upper limit of charge is not limited, the cost is saved, and the reuse rate is improved to the maximum extent.
For example, the obtaining of the service requirement, searching for target transaction data matched with the service requirement from the preset structured table data queue, and sending the target transaction data to a corresponding target service system end for data processing specifically includes:
s31, analyzing the acquired service requirement, and determining the transaction identification information of the target service system end corresponding to the service requirement;
s32, determining whether target transaction data identical to the transaction identification information exists in the structured table data queue or not based on the transaction identification information;
s33, if target transaction data identical to the transaction identification information exist, the target transaction data are sent to a target service system end corresponding to the transaction identification information;
and S34, processing the target transaction data through the target service system terminal.
The transaction identification information includes a user ID or a device identification. Further, the determining whether the target transaction data identical to the transaction identification information exists in the structured table data queue represents looking up the target transaction data identical to the user ID or the device identification in the structured table data queue.
Thus, based on the steps S10-S30, the invention aims to collect the data of each transaction system into one queue in a unified standard manner, acquire the data of the transaction systems from the same queue in real time aiming at diversified service requirements and expansion, isolate each transaction system, and not increase load pressure and statistical pressure on the database of the transaction system, thereby improving the data processing efficiency, reducing the development and maintenance difficulty, and facilitating the user.
In the above embodiments, all or part of the implementation may be realized by software, hardware, firmware, or any combination thereof. When implemented using a software program, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, cause the processes or functions described in accordance with the embodiments of the invention to occur, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that incorporates one or more of the available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.
Based on the method, the application also provides a server cluster. In one embodiment, as shown in FIG. 3, FIG. 3 illustrates a block diagram of a multi-server cluster. Which includes at least one processor (processor) 20; a display screen 21; and a memory (memory)22, and may further include a communication Interface (Communications Interface)23 and a bus 24. The processor 20, the display 21, the memory 22 and the communication interface 23 can communicate with each other through the bus 24. The display screen 21 is configured to display a user guidance interface preset in the initial setting mode. The communication interface 23 may transmit information. The processor 20 may call logic instructions in the memory 22 to perform the methods in the embodiments described above.
Furthermore, the logic instructions in the memory 22 may be implemented in software functional units and stored in a computer readable storage medium when sold or used as a stand-alone product.
The memory 22, which is a computer-readable storage medium, may be configured to store a software program, a computer-executable program, such as program instructions or modules corresponding to the methods in the embodiments of the present invention. The processor 20 executes the functional application and data processing, i.e. implements the method in the above-described embodiments, by executing the software program, instructions or modules stored in the memory 22.
The memory 22 may include a storage program area and a storage data area, wherein the storage program area may store an application program required for operating the voice customer service system, at least one function; the storage data area may store data created according to the use of the terminal device, and the like. Further, the memory 22 may include a high speed random access memory and may also include a non-volatile memory. For example, a variety of media that can store program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk, may also be transient storage media.
The above disclosure is only for the purpose of illustrating the preferred embodiments of the present invention, and it is therefore to be understood that the invention is not limited by the scope of the appended claims.
Claims (10)
1. A multi-source data processing method comprises the following steps:
collecting a plurality of transaction data corresponding to a plurality of target service system ends;
preprocessing the transaction data, and storing the preprocessed transaction data in a preset structured table data queue;
and acquiring a service requirement, searching target transaction data matched with the service requirement from the preset structured table data queue, and sending the target transaction data to a corresponding target service system end for data processing.
2. The multi-source data processing method according to claim 1, wherein the collecting of the transaction data corresponding to the target business system terminals specifically comprises:
determining a plurality of target transaction system ends corresponding to a plurality of target service system ends;
and inquiring databases of respective target transaction system ends, and extracting corresponding transaction data belonging to the target service system end from the databases of the respective target transaction system ends.
3. The method for processing multi-source data according to claim 2, further comprising:
monitoring whether data files stored in databases of respective target transaction system ends are changed;
if the situation that the data files stored in the database of at least one target transaction system end are changed is monitored, the changed data files are obtained, and the changed data files are preprocessed;
and storing the preprocessed changed data file in a preset structured data queue.
4. The method for processing multi-source data according to claim 3, wherein the monitoring whether the data files stored in the database of the respective target transaction system end are changed specifically comprises:
starting the canal service of each target transaction system end;
and monitoring Mysqlbinlog binary log files in the databases of the respective target transaction system ends to determine whether the data files stored in the respective databases are changed or not through the respective Mysqlbinlog binary log files.
5. The multi-source data processing method according to claim 1 or 3, wherein the preprocessing the transaction data and storing the preprocessed transaction data in a preset structured table data queue specifically comprises:
acquiring a preset structured table data queue; the structured table data queue is configured with a standard format;
respectively converting the transaction data into unified standard format data by adopting a parallel processing mode through canal service;
and storing the converted data in the standard format in a preset structured table data queue.
6. The method of claim 5, wherein the standard format is json format.
7. The method for processing multi-source data according to claim 1, wherein the acquiring of the business demand, searching for the target transaction data matched with the business demand from the preset structured table data queue, and sending the target transaction data to a corresponding target business system side for data processing specifically comprises:
analyzing the acquired service requirement, and determining the transaction identification information of a target service system end corresponding to the service requirement;
determining whether target transaction data identical to the transaction identification information exists in the structured table data queue or not based on the transaction identification information;
if target transaction data identical to the transaction identification information exists, the target transaction data are sent to a target service system end corresponding to the transaction identification information;
and carrying out data processing on the target transaction data through the target service system end.
8. A multi-source data processing system is characterized by comprising a plurality of business system ends, a preprocessing server end and a plurality of transaction system ends, wherein the business system ends and the transaction system ends are connected with the preprocessing server end, and the multi-source data processing system is used for realizing the steps of the multi-source data processing method as claimed in any one of claims 1 to 7.
9. A server cluster, comprising:
a memory and a processor; wherein the memory is used for storing a computer program, and the processor is used for implementing the steps of the multi-source data processing method according to any one of claims 1 to 7 when executing the computer program.
10. A computer temporary storage medium, characterized in that the storage medium stores one or more programs, which are executed by a processor to implement the steps in the method of processing multi-source data according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011180781.9A CN112270600A (en) | 2020-10-29 | 2020-10-29 | Multi-source data processing method, system and related device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011180781.9A CN112270600A (en) | 2020-10-29 | 2020-10-29 | Multi-source data processing method, system and related device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112270600A true CN112270600A (en) | 2021-01-26 |
Family
ID=74345330
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011180781.9A Pending CN112270600A (en) | 2020-10-29 | 2020-10-29 | Multi-source data processing method, system and related device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112270600A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114841570A (en) * | 2022-05-07 | 2022-08-02 | 金腾科技信息(深圳)有限公司 | Data processing method, device, equipment and medium for customer relationship management system |
| CN115115457A (en) * | 2022-08-29 | 2022-09-27 | 天津金城银行股份有限公司 | Method, system and medium for processing business transaction flow |
| CN115599797A (en) * | 2022-11-29 | 2023-01-13 | 广东通莞科技股份有限公司（Cn） | Bidirectional synchronization method and device based on operation log and electronic equipment |
| CN118014732A (en) * | 2024-04-10 | 2024-05-10 | 深圳华锐分布式技术股份有限公司 | Data return method, device, equipment and medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8671001B1 (en) * | 2010-12-30 | 2014-03-11 | Eventbrite, Inc. | Real-time attendance reporting |
| CN110489475A (en) * | 2019-08-14 | 2019-11-22 | 广东电网有限责任公司 | A kind of multi-source heterogeneous data processing method, system and relevant apparatus |
| CN111124679A (en) * | 2019-12-19 | 2020-05-08 | 南京莱斯信息技术股份有限公司 | Time-limited automatic processing method for multi-source heterogeneous mass data |
-
2020
- 2020-10-29 CN CN202011180781.9A patent/CN112270600A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8671001B1 (en) * | 2010-12-30 | 2014-03-11 | Eventbrite, Inc. | Real-time attendance reporting |
| CN110489475A (en) * | 2019-08-14 | 2019-11-22 | 广东电网有限责任公司 | A kind of multi-source heterogeneous data processing method, system and relevant apparatus |
| CN111124679A (en) * | 2019-12-19 | 2020-05-08 | 南京莱斯信息技术股份有限公司 | Time-limited automatic processing method for multi-source heterogeneous mass data |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114841570A (en) * | 2022-05-07 | 2022-08-02 | 金腾科技信息(深圳)有限公司 | Data processing method, device, equipment and medium for customer relationship management system |
| CN115115457A (en) * | 2022-08-29 | 2022-09-27 | 天津金城银行股份有限公司 | Method, system and medium for processing business transaction flow |
| CN115599797A (en) * | 2022-11-29 | 2023-01-13 | 广东通莞科技股份有限公司（Cn） | Bidirectional synchronization method and device based on operation log and electronic equipment |
| CN118014732A (en) * | 2024-04-10 | 2024-05-10 | 深圳华锐分布式技术股份有限公司 | Data return method, device, equipment and medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112270600A (en) | Multi-source data processing method, system and related device | |
| CN101611402B (en) | System and method for optimizing changes of data sets | |
| CN111597257A (en) | Database synchronization method and device, storage medium and terminal | |
| CN112527649A (en) | Test case generation method and device | |
| CN108984553B (en) | Caching method and device | |
| CN112434039A (en) | Data storage method, device, storage medium and electronic device | |
| CN111639138A (en) | Data processing method, device, equipment and storage medium | |
| CN111090268B (en) | Data acquisition method and device based on thread division and data acquisition equipment | |
| EP3937022B1 (en) | Method and apparatus of monitoring interface performance of distributed application, device and storage medium | |
| CN106709805B (en) | User income data acquisition method and system | |
| CN104881454A (en) | Updating method and system of parameter | |
| CN111339743B (en) | Account number generation method and device | |
| CN114398520A (en) | Data retrieval method, system, device, electronic equipment and storage medium | |
| CN110222046B (en) | List data processing method, device, server and storage medium | |
| CN116308400A (en) | Customer maintenance method and system based on user code scanning verification and verification behavior track analysis | |
| CN111046077A (en) | Data acquisition method and device, storage medium and terminal | |
| KR20210000041A (en) | Method and apparatus for analyzing log data in real time | |
| CN116016561A (en) | Data synchronization method and device | |
| CN112764988B (en) | Data segment acquisition method and device | |
| CN110569424A (en) | Information recommendation method and device | |
| CN111061719B (en) | Data collection method, device, equipment and storage medium | |
| CN113326333A (en) | Data processing method, system, computer device and computer storage medium | |
| CN111401819A (en) | Intersystem data pushing method and system | |
| CN112783500A (en) | Compiling optimization information generation method and device and electronic equipment | |
| CN111695984A (en) | Account balance processing method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210126 |