CN109800265B - Data loading method, device, equipment and computer readable storage medium - Google Patents
Data loading method, device, equipment and computer readable storage medium Download PDFInfo
- Publication number
- CN109800265B CN109800265B CN201811629551.9A CN201811629551A CN109800265B CN 109800265 B CN109800265 B CN 109800265B CN 201811629551 A CN201811629551 A CN 201811629551A CN 109800265 B CN109800265 B CN 109800265B
- Authority
- CN
- China
- Prior art keywords
- data
- loading
- service data
- service
- real
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention belongs to the technical field of big data, and discloses a data loading method, which comprises the following steps: receiving service data pushed by an upstream system, and loading the service data into a temporary table based on a preset data processing model; obtaining a mapping document, and determining a mapping relation corresponding to the service data based on the mapping document; and loading the business data in the temporary table into a real-time table based on the mapping relation. The invention also discloses a data loading device, equipment and a computer readable storage medium. The invention improves the speed and accuracy of data loading.
Description
Technical Field
The present invention relates to the field of big data technologies, and in particular, to a data loading method, apparatus, device, and computer readable storage medium.
Background
The upstream system pushes the business data of different business lines to the downstream system every day so that the downstream system sorts the business data and loads the business data into a corresponding business table, and finally the business data are used for a business operator. However, the data of different service lines are different, the data are loaded into the database through the same flow, the loaded data are huge in volume and complex in process, and the accuracy of data loading is difficult to ensure.
Disclosure of Invention
The invention mainly aims to provide a data loading method, a device, equipment and a computer readable storage medium, which aim to solve the technical problems of complicated data loading process and low accuracy caused by large data volume when loading data of different service lines.
In order to achieve the above object, the present invention provides a data loading method, including:
receiving service data pushed by an upstream system, and loading the service data into a temporary table based on a preset data processing model;
obtaining a mapping document, and determining a mapping relation corresponding to the service data based on the mapping document;
and loading the business data in the temporary table into a real-time table based on the mapping relation.
Optionally, the step of receiving service data pushed by the upstream system and loading the service data into the temporary table based on a preset data processing model includes:
receiving service data pushed by an upstream system through a data loading tool, and loading the service data to a corresponding interface table based on the service type of the service data;
and acquiring a preset data processing model corresponding to the interface table, and processing the business processing in the interface table based on the preset data processing model so as to load the processed business data into a corresponding temporary table.
Optionally, before the step of receiving the service data pushed by the upstream system and loading the service data into the temporary table based on the preset data processing model, the method further includes:
making a mapping document, wherein the mapping document comprises an interface area, a temporary area and a real-time area, and the interface area, the temporary area and the real-time area record storage position fields of an interface table, a temporary table and a real-time table corresponding to the service data;
and storing the mapping document.
Optionally, the step of determining the mapping relation corresponding to the service data based on the mapping document includes:
and generating a mapping relation corresponding to the temporary zone and the real-time zone based on the storage position fields respectively recorded in the temporary zone and the real-time zone in the mapping document, and determining the mapping relation as the mapping relation corresponding to the service data.
Optionally, the step of loading the service data in the temporary table into a real-time table based on the mapping relationship includes:
acquiring current time and judging whether the current time meets a preset time condition or not;
if yes, judging whether the service data meets the loading condition;
if the service data meets the loading condition, loading the service data to a corresponding real-time table based on the mapping relation;
the step of loading the service data into the corresponding real-time table based on the mapping relation comprises the following steps:
creating a plurality of parallel data loading tasks, and distributing the service data to the data loading tasks based on the service type of the service data;
and starting the data loading task, and loading the business data in the data loading task to a corresponding real-time table based on the mapping relation.
Optionally, after the step of loading the service data in the temporary table into the real-time table based on the mapping relationship, the method further includes:
acquiring a preset service distribution rule, and distributing service data in the real-time table to different salesman systems according to the service distribution rule.
Optionally, after the step of loading the service data in the temporary table into the real-time table based on the mapping relationship, the method further includes:
judging whether the service data loading is completed or not;
if yes, the service data in the temporary table is backed up, and the service data in the temporary table is deleted after the backup is completed.
In addition, to achieve the above object, the present invention also provides a data loading device, including:
the data receiving module is used for receiving service data pushed by the upstream system and loading the service data into the temporary table based on a preset data processing model;
the mapping relation acquisition module is used for acquiring a mapping document and determining a mapping relation corresponding to the service data based on the mapping document;
and the data loading module is used for loading the business data in the temporary table into a real-time table based on the mapping relation.
In addition, to achieve the above object, the present invention also provides a data loading apparatus including: the system comprises a memory, a processor and a data loading program stored in the memory and capable of running on the processor, wherein the data loading program realizes the steps of the data loading method when being executed by the processor.
In addition, in order to achieve the above object, the present invention also provides a computer-readable storage medium having stored thereon a data loading program which, when executed by a processor, implements the steps of the data loading method as described above.
The invention provides a data loading method, which comprises the steps of firstly, receiving service data pushed by an upstream system, loading the received service data into a temporary table through a preset data processing model, further, obtaining a mapping document to determine a mapping relation corresponding to the service data, and finally loading the service data in the temporary table into a real-time table according to the mapping relation. According to the data loading method, the temporary table is established to receive the service data pushed by the upstream system, the mapping relation is determined according to the mapping document, and finally the service data in the temporary table is mapped into the real-time table, so that the accuracy of data loading is improved, and the process optimization of data loading of different service lines is realized.
Drawings
FIG. 1 is a schematic diagram of a hardware structure of a data loading device according to an embodiment of the present invention;
FIG. 2 is a flowchart of a data loading method according to a first embodiment of the present invention;
fig. 3 is a schematic diagram of a refinement flow of step S10 in fig. 2;
FIG. 4 is a flowchart of a second embodiment of the data loading method of the present invention;
fig. 5 is a schematic functional block diagram of a first embodiment of the data loading device of the present invention.
The achievement of the objects, functional features and advantages of the present invention will be further described with reference to the accompanying drawings, in conjunction with the embodiments.
Detailed Description
It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.
The main solutions of the embodiments of the present invention are: receiving service data pushed by an upstream system, and loading the service data into a temporary table based on a preset data processing model; obtaining a mapping document, and determining a mapping relation corresponding to the service data based on the mapping document; and loading the business data in the temporary table into a real-time table based on the mapping relation. By the technical scheme of the embodiment of the invention, the technical problems of complicated data loading process and low accuracy caused by large data volume during data loading of different service lines are solved.
As shown in fig. 1, fig. 1 is a schematic hardware structure of a data loading device according to an embodiment of the present invention.
The data loading method related to the embodiment of the invention is mainly applied to data loading equipment, and the data loading equipment can be equipment with display and processing functions such as a PC (personal computer), a portable computer, a mobile terminal and the like.
As shown in fig. 1, the data loading device may include: a processor 1001, such as a CPU, a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Wherein the communication bus 1002 is used to enable connected communication between these components. The user interface 1003 may include a Display, an input unit such as a Keyboard (Keyboard), and the optional user interface 1003 may further include a standard wired interface, a wireless interface. The network interface 1004 may optionally include a standard wired interface, a wireless interface (e.g., WI-FI interface). The memory 1005 may be a high-speed RAM memory or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also optionally be a storage device separate from the processor 1001 described above.
Optionally, the data loading device may also include a camera, an RF (Radio Frequency) circuit, a sensor, an audio circuit, a Wi-Fi module, and so forth. Among other sensors, such as light sensors, motion sensors, and other sensors. Of course, the data loading device may also be configured with other sensors such as a gyroscope, a barometer, a hygrometer, a thermometer, an infrared sensor, and the like, which are not described herein.
Those skilled in the art will appreciate that the data loading device structure shown in FIG. 1 does not constitute a limitation of the data loading device and may include more or fewer components than shown, or may combine certain components, or a different arrangement of components.
As shown in fig. 1, an operating system, a network communication module, a user interface module, and a data loading program may be included in the memory 1005, which is a type of computer storage medium. In fig. 1, the network interface 1004 is mainly used for connecting to a background server, and is in data communication with the background server; the user interface 1003 is mainly used for connecting a client (user side) and performing data communication with the client; and the processor 1001 and the memory 1005 may be provided in a data loading device, which calls a data loading program stored in the memory 1005 through the processor 1001 and performs the data loading method provided by the embodiment of the present invention.
According to the scheme provided by the embodiment, firstly, service data pushed by an upstream system are received, the received service data are loaded into a temporary table through a preset data processing model, further, a mapping document is obtained to determine a mapping relation corresponding to the service data, and finally, the service data in the temporary table are loaded into a real-time table according to the mapping relation. According to the data loading method, the temporary table is established to receive the service data pushed by the upstream system, the mapping relation is determined according to the mapping document, and finally the service data in the temporary table is mapped into the real-time table, so that the accuracy of data loading is improved, and the process optimization of data loading of different service lines is realized.
Based on the hardware structure, the embodiment of the data loading method is provided.
Referring to fig. 2, fig. 2 is a flow chart of a first embodiment of a data loading method according to the present invention, in which the method includes:
step S10, receiving service data pushed by an upstream system, and loading the service data into a temporary table based on a preset data processing model;
in order to optimize the data loading flow when the data of different service lines are loaded to a downstream system, the invention provides a data loading method, all the service data pushed by the upstream system are loaded to a temporary table through a preset data processing model, and then a mapping relation is established through a mapping document, so that the service data in the temporary table are mapped to a real-time table, and the flow optimization of the data loading of different service lines is realized.
Specifically, the upstream banking system pushes service data of different service lines, such as credit card, retail, one account, etc., each day, as shown in fig. 3, the step S10 specifically includes:
step S11, receiving service data pushed by an upstream system through a data loading tool, and loading the service data to a corresponding interface table based on the service type of the service data;
firstly, these original service data pushed by the upstream system are all loaded into the interface table by the data loading tool, for example, an ETL (Extract-Transform-Load) loading tool may be used, and when the service data is pushed from the upstream system, the service data is loaded into the interface table through steps of extraction (Extract), cross conversion (Transform), loading (Load), and the like.
It can be understood that the data sources of different service lines are different, so that the correspondingly stored interface tables are also different, and in this embodiment, different service data can be pushed to the downstream system by adopting a mode agreed with the upstream system based on different servers in different modes, so that the received service data of which service line can be determined by the different servers or pushing modes of the service data. Further, after receiving the service data, an identification information may be further set for the service data, so as to distinguish different service data, for example, the service data of the credit card is identified by the CC.
Step S12, a preset data processing model corresponding to the interface table is obtained, and service processing in the interface table is processed based on the preset data processing model so as to load the processed service data into a corresponding temporary table.
Further, data processing models corresponding to different interface tables are obtained, namely different business data correspond to different data processing models, the business data are processed into data which can be analyzed in a standard form based on the data processing models, and the data are loaded into the temporary tables. Correspondingly, the data of different service lines are stored in different temporary tables, and are distinguished by the identification information.
Step S20, a mapping document is obtained, and a mapping relation corresponding to the service data is determined based on the mapping document;
in this embodiment, the mapping relationship is determined based on a mapping document, which is created before the business data is loaded. Specifically, three workspaces are included in the mapping document: the system comprises an interface area, a temporary area and a real-time area, wherein service data pushed by an upstream system is initially stored in the interface table, the interface area records the storage position of the interface table, the specific expression form is a storage position field, the temporary area records the storage position field of the temporary table, and the real-time area records the storage position field of the real-time table. For example, the storage location field corresponding to the service data recorded in the temporary table in the interface area is an a field, the temporary table in the temporary area is a B field, and the real-time table in the real-time area is a C field. It will be appreciated that the mapping documents corresponding to different business data are different.
And step S30, based on the mapping relation, loading the business data in the temporary table into a real-time table.
Further, after determining the mapping relationship between the temporary table and the real-time table, the service data in the temporary table can be loaded into the real-time table based on the mapping relationship, specifically, the flow of data loading according to the mapping relationship is as follows:
firstly, the data loading process in this embodiment may set a timing task, and start the data loading process at regular time every day, for example, may set starting the data loading process every three minutes from 3 a.m. to 20 a.m. every day, so as to first determine whether the current time meets a preset time condition for starting the data loading process; if yes, further analyzing the service data to be loaded, and judging whether the service data meets the loading condition, wherein the analysis process before data loading is as follows:
firstly, judging whether the service data is already loaded, specifically, in the embodiment, the service data which is completely loaded is provided with an identifier of the completion of loading, so if the service data which is already loaded is judged, the repeated loading is not carried out, and the service data is removed; otherwise, if the service data is not loaded yet, judging whether the service data to be loaded is new data or old data, wherein the judging basis is to compare the service data to be loaded with the original data in the real-time table, judging whether the data exists in the real-time table, if so, the data to be loaded is old data, and loading the data to be loaded is to update the old data actually; if not, the data to be loaded is new data, and the subsequent operation is to load the data into the real-time table. Finally, carrying out data integrity check on the data to be loaded, wherein the integrity check steps are as follows: deleting repeated data in the data to be loaded, including repeated information in account information, customer information and debt information, and then checking the accuracy of the account and the customer data in the data to be loaded, wherein the method is characterized in that a new account must have a corresponding customer number, a bill postal code cannot be empty, the customer must have corresponding account information, and the like, and if the data check fails, the data are also rejected.
After the analysis process before loading the data is completed, the business data passing through the data analysis process is further loaded. Firstly, creating a plurality of parallel data loading tasks to reduce loading time; receiving data, judging whether the data is in a state of completing analysis to be loaded, if so, distributing the service data to be loaded into the plurality of parallel data loading tasks according to different service types, and of course, distributing according to other conditions besides distributing according to the service types; and finally, starting the parallel data loading task, loading information such as account, client, debt, transaction information, address, telephone information and the like contained in the data to be loaded into a real-time table based on the determined mapping relation, and storing the client relation contained in the information into other client relation tables, so that the business data is completely loaded into the real-time table from the temporary table.
In this embodiment, firstly, service data pushed by an upstream system is received, the received service data is loaded into a temporary table through a preset data processing model, further, a mapping document is obtained to determine a mapping relationship corresponding to the service data, and finally, the service data in the temporary table is loaded into a real-time table according to the mapping relationship. According to the data loading method, the temporary table is established to receive the service data pushed by the upstream system, the mapping relation is determined according to the mapping document, and finally the service data in the temporary table is mapped into the real-time table, so that the accuracy of data loading is improved, and the process optimization of data loading of different service lines is realized.
Further, referring to fig. 4, based on the foregoing embodiment, a second embodiment of the data loading method of the present invention is provided, and in this embodiment, after the step S30, the method further includes:
step S40, a preset service distribution rule is obtained, and service data in the real-time table are distributed to different salesman systems according to the service distribution rule.
In this embodiment, after the loading process of the service data from the upstream system to the downstream system is completed, different service data loaded into the real-time table may be allocated to different operators for use based on a preset service allocation rule, so as to count the work performance of the operators.
Specifically, the service allocation rule may be to allocate the service allocation rule to different operators based on different service data types, or to allocate the service allocation rule based on client information in the service data, etc., and may be set according to actual situations.
In this embodiment, how to calculate the work performance of the salesman through the business data is described by taking the customer repayment and collection as an example. Specifically, after the corresponding service data is distributed to the corresponding service personnel system for processing by the service personnel, if the corresponding service update data is arranged in the follow-up sequence of the service data, a transaction information table in the service update data is obtained, and the work performance of the service personnel is determined according to the repayment information of the customer in the transaction information table. For example, if the transaction information table contains customer repayment information, the KP (key process, compliance commitment) times in the business data of the salesman is increased by 1; if there is no repayment information or the repayment amount is insufficient, the BP (break commitment) number is increased by 1, for example, the service data loaded into the real-time table indicates that the client commits 100 yuan for repayment, and the subsequent transaction information table indicates that the client pays 90 yuan, which indicates that the client complies with the repayment commitment, if the repayment amount of the client is less than 90 yuan or there is no repayment data, which indicates that the client does not comply with the repayment commitment, and the salesman does not complete the collect-promoting service.
Further, after the step S30, the method may further include:
step a, judging whether the business data loading is completed or not; if yes, executing the step b;
and b, backing up the service data in the temporary table, and deleting the service data in the temporary table after the backup is completed.
After the loading process of the business data from the temporary table to the real-time table is completed, the data in the temporary table can be backed up, and the business data in the temporary table is deleted after the backup is completed, because the temporary table is only used for storing, analyzing and loading the business data loaded from the upstream system to the real-time table every day, the temporary table is emptied after the data loading is completed, and the next round of data loading process is performed.
Specifically, before the backup is executed, whether the data loading process in the temporary table is completed is firstly judged, if yes, information such as account, customer, debt, transaction information, address, telephone information and the like is backed up in the corresponding backup table, and all data in the temporary table is deleted.
In this embodiment, after the loading process of the service data from the temporary table to the real-time table is completed, different service data loaded into the real-time table can be allocated to different operators for use based on a preset service allocation rule, so as to count the work performance of the operators, and facilitate the management of the service system.
In addition, the embodiment of the invention also provides a data loading device.
Referring to fig. 5, fig. 5 is a schematic functional block diagram of a data loading device according to a first embodiment of the present invention.
In this embodiment, the data loading device includes:
the data receiving module 10 is configured to receive service data pushed by an upstream system, and load the service data into a temporary table based on a preset data processing model;
the mapping relation obtaining module 20 is configured to obtain a mapping document, and determine a mapping relation corresponding to the service data based on the mapping document;
and the data loading module 30 is configured to load the service data in the temporary table into a real-time table based on the mapping relationship.
Further, the data receiving module 10 specifically includes:
the data receiving unit is used for receiving service data pushed by an upstream system through a data loading tool and loading the service data to a corresponding interface table based on the service type of the service data;
the data processing unit is used for acquiring a preset data processing model corresponding to the interface table, and processing the business processing in the interface table based on the preset data processing model so as to load the processed business data to the corresponding temporary table.
Further, the data loading device further includes:
the mapping document production unit is used for producing a mapping document, wherein the mapping document comprises an interface area, a temporary area and a real-time area, and the interface area, the temporary area and the real-time area record storage position fields of an interface table, a temporary table and a real-time table corresponding to the service data;
and the mapping document storage unit is used for storing the mapping document.
Further, the mapping relationship obtaining module 20 specifically includes:
and the mapping relation determining unit is used for generating mapping relation corresponding to the temporary zone and the real-time zone based on the storage position fields respectively recorded in the temporary zone and the real-time zone in the mapping document, and determining the mapping relation corresponding to the service data.
Further, the data loading module 30 specifically includes:
the time judging unit is used for acquiring the current time and judging whether the current time meets the preset time condition or not;
the condition judging unit is used for judging whether the service data meets the loading condition or not if yes;
the data loading unit is used for loading the service data to the corresponding real-time table based on the mapping relation if the service data meets the loading condition;
the data loading unit specifically includes:
a data loading task creation subunit, configured to create a plurality of parallel data loading tasks, and allocate the service data to the data loading tasks based on a service type of the service data;
and the data loading task starting sub-unit is used for starting the data loading task and loading the business data in the data loading task to the corresponding real-time table based on the mapping relation.
Further, the data loading device further includes:
the service distribution unit is used for acquiring a preset service distribution rule and distributing the service data in the real-time table to different salesman systems according to the service distribution rule.
Further, the data loading device further includes:
the loading completion judging unit is used for judging whether the loading of the service data is completed or not;
and the data backup unit is used for backing up the service data in the temporary table if yes, and deleting the service data in the temporary table after the backup is completed.
The modules in the data loading device correspond to the steps in the data loading method embodiment, and the functions and implementation processes of the modules are not described in detail herein.
In addition, the embodiment of the invention also provides a computer readable storage medium, wherein the computer readable storage medium stores a data loading program, and the data loading program realizes the steps of the data loading method when being executed by a processor.
The method implemented when the data loading program is executed may refer to various embodiments of the data loading method of the present invention, which are not described herein.
According to the scheme provided by the embodiment, firstly, service data pushed by an upstream system are received, the received service data are loaded into a temporary table through a preset data processing model, further, a mapping document is obtained to determine a mapping relation corresponding to the service data, and finally, the service data in the temporary table are loaded into a real-time table according to the mapping relation. According to the data loading method, the temporary table is established to receive the service data pushed by the upstream system, the mapping relation is determined according to the mapping document, and finally the service data in the temporary table is mapped into the real-time table, so that the accuracy of data loading is improved, and the process optimization of data loading of different service lines is realized.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or system. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or system that comprises the element.
The foregoing embodiment numbers of the present invention are merely for the purpose of description, and do not represent the advantages or disadvantages of the embodiments.
From the above description of the embodiments, it will be clear to those skilled in the art that the above-described embodiment method may be implemented by means of software plus a necessary general hardware platform, but of course may also be implemented by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art in the form of a software product stored in a storage medium (e.g. ROM/RAM, magnetic disk, optical disk) as described above, comprising instructions for causing a terminal device (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to perform the method according to the embodiments of the present invention.
The foregoing description is only of the preferred embodiments of the present invention, and is not intended to limit the scope of the invention, but rather is intended to cover any equivalents of the structures or equivalent processes disclosed herein or in the alternative, which may be employed directly or indirectly in other related arts.
Claims (6)
1. A data loading method, characterized in that the data loading method comprises the steps of:
receiving service data pushed by an upstream system, and loading the service data into a temporary table based on a preset data processing model;
obtaining a mapping document, and determining a mapping relation corresponding to the service data based on the mapping document;
based on the mapping relation, loading the business data in the temporary table into a real-time table;
the step of receiving the service data pushed by the upstream system and loading the service data into the temporary table based on a preset data processing model comprises the following steps:
receiving service data pushed by an upstream system through a data loading tool, and loading the service data to a corresponding interface table based on the service type of the service data;
acquiring a preset data processing model corresponding to the interface table, and processing service processing in the interface table based on the preset data processing model so as to load the processed service data into a corresponding temporary table;
before the step of receiving the service data pushed by the upstream system and loading the service data into the temporary table based on the preset data processing model, the method further comprises the following steps:
making a mapping document, wherein the mapping document comprises an interface area, a temporary area and a real-time area, and the interface area, the temporary area and the real-time area record storage position fields of an interface table, a temporary table and a real-time table corresponding to the service data;
saving the mapping document;
the step of determining the mapping relation corresponding to the service data based on the mapping document comprises the following steps:
generating a mapping relation corresponding to the temporary zone and the real-time zone based on the storage position fields respectively recorded in the temporary zone and the real-time zone in the mapping document, and determining the mapping relation as the mapping relation corresponding to the service data;
the step of loading the business data in the temporary table into the real-time table based on the mapping relation comprises the following steps:
acquiring current time and judging whether the current time meets a preset time condition or not;
if yes, judging whether the service data meets the loading condition;
if the service data meets the loading condition, loading the service data to a corresponding real-time table based on the mapping relation;
the step of loading the service data into the corresponding real-time table based on the mapping relation comprises the following steps:
creating a plurality of parallel data loading tasks, and distributing the service data to the data loading tasks based on the service type of the service data;
and starting the data loading task, and loading the business data in the data loading task to a corresponding real-time table based on the mapping relation.
2. The data loading method according to claim 1, wherein after the step of loading the service data in the temporary table into the real-time table based on the mapping relationship, the method further comprises:
acquiring a preset service distribution rule, and distributing service data in the real-time table to different salesman systems according to the service distribution rule.
3. The data loading method according to claim 2, wherein after the step of loading the service data in the temporary table into the real-time table based on the mapping relationship, the method further comprises:
judging whether the service data loading is completed or not;
if yes, the service data in the temporary table is backed up, and the service data in the temporary table is deleted after the backup is completed.
4. A data loading device, the data loading device comprising:
the data receiving module is used for receiving service data pushed by the upstream system and loading the service data into the temporary table based on a preset data processing model;
the mapping relation acquisition module is used for acquiring a mapping document and determining a mapping relation corresponding to the service data based on the mapping document;
the data loading module is used for loading the business data in the temporary table into a real-time table based on the mapping relation;
the data receiving module is further used for receiving service data pushed by an upstream system through a data loading tool and loading the service data to a corresponding interface table based on the service type of the service data;
acquiring a preset data processing model corresponding to the interface table, and processing service processing in the interface table based on the preset data processing model so as to load the processed service data into a corresponding temporary table;
the data receiving module is further used for manufacturing a mapping document, wherein the mapping document comprises an interface area, a temporary area and a real-time area, and storage position fields of an interface table, a temporary table and a real-time table corresponding to the service data are recorded in the interface area, the temporary area and the real-time area;
saving the mapping document;
the mapping relation acquisition module is further used for generating mapping relation corresponding to the temporary zone and the real-time zone based on the storage position fields respectively recorded in the temporary zone and the real-time zone in the mapping document, and determining the mapping relation as the mapping relation corresponding to the service data;
the data loading module is further used for acquiring current time and judging whether the current time meets a preset time condition or not;
if yes, judging whether the service data meets the loading condition;
if the service data meets the loading condition, loading the service data to a corresponding real-time table based on the mapping relation;
wherein the loading the service data into the corresponding real-time table based on the mapping relation comprises:
creating a plurality of parallel data loading tasks, and distributing the service data to the data loading tasks based on the service type of the service data;
and starting the data loading task, and loading the business data in the data loading task to a corresponding real-time table based on the mapping relation.
5. A data loading device, the data loading device comprising: a memory, a processor and a data loading program stored on the memory and executable on the processor, the data loading program when executed by the processor implementing the steps of the data loading method according to any one of claims 1 to 3.
6. A computer readable storage medium, characterized in that the computer readable storage medium has stored thereon a data loading program, which when executed by a processor, implements the steps of the data loading method according to any of claims 1 to 3.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811629551.9A CN109800265B (en) | 2018-12-28 | 2018-12-28 | Data loading method, device, equipment and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811629551.9A CN109800265B (en) | 2018-12-28 | 2018-12-28 | Data loading method, device, equipment and computer readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109800265A CN109800265A (en) | 2019-05-24 |
| CN109800265B true CN109800265B (en) | 2023-07-25 |
Family
ID=66558004
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811629551.9A Active CN109800265B (en) | 2018-12-28 | 2018-12-28 | Data loading method, device, equipment and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109800265B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111538774B (en) * | 2020-04-23 | 2023-04-18 | 上海携程商务有限公司 | Data storage and display method, system, equipment and storage medium |
| CN112541693A (en) * | 2020-12-21 | 2021-03-23 | 携程计算机技术（上海）有限公司 | Performance data generation method, system, electronic device and storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102508743A (en) * | 2011-11-10 | 2012-06-20 | 武汉噢易科技有限公司 | Multi-point backup and real-time protection method for Linux file system |
| CN103955863A (en) * | 2014-04-10 | 2014-07-30 | 中国南方电网有限责任公司超高压输电公司检修试验中心 | Method for processing power network monitoring device data |
| WO2016060547A1 (en) * | 2014-10-13 | 2016-04-21 | Kim Seng Kee | Emulating manual system of filing using electronic document and electronic file |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7940899B2 (en) * | 2006-10-06 | 2011-05-10 | Pricewaterhousecoopers Llp | Fraud detection, risk analysis and compliance assessment |
| US9959268B2 (en) * | 2014-12-01 | 2018-05-01 | Environmental Systems Research Institute, Inc. | Semantic modeling of geographic information in business intelligence |
-
2018
- 2018-12-28 CN CN201811629551.9A patent/CN109800265B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102508743A (en) * | 2011-11-10 | 2012-06-20 | 武汉噢易科技有限公司 | Multi-point backup and real-time protection method for Linux file system |
| CN103955863A (en) * | 2014-04-10 | 2014-07-30 | 中国南方电网有限责任公司超高压输电公司检修试验中心 | Method for processing power network monitoring device data |
| WO2016060547A1 (en) * | 2014-10-13 | 2016-04-21 | Kim Seng Kee | Emulating manual system of filing using electronic document and electronic file |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109800265A (en) | 2019-05-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110889134B (en) | Data desensitizing method and device and electronic equipment | |
| CN105187676B (en) | Method and device for processing call request | |
| CN108416485B (en) | User identity recognition method, electronic device and computer readable storage medium | |
| US10963888B2 (en) | Payment complaint method, device, server and readable storage medium | |
| CN109951512B (en) | User preference determination method, system, electronic device and storage medium | |
| CN108241529B (en) | Salary calculation method, application server and computer readable storage medium | |
| CN112613917A (en) | Information pushing method, device and equipment based on user portrait and storage medium | |
| WO2019095670A1 (en) | Sales performance tracking method, application server and computer-readable storage medium | |
| CN110716966A (en) | Data visualization processing method and system, electronic device and storage medium | |
| CN110956269A (en) | Data model generation method, device, equipment and computer storage medium | |
| CN109800265B (en) | Data loading method, device, equipment and computer readable storage medium | |
| CN111324786A (en) | Method and device for processing consultation problem information | |
| CN111552712A (en) | Report data extraction method and device and computer equipment | |
| CN110782142A (en) | Daily birth daily clearing accounting method, device, equipment and computer readable storage medium | |
| CN111475468A (en) | Log access method, device, equipment and storage medium of newly added system | |
| CN111026613A (en) | Log processing method and device | |
| CN111158827A (en) | Method and device for graphic configuration tool correlation calculation value information | |
| CN115905371A (en) | Data trend analysis method, device and equipment and computer readable storage medium | |
| CN111459737B (en) | Problem positioning method, device, computer equipment and storage medium | |
| CN111222739B (en) | Nuclear power station task allocation method and nuclear power station task allocation system | |
| CN110515946B (en) | Data extraction method, device, equipment and computer readable storage medium | |
| CN112860456A (en) | Log processing method and device | |
| CN112541730A (en) | Project working hour statistical method, device and storage medium | |
| CN110019547B (en) | Method, device, equipment and medium for acquiring association relation between clients | |
| CN110737693A (en) | Data mining processing method, device, equipment and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |