Specific embodiment
It is first before introducing the technical solution of the application for the elaboration being purged convenient for the technical solution to the application
First it is introduced for some contents in current data warehouse:
(1) table
Table is the most important component part of data warehouse.One table record is measured by key, attribute data composition (such as member
Work table is by employee number (key), employee name, age etc. employee's attribute data composition).In the technical solution of the application, number
According to warehouse construction there are the tables of following two type:
Increment list: in order to improve performance, timestamp field (usually gmt_ is changed according to record for large data volume table
Modify increment synchronization) is used, each snapshot of increment list retains a incremental data, and table naming method is tablename_
{ yyyymmdd } _ delta or tablename_delta (subregion field dt=yyyymmdd);
Full dose table: each snapshot can retain a full dose table, which can be synchronizes from storage facility located at processing plant full dose,
Snapshot data yesterday for being also possible to the incremental data that will be synchronized from storage facility located at processing plant with full dose table carries out full outer join
Afterwards, retain a newest full dose, the structure of full dose table is consistent with increment list, and table naming method is tablename_ { yyyymmdd }
Or tablename (subregion field dt=yyyymmdd).
(2) view
View is a Virtual table, and content is by query-defined.The same with true table, view includes that a series of bands are famous
The columns and rows data of title.It should be noted that view does not exist in the form of the data value collection of storage in the database.
(3) metadata
Metadata is to describe the data of data, is in the nature the descriptive information to data and information resources, including traffic table
Structural information, data warehouse table structure information etc..Wherein metadata, SQL are dispatched in more important having in Construction of Data Warehouse
Execution journal metadata, table structure metadata, synchronous center metadata, timed task metadata etc..
(4) Merge task
Merge task is mode indispensable in Construction of Data Warehouse, its role is to by the data of increment list with
Snapshot data yesterday of full dose table merges, and generates a newest full dose snapshot table data.
(5) synchronous center
Synchronous center is that creation data is synchronized to data warehouse or data warehouse data flows back the dress to production system
It sets or equipment.
Background technique based on above content and the application, in Construction of Data Warehouse or restructuring procedure, according to data
Model construction framework needs to establish basal layer, some incremental data tables synchronized from storage facility located at processing plant are needed in data warehouse
ODS is laminated and a full dose table, and to retain a newest full dose snapshot data, and this process is related to generating full dose table and builds predicative
The operations such as sentence, merge mission script, scheduling dependence, data initialization, publication.And the technical solution of the application is then intended to pass through
Merge task realizes Mass production base layer data table (full dose table), to reduce data under the premise of safeguard work quality
The complexity of construction of warehouse promotes development efficiency.
As shown in Figure 1, a kind of flow diagram of the data table generating method proposed for the application, comprising the following steps:
S101 generates current tables of data according to the structural metadata information of increment list and preset task template and appoints
Business.
As stated in the background art, the full dose table construction demand of basal layer is more in existing database and configuration is cumbersome, therefore
The tables of data that the application is directed to layer full dose table based on type in a preferred embodiment carries out generation processing.In order to it is subsequent can
The production operation of tables of data is rapidly carried out, technical staff can carry out the initialization of uniform data table generation before this step
Operation builds table statement and the data initialization according to structural metadata information generation is corresponding with the tables of data
Script.
In specific application scenarios, incremental data table naming method are as follows: ods_ { source system table name } _ delta;And full dose
The naming method of tables of data are as follows: ods_ { source system table name }.
It is realized since existing basal layer full dose table is most of by full dose table merge task, the application's
In one preferred embodiment, task template can be specifically configured to merge task template, and root in this step by technical staff
According to increment list structural metadata information and batch merge task template at merge task code, then by merge task code
Preset code library is uploaded to as the tables of data task.In specific application scenarios, Mass production basal layer full dose table
Merge task template format is as shown in table 1 below:
Table 1
Further, since the method for building up and device of ODS layers of full dose table of data warehouse are a highly integrated schemes, wherein
Relate to metadata, beyond the clouds code library, scheduling dependences, data merge task, data initialization, issue it is online etc. a series of
Process.It therefore, can be in the structural metadata according to increment list in order to further improve automatization level and treatment effeciency
Information and preset task template are directed to increment list structural metadata information before generating current tables of data task carries out
Pretreatment.Specifically, in the preferred embodiment of the application, increment list is synchronized to data warehouse before this step,
And the structural metadata information of increment list is obtained, the metadata clothes for the table structure information for being capable of providing data warehouse are established with this
Subsequent processing is convenient in business.
As shown in Fig. 2, the flow diagram of the data table generating method proposed by the application specific embodiment, in the figure
Other than comprising synchronization center module described above, meta data block, also relate to lower module:
Middle layer: integrate using the base layer data synchronized a data of precipitating in Construction of Data Warehouse
Layer, it is therefore intended that subsequent applications is facilitated to use business datum;
Beyond the clouds: the Integrated Development Environment of data warehouse exploitation, by the way that the synchronous configuration of data, model can be carried out beyond the clouds
The operation such as design, ETL exploitation, unit testing, task publication, O&M.
Scheduling: data warehouse task is carried out to have adjusted the system executed automatically according to configuration.
Based on above-mentioned module, which executes following steps in early period:
Step 1: by synchronous central synchronous incremental data to data warehouse;
Step 2.1: meta data block generates full dose table according to increment list structural information and builds table statement, and sends it to
Cloud module;
Step 2.2: meta data block is according to template and increment list structural generation Merge task.
In above-mentioned steps, the name format of incremental data table are as follows: ods_ { source system table name } _ delta;Full dose tables of data
Name format are as follows: ods_ { source system table name }.Modules are established and are inquired to table with this.In synchronous central synchronous
During, the name format of synchronous task are as follows: imp_ { table name of ODPS };The name format of merging task are as follows: mrg_ { ODPS
Table name.
Since meta data block is capable of providing the table structure information of data warehouse, and code library module being capable of basis beyond the clouds
Increment list structural metadata information and batch merge task template generate merge task code and are taken by code library beyond the clouds
Business submits code library to save, therefore after safeguarding information according to Mass production basal layer full dose table merge task template
The system for being exclusively used in tables of data generation will be uploaded to by Excel file, and by calling meta data block and according to increment list
Structural information generate basal layer full dose table build table statement and data initializtion script, call beyond the clouds code library module generate
Merge task.
S102 is configured according to scheduling information of the task template to the tables of data task.
The early-stage preparations generated for tables of data are completed by the processing of S101 and S102, which is mainly used for
It is configured for contents such as node, scripts required for specific generate.In order to realize the scheduling configuration for being directed to tables of data and generating
Service, in the application preferred embodiment, this step generates and the tables of data task pair according to the merge task template
The preposition dependence node answered, task output name, scheduler task baseline, scheduler task owner.
In the specific embodiment of Fig. 2, after having received Merge task by step 2.2, module is in step beyond the clouds
In 2.3 according to template configuration scheduling information be Merge task configure relevant information, and call scheduling configuration service creation scheduling according to
Rely and merge task exports title.For the content shown in the table 1, according to batch merge task mould in the specific embodiment
Plate calls the preposition dependence node of scheduling configuration service creation merge task, task output name, scheduler task baseline, dispatches and appoint
Be engaged in owner.
S103 builds table statement and initializtion script according to the tables of data task and scheduling information execution, with
Generate tables of data.
After generating tables of data task by S101 and finishing scheduling information by S102 configuration, scheduler module exists
Indicate that system will build table statement, initializtion script, merge task and be packaged and be published on line in step 3.Subsequent module beyond the clouds
Table statement is built by the execution of step 4.1 instruction system, after table statement and initializtion script are built in system execution, module beyond the clouds
Initialization data processing is carried out by step 4.2.
By the above process, quickly generating and creating for tables of data is realized, builds table instead of traditional manual preparation
Sentence, hand-coding basal layer full dose table merge task, manual configuration scheduling dependence, hand-coding data initialization script, hand
Work is packaged, by hand publication, execute by hand build table statement and initializtion script etc. the work that is accomplished manually of needs, realize data
Table automatically generates, and improves the working efficiency of data warehouse.
To reach the above technical purpose, the application also proposed a kind of tables of data generating device, as shown in Figure 2, comprising:
Generation module 210, it is current for being generated according to the structural metadata information of increment list and preset task template
Tables of data task;
Configuration module 220, for being configured according to scheduling information of the task template to the tables of data task;
Execution module 230, for according to the tables of data task and the scheduling information execution build table statement and just
Beginningization script, to generate tables of data.
In specific application scenarios, the tables of data is specially basal layer full dose table, further includes:
Initialization module, for according to the structural metadata information generate it is corresponding with the tables of data build table statement with
And the data initialization script.
In specific application scenarios, further includes:
Synchronization module for the increment list to be synchronized to data warehouse, and obtains the structural elements of the increment list
Data information.
In specific application scenarios, the generation module is specifically used for:
According to increment list structural metadata information and batch merge task template at merge task code, and will be described
Merge task code is uploaded to preset code library as the tables of data task.
In specific application scenarios, the configuration module is specifically used for:
Preposition dependence node corresponding with the tables of data task, task output are generated according to the merge task template
Name, scheduler task baseline, scheduler task owner.
Through the above description of the embodiments, those skilled in the art can be understood that the application can lead to
Hardware realization is crossed, the mode of necessary general hardware platform can also be added to realize by software.Based on this understanding, this Shen
Technical solution please can be embodied in the form of software products, which can store in a non-volatile memories
In medium (can be CD-ROM, USB flash disk, mobile hard disk etc.), including some instructions are used so that a computer equipment (can be
Personal computer, server or network equipment etc.) execute method described in each implement scene of the application.
It will be appreciated by those skilled in the art that the accompanying drawings are only schematic diagrams of a preferred implementation scenario, module in attached drawing or
Process is not necessarily implemented necessary to the application.
It will be appreciated by those skilled in the art that the module in device in implement scene can be described according to implement scene into
Row is distributed in the device of implement scene, can also be carried out corresponding change and is located at the one or more dresses for being different from this implement scene
In setting.The module of above-mentioned implement scene can be merged into a module, can also be further split into multiple submodule.
Above-mentioned the application serial number is for illustration only, does not represent the superiority and inferiority of implement scene.
Disclosed above is only several specific implementation scenes of the application, and still, the application is not limited to this, Ren Heben
What the technical staff in field can think variation should all fall into the protection scope of the application.