CN112965962A - Industry website data conversion method and device and industry website - Google Patents
Industry website data conversion method and device and industry website Download PDFInfo
- Publication number
- CN112965962A CN112965962A CN202110149446.0A CN202110149446A CN112965962A CN 112965962 A CN112965962 A CN 112965962A CN 202110149446 A CN202110149446 A CN 202110149446A CN 112965962 A CN112965962 A CN 112965962A
- Authority
- CN
- China
- Prior art keywords
- data
- source
- standard
- database
- standard data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/258—Data format conversion from or to a database
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses an industry website data conversion method, an industry website data conversion device and an industry website, and belongs to the field of data processing; firstly, acquiring source data in a source database, and then converting the source data into standard data according to a preset accompanying database; and finally, loading the standard data into a target database. The scheme of the application converts source data in various formats and forms into unified standard data through coding; when data needs to be changed or added, only the code corresponding to the source data in the configuration database needs to be changed, and the data does not need to be developed or the existing content needs to be changed, so that the development rate is greatly increased, and the development cost is saved; while at the same time increasing the stability of the developed system.
Description
Technical Field
The present invention relates to data processing technologies, and in particular, to an industry website data conversion method, an industry website data conversion device, and an industry website.
Background
The purpose of the industry website is to provide industry consultations and industry data for users, so that information support is provided for activities such as industry marketing and purchasing, and the like, and the user operation benefit is improved. The existing service scheme takes a name set by a data release mechanism as a data name, when new data or a data name is changed, developers are required to develop the data again and change the content of the existing system, the development efficiency is low, the stability is poor, and with the increase of data change, the system versions are more and more, and the development cost is higher.
Disclosure of Invention
In order to overcome the defects of the prior art, the invention provides an industry website data conversion method, an industry website data conversion device and an industry website, and aims to solve the problems of low development efficiency, poor stability and high maintenance cost of the existing service method when new data or data names are changed.
The technical scheme adopted by the invention for solving the technical problems is as follows:
in a first aspect,
a method of data conversion comprising the steps of:
acquiring source data in a source database, wherein the source data comprises a data table, report data, news information and contents of an electronic periodical;
converting the source data into standard data according to a preset configuration database, wherein the configuration database comprises codes corresponding to the data, and the codes corresponding to any two kinds of data are different;
and loading the standard data into a target database.
Further, the obtaining the source data in the source database includes:
and extracting the data in the source database according to a preset extraction rule to obtain source data.
Further, the converting the source data into standard data according to a preset configuration database includes:
converting the source data into corresponding codes;
generating a temporary file according to the codes;
and converting the temporary file into a standard format to obtain standard data.
Further, before loading the standard data into the target database, the method further includes:
and performing data cleaning on the standard data, wherein the data cleaning comprises checking the consistency of the standard data and processing invalid values and missing values of the standard data.
Further, the loading the standard data into the target database includes:
classifying the standard data by a data distributor;
and distributing the standard data to a corresponding target database according to the classification result.
Further, still include:
all operations of the data conversion process are logged for review.
In a second aspect of the present invention,
a data conversion apparatus comprising:
the data acquisition module is used for acquiring source data in a source database, wherein the source data comprises a data table, report data, news information and contents of an electronic periodical;
the data conversion module is used for converting the source data into standard data according to a preset configuration database, wherein the configuration database comprises codes corresponding to the data, and the codes corresponding to any two kinds of data are different;
and the data loading module is used for loading the standard data into the target database.
In a third aspect,
an industry website comprising:
the data uploading platform and the data acquisition unit are used for receiving uploaded source data or acquiring the source data from a source database, wherein the source data comprises a data table, report data, news consultation and an electronic journal;
the configuration database is used for storing codes corresponding to the data, and the codes corresponding to any two kinds of data are different;
the data conversion server is used for converting the source data into standard data according to the configuration database and sending the standard data to the target database;
and the target database is used for receiving and storing the standard data converted by the data conversion server.
Further, the data conversion server comprises a data cleaning unit, wherein the data cleaning unit is used for checking the consistency of the standard data and processing invalid values and missing values of the standard data.
Further, still include:
and the log unit is used for storing all operations of the industry website.
This application adopts above technical scheme, possesses following beneficial effect at least:
the technical scheme of the application provides an industry website data conversion method, an industry website data conversion device and an industry website, wherein source data in a source database are obtained firstly, and then the source data are converted into standard data according to a preset accompanying database; and finally, loading the standard data into a target database. The scheme of the application converts source data in various formats and forms into unified standard data through coding; when data needs to be changed or added, only the code corresponding to the source data in the configuration database needs to be changed, and the data does not need to be developed or the existing content needs to be changed, so that the development rate is greatly increased, and the development cost is saved; while at the same time increasing the stability of the developed system.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
FIG. 1 is a flow chart of a data conversion method according to an embodiment of the present invention;
FIG. 2 is a flow chart of a specific method for data transformation according to an embodiment of the present invention;
fig. 3 is a block diagram of a data conversion apparatus according to an embodiment of the present invention;
fig. 4 is a schematic diagram of an industry website structure according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the following detailed description of the technical solutions of the present invention is provided with reference to the accompanying drawings and examples. It is to be understood that the embodiments described are only a few embodiments of the present application and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the examples given herein without making any creative effort, shall fall within the protection scope of the present application.
Referring to fig. 1, an embodiment of the present invention provides a data conversion method, including the following steps:
acquiring source data in a source database, wherein the source data comprises a data table, report data, news information and contents of an electronic periodical;
converting source data into standard data according to a preset configuration database, wherein the configuration database comprises codes corresponding to the data, and the codes corresponding to any two kinds of data are different;
and loading the standard data into the target database.
The embodiment of the invention provides an industry website data conversion method, which comprises the steps of firstly obtaining source data in a source database, and then converting the source data into standard data according to a preset accompanying database; and finally, loading the standard data into a target database. The embodiment of the invention converts source data in various formats and forms into unified standard data through coding; when data needs to be changed or added, only the code corresponding to the source data in the configuration database needs to be changed, and the data does not need to be developed or the existing content needs to be changed, so that the development rate is greatly increased, and the development cost is saved; while at the same time increasing the stability of the developed system.
As a supplementary explanation to the above embodiment, as shown in fig. 2, acquiring a source data in a source database includes: and extracting the data in the source database according to a preset extraction rule to obtain source data. It can be understood that there are many source data in the source database, and some data may not be needed, so when the source data is acquired, only the needed source data is extracted according to the preset extraction rule.
In some optional embodiments, converting the source data into the standard data according to a preset configuration database includes: converting the source data into corresponding codes; generating a temporary file according to the codes; and converting the temporary file into a standard format to obtain standard data. Optionally, before loading the standard data into the target database, the method further includes: and performing data cleaning on the standard data, wherein the data cleaning comprises checking the consistency of the standard data and processing invalid values and missing values of the standard data. It should be noted that, because the source data includes multiple formats or the same data has different names in different source databases, all the formats need to be converted into the same format and the same name in the conversion process. Therefore, the existing coding mode is adopted, each kind of data adopts one code, the data which is not in the source database is also provided with the codes, and if the data needs to be added newly, the codes are directly added. If the data type is to be changed, only the corresponding code in the configuration database needs to be changed. No changes to the data are required. The method is simpler and quicker when upgrading is carried out, and the maintenance cost can be reduced. The development efficiency is improved.
As an optional implementation manner of the embodiment of the present invention, loading the standard data into the target database includes: classifying the standard data through a data distributor; and distributing the standard data to the corresponding target database according to the classification result.
During actual use, all operations of the data conversion process are also logged for review.
In one embodiment, the present invention provides a data conversion apparatus, as shown in fig. 3, including:
the data acquisition module 31 is configured to acquire source data in a source database, where the source data includes data tables, report data, news information, and contents of an electronic periodical; specifically, the data acquisition module extracts data in the source database according to a preset extraction rule to obtain source data.
The data conversion module 32 is configured to convert the source data into standard data according to a preset configuration database, where the configuration database includes codes corresponding to the data, and the codes corresponding to any two kinds of data are different; specifically, the data conversion module converts the source data into corresponding codes; generating a temporary file according to the codes; and converting the temporary file into a standard format to obtain standard data. Optionally, the data conversion module performs data cleaning on the standard data, where the data cleaning includes checking consistency of the standard data and processing invalid values and missing values of the standard data.
And a data loading module 33, configured to load the standard data into the target database. Specifically, the data loading module classifies the standard data through the data distributor; and distributing the standard data to the corresponding target database according to the classification result.
And a logging module 34 for logging all operations of the data conversion process for review.
In the data conversion apparatus provided in the embodiment of the present invention, the data acquisition module acquires source data in a source database; the data conversion module converts the source data into standard data according to a preset configuration database; and the data loading module loads the standard data into the target database. The logging module logs all operations of the data conversion process for review. The conversion device provided by the embodiment of the invention can standardize the data in the source database and obtain the standard data from the source data in various formats or forms. The method is beneficial to diversified application of data, simultaneously adopts the same code, has more controllable data service and higher accuracy, can improve the development efficiency and save the development cost.
In one embodiment, an embodiment of the present invention provides an industry website, as shown in fig. 4, including:
a data uploading platform and data collector 41, configured to receive uploaded source data or obtain source data from a source database 42, where the source data includes data tables, report data, news advisories, and electronic periodicals;
a configuration database 43 for storing codes corresponding to data, wherein the codes corresponding to any two kinds of data are different;
a data conversion server 44 for converting the source data into standard data according to the configuration database and transmitting the standard data to the target database; optionally, the data conversion server comprises a data cleansing unit 47 for checking the consistency of the standard data, processing invalid values and missing values of the standard data. Preferably, the data conversion server employs an ETL server.
And the target database 45 is used for receiving and storing the standard data converted by the data conversion server.
And the log unit 46 is used for storing all operations of the industry website.
The industrial website provided by the embodiment of the invention acquires data of contents such as data tables, report data, news consultation, electronic periodicals and the like through the data uploading platform and the data acquisition unit, presets codes corresponding to the data in the configuration database, standardizes the data format and the data through the data conversion server, and finally distributes the data to the target database. Providing standardized service for industry data. Therefore, the stability and controllability of the data can be ensured, and various conversion requirements and display services for the data can be met. Compared with the traditional industry website, the embodiment is easy to expand to diversified application of data; the data service is more controllable and the accuracy is higher; the data versions can be rapidly iterated, development efficiency is improved, and development cost is saved.
It is understood that the same or similar parts in the above embodiments may be mutually referred to, and the same or similar parts in other embodiments may be referred to for the content which is not described in detail in some embodiments.
It should be noted that, in the description of the present application, the terms "first", "second", etc. are used for descriptive purposes only and are not to be construed as indicating or implying relative importance. Further, in the description of the present application, the meaning of "a plurality" means at least two unless otherwise specified.
Any process or method descriptions in flow charts or otherwise described herein may be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or steps of the process, and the scope of the preferred embodiments of the present application includes other implementations in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those reasonably skilled in the art of the present application.
It should be understood that portions of the present application may be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the various steps or methods may be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one or combination of the following techniques, which are known in the art, may be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application specific integrated circuit having an appropriate combinational logic gate circuit, a Programmable Gate Array (PGA), a Field Programmable Gate Array (FPGA), or the like.
It will be understood by those skilled in the art that all or part of the steps carried by the method for implementing the above embodiments may be implemented by hardware related to instructions of a program, which may be stored in a computer readable storage medium, and when the program is executed, the program includes one or a combination of the steps of the method embodiments.
In addition, functional units in the embodiments of the present application may be integrated into one processing module, or each unit may exist alone physically, or two or more units are integrated into one module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode. The integrated module, if implemented in the form of a software functional module and sold or used as a stand-alone product, may also be stored in a computer readable storage medium.
The storage medium mentioned above may be a read-only memory, a magnetic or optical disk, etc.
In the description herein, reference to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the application. In this specification, the schematic representations of the terms used above do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
Although embodiments of the present application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present application, and that variations, modifications, substitutions and alterations may be made to the above embodiments by those of ordinary skill in the art within the scope of the present application.
Claims (10)
1. A method of data conversion, comprising the steps of:
acquiring source data in a source database, wherein the source data comprises a data table, report data, news information and contents of an electronic periodical;
converting the source data into standard data according to a preset configuration database, wherein the configuration database comprises codes corresponding to the data, and the codes corresponding to any two kinds of data are different;
and loading the standard data into a target database.
2. The method of claim 1, wherein: the obtaining of the source data in the source database includes:
and extracting the data in the source database according to a preset extraction rule to obtain source data.
3. The method of claim 1, wherein: the converting the source data into standard data according to a preset configuration database includes:
converting the source data into corresponding codes;
generating a temporary file according to the codes;
and converting the temporary file into a standard format to obtain standard data.
4. The method of claim 1, wherein: before the loading the standard data into the target database, the method further comprises:
and performing data cleaning on the standard data, wherein the data cleaning comprises checking the consistency of the standard data and processing invalid values and missing values of the standard data.
5. The method of claim 1, wherein: the loading the standard data into the target database comprises:
classifying the standard data by a data distributor;
and distributing the standard data to a corresponding target database according to the classification result.
6. The method of claim 1, further comprising:
all operations of the data conversion process are logged for review.
7. A data conversion apparatus, comprising:
the data acquisition module is used for acquiring source data in a source database, wherein the source data comprises a data table, report data, news information and contents of an electronic periodical;
the data conversion module is used for converting the source data into standard data according to a preset configuration database, wherein the configuration database comprises codes corresponding to the data, and the codes corresponding to any two kinds of data are different;
and the data loading module is used for loading the standard data into the target database.
8. An industry website, comprising:
the data uploading platform and the data acquisition unit are used for receiving uploaded source data or acquiring the source data from a source database, wherein the source data comprises a data table, report data, news consultation and an electronic journal;
the configuration database is used for storing codes corresponding to the data, and the codes corresponding to any two kinds of data are different;
the data conversion server is used for converting the source data into standard data according to the configuration database and sending the standard data to the target database;
and the target database is used for receiving and storing the standard data converted by the data conversion server.
9. The industry website of claim 8, wherein: the data conversion server comprises a data cleaning unit, and the data cleaning unit is used for checking the consistency of the standard data and processing invalid values and missing values of the standard data.
10. The industry website of claim 8, further comprising:
and the log unit is used for storing all operations of the industry website.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110149446.0A CN112965962A (en) | 2021-02-03 | 2021-02-03 | Industry website data conversion method and device and industry website |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110149446.0A CN112965962A (en) | 2021-02-03 | 2021-02-03 | Industry website data conversion method and device and industry website |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112965962A true CN112965962A (en) | 2021-06-15 |
Family
ID=76274288
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110149446.0A Pending CN112965962A (en) | 2021-02-03 | 2021-02-03 | Industry website data conversion method and device and industry website |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112965962A (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20050131970A1 (en) * | 2003-12-15 | 2005-06-16 | International Business Machines Corporation | Customizable data translation method and system |
| CN108984652A (en) * | 2018-06-27 | 2018-12-11 | 北京圣康汇金科技有限公司 | A kind of configurable data cleaning system and method |
| CN110781230A (en) * | 2019-09-12 | 2020-02-11 | 腾讯大地通途（北京）科技有限公司 | Data access method, device and equipment |
| CN111241182A (en) * | 2020-01-19 | 2020-06-05 | 北京奇艺世纪科技有限公司 | Data processing method and apparatus, storage medium, and electronic apparatus |
-
2021
- 2021-02-03 CN CN202110149446.0A patent/CN112965962A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20050131970A1 (en) * | 2003-12-15 | 2005-06-16 | International Business Machines Corporation | Customizable data translation method and system |
| CN108984652A (en) * | 2018-06-27 | 2018-12-11 | 北京圣康汇金科技有限公司 | A kind of configurable data cleaning system and method |
| CN110781230A (en) * | 2019-09-12 | 2020-02-11 | 腾讯大地通途（北京）科技有限公司 | Data access method, device and equipment |
| CN111241182A (en) * | 2020-01-19 | 2020-06-05 | 北京奇艺世纪科技有限公司 | Data processing method and apparatus, storage medium, and electronic apparatus |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN101040444B (en) | Method and apparatus for compressing structuring document | |
| US7320003B2 (en) | Method and system for storing and retrieving document data using a markup language string and a serialized string | |
| CN113518094B (en) | Data processing method, device, robot and storage medium | |
| US8880463B2 (en) | Standardized framework for reporting archived legacy system data | |
| CN110737689B (en) | Data standard compliance detection method, device, system and storage medium | |
| CN111370132B (en) | Electronic file analysis method and device, computer equipment and storage medium | |
| CN108228755A (en) | The data of MySQL database based on daily record analytic technique to Hadoop platform synchronize clone method | |
| US20020178192A1 (en) | Data integrate system and data integrate method | |
| CN108228756A (en) | Data based on the PG databases of daily record analytic technique to Hadoop platform synchronize clone method | |
| CN100382075C (en) | Hardware-level based test script automatic generating system and method | |
| Nour | Performance of different (BIM/IFC) exchange formats within private collaborative workspace for collaborative work | |
| CN112965962A (en) | Industry website data conversion method and device and industry website | |
| CN112699183A (en) | Data processing method, system, readable storage medium and computer equipment | |
| CN110765196A (en) | Method and equipment for generating and executing ETL task | |
| CN113659993B (en) | Immune batch data processing method and device, terminal and readable storage medium | |
| KR101508068B1 (en) | Apparatus and method for data de-duplication | |
| CN110957016B (en) | Physical examination data intelligent identification system and method based on health cloud management platform | |
| CN102310950B (en) | Experiment intelligent interpretation system and method | |
| CN106230646B (en) | A kind of method and test macro obtaining network management alarm information | |
| CN115357553A (en) | Internet of things client data processing method and device | |
| CN113515515B (en) | Method, device, equipment and medium for fusing customer data and power grid equipment data | |
| CN113778886B (en) | Processing method and device for test cases | |
| CN115269708A (en) | Data reconciliation processing method and device | |
| CN117520324A (en) | Government affair data cleaning method and device, electronic equipment and storage medium | |
| CN112650809A (en) | Method and device for formatting tree structure data and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210615 |
|
| RJ01 | Rejection of invention patent application after publication |