CN111124427A - Method, system and equipment for extracting and integrating data - Google Patents
Method, system and equipment for extracting and integrating data Download PDFInfo
- Publication number
- CN111124427A CN111124427A CN201911107150.1A CN201911107150A CN111124427A CN 111124427 A CN111124427 A CN 111124427A CN 201911107150 A CN201911107150 A CN 201911107150A CN 111124427 A CN111124427 A CN 111124427A
- Authority
- CN
- China
- Prior art keywords
- data
- extracting
- server
- program
- integrating
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 30
- 238000004590 computer program Methods 0.000 claims description 6
- 238000000605 extraction Methods 0.000 claims description 6
- 230000006870 function Effects 0.000 claims description 4
- 230000002776 aggregation Effects 0.000 claims description 2
- 238000004220 aggregation Methods 0.000 claims description 2
- 238000013075 data extraction Methods 0.000 abstract description 7
- 238000013523 data management Methods 0.000 abstract description 3
- 238000012423 maintenance Methods 0.000 abstract description 3
- 230000008878 coupling Effects 0.000 description 5
- 238000010168 coupling process Methods 0.000 description 5
- 238000005859 coupling reaction Methods 0.000 description 5
- 230000010354 integration Effects 0.000 description 4
- 238000012545 processing Methods 0.000 description 4
- 230000008569 process Effects 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 239000000284 extract Substances 0.000 description 2
- 238000011161 development Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000000547 structure data Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/60—Software deployment
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/547—Remote procedure calls [RPC]; Web services
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Stored Programmes (AREA)
Abstract
The invention provides a method, a system and equipment for extracting and integrating data, which comprises the following steps: deploying a preset interface program to a server IIS; accessing a preset interface program by configuring related IP and ports; reading and executing a preset extraction program on a storage server; and extracting data on the data server and transmitting the data to the storage server. The method for extracting the integrated data through the interface deployment ensures the integrity of data extraction, reduces the difficulty of system maintenance and improves the capability of data management.
Description
Technical Field
The invention relates to the technical field of computers, in particular to a method, a system and equipment for extracting and integrating data.
Background
Computer software data extraction integration, including data integration and data extraction; data integration is the logical or physical organic collection of data of different sources, formats, and characteristic properties. Data integration throughThe data exchange between applications is integrated, mainly solving the problem of data distribution and heterogeneity, and its premise is that the integrated application must disclose data structure, i.e. must disclose table structure, relationship between tables, meaning of code, etc., and data extraction is implemented by using data extracted from the table structureData sourceIn the process of extracting data, in practical application, a relational database is frequently adopted as a data source.
Data extraction is the most common in the field of computer software, when one server side simultaneously extracts N data server sides and uniformly extracts data to a storage server side, the data quantity of each data server side is very large, and at the moment, if the server configuration is poor, data loss cannot be avoided, so that great inconvenience is brought.
Disclosure of Invention
In view of the above problems, an object of the present invention is to provide a method, a system, and a device for extracting and integrating data, which can deploy a method for extracting and integrating integrated data through an interface, thereby ensuring the integrity of data extraction, reducing the difficulty of system maintenance, and improving the capability of data management.
In order to achieve the purpose, the invention is realized by the following technical scheme: a method of extracting and integrating data, comprising the steps of:
step 1: deploying a preset interface program to a server IIS;
step 2: accessing a preset interface program by configuring related IP and ports;
and step 3: reading and executing a preset extraction program on a storage server;
and 4, step 4: and extracting data on the data server and transmitting the data to the storage server.
Further, in the step 1, the Nancy framework + IIS server is adopted as the preset interface program.
Further, the step 2 further comprises:
by accessing a preset interface program, a universal interface system is used for realizing the function of operating data by a client through an interface; the general interface system comprises a main server for extracting the aggregation library and a sub-server for deploying the interface; the main server is provided with a data platform data extraction program, and the sub-servers are provided with a large data platform general interface program.
Further, in the step 3, a Winform + NHibernate framework is adopted in a preset extraction program.
Further, in the step 3, the data is transmitted in a Json format and read in a streaming manner.
Further, step 4 further comprises: and the storage server side respectively calls the data on the data server in a multithreading mode.
Correspondingly, the invention also discloses a system for extracting and integrating data, which comprises:
the deployment unit is used for deploying the preset interface program to the server IIS;
the access unit is used for accessing the preset interface program by configuring related IP and ports;
the execution unit is used for reading and executing a preset extraction program on the storage server;
and the extraction unit is used for extracting the data on the data server and transmitting the data to the storage server.
Correspondingly, the invention also discloses a device for extracting and integrating data, which comprises:
a memory for storing a computer program;
a processor for implementing the method steps of extracting and integrating data as described in any of the above when said computer program is executed.
Compared with the prior art, the invention has the beneficial effects that: the invention provides a method, a system and equipment for extracting and integrating data, wherein the method for extracting and integrating data is deployed through an interface, so that the coupling is reduced, a certain module or function can be repeatedly used, the working efficiency is improved, the data extraction integrity is realized, the system maintenance difficulty is greatly reduced, and the data management capability is improved.
Therefore, compared with the prior art, the invention has prominent substantive features and remarkable progress, and the beneficial effects of the implementation are also obvious.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
FIG. 1 is a flow chart of the method of the present invention.
Fig. 2 is a system block diagram of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made with reference to the accompanying drawings.
A method of extracting and integrating data as shown in fig. 1, comprising the steps of:
step 1: and deploying the preset interface program to the server IIS.
The preset interface program is independently researched and developed, an Mvc lightweight Nancy framework + IIS server is adopted, loose coupling is achieved based on the independently researched and developed interface specification by utilizing the development language support capability, and a data extraction system at the data end can be kept stable. The deployment interface can also provide purposes for other developers, and the other developers can conveniently inquire related data through the interface.
The call is carried out through the interface program of each data server side. For example:
step 2: and accessing the preset interface program by configuring the related IP and the port.
In step 2, the preset interface program can be normally accessed through setting the relevant IP and the port, the universal interface system realizes the function of operating data by the client through the interface, and the system consists of a convergence library extraction main server and an interface deployment sub-server. The data platform data extraction program is installed in the main service, and the large data platform general interface program is configured in the sub-servers, so that the process of extracting data from each sub-server to the main server is realized.
And step 3: and reading and executing a preset extraction program on the storage server.
In step 3, the preset extraction program is independently developed, Winform + lightweight ORM frame NHibernate is adopted, and when the data extraction program is opened, the data of the configured interface address server can be completely extracted and transmitted to the storage server in a button clicking mode.
In step 3, the data is transmitted in Json format and read in streaming fashion. For example:
and 4, step 4: and extracting data on the data server and transmitting the data to the storage server.
In step 4, the storage server side calls the data of the data server respectively in a multithreading mode, so that the condition that the process is blocked and the false death of the storage server extraction system is avoided. For example:
correspondingly, as shown in fig. 2, the present invention also discloses a system for extracting and integrating data, comprising: the deployment unit is used for deploying the preset interface program to the server IIS;
the access unit is used for accessing the preset interface program by configuring related IP and ports;
the execution unit is used for reading and executing a preset extraction program on the storage server;
and the extraction unit is used for extracting the data on the data server and transmitting the data to the storage server.
Correspondingly, the invention also discloses a device for extracting and integrating data, which comprises:
a memory for storing a computer program;
a processor for implementing the method steps of extracting and integrating data as described in any of the above when said computer program is executed.
Those skilled in the art will readily appreciate that the techniques of the embodiments of the present invention may be implemented as software plus a required general purpose hardware platform. Based on such understanding, the technical solutions in the embodiments of the present invention may be embodied in the form of a software product, where the computer software product is stored in a storage medium, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and the like, and the storage medium can store program codes, and includes instructions for enabling a computer terminal (which may be a personal computer, a server, or a second terminal, a network terminal, and the like) to perform all or part of the steps of the method in the embodiments of the present invention. The same and similar parts in the various embodiments in this specification may be referred to each other. Especially, for the terminal embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant points can be referred to the description in the method embodiment.
In the embodiments provided by the present invention, it should be understood that the disclosed system, system and method can be implemented in other ways. For example, the above-described system embodiments are merely illustrative, and for example, the division of the units is only one logical functional division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, systems or units, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional modules in the embodiments of the present invention may be integrated into one processing unit, or each module may exist alone physically, or two or more modules are integrated into one unit.
Similarly, each processing unit in the embodiments of the present invention may be integrated into one functional module, or each processing unit may exist physically, or two or more processing units are integrated into one functional module.
The invention is further described with reference to the accompanying drawings and specific embodiments. It should be understood that these examples are for illustrative purposes only and are not intended to limit the scope of the present invention. Further, it should be understood that various changes or modifications of the present invention may be made by those skilled in the art after reading the teaching of the present invention, and these equivalents also fall within the scope of the present application.
Claims (8)
1. A method of extracting and integrating data, comprising the steps of:
step 1: deploying a preset interface program to a server IIS;
step 2: accessing a preset interface program by configuring related IP and ports;
and step 3: reading and executing a preset extraction program on a storage server;
and 4, step 4: and extracting data on the data server and transmitting the data to the storage server.
2. The method for extracting and integrating data as claimed in claim 1, wherein in the step 1, the default interface program employs a Nancy framework + IIS server.
3. The method of extracting and integrating data as claimed in claim 1, wherein said step 2 further comprises:
by accessing a preset interface program, a universal interface system is used for realizing the function of operating data by a client through an interface; the general interface system comprises a main server for extracting the aggregation library and a sub-server for deploying the interface; the main server is provided with a data platform data extraction program, and the sub-servers are provided with a large data platform general interface program.
4. The method for extracting and integrating data as claimed in claim 1, wherein in the step 3, the default extraction procedure adopts Winform + NHibernate framework.
5. The method of extracting and integrating data as claimed in claim 1, wherein in said step 3, the data is transmitted in a Json format and read in a streaming manner.
6. The method for extracting and integrating data as claimed in claim 1, further comprising at said step 4: and the storage server side respectively calls the data on the data server in a multithreading mode.
7. A system for extracting and integrating data, comprising:
the deployment unit is used for deploying the preset interface program to the server IIS;
the access unit is used for accessing the preset interface program by configuring related IP and ports;
the execution unit is used for reading and executing a preset extraction program on the storage server;
and the extraction unit is used for extracting the data on the data server and transmitting the data to the storage server.
8. An apparatus for extracting and integrating data, comprising:
a memory for storing a computer program;
processor for implementing the method steps of extracting and integrating data according to any one of claims 1 to 6 when executing said computer program.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911107150.1A CN111124427A (en) | 2019-11-13 | 2019-11-13 | Method, system and equipment for extracting and integrating data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911107150.1A CN111124427A (en) | 2019-11-13 | 2019-11-13 | Method, system and equipment for extracting and integrating data |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111124427A true CN111124427A (en) | 2020-05-08 |
Family
ID=70495596
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911107150.1A Pending CN111124427A (en) | 2019-11-13 | 2019-11-13 | Method, system and equipment for extracting and integrating data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111124427A (en) |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20040010796A1 (en) * | 2002-07-09 | 2004-01-15 | Santanu Paul | Adaptive content platform and method of using same |
| CN101562606A (en) * | 2008-04-17 | 2009-10-21 | 鸿富锦精密工业（深圳）有限公司 | Open type data calling system and method |
| CN102902750A (en) * | 2012-09-20 | 2013-01-30 | 浪潮齐鲁软件产业有限公司 | Universal data extraction and conversion method |
| CN107291957A (en) * | 2017-07-31 | 2017-10-24 | 袁佳宁 | A kind of miniature service system and its implementation based on data message intercommunication |
| CN107957830A (en) * | 2016-10-14 | 2018-04-24 | 富士通株式会社 | Development support system, development supporting apparatus, response control mehtod and responsive control device |
| CN108415710A (en) * | 2018-03-14 | 2018-08-17 | 苏州思必驰信息科技有限公司 | The method and system of API is issued, called in Intelligent dialogue development platform |
| CN109525474A (en) * | 2018-12-06 | 2019-03-26 | 连云港杰瑞深软科技有限公司 | A kind of multiplex roles protocol-intelligent converter and conversion method based on Linux platform |
| CN110262904A (en) * | 2019-05-17 | 2019-09-20 | 北京达佳互联信息技术有限公司 | Collecting method and device |
| WO2019211122A1 (en) * | 2018-05-02 | 2019-11-07 | Robert Bosch Gmbh | Feature development framework and feature integration framework for implementing physical functional features in a target device |
-
2019
- 2019-11-13 CN CN201911107150.1A patent/CN111124427A/en active Pending
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20040010796A1 (en) * | 2002-07-09 | 2004-01-15 | Santanu Paul | Adaptive content platform and method of using same |
| CN101562606A (en) * | 2008-04-17 | 2009-10-21 | 鸿富锦精密工业（深圳）有限公司 | Open type data calling system and method |
| CN102902750A (en) * | 2012-09-20 | 2013-01-30 | 浪潮齐鲁软件产业有限公司 | Universal data extraction and conversion method |
| CN107957830A (en) * | 2016-10-14 | 2018-04-24 | 富士通株式会社 | Development support system, development supporting apparatus, response control mehtod and responsive control device |
| CN107291957A (en) * | 2017-07-31 | 2017-10-24 | 袁佳宁 | A kind of miniature service system and its implementation based on data message intercommunication |
| CN108415710A (en) * | 2018-03-14 | 2018-08-17 | 苏州思必驰信息科技有限公司 | The method and system of API is issued, called in Intelligent dialogue development platform |
| WO2019211122A1 (en) * | 2018-05-02 | 2019-11-07 | Robert Bosch Gmbh | Feature development framework and feature integration framework for implementing physical functional features in a target device |
| CN109525474A (en) * | 2018-12-06 | 2019-03-26 | 连云港杰瑞深软科技有限公司 | A kind of multiplex roles protocol-intelligent converter and conversion method based on Linux platform |
| CN110262904A (en) * | 2019-05-17 | 2019-09-20 | 北京达佳互联信息技术有限公司 | Collecting method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN206249316U (en) | A kind of Data Centre in Hospital plateform system based on high in the clouds | |
| CN103581250B (en) | Data processing, publication, acquisition methods and system | |
| CN107256206A (en) | The method and apparatus of character stream format conversion | |
| CN109471893B (en) | Network data query method, equipment and computer readable storage medium | |
| CN110019539A (en) | A kind of method and apparatus that the data of data warehouse are synchronous | |
| CN110795443A (en) | Method, device, equipment and computer readable medium for data synchronization | |
| CN113641700A (en) | Data processing method and device based on Spring boot frame | |
| CN110502560A (en) | A kind of method and server of Database Connection Parameters encapsulation | |
| CN110008284A (en) | Method for synchronizing data of database and equipment based on data page preloading and rollback | |
| CN111124427A (en) | Method, system and equipment for extracting and integrating data | |
| CN107729394A (en) | Data Mart management system and its application method based on Hadoop clusters | |
| CN108984221B (en) | Method and device for acquiring multi-platform user behavior logs | |
| US20120317073A1 (en) | Replication Support for Procedures with Arguments of Unsupported Types | |
| WO2016086700A1 (en) | Rack and communication method | |
| CN115859431A (en) | Linkage method, device and equipment of three-dimensional building model and two-dimensional drawing | |
| CN205620854U (en) | Motherboard assembly and server | |
| CN114238335A (en) | Buried point data generation method and related equipment thereof | |
| CN105376273A (en) | Standard cloud service interface and standard cloud service method | |
| CN106156339B (en) | The access method of declaration form management system, apparatus and system | |
| US9588926B2 (en) | Input/output swtiching module interface identification in a multi-server chassis | |
| CN114218224A (en) | Data processing method and device in robot service scene and electronic equipment | |
| CN117194569A (en) | Database data synchronization method and device | |
| US20240054107A1 (en) | File processing method, electronic device, and storage medium | |
| CN115086441B (en) | Information transmission method, device, electronic equipment and storage medium | |
| CN107665241A (en) | A kind of real time data various dimensions De-weight method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200508 |
|
| RJ01 | Rejection of invention patent application after publication |