CN109240757A - Configuration management system and method in a kind of big data component set - Google Patents
Configuration management system and method in a kind of big data component set Download PDFInfo
- Publication number
- CN109240757A CN109240757A CN201810812933.9A CN201810812933A CN109240757A CN 109240757 A CN109240757 A CN 109240757A CN 201810812933 A CN201810812933 A CN 201810812933A CN 109240757 A CN109240757 A CN 109240757A
- Authority
- CN
- China
- Prior art keywords
- component
- configuration
- big data
- data component
- module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/445—Program loading or initiating
- G06F9/44505—Configuring for program initiating, e.g. using registry, configuration files
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present invention relates to configuration management system and methods in a kind of big data component set.The system includes instantiation module, version control module and configuration distribution module；The instantiation module, for component configuration parameter item based on the received and built-in Global Component configuration template formation component profile instance；The version control module, for generate the component Configuration example version number and corresponding version file；The configuration distribution module makes the component on the big data component server complete configuration on line and updates and come into force for the component Configuration example to be distributed to the component on big data component server.Technical solution of the present invention can be improved the working efficiency of big data system configuration management and reduce Technology Selection cost.
Description
Technical field
The present invention relates to big data technical fields, and in particular to configuration management system and side in a kind of big data component set
Method.
Background technique
A set of big data system is built by enterprise, needs to be related to very more big data open source components.At big data
Process is managed, is generally divided into data collection section (Common Component has Sqoop, Flume, Logstash and Kakfa etc.), data storage
Partially (Common Component has HDFS, Ceph, HBase, MongoDB, Cassandra and Neo4j etc.), data calculating section are (common
Component has Hadoop, Spark and Storm etc.), scheduling of resource part (Common Component has Yarn, Mesos and Zookeeper etc.),
Data analysis component (Common Component has Hive, Pig, Impala, Presto, Spark and Kylin etc.), data visualization part
(Common Component has Tableau, Zeppelin, Qlikview and PowrerBI etc.).
In traditional big data system component configuration management mode, system operation maintenance personnel needs to log in different components respectively
Place host is modified and issues configuration, faces more and more big data components, and this mode not only needs to consume O&M
The great effort of personnel, leads to inefficiency, and real-time is insufficient, once and the configuration of the system component under production environment occur
The case where mistake, operation maintenance personnel would become hard to rapidly and accurately restore system, and big data system is likely to occur paralysis, in turn
Cause more serious consequence.
Summary of the invention
In view of the deficiencies of the prior art, the present invention provides configuration management system and method in a kind of big data component set.
On the one hand, the present invention provides configuration management system in a kind of big data component set, which includes instantiation mould
Block, version control module and configuration distribution module.
The instantiation module, it is raw for component configuration parameter item based on the received and built-in Global Component configuration template
At component profile instance.
The version control module, for generate the component Configuration example version number and corresponding version file.
The configuration distribution module, for being distributed to the component Configuration example on big data component server
On component, so that the component on the big data component server is completed configuration on line and update and come into force.
On the other hand, the present invention provides configuring management methods in a kind of big data component set, this method comprises:
Step 1, component configuration parameter item and the configuration of built-in Global Component configuration template formation component are real based on the received
Example.
Step 2, generate the component Configuration example version number and corresponding version file.
Step 3, the component Configuration example is distributed on the component on big data component server, is made described big
Component on data package server completes configuration on line and updates and come into force.
The beneficial effect of configuration management system and method is in big data component set provided by the invention, when needs are to big number
When carrying out configuration update according to the one or more components of system, it can be controlled for example, by the configuration management of Web operation interface form
The configuration parameter item of platform editor's difference component generates the configuration for being applicable to global all components based on Global Component configuration template
Example.On the one hand, since the receiving time of the configuration parameter item of multiple batches or configuration object are different, profile instance will have not
Same version, generated by version control module and record profile instance each current or old version file and corresponding version
Number, current component configure break down when, can by way of calling respective version file rapid recovery system, avoid system
The serious consequences such as paralysis.On the other hand, profile instance is distributed to positioned at big data component clothes automatically by configuring distribution module
It is engaged in the one or more components on device, so that corresponding assembly is automatically performed configuration on line and update and come into force, avoid operation maintenance personnel point
It is other that different components are configured, working efficiency is greatly improved, and reduce the Technology Selection for component Configuration management aspect
Cost.
Detailed description of the invention
In order to more clearly explain the embodiment of the invention or the technical proposal in the existing technology, to embodiment or will show below
There is attached drawing needed in technical description to be briefly described, it should be apparent that, the accompanying drawings in the following description is this hair
Bright some embodiments for those of ordinary skill in the art without any creative labor, can be with
It obtains other drawings based on these drawings.
Fig. 1 is the structural block diagram of configuration management system in a kind of big data component set of the embodiment of the present invention；
Fig. 2 is the structural block diagram of configuration management system in a kind of big data component set of another embodiment of the present invention；
Fig. 3 is the flow diagram of configuring management method in a kind of big data component set of the embodiment of the present invention.
Specific embodiment
The principle and features of the present invention will be described below with reference to the accompanying drawings, and the given examples are served only to explain the present invention, and
It is non-to be used to limit the scope of the invention.
As shown in Figure 1, in a kind of big data component set of the embodiment of the present invention configuration management system include instantiation module,
Version control module and configuration distribution module.
The instantiation module, it is raw for component configuration parameter item based on the received and built-in Global Component configuration template
At component profile instance.
The version control module, for generate the component Configuration example version number and corresponding version file.
The configuration distribution module, for being distributed to the component Configuration example on big data component server
On component, so that the component on the big data component server is completed configuration on line and update and come into force.
In the present embodiment, when needing the one or more components to big data system to carry out configuration update, can pass through
Such as the configuration parameter item of configuration management console editor's difference component of Web operation interface form, it is configured based on Global Component
Template generation is applicable to the profile instance of global all components.On the one hand, due to the reception of the configuration parameter item of multiple batches
Time or configuration object are different, and profile instance will have different editions, and be generated by version control module and record profile instance
Each current or old version file and corresponding version number can pass through when current component is configured and broken down and call phase
The rapid recovery system of the mode of version file is answered, the serious consequences such as systemic breakdown are avoided.On the other hand, by configuring distribution module
Automatically profile instance is distributed in the one or more components on big data component server, keeps corresponding assembly automatically complete
It updates and comes into force at configuration on line, avoid operation maintenance personnel from configuring respectively to different components, greatly improve working efficiency, and subtract
The Technology Selection cost for component Configuration management aspect is lacked.
It should be noted that configuration management console is as an entrance, it can be achieved that the increasing of component Configuration, deleting, changing, looking into
Deng operation.In addition, system manager can be the operating right that user configuration is different on configuration management console, all users'
Operation note is recorded in the database built in console, provides support for system audit.Pass through unified Web operation interface
Entrance manages the various configurations of big data component, so that big data system development, operation maintenance personnel carry out hand without login service device
The dynamic various configuration files of management, improve the efficiency and correctness of big data system development, O&M.
The client of the settable agent software form of big data component server receives profile instance and is deployed to line
It is allowed to come into force in the big data component of upper operation.Enabling configuration management system in the big data component set in embodiment is server-side,
Namely user can be configured server-side by configuration management console, generated profile instance by server-side and be sent to
To client, it is finally completed the deployment of configuration.After the new profile instance that client receives server-side distribution, a component
It is realized using the process that (S) deployment newly configures are as follows:
While not influencing component Configuration example (T0) operation of currently used old configuration (C0), new configuration (C1) is used
Start a New Parent profile instance (T1).
After New Parent profile instance (T1) normally starts, the component Configuration example (T0) of the old configuration of end use, new group
Part profile instance (T1) becomes running example on the current line of the component, and new configuration (C1) becomes to be configured on current line, client
It holds the result notice server-side of deployment success.
When New Parent profile instance (T1) can not normally start, terminate New Parent profile instance (T1), client is by portion
Affix one's name to the result notice server-side of failure.
Wherein, using also referred to as Service, configuration is also referred to as Configuration, and example is also referred to as
Task。
Preferably, as shown in Fig. 2, the system also includes configuration backup modules.The configuration backup module, for according to
The component Configuration example is backed up in backup strategy and executive plan.
Configuration backup module is realized based on file backup, is responsible for all profile instance file backups of server-side to target
Server.By the module, system manager can on operating console configuration backup destination server address and backup plan
Slightly, and backup result is checked.User can login configurations manage console, check component Configuration backup operating condition and backup match
It sets, and configuration backup strategy and executive plan and submits, backup services in hours or in terms of day according to for example being operated
Backup strategy and executive plan carry out.
Since all big data component Configurations can be backed up automatically by background system, change and version are automatically recorded in
Case enhances the ability of the audit of big data system configuration, version management and diagnosis etc..
Preferably, the instantiation module is specifically used for: by the way of template replacement, the Global Component being configured mould
Parameter item in plate replaces with the corresponding component configuration parameter item, obtains the component Configuration example.
Instantiation module receives each component configuration parameter item that user submits in configuration management console, and user only needs to fill in
A small amount of configuration item, the Global Component configuration template in conjunction with built in server-side are replaced using the template based on Mustache grammer
Mode, generate a profile instance of each component, that is, by template blank or existing parameter item replace with accordingly
Component configuration parameter item, being formed can be towards a component Configuration example of all components, or passes through the parameter item of replacement template
Generate different component Configuration examples.Wherein Mustache is a kind of implementation of template system in JavaScript language.
Preferably, the version control module is also used to: the corresponding version file is called according to the version number, and
The version file is distributed on the component on the big data component server via configuration distribution module.
Version control module is generated using document control (Documentation Control) and records this part of profile instance
Version number, and all configurations realities are recorded based on concurrent edition system (CVS, Concurrent Version System) realization
The current version and old version of example.
User can inquire and compare the variation of all components profile instance different editions on configuration management console.?
When there is such as current component configuration failure, configuration management console can be first logged into, is checked by version control module
The old version of component Configuration example, that is, the profile instance of some period application select some old version and call to match
The problems such as setting distribution module to be distributed, realizing the switching or rollback to component Configuration on line, avoid the paralysis of big data system.
Preferably, the configuration distribution module is specifically used for: being based on SSH and multi-process technology, parallel remote distribution is different
The component Configuration example of component.
Configuration distribution module is responsible for receiving a New Parent profile instance of instantiation module generation and is distributed to line
The big data component Configuration client of upper all operations.It configures distribution module and is based on SSH (Secure Shell) and multi-process
(Multi-processing) technology is realized, the profile instance of different components is distributed by parallel remote mode, can be greatly improved
The efficiency of component Configuration distribution.
As shown in figure 3, configuring management method includes: in a kind of big data component set of the embodiment of the present invention
Step 1, component configuration parameter item and the configuration of built-in Global Component configuration template formation component are real based on the received
Example.
Step 2, generate the component Configuration example version number and corresponding version file.
Step 3, the component Configuration example is distributed on the component on big data component server, is made described big
Component on data package server completes configuration on line and updates and come into force.
Preferably, the method also includes:
Step 4, the component Configuration example is backed up according to backup strategy and executive plan.
Preferably, the specific implementation of the step 1 are as follows: by the way of template replacement, the Global Component is configured into mould
Parameter item in plate replaces with the corresponding component configuration parameter item, obtains the component Configuration example.
Preferably, the method also includes:
Step 5, the corresponding version file is called according to the version number, and by the version file via configuration point
Hair module is distributed on the component on the big data component server.
Preferably, the specific implementation of the step 3 are as follows: be based on SSH and multi-process technology, parallel remote distribution difference component
The component Configuration example.
The embodiment of the present invention also provides a kind of computer equipment, including memory and processor.
The memory, for storing computer program.
The processor, for executing above-mentioned big number when reading the computer program of the memory storage
According to configuring management method in component set.
The embodiment of the present invention also provides a kind of computer readable storage medium, for storing computer program, when the meter
When calculation machine program is executed by processor, configuring management method in above-mentioned big data component set is realized.
Below with reference to specific production environment, it is illustrated by taking a typical big data system as an example.
One big data ETL (Extract- built using components such as Hadoop/Hive/Kafka/MySQL/Oracle
Transform-Load) pipeline system.In traditional configuration management mode, system operation maintenance personnel needs to log in respectively
Host where the components such as Hadoop/Hive/Kafka/MySQL/Oracle is changed and issues configuration, and system manager does not have
The configuration of unified entrance management or backup different editions.After using this configuration management system, system operation maintenance personnel is not necessarily to
Log in each component host, all updates, publication configuration can all be completed by this system, system manager can also manage concentratedly and
Back up the component Configuration of all versions.When the big data component of the system integration it is more when, this system safety, efficiency and
Effect in terms of system robustness can be more significant.
Reader should be understood that in the description of this specification reference term " one embodiment ", " is shown " some embodiments "
The description of example ", specific examples or " some examples " etc. mean specific features described in conjunction with this embodiment or example, structure,
Material or feature are included at least one embodiment or example of the invention.In the present specification, above-mentioned term is shown
The statement of meaning property need not be directed to identical embodiment or example.Moreover, particular features, structures, materials, or characteristics described
It may be combined in any suitable manner in any one or more of the embodiments or examples.In addition, without conflicting with each other, this
The technical staff in field can be by the spy of different embodiments or examples described in this specification and different embodiments or examples
Sign is combined.
Although the embodiments of the present invention has been shown and described above, it is to be understood that above-described embodiment is example
Property, it is not considered as limiting the invention, those skilled in the art within the scope of the invention can be to above-mentioned
Embodiment is changed, modifies, replacement and variant.
Claims (10)
1. configuration management system in a kind of big data component set, which is characterized in that the system comprises instantiation modules, version control
Molding block and configuration distribution module；
The instantiation module, for component configuration parameter item based on the received and built-in Global Component configuration template generation group
Part profile instance；
The version control module, for generate the component Configuration example version number and corresponding version file；
The configuration distribution module, for the component Configuration example to be distributed to the component on big data component server
On, so that the component on the big data component server is completed configuration on line and updates and come into force.
2. configuration management system in big data component set according to claim 1, which is characterized in that the system also includes
Configuration backup module,
The configuration backup module, for being backed up according to backup strategy and executive plan to the component Configuration example.
3. configuration management system in big data component set according to claim 2, which is characterized in that the instantiation module
It is specifically used for:
By the way of template replacement, the parameter item in the Global Component configuration template is replaced with into the corresponding component and is matched
Parameter item is set, the component Configuration example is obtained.
4. configuration management system in big data component set according to claim 3, which is characterized in that the Version Control mould
Block is also used to:
The corresponding version file is called according to the version number, and by the version file via configuration distribution module distribution
To the component being located on the big data component server.
5. configuration management system in big data component set according to any one of claims 1 to 4, which is characterized in that described
Configuration distribution module is specifically used for:
Based on SSH and multi-process technology, the component Configuration example of parallel remote distribution difference component.
6. configuring management method in a kind of big data component set, which is characterized in that the described method includes:
Step 1, component configuration parameter item and built-in Global Component configuration template formation component profile instance based on the received；
Step 2, generate the component Configuration example version number and corresponding version file；
Step 3, the component Configuration example is distributed on the component on big data component server, makes the big data
Component on component server completes configuration on line and updates and come into force.
7. configuring management method in big data component set according to claim 6, which is characterized in that the method is also wrapped
It includes:
Step 4, the component Configuration example is backed up according to backup strategy and executive plan.
8. configuring management method in big data component set according to claim 7, which is characterized in that the tool of the step 1
Body is realized are as follows: by the way of template replacement, the parameter item in the Global Component configuration template is replaced with corresponding described
Component configuration parameter item obtains the component Configuration example.
9. configuring management method in big data component set according to claim 8, which is characterized in that the method is also wrapped
It includes:
Step 5, the corresponding version file is called according to the version number, and the version file is distributed into mould via configuration
Block is distributed on the component on the big data component server.
10. according to configuring management method in the described in any item big data component set of claim 6 to 9, which is characterized in that described
The specific implementation of step 3 are as follows: be based on SSH and multi-process technology, the component Configuration example of parallel remote distribution difference component.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810812933.9A CN109240757A (en) | 2018-07-23 | 2018-07-23 | Configuration management system and method in a kind of big data component set |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810812933.9A CN109240757A (en) | 2018-07-23 | 2018-07-23 | Configuration management system and method in a kind of big data component set |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109240757A true CN109240757A (en) | 2019-01-18 |
Family
ID=65072844
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810812933.9A Pending CN109240757A (en) | 2018-07-23 | 2018-07-23 | Configuration management system and method in a kind of big data component set |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109240757A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109961230A (en) * | 2019-03-28 | 2019-07-02 | 金瓜子科技发展（北京）有限公司 | A kind of configuration method of index, device, storage medium and electronic equipment |
| CN112988130A (en) * | 2021-02-24 | 2021-06-18 | 恒安嘉新（北京）科技股份公司 | Visual modeling method, device, equipment and medium based on big data |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1547700A (en) * | 2001-08-30 | 2004-11-17 | �Ҵ���˾ | Integrated management system and method for distributing software |
| CN102455915A (en) * | 2010-10-18 | 2012-05-16 | 卓望数码技术（深圳）有限公司 | Method for automatically configuring distributed system, and server |
| US20130042239A1 (en) * | 2011-04-15 | 2013-02-14 | Oracle International Corporation | System and method for use of a virtual assembly builder |
| CN104317610A (en) * | 2014-10-11 | 2015-01-28 | 福建新大陆软件工程有限公司 | Method and device for automatic installation and deployment of hadoop platform |
| CN105224351A (en) * | 2014-06-17 | 2016-01-06 | 中兴通讯股份有限公司 | Rapid configuration implementation method and rapid configuration server |
| CN105681547A (en) * | 2015-12-31 | 2016-06-15 | 努比亚技术有限公司 | Middleware instance management device and method |
| CN106610853A (en) * | 2015-10-26 | 2017-05-03 | 阿里巴巴集团控股有限公司 | Application deployment method and device |
-
2018
- 2018-07-23 CN CN201810812933.9A patent/CN109240757A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1547700A (en) * | 2001-08-30 | 2004-11-17 | �Ҵ���˾ | Integrated management system and method for distributing software |
| CN102455915A (en) * | 2010-10-18 | 2012-05-16 | 卓望数码技术（深圳）有限公司 | Method for automatically configuring distributed system, and server |
| US20130042239A1 (en) * | 2011-04-15 | 2013-02-14 | Oracle International Corporation | System and method for use of a virtual assembly builder |
| CN105224351A (en) * | 2014-06-17 | 2016-01-06 | 中兴通讯股份有限公司 | Rapid configuration implementation method and rapid configuration server |
| CN104317610A (en) * | 2014-10-11 | 2015-01-28 | 福建新大陆软件工程有限公司 | Method and device for automatic installation and deployment of hadoop platform |
| CN106610853A (en) * | 2015-10-26 | 2017-05-03 | 阿里巴巴集团控股有限公司 | Application deployment method and device |
| CN105681547A (en) * | 2015-12-31 | 2016-06-15 | 努比亚技术有限公司 | Middleware instance management device and method |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109961230A (en) * | 2019-03-28 | 2019-07-02 | 金瓜子科技发展（北京）有限公司 | A kind of configuration method of index, device, storage medium and electronic equipment |
| CN112988130A (en) * | 2021-02-24 | 2021-06-18 | 恒安嘉新（北京）科技股份公司 | Visual modeling method, device, equipment and medium based on big data |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10977274B2 (en) | Unified replication and recovery | |
| CN110609675B (en) | Workflow modeling method and device and computer readable storage medium | |
| CN108768728A (en) | O&M task processing method, device, computer equipment and storage medium | |
| CN106210124B (en) | Unified cloud data center monitoring system | |
| CN107844343A (en) | The upgrade-system and method of a kind of complex services end application system | |
| CN104281906A (en) | Process engine achieving method and system in service process | |
| CN105843182A (en) | Power dispatching accident handling scheme preparing system and power dispatching accident handling scheme preparing method based on OMS | |
| CN103412768A (en) | Zookeeper cluster automatic-deployment method based on script program | |
| CN103677973A (en) | Distributed multi-task scheduling management system | |
| CN107783816A (en) | The method and device that creation method and device, the big data cluster of virtual machine create | |
| CN111176645A (en) | Power grid big data application-oriented data integration management system and implementation method thereof | |
| CN107294771A (en) | A kind of efficient deployment system and application method suitable for big data cluster | |
| Erraissi et al. | Meta-modeling of Zookeeper and MapReduce processing | |
| WO2019223178A1 (en) | Cross-platform task scheduling method and system, computer device, and storage medium | |
| CN109669976A (en) | Data service method and equipment based on ETL | |
| CN109992278A (en) | A kind of application issued method and device based on container | |
| CN110019138A (en) | A kind of transmission table space Autonomic Migration Framework method and system based on Zabbix | |
| CN110569113A (en) | Method and system for scheduling distributed tasks and computer readable storage medium | |
| DE112020004967T5 (en) | CHANGE MANAGEMENT AND ANALYTICS FOR MICROSERVICES | |
| CN115687486B (en) | Light-weight data acquisition method and device based on keyle | |
| CN109240757A (en) | Configuration management system and method in a kind of big data component set | |
| CN111158743B (en) | Big data operation and maintenance management platform | |
| US11663349B2 (en) | System and method for managing data object creation | |
| CN109445800A (en) | A kind of version automatic deployment method and system based on distributed system | |
| CN110162312A (en) | A kind of BeeGFS configuration method and device based on IML |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190118 |