CN107016101B

CN107016101B - Data management method, device and system

Info

Publication number: CN107016101B
Application number: CN201710235499.8A
Authority: CN
Inventors: 刘胜旺; 舒羿宁
Original assignee: Global Big Data Technology Co ltd
Current assignee: Global Big Data Technology Co ltd
Priority date: 2017-04-12
Filing date: 2017-04-12
Publication date: 2020-06-30
Anticipated expiration: 2037-04-12
Also published as: CN107016101A

Abstract

The embodiment of the invention provides a data management method, device and system. Initializing a system, wherein the initialized data directory at least comprises a production data storage directory and an event data storage directory; when a business transaction operation is monitored, the operation is executed by the system through the entrusted transaction model. Through the operation, the risk possibly caused by directly modifying the production data by the service is avoided, and the data safety is enhanced.

Description

Data management method, device and system

Technical Field

The present invention relates to the field of data management, and more particularly, to the manipulation of business data of data.

Background

The operation state of the power grid changes instantly, the requirement on real-time performance is high, and the intelligent degree of the active power distribution network is low for a long time, so that the emergency response capability of the system cannot meet the operation requirement, calculation delay of only a few seconds is afraid, and catastrophic influence is brought to a power grid production system. Therefore, the intelligent scheduling of the active power distribution network is usually to simulate various abnormal conditions of the power grid through simulation calculation (based on actual operation data of the power grid), find solutions for the various abnormal conditions, and set the solutions in the production system of the power grid in advance. When the production system encounters a certain abnormal condition, the corresponding solution is automatically switched to, so that the decision-making time for keeping the cost is saved, and the method is a space-to-time method. The situation of simulating various operating conditions of the power grid through the simulation environment is called multi-scene multi-state. The multi-scenario polymorphism is the basis for performing simulation calculation, because the simulation calculation must be performed before the simulation environment.

The multi-scenario is a concept that individuals are divided into different scenarios, such as a safety scenario, an abnormal scenario, a specific event scenario and the like, various selection schemes exist in each scenario, and are called as a multi-scenario, the multi-scenario and the multi-scenario are shown in fig. 1, a multi-scenario database is generated based on a production library, and the production library can generate a plurality of multi-scenario database versions according to different research purposes and is respectively used for meeting different simulation test occasions. Polymorphic databases are just a few data files, and the polymorphic example refers to a database service process loading a polymorphic database file to generate a specific executable database service.

In the current electric power industry, polymorphic data extraction and example deployment are manually operated, the automation degree is low, the efficiency is not high, and one example deployment usually needs one day or even several days.

The existing polymorphic scheme has low resource sharing rate, one polymorphic instance usually needs to monopolize a batch of server resources, and after verification is completed, the release of the resources is also a great problem, and generally a system needs to be deleted, cleaned or reinstalled manually.

In the existing polymorphic system, after an instance of a polymorphic database is deployed online, if a business system needs to access the instance, an IP address and a port corresponding to the instance database need to be manually configured in the business system, and an effective automatic service registration and discovery mechanism is lacked.

In the prior art, the transaction operation can directly operate production data, is mainly completed manually, has large workload and low efficiency, and is easy to cause data confusion and higher risk because the transaction operation can relate to the management of a large number of polymorphic versions and the change of data.

Disclosure of Invention

The embodiment of the invention provides a data management method, which is characterized by comprising the following steps:

initializing a system, wherein the initialized data directory at least comprises a production data storage directory and an event data storage directory;

when a business transaction operation is monitored, the operation is executed by the system through the entrusted transaction model.

Because the transaction operation can cause the change of data, the risk is high, so the business is not allowed to directly operate the production data, and the production data is handed to the system to be executed through the entrustment transaction model.

Further, the method further comprises:

when a query request is received, the query request is executed.

Because the query does not cause damage to the production data, the licensing service queries the production data directly.

Further, the business transaction operation includes: added, deleted, or modified.

Further, the production data directory and the event directory comprise three layers of structures of regions, scenes and examples.

Further, the data stored in the production data catalog are the identification and the interface of the production data.

In one embodiment of the invention, the data management method comprises three layers of structures, namely regions, scenes and instances. The business transaction operation comprises the following operations:

1) and (3) regional management: management of new instances of the region: when a regional service triggering event is monitored, a new instance database is deployed by the system through a entrustment transaction model, and a relevant directory is automatically configured; then, carrying out system backfill operation, and giving a result to be examined;

management of deleted zone instances: when a regional service triggering event is monitored, a transaction entrusting model is used for handing over the system to execute deletion operation of data; and then carrying out system backfill operation, and giving a result to be examined.

2) Scene management: managing the newly added scene instances: when a scene service triggering event is monitored, a system deploys a new instance database through a entrustment transaction model, and a relevant directory is automatically configured; then, carrying out system backfill operation, and giving a result to be examined;

management of deleted scene instances: when a scene service triggering event is monitored, a transaction entrusting model is handed to a system to execute deletion operation of data; and then carrying out system backfill operation, and giving a result to be examined.

3) Instance management: management of newly added instances: the system adds an instance process, when a service triggering event is monitored, a new instance database is deployed by the system through a entrusted transaction model, and a relevant directory is automatically configured; then, carrying out system backfill operation, and giving a result to be examined;

management of deleted instances: when a business triggering event is monitored, a transaction entrusting model is used for handing over the system to execute the deletion operation of the data; and then carrying out system backfill operation, and giving a result to be examined.

In one embodiment of the invention, the data management method comprises three layers of structures, namely regions, scenes and instances. The query operation comprises the following operations:

1) inquiring all regions;

2) inquiring all scenes in a certain area;

3) inquiring all examples in a certain area and a certain scene;

4) inquiring a production library interface in a certain area;

5) and inquiring the interface of a certain area, a certain scene and a certain instance.

An embodiment of the present invention further provides a data management apparatus, including:

the initialization module is used for initializing the system, and the initialized data directory at least comprises a production data storage directory and an event monitoring directory;

and the monitoring module is used for handing over the operation to the system to execute the operation through the entrusted transaction model when monitoring the operation of the business transaction.

Further, the apparatus further comprises:

and the query module is used for executing the query request when receiving the query request.

An embodiment of the present invention further provides a data management system, which includes:

the storage module is used for storing production data and event data;

the initialization module is used for initializing the system, so that the data directory in the storage module at least comprises a production data storage directory and an event data storage directory;

the monitoring module is used for handing over the business transaction operation to the system to execute the operation through the entrusted transaction model when the business transaction operation is monitored;

Further: the storage module is an ETCD cluster.

Compared with the prior art, the technical scheme of the invention is characterized in that: 1) supporting a three-layer architecture model configured according to regions, scenes and instances; 2) supporting an event-based delegated transaction model; 3) a safety model supporting isolation of production data from business operations; 4) and the data storage and processing model is based on the ETCD cluster. The method can rapidly and efficiently complete intelligent multi-scene multi-state configuration based on the container cloud.

Drawings

The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the invention and not to limit the invention. In the drawings:

FIG. 1 is a diagram of a multi-scenario polymorphism in the prior art;

FIG. 2 is a schematic diagram of a data management flow in an embodiment of the invention;

FIG. 3 is a diagram illustrating a directory structure after initialization in an embodiment of the present invention;

FIG. 4 is a diagram illustrating a directory structure after adding new regions in region management according to an embodiment of the present invention;

fig. 5 is a diagram illustrating a directory structure after a region is deleted in the region management in the embodiment of the present invention;

FIG. 6 is a diagram illustrating a directory structure after adding a new scene in scene management according to an embodiment of the present invention;

FIG. 7 is a diagram illustrating a directory structure after a scene is deleted in scene management according to an embodiment of the present invention;

FIG. 8 is a diagram illustrating a directory structure after an instance is newly added in instance management according to an embodiment of the present invention;

FIG. 9 is a diagram illustrating a directory structure after an instance is deleted in instance management, in accordance with an embodiment of the present invention;

FIG. 10 is a schematic diagram of a data management apparatus in an embodiment of the present invention;

FIG. 11 is a schematic diagram of a data management system in an embodiment of the invention;

FIG. 12 is a schematic diagram of a data model design in an embodiment of the invention.

Detailed Description

As used in the specification and in the claims, certain terms are used to refer to particular components. As one skilled in the art will appreciate, manufacturers may refer to a component by different names. This specification and claims do not intend to distinguish between components that differ in name but not function. In the following description and in the claims, the terms "include" and "comprise" are used in an open-ended fashion, and thus should be interpreted to mean "include, but not limited to. "substantially" means within an acceptable error range, and a person skilled in the art can solve the technical problem within a certain error range to substantially achieve the technical effect. The following description is of the preferred embodiment for carrying out the invention, and is made for the purpose of illustrating the general principles of the invention and not for the purpose of limiting the scope of the invention. The scope of the present invention is defined by the appended claims.

The embodiment of the invention provides a data management method as shown in fig. 2:

step 210, initializing the system, wherein the initialized data directory at least comprises a production data storage directory and an event data storage directory;

in this step, the storage structure after system initialization is shown in fig. 3, where 'multi-scenes' is a root directory, and there are two subdirectories 'distribute' and 'listener' under the root directory. The 'discrete' mainly stores production data and is used for business query; the 'listener' directory is used for event monitoring, the event is entrusted through service setting, and the system agent service realizes transactional operation, such as addition and deletion of a data structure in the 'district' directory.

Step 220, when the business transaction operation is monitored, the operation is executed by the system through the entrusted transaction model;

In this embodiment, the multi-state multi-scene model includes three layers of structures, namely, a region, a scene, and an instance. The business transaction operation comprises the following operations:

1. regional management

1.1 adding a new area process (assuming that the area number is 'beijing-001'):

1) a service triggers a monitoring event;

setting a key value:

the bond name is/multi-genes/lithium/discrete-lithium/beijing-001

The key value is: new;

2) system monitoring

The change of the'/multi-scenes/list/discrete-list' directory obtains the corresponding key value, the proxy service executes the entrusting transaction and automatically configures the related directory.

After completion, the etcd directory structure is shown in fig. 4, where 410 and 420 are added parts, the data structure of the part is automatically maintained by the system, and part of the data needs to be extracted from the deployed production library.

3) Starting a new listener

Two listeners are added under the new area:

multi-scenes/ristener/beijing-001/scene-ristener for managing multiple scenes;

multi-scenes/listener/beijing-001/instance-listener for polymorphic (instance) management;

4) system backfill operation result (for service detection)

Bond: multi-scenes/lithium/discrete-lithium/beijing-001

The value: ok indicates success, and failure sets the reason for failure.

1.2 delete region flow (assuming region number 'beijing-001'):

1) service triggered listening events

Setting a key value:

the bond name is/multi-genes/lithium/discrete-lithium/beijing-001

The key value is: delete;

2) system monitoring

The change of the'/multi-scenes/list/discard-list' directory takes the corresponding key value to clear the relevant directory, and the structure of the etcd directory is shown in FIG. 5, wherein the broken-line frame part in the figure represents the deleted part.

3) Delete listener

The following two listeners are deleted:

/multi-scenes/listener/beijing-001/scene-listener；

/multi-scenes/listener/beijing-001/instance-listener；

4) results of system backfill operations

Bond: multi-scenes/lithium/discrete-lithium/beijing-001

The value: ok indicates success, and failure sets the reason for failure.

2. Scene management

2.1 adding a new scene process (assuming that the area number is 'beijing-001', and the name of the new scene is 'middle planning'):

1) service triggered listening events

Setting a key value:

the key name is/multi-genes/lithium/beijing-001/gene-lithium/metaphase plan;

the key value is: new;

2) system monitoring

The change of the'/multi-directories/listing/following-001/hierarchy-listing' directory obtains the corresponding key value to automatically configure the related directory, and the structure of the etcd directory is shown in fig. 6, in which 610 and 620 are newly added parts.

3) Results of system backfill operations

Bond: multi-scenes/ristener/beijing-001/scene-ristener/metaphase plan

The value: ok indicates success, and failure sets the reason for failure.

2.2 delete scenario flow (assuming the area number is 'beijing-001', new scenario name is 'middle planning'):

1) service triggered listening events

Setting a key value:

the key name is/multi-genes/lithium/beijing-001/gene-lithium/metaphase plan;

the key value is: delete;

2) system monitoring

The change of the'/multi-scenes/list/following-001/scene-list' directory, obtaining the corresponding key value, deleting the related directory, if the scene contains the instance, deleting all instances in cascade, and the etcd directory structure is shown in fig. 7, wherein the dashed box part in the figure represents the deleted part.

3) Results of system backfill operations

Bond: multi-scenes/ristener/beijing-001/scene-ristener/metaphase plan

The value: ok indicates success, and failure sets the reason for failure.

3. Instance management

3.1 adding an example process (assuming that the area number is 'beijing-001', the scene name is 'middle planning', and the example name is 'safety analysis'):

1) service triggered listening events

Setting a key value:

the key name is/multi-genes/listener/beijing-001/instance-listener/metaphase planning/safety analysis

The key value is: new;

2) system monitoring

'/multi-scenes/list/following-001/instance-list/middle planning' catalog change, obtain corresponding key values, deploy new instance database, automatically configure related catalog, at this time the etcd catalog structure is as shown in FIG. 8, where 810 is the added part.

3) Results of system backfill operations

Bond: multi-scenes/listener/beijing-001/instance-listener/metaphase planning/safety analysis

The value: ok indicates success, and failure sets the reason for failure.

3.2 delete instance flow (assuming the area number is 'beijing-001', the scene name is 'middle planning', and the instance name is 'safety analysis'):

1) service triggered listening events

Setting a key value:

the key name is/multi-scenes/listener/beijing-001/instance-listener/middle-term planning/safety analysis;

the key value is: delete;

2) system monitoring

'/multi-scenes/list/following-001/instance-list/middle plan' change of directory, get corresponding key value, unload corresponding instance database, clean up relevant directory, at this time etcd directory structure is shown in FIG. 9, wherein the dashed box part in the figure represents the deleted part.

3) Results of system backfill operations

Bond: multi-scenes/listener/beijing-001/instance-listener/interim planning/safety analysis;

the value: ok indicates success, and failure sets the reason for failure.

When a query request is received, the query request is executed, step 230.

In this embodiment, the query operation may include, for example, the following operations:

1) querying all regions

Searching all subdirectories under'/multi-scenes/distribute';

2) query all scenes of a certain area (assuming area number 'beijing-001')

Searching all subdirectories under'/multi-scenes/district/Beijing-001/scenes';

3) inquiring all examples of a certain area and a certain scene (assuming that the area number is 'beijing-001' and the scene is 'middle-term planning')

Searching all subdirectories under'/multi-scenes/distribute/beijing-001/scenes/medium planning';

4) query some area production library interface (assume area number is 'beijing-001')

The key value is searched for '/multi-genes/discrete/beijing-001/default/online';

5) inquiring interfaces of a certain area, a certain scene and a certain example (assuming that the area number is 'beijing-001', the scene is 'middle-term planning', and the example is 'safety analysis')

The key value is searched for '/multi-genes/discrete/beijing-001/genes/metaphase planning/safety analysis';

to solve the technical problem to be solved by the present invention, an embodiment of the present invention further provides a multi-scenario multi-state model data management apparatus, as shown in fig. 10, including:

To solve the technical problem to be solved by the present invention, an embodiment of the present invention further provides a data management system as shown in fig. 11, including:

the storage module is used for storing production data and event data;

The embodiment of the invention adopts an ETCD cluster to store production and event data, the ETCD adopts a directory structure to store data, and the data model design of the invention is shown in figure 11.

As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, apparatus, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

While the foregoing specification illustrates and describes several particular embodiments of the invention, it is to be understood, as noted above, that the invention is not limited to the forms disclosed herein, but is not intended to be exhaustive of other embodiments and may be used in various other combinations, modifications, and environments and is capable of changes within the scope of the inventive concept as described herein, commensurate with the above teachings, or the skill or knowledge of the relevant art. And that modifications and variations may be effected by those skilled in the art without departing from the spirit and scope of the invention as defined by the appended claims.

Claims

1. A method for managing data, comprising:

when the business transaction operation is monitored, the operation is executed by the system through the entrusted transaction model; the business transaction operation comprises: add, delete, or modify;

the production data storage catalog is used for business query and can be directly searched by businesses;

the event data storage directory is used for the business transaction operation, and when the business transaction operation is monitored, the business transaction operation is executed by a system through a commission transaction model, and the method specifically comprises the following steps:

the business triggers a snoop event to set a key in the event data storage directory,

the system configures a production data storage directory and an event data storage directory upon detecting a change in a key value in the event data storage directory,

and backfilling the operation result of the system.

2. The method of claim 1, further comprising:

when a query request is received, the query request is executed.

3. The method of claim 1, wherein:

the production data storage directory and the event data storage directory comprise three layers of structures including regions, scenes and examples.

4. The method of claim 1, wherein:

and the data stored in the production data storage catalog are the identification and the interface of the production data.

5. A data management apparatus, comprising:

the initialization module is used for initializing the system, and the initialized data directory at least comprises a production data storage directory and an event data storage directory;

the monitoring module is used for handing over the business transaction operation to the system to execute the operation through the entrusted transaction model when the business transaction operation is monitored; the business transaction operation comprises: adding, deleting or modifying

and backfilling the operation result of the system.

6. The apparatus of claim 5, further comprising:

7. The apparatus of claim 5, wherein:

8. The apparatus of claim 5, wherein:

9. A data management system, comprising:

the storage module is used for storing production data and event data;

the query module is used for executing the query request when receiving the query request;

and backfilling the operation result of the system.

10. The system of claim 9, wherein:

the storage module is an ETCD cluster.