WO2021032146A1

WO2021032146A1 - Metadata management method and apparatus, device, and storage medium

Info

Publication number: WO2021032146A1
Application number: PCT/CN2020/110167
Authority: WO
Inventors: 吕燕
Original assignee: 中兴通讯股份有限公司
Priority date: 2019-08-22
Filing date: 2020-08-20
Publication date: 2021-02-25
Also published as: CN112416923A

Abstract

A metadata management method and apparatus, a device, and a computer readable storage medium. The metadata management apparatus (100) comprises a metadata type management module (101), a metadata collection module (102), and a metadata storage index module (103); the metadata type management module (101) is configured to load a metadata type based on object-oriented mode definition and record the metadata type in the metadata storage index module (103); the metadata collection module (102) is configured to collect a metadata instance corresponding to the metadata type according to the metadata type, and store the metadata instance in the metadata storage index module (103); the metadata storage index module (103) is configured to store the metadata type and the metadata instance.

Description

元数据管理方法和装置、设备及存储介质Metadata management method and device, equipment and storage medium

本申请要求在2019年08月22日提交中国专利局、申请号为201910780607.9的中国专利申请的优先权，该申请的全部内容通过引用结合在本申请中。This application claims the priority of a Chinese patent application filed with the Chinese Patent Office with application number 201910780607.9 on August 22, 2019. The entire content of this application is incorporated into this application by reference.

技术领域Technical field

本公开涉及数据管理技术领域，例如涉及一种元数据管理方法和装置、设备及计算机可读存储介质。The present disclosure relates to the technical field of data management, for example, to a metadata management method and device, equipment, and computer-readable storage medium.

背景技术Background technique

当今社会中，数据量高速膨胀，数据正成为政府、企业的核心竞争力，人们通过数据分析，挖掘数据价值为管理决策者提供准确的判断依据。In today's society, the amount of data is rapidly expanding, and data is becoming the core competitiveness of governments and enterprises. People use data analysis to mine the value of data to provide management decision-makers with accurate judgments.

然而，政府、企业等在电子信息化的过程中，拥有众多、异构的数据***，这些海量的、分散在不同***的数据导致了数据资源利用的复杂度和管理的高难度。管理决策者无法从统一的业务视角去概览内部的数据信息，***与***之间的关系。为达到挖掘数据价值的目标，需要先进行元数据管理，建立全局数据地图，统一数据口径、标明数据方位、分析数据关系、管理模型变更，帮助管理者从容分析数据仓库每个局部与全局上下文的关系，真正做到观局部而知全局。大数据相关技术让政府、企业数据的价值能被充分挖掘，但是大数据往往意味着许多数据源之间的采集、传播和共享，例如移动个人数据、社交网络数据、公共数据、物联网数据等，这些过程需要基于大数据的元数据管理的支撑。However, in the process of electronic informationization, governments and enterprises have numerous and heterogeneous data systems. These massive amounts of data scattered in different systems have led to the complexity of data resource utilization and the high difficulty of management. Management decision makers cannot overview the internal data and the relationship between systems from a unified business perspective. In order to achieve the goal of mining data value, it is necessary to first conduct metadata management, establish a global data map, unify data caliber, indicate data location, analyze data relationships, and manage model changes to help managers calmly analyze each local and global context of the data warehouse. Relations, truly understand the overall situation while observing the part. Big data-related technologies allow the value of government and enterprise data to be fully tapped, but big data often means the collection, dissemination and sharing of many data sources, such as mobile personal data, social network data, public data, Internet of Things data, etc. These processes need the support of metadata management based on big data.

国内外对元数据管理的需求日益增加，元数据管理是政府、企业数据治理的重要手段。元数据是描述数据的数据，主要是描述数据属性的信息。相关技术中的元数据管理产品中，采用传统的数据字典建模方法，从数据集、字段、元素和代码集4个维度定义元数据类型，元数据类型冗余，且不能设计和管理非结构化元数据。因此，有必要进行改进。There is an increasing demand for metadata management at home and abroad, and metadata management is an important means of government and enterprise data governance. Metadata is data describing data, mainly information describing data attributes. Metadata management products in related technologies use traditional data dictionary modeling methods to define metadata types from four dimensions: data sets, fields, elements, and code sets. The metadata types are redundant, and non-structures cannot be designed and managed. Metadata. Therefore, it is necessary to make improvements.

发明内容Summary of the invention

本公开提供了一种元数据管理方法和装置、设备及计算机可读存储介质，实现对多种元数据类型的支持。The present disclosure provides a metadata management method and device, equipment, and computer-readable storage medium to realize support for multiple metadata types.

提供一种元数据管理装置，包括元数据类型管理模块、元数据采集模块和元数据存储索引模块，其中：A metadata management device is provided, which includes a metadata type management module, a metadata collection module, and a metadata storage index module, wherein:

所述元数据类型管理模块设置为，加载基于面向对象模式定义的元数据类型，将所述元数据类型记录在所述元数据存储索引模块；The metadata type management module is configured to load the metadata type defined based on the object-oriented mode, and record the metadata type in the metadata storage index module;

所述元数据采集模块设置为，根据所述元数据类型获取所述元数据类型对应的元数据实例，将所述元数据实例存储在所述元数据存储索引模块；The metadata collection module is configured to obtain a metadata instance corresponding to the metadata type according to the metadata type, and store the metadata instance in the metadata storage index module;

所述元数据存储索引模块设置为，存储所述元数据类型和所述元数据实例。The metadata storage index module is configured to store the metadata type and the metadata instance.

还提供一种元数据管理方法，包括：A metadata management method is also provided, including:

加载基于面向对象模式定义的元数据类型，存储所述元数据类型；Load the metadata type defined based on the object-oriented mode, and store the metadata type;

根据所述元数据类型获取所述元数据类型对应的元数据实例，存储所述元数据实例。Obtain the metadata instance corresponding to the metadata type according to the metadata type, and store the metadata instance.

还提供一种元数据管理设备，包括存储器和处理器，所述存储器存储有程序，所述程序在被所述处理器读取执行时，实现本公开所述的元数据管理方法。A metadata management device is also provided, which includes a memory and a processor, the memory stores a program, and when the program is read and executed by the processor, the metadata management method described in the present disclosure is implemented.

还提供一种计算机可读存储介质，所述计算机可读存储介质存储有一个或者多个程序，所述一个或者多个程序可被一个或者多个处理器执行，以实现本公开所述的元数据管理方法。A computer-readable storage medium is also provided. The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to realize the elements described in the present disclosure. Data management methods.

附图说明Description of the drawings

附图用来提供对本公开技术方案的理解，并且构成说明书的一部分，与本公开的实施例一起用于解释本公开的技术方案，并不构成对本公开技术方案的限制。The accompanying drawings are used to provide an understanding of the technical solution of the present disclosure, and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the technical solution of the present disclosure, and do not constitute a limitation to the technical solution of the present disclosure.

图1是本发明一实施例提供的元数据管理模块组成图；FIG. 1 is a composition diagram of a metadata management module provided by an embodiment of the present invention;

图2是本发明一实施例提供的元数据类型管理模块示意图；2 is a schematic diagram of a metadata type management module provided by an embodiment of the present invention;

图3是本发明一实施例提供的元数据类型采集模块示意图；3 is a schematic diagram of a metadata type collection module provided by an embodiment of the present invention;

图4是本发明一实施例提供的元数据可视化维护模块示意图；4 is a schematic diagram of a metadata visual maintenance module provided by an embodiment of the present invention;

图5是本发明一实施例提供的元数据类型管理子界面示意图；5 is a schematic diagram of a metadata type management sub-interface provided by an embodiment of the present invention;

图6是本发明一实施例提供的元数据通用可视化维护界面图示意图；6 is a schematic diagram of a general visual maintenance interface for metadata provided by an embodiment of the present invention;

图7是本发明另一实施例提供的元数据管理模块组成图；FIG. 7 is a composition diagram of a metadata management module provided by another embodiment of the present invention;

图8是本发明一实施例提供的元数据管理方法流程图；FIG. 8 is a flowchart of a metadata management method provided by an embodiment of the present invention;

图9是本发明一实施例提供的数据共享领域实现流程图；FIG. 9 is a flowchart of the realization of the data sharing field provided by an embodiment of the present invention;

图10是本发明一实施例提供的应用元数据管理装置进行数据标准管理的流程图；Fig. 10 is a flowchart of data standard management performed by an application metadata management device provided by an embodiment of the present invention;

图11是本发明一实施例提供的应用元数据管理装置进行数据治理的流程图；11 is a flowchart of data management performed by the application metadata management device provided by an embodiment of the present invention;

图12是本发明一实施例提供的应用元数据管理装置提供数据服务的流程图；FIG. 12 is a flowchart of data services provided by an application metadata management apparatus according to an embodiment of the present invention;

图13是本发明一实施例提供的元数据管理设备框图；Figure 13 is a block diagram of a metadata management device provided by an embodiment of the present invention;

图14是本发明一实施例提供的计算机可读存储介质框图。Fig. 14 is a block diagram of a computer-readable storage medium according to an embodiment of the present invention.

具体实施方式detailed description

下文中将结合附图对本发明的实施例进行说明。需要说明的是，在不冲突的情况下，本申请中的实施例及实施例中的特征可以相互任意组合。Hereinafter, the embodiments of the present invention will be described with reference to the accompanying drawings. It should be noted that the embodiments in this application and the features in the embodiments can be combined with each other arbitrarily if there is no conflict.

在附图的流程图示出的步骤可以在诸如一组计算机可执行指令的计算机***中执行。并且，虽然在流程图中示出了逻辑顺序，但是在一些情况下，可以以不同于此处的顺序执行所示出或描述的步骤。The steps shown in the flowchart of the drawings may be executed in a computer system such as a set of computer-executable instructions. Also, although a logical sequence is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than here.

很多政府、企业内部数据形态多样，标准不统一，这需要一个可扩展、可管理任意类型的元数据管理的支撑。传统的数据字典建模方法，很难设计元数据类型之间的关联、继承关系，不能从对象继承扩展角度出发定义元数据类型。本申请中，采用基于面向对象模式定义元数据，从而可以设计元数据类型之间的关联关系、继承关系、包容关系。Many governments and enterprises have various internal data forms and non-uniform standards. This requires the support of an extensible and manageable metadata management of any type. The traditional data dictionary modeling method is difficult to design the association and inheritance relationship between metadata types, and cannot define metadata types from the perspective of object inheritance and expansion. In this application, metadata is defined based on an object-oriented model, so that association relationships, inheritance relationships, and tolerance relationships between metadata types can be designed.

如图1所示，本发明一实施例提供一种元数据管理装置100，包括：元数据类型管理模块101、元数据采集模块102和元数据存储索引模块103。As shown in FIG. 1, an embodiment of the present invention provides a metadata management device 100, which includes: a metadata type management module 101, a metadata collection module 102, and a metadata storage index module 103.

所述元数据类型管理模块101设置为，加载基于面向对象模式定义的元数据类型，将所述元数据类型记录在所述元数据存储索引模块103。The metadata type management module 101 is configured to load the metadata type defined based on the object-oriented mode, and record the metadata type in the metadata storage index module 103.

每个元数据类型有多个属性，元数据类型之间存在关系时，可进行定义，所述关系可以有继承关系、关联关系、包容关系等。与相关技术中的数据字典定义方式定义的元数据相比，本实施例提供的元数据类型，可以定义结构化元数据，也可以定义非结构化数据，具有很强的灵活性和通用性。另外，还可以描述元数据类型彼此之间的关系。Each metadata type has multiple attributes. When there are relationships between metadata types, they can be defined. The relationships may include inheritance relationships, association relationships, and containment relationships. Compared with the metadata defined in the data dictionary definition method in the related technology, the metadata type provided in this embodiment can define structured metadata or unstructured data, which has strong flexibility and versatility. In addition, you can also describe the relationship between metadata types.

加载元数据类型即在元数据存储索引模块103中新增元数据类型。所加载的元数据类型可以来自于外部输入的文件等，也可以由用户通过操作界面输入。Loading the metadata type means adding a metadata type to the metadata storage index module 103. The type of metadata loaded can come from external input files, etc., or can be input by the user through the operation interface.

所述元数据采集模块102设置为，根据所述元数据类型获取所述元数据类型对应的元数据实例，将所述元数据实例存储在所述元数据存储索引模块103。The metadata collection module 102 is configured to obtain a metadata instance corresponding to the metadata type according to the metadata type, and store the metadata instance in the metadata storage index module 103.

所述元数据存储索引模块103设置为，存储所述元数据类型和所述元数据实例。The metadata storage index module 103 is configured to store the metadata type and the metadata instance.

本实施例提供的方案，打破了常规的数据字典定义方式，元数据类型基于面向对象模式定义，设计灵活，语义简单，易于复用，扩展性好，可维护性强，可以实现对任意类型的元数据建模，能适用于多个业务领域，具备很强的灵活性和通用性。The solution provided in this embodiment breaks the conventional data dictionary definition method. The metadata type is defined based on the object-oriented model, with flexible design, simple semantics, easy reuse, good scalability, strong maintainability, and can be implemented for any type of Metadata modeling can be applied to multiple business fields, with strong flexibility and versatility.

在一实施例中，定义元数据类型的步骤如下：In one embodiment, the steps of defining the metadata type are as follows:

(a)设计基本类型，包括：枚举类型，结构、标签类型；(a) Design basic types, including: enumeration type, structure, label type;

(b)设计对象属性类型，属性有唯一性标识、必选标识、类型标识、个数标识、关系标识、默认值标识等；(b) The attribute type of the design object, the attribute has unique identification, mandatory identification, type identification, number identification, relationship identification, default value identification, etc.;

(c)设计对象类型，它可以包含多个对象属性；(c) Design object type, which can contain multiple object attributes;

(d)设计对象类型的继承关系，一种对象类型可以继承多个父类。(d) Design the inheritance relationship of object types. One object type can inherit multiple parent classes.

上述步骤仅为示例，本申请对此不作限定，基于面向对象模式定义的元数据类型即可。The above-mentioned steps are only examples, and this application does not limit this, as long as the metadata types are defined based on the object-oriented mode.

本发明实施例提供的方案，抽象度高、通用性强，适用于任何与数据使用相关的应用领域。The scheme provided by the embodiment of the present invention has high abstraction and strong versatility, and is suitable for any application field related to data use.

在一实施例中，所述元数据类型管理模块101还设置为，实现元数据类型的删除、更新、查询。In an embodiment, the metadata type management module 101 is further configured to implement deletion, update, and query of metadata types.

在一实施例中，如图2所示，所述元数据类型管理模块101包括元数据类型加载子模块1011和元数据类型查询子模块1012。In an embodiment, as shown in FIG. 2, the metadata type management module 101 includes a metadata type loading submodule 1011 and a metadata type query submodule 1012.

所述元数据类型加载子模块1011设置为，实现元数据类型的新增、更新、删除。The metadata type loading submodule 1011 is configured to implement addition, update, and deletion of metadata types.

在一实施例中，所述元数据类型加载子模块1011中预先加载多个抽象元数据类型例如数据集、数据对象、表、字段、处理过程等，所述元数据类型可继承。In an embodiment, the metadata type loading submodule 1011 preloads multiple abstract metadata types such as data sets, data objects, tables, fields, processing procedures, etc., and the metadata types can be inherited.

在一实施例中，加载新的元数据类型后，首先解析枚举、结构、标签定义，创建基础元数据类型实例；然后解析继承关系，创建抽象元数据类型实例；然后解析对象类型，创建元数据类型实例；然后解析对象类型之间的关联、包容、继承关系，向元数据存储索引模块103发起存储请求，创建多个元数据类型节点，并且在元数据类型节点之间增加边，建立图谱关系。In one embodiment, after loading a new metadata type, first parse the enumeration, structure, and label definitions to create basic metadata type instances; then parse inheritance relationships to create abstract metadata type instances; then parse the object types to create metadata Data type instance; then analyze the association, tolerance, and inheritance relationship between object types, initiate a storage request to the metadata storage index module 103, create multiple metadata type nodes, and add edges between metadata type nodes to build a graph relationship.

元数据类型查询子模块1012设置为，与所述元数据存储索引模块103交互，实现对元数据类型的查看，包括对元数据类型的信息、关系图谱的查看。The metadata type query submodule 1012 is configured to interact with the metadata storage index module 103 to realize the viewing of the metadata type, including viewing the information of the metadata type and the relationship graph.

在一实施例中，所述元数据采集模块102中设计一套插件式的元数据采集架构，桥接业界主流数据库、大数据平台、消息接口等，自动获取零散分布的元数据。如图3所示，所述元数据采集模块102包括元数据采集器子模块1021和元数据采集任务子模块1022。In one embodiment, a plug-in metadata collection architecture is designed in the metadata collection module 102 to bridge mainstream databases, big data platforms, message interfaces, etc. in the industry to automatically obtain scattered metadata. As shown in FIG. 3, the metadata collection module 102 includes a metadata collector sub-module 1021 and a metadata collection task sub-module 1022.

所述元数据采集器子模块1021设置为，提供多种采集器，包括但不限于：为多种常用、标准化的元数据类型实现自动采集的采集器，例如关系数据库表、大数据库表、简单对象访问协议(Simple Object Access Protocol，SOAP)、表述性状态转移(Representational State Transfer，REST)、Java消息服务(Java Message Service，JMS)、Elastic Search、Kafka等；实现外部同步元数据采集；自定义采集器接口，允许应用自定义采集器接口实现元数据自动采集。The metadata collector submodule 1021 is configured to provide a variety of collectors, including but not limited to: collectors that realize automatic collection for a variety of commonly used and standardized metadata types, such as relational database tables, large database tables, simple Object Access Protocol (Simple Object Access Protocol, SOAP), Representational State Transfer (REST), Java Message Service (Java Message Service, JMS), Elastic Search, Kafka, etc.; realize external synchronization metadata collection; custom The collector interface allows the application of a custom collector interface to realize automatic metadata collection.

所述元数据采集任务子模块1022设置为，使用所述采集器实现自动采集任务、外部同步任务的维护。The metadata collection task sub-module 1022 is configured to use the collector to implement maintenance of automatic collection tasks and external synchronization tasks.

当用户配置了一资源接口(注：资源接口也是一种类型的元数据实例)子元数据实例的自动采集功能，启动相应元数据类型的采集器，该采集器对接资源接口自动采集元数据实例，向元数据存储索引模块103发起存储请求，创建元数据实例节点，根据元数据实例所属元数据类型之间的关联关系、包容关系，在元数据实例节点之间增加边，建立图谱关系。When the user configures the automatic collection function of a sub-metadata instance of a resource interface (Note: the resource interface is also a type of metadata instance), the collector of the corresponding metadata type is started, and the collector is connected to the resource interface to automatically collect the metadata instance , Initiate a storage request to the metadata storage index module 103, create metadata instance nodes, and add edges between metadata instance nodes according to the association relationship and tolerance relationship between the metadata types to which the metadata instance belongs to establish a graph relationship.

本实施例提供的方案，依据元数据标准规范，实现了常用的元数据类型自动采集，降低了运维人员人工收集元数据实例和变更的工作量。The solution provided in this embodiment implements automatic collection of commonly used metadata types in accordance with metadata standards and specifications, and reduces the workload of operation and maintenance personnel for manually collecting metadata instances and changing them.

在一实施例中，所述元数据存储索引模块103还设置为，存储所述元数据类型时，还存储所述元数据类型之间的关系，以及，在存储所述元数据实例时，还存储所述元数据实例之间的关系。所述关系包括关联关系、继承关系、包容关系等。In an embodiment, the metadata storage index module 103 is further configured to store the relationship between the metadata types when storing the metadata types, and when storing the metadata instances, also Store the relationship between the metadata instances. The relationship includes association relationship, inheritance relationship, tolerance relationship and so on.

在一实施例中，使用图数据库存储所述元数据类型之间的关系，以及，使用图数据库存储所述元数据实例之间的关系。本实施例中，使用图数据库存储元数据类型之间的关系和元数据实例之间的关系，查询速度快速，展示效果清晰。另外，使用图搜索能够灵活、高效地搜索元数据实例及其关联关系，对外提供丰富的查询服务，增强了元数据血缘关系追溯能力。In one embodiment, a graph database is used to store the relationship between the metadata types, and a graph database is used to store the relationship between the metadata instances. In this embodiment, the graph database is used to store the relationship between metadata types and the relationship between metadata instances, the query speed is fast, and the display effect is clear. In addition, the use of graph search can flexibly and efficiently search for metadata instances and their associated relationships, provide rich query services to the outside, and enhance the ability to trace metadata blood relationships.

在一实施例中，所述元数据存储索引模块103是设置为：使用列式存储数据库存储所述元数据类型和所述元数据实例。列式存储数据库比如为Hadoop数据库(Hadoop database，HBASE)。利用大数据无固定列、可横向扩展、高实时并发的存储计算特点，将所有元数据实例、元数据实例关系通过图数据库引擎统一存储到一张表中(例如：HBASE，HBASE仅为示例，也可使用其他列式存储数据库)，减少了传统方式的数据表定义的工作量。本实施例提供的方案，能够存储大规模的元数据，提供大规模元数据的服务能力，方便建立统一的元数据视图，为大数据处理建立了统一，稳定的数据仓库，为提高数据治理的能力和效率提供了强大的基础支撑。In an embodiment, the metadata storage index module 103 is configured to store the metadata type and the metadata instance using a columnar storage database. The columnar storage database is, for example, a Hadoop database (HBASE). Utilizing the storage and computing features of big data without fixed columns, horizontal scalability, and high real-time concurrency, all metadata instances and metadata instance relationships are stored in a single table through the graph database engine (for example: HBASE, HBASE are only examples, but also Use other columnar storage databases), reducing the workload of traditional data table definition. The solution provided in this embodiment can store large-scale metadata, provide large-scale metadata service capabilities, facilitate the establishment of a unified metadata view, establish a unified and stable data warehouse for big data processing, and improve data governance. Ability and efficiency provide a strong basic support.

在一实施例中，所述元数据存储索引模块103在存储元数据实例的同时，提取元数据实例中的信息创建元数据索引以便于高效查询(例如：使用SOLR、ElasticSearch等搜索引擎)。本实施例提供的方案，基于大数据存储元数据实例和对元数据实例创建索引，存储和计算能力可扩展，元数据管理可扩展性强。In one embodiment, the metadata storage index module 103 extracts information in the metadata instance while storing the metadata instance to create a metadata index for efficient query (for example, using search engines such as SOLR and ElasticSearch). The solution provided by this embodiment is based on storing metadata instances of big data and creating indexes for metadata instances, the storage and computing capabilities are expandable, and metadata management is highly scalable.

在一实施例中，所述元数据采集模块102是设置为以下至少之一：所述元数据采集模块102使用所述元数据类型对应的采集器采集外部信息，并根据所述外部信息创建所述元数据类型对应的元数据实例；所述元数据采集模块102根据所述元数据类型接收外部同步的信息，并根据所述外部同步的信息创建所述元数据类型对应的元数据实例。In an embodiment, the metadata collection module 102 is set to at least one of the following: the metadata collection module 102 uses the collector corresponding to the metadata type to collect external information, and creates a collection of information based on the external information. The metadata instance corresponding to the metadata type; the metadata collection module 102 receives externally synchronized information according to the metadata type, and creates a metadata instance corresponding to the metadata type according to the externally synchronized information.

外部信息可以是零散的元数据实例，也可以是存储在数据库服务器等的信息，比如关系数据库、文件传输协议(File Transfer Protocol，FTP)文件服务器、WEB服务器等的信息。The external information may be scattered metadata instances, or information stored in a database server, etc., such as information about relational databases, File Transfer Protocol (FTP) file servers, and WEB servers.

在一实施例中，所述元数据管理装置还包括元数据可视化维护模块104。所述元数据可视化维护模块104设置为，提供与所述元数据管理装置交互的操作界面，所述操作界面上包括对所述元数据实例进行管理的子界面。In an embodiment, the metadata management device further includes a metadata visualization maintenance module 104. The metadata visualization maintenance module 104 is configured to provide an operation interface for interacting with the metadata management device, and the operation interface includes a sub-interface for managing the metadata instance.

即提供一交互平台，便于用户对元数据类型和元数据实例进行管理。所述管理包括新增、修改、删除、查询元数据类型和元数据实例。还可根据需要在操作界面上提供其他功能单元。That is, an interactive platform is provided to facilitate users to manage metadata types and metadata instances. The management includes adding, modifying, deleting, and querying metadata types and metadata instances. Other functional units can also be provided on the operation interface as required.

在一实施例中，所述元数据可视化维护模块104还设置为，提供对所述元数据类型进行管理的界面。In an embodiment, the metadata visualization maintenance module 104 is further configured to provide an interface for managing the metadata type.

在一实施例中，所述对所述元数据类型进行管理的界面基于所述元数据类型的属性及属性扩展规则生成。In an embodiment, the interface for managing the metadata type is generated based on the attributes of the metadata type and attribute expansion rules.

通过定义元数据类型的属性扩展规则，实现特性化的可视化处理，所述属性扩展规则包括：By defining the attribute expansion rules of the metadata type, characteristic visualization processing is realized, and the attribute expansion rules include:

(a)属性可视化规则，例如：可编辑规则，掩码规则，显示规则，属性值变更操作规则，界面属性离开操作规则等；(a) Attribute visualization rules, such as editable rules, mask rules, display rules, attribute value change operation rules, interface attribute leave operation rules, etc.;

(b)属性值域校验规则，例如：字符串长度规则，数值型范围规则，日期型范围规则，正则规则，自定义服务检查规则等；(b) Attribute value range verification rules, such as: string length rules, numeric range rules, date range rules, regular rules, custom service inspection rules, etc.;

(c)属性值域规则，例如：单选指定类型的元数据实例，多选指定类型的元数据实例，调用服务获取值域(支持多个属性值作为参数)，单选父元数据属性值，多选父元数据属性值，单选同级元数据属性值，单选同级元数据属性的子元数据，值域格式规则(支持多个属性值作为参数)，自定义值域规则等。(c) Attribute value range rules, such as: single-select metadata instances of the specified type, multiple select metadata instances of the specified type, call the service to obtain the value range (support multiple attribute values as parameters), single-select the parent metadata attribute value , Multiple choice of parent metadata attribute value, single choice of the same level metadata attribute value, single choice of the child metadata of the same level metadata attribute, value range format rules (supporting multiple attribute values as parameters), custom range rules, etc. .

在一实施例中，如图4所示，所述元数据可视化维护模块104包括：通用维护界面生成子模块1041，还可包括以下至少之一：元数据导入导出子模块1042、元数据分类子模块1043、元数据变更子模块1044和元数据搜索子模块1045。In an embodiment, as shown in FIG. 4, the metadata visualization maintenance module 104 includes: a general maintenance interface generation submodule 1041, and may also include at least one of the following: a metadata import/export submodule 1042, a metadata classification submodule Module 1043, metadata change submodule 1044, and metadata search submodule 1045.

所述界面生成子模块1041设置为，根据元数据类型属性及其扩展属性规则，元数据类型之间的包容关系，元数据类型之间的关联关系，生成操作界面。The interface generation submodule 1041 is configured to generate an operation interface according to the metadata type attribute and its extended attribute rules, the inclusive relationship between the metadata types, and the association relationship between the metadata types.

当用户进入元数据可视化维护模块104维护元数据实例时，通用维护界面生成子模块1041解析元数据类型属性及其扩展规则，创建操作界面，用户的配置操作遵从属性扩展规则；通用维护界面生成子模块1041解析元数据类型包容关系，创建钻取界面，用户可钻取式查看子元数据；通用维护界面生成子模块1041解析元数据类型关联关系，创建关联界面。通用维护界面生成子模块1041与元数据存储索引模103块交互，实现元数据实例的新增、修改、删除和查看。When the user enters the metadata visualization maintenance module 104 to maintain the metadata instance, the general maintenance interface generation sub-module 1041 parses the metadata type attributes and their extension rules, creates an operation interface, and the user’s configuration operation complies with the attribute extension rules; the general maintenance interface generates submodules The module 1041 parses the metadata type containment relationship and creates a drill interface where the user can drill to view the sub-metadata; the general maintenance interface generation sub-module 1041 parses the metadata type association relationship and creates an association interface. The general maintenance interface generation submodule 1041 interacts with the metadata storage index module 103 to realize the addition, modification, deletion, and viewing of metadata instances.

所述元数据导入导出子模块1042设置为，实现多种格式的元数据文件的导入和导出，实现元数据库的备份和恢复。该模块可选。The metadata import and export submodule 1042 is configured to realize the import and export of metadata files in multiple formats, and realize the backup and recovery of the metadata database. This module is optional.

所述元数据分类子模块1043设置为，实现给元数据实例动态添加标签，删除标签，以及，实现根据标签查询元数据实例。该模块可选。The metadata classification sub-module 1043 is configured to dynamically add tags to metadata instances, delete tags, and query metadata instances based on tags. This module is optional.

所述元数据变更子模块1044设置为，实现根据元数据实例查询元数据实例变更历史，实现元数据实例变更统计等。该模块可选。The metadata change sub-module 1044 is configured to query metadata instance change history according to metadata instances, and implement metadata instance change statistics. This module is optional.

所述元数据搜索子模块1045设置为，实现通过文本搜索元数据实例。该模块可选。The metadata search submodule 1045 is configured to search metadata instances through text. This module is optional.

本实施例提供了一套可视化元数据实例维护规则，打破常规的定制化界面维护元数据的方式，提供了通用的可视化元数据维护方法，提供随时新增或者变更元数据类型的能力，可以可视化维护任意类型的元数据实例，元数据维护管理通用性高。This embodiment provides a set of visual metadata instance maintenance rules, breaks the conventional customized interface to maintain metadata, provides a general visual metadata maintenance method, and provides the ability to add or change metadata types at any time, which can be visualized Maintain any type of metadata instance, with high versatility for metadata maintenance and management.

在一实施例中，所述元数据管理装置100还包括元数据服务模块105，所述元数据服务模块105设置为提供对所述元数据实例的查询服务，所述查询服务包括对所述元数据实例的血缘关系的查询。元数据服务模块105与元数据存储索引模块103交互，实现多样化的元数据查询服务。本实施例中，提供一种元数据血缘关系追溯增强方法，提供粗、细粒度级别的元数据血缘关系追溯。In an embodiment, the metadata management apparatus 100 further includes a metadata service module 105 configured to provide a query service for the metadata instance, and the query service includes a metadata service module 105 Query the blood relationship of the data instance. The metadata service module 105 interacts with the metadata storage index module 103 to implement diversified metadata query services. In this embodiment, an enhanced method for tracing metadata blood relationship is provided, which provides coarse and fine-grained metadata blood relationship tracing.

在一实施例中，元数据服务模块105与元数据存储索引模块103交互，从指定元数据实例节点沿着输出边获取指定深度路径；从指定元数据实例节点沿着输入边获取指定深度路径；依据元数据类型包容关系，在血缘路径上的父节点上增加折叠标记，依此创建元数据血缘关系图并输出。本实施例提供的方案，血缘关系图从粗粒度展开到细粒度，从细粒度折叠回粗粒度，丰富地展现元数据如何产生、加工使用的历程。In one embodiment, the metadata service module 105 interacts with the metadata storage index module 103 to obtain a specified depth path from the specified metadata instance node along the output edge; obtain the specified depth path from the specified metadata instance node along the input edge; According to the inclusive relationship of the metadata type, a folding mark is added to the parent node on the kinship path, and the metadata kinship diagram is created and output accordingly. In the solution provided in this embodiment, the blood relationship graph expands from coarse-grained to fine-grained, and collapses from fine-grained to coarse-grained, which richly shows how metadata is generated, processed and used.

为实现粗、细粒度元数据血缘追溯，在定义元数据类型时：To achieve coarse and fine-grained metadata tracing, when defining metadata types:

(a)元数据类型泛化(a) Generalization of metadata types

当多种元数据类型具备相同的语义和大部分的属性，可以将其泛化，定义为父类型。When multiple metadata types have the same semantics and most of the attributes, they can be generalized and defined as a parent type.

例如：将数据集定义为一种元数据父类型，扩展数据集定义为数据表、结构化数据文件等元数据类型，即数据集(父类型)-数据表、结构化数据文件(子类型)；将字段定义为一种元数据父类型，扩展字段定义为表字段、结构化数据文件字段等元数据类型。即字段(父类型)-表字段、结构化数据文件字段(子类型)。For example: define a data set as a metadata parent type, and an extended data set as a metadata type such as a data table and a structured data file, that is, data set (parent type)-data table, structured data file (subtype) ; The field is defined as a metadata parent type, and the extension field is defined as a metadata type such as a table field and a structured data file field. Namely field (parent type)-table field, structured data file field (subtype).

(b)元数据类型细化(b) Refinement of metadata types

细化元数据类型，可以将它的部分属性进一步模型化，建立子元数据类型。To refine the metadata type, you can further model some of its attributes to establish sub-metadata types.

例如：抽取-转换-装载(Extract-Transform-Load，ETL)数据集成作业，数据集成作业的输入数据表、输出数据表、文件、接口等属性设置为数据集元数据类型，对数据集成作业的元数据类型进一步细化，将数据处理的表字段、文件字段等属性设置为字段元数据类型。这样进行数据追溯，可以从数据集层面细化到字段层面。即表字段、文件字段(字段元数据类型，为子类型)-输入数据表、输出数据表、文件、接口(数据集元数据类型，为父类型)。For example: Extract-Transform-Load (ETL) data integration job, the input data table, output data table, file, interface and other attributes of the data integration job are set to the metadata type of the data set. Metadata types are further refined, and attributes such as table fields and file fields for data processing are set as field metadata types. Data traceability in this way can be refined from the data set level to the field level. That is, table fields, file fields (field metadata type, subtype)-input data table, output data table, file, interface (data set metadata type, which is the parent type).

本发明一实施例中，所述元数据管理装置多个模块采用组件化设计，接***互。In an embodiment of the present invention, multiple modules of the metadata management device adopt componentized design and interface interaction.

本发明实施例提供的元数据管理装置，能够在大数据环境下，管理任意类型的元数据，打破了定制化的元数据维护模式，支持自动获取零散分布的元数据，提供大规模元数据的存储能力，从而建立统一的元数据视图。在统一的元数据视图之上，可以进一步地获知元数据血缘，方便地进行元数据搜索。The metadata management device provided by the embodiment of the present invention can manage any type of metadata in a big data environment, breaks the customized metadata maintenance mode, supports automatic acquisition of scattered metadata, and provides large-scale metadata. Storage capacity to establish a unified metadata view. On top of the unified metadata view, you can further know the blood relationship of the metadata and facilitate metadata search.

如图5所示，本发明一实施例提供一种用于元数据类型维护的界面，该子界面中，可对元数据类型的属性进行定义，可设置元数据类型继承的父类型，可对属性扩展规则进行定义等。As shown in Figure 5, an embodiment of the present invention provides an interface for metadata type maintenance. In this sub-interface, the attributes of the metadata type can be defined, the parent type inherited by the metadata type can be set, and the metadata type can be Attribute expansion rules are defined, etc.

如图6所示，本发明一实施例提供一种可视化的操作界面。该操作界面中，左侧为元数据类型列表，右侧包括：对元数据实例进行管理的子界面(图6中的“元数据信息”菜单)，对元数据实例变更历史进行查询的子界面(图6中的“变更历史”菜单)，查询血缘关系的子界面(图6中的“血缘影响”菜单)，给元数据实例添加标签和删除标签的子界面(图6中的“分类标签”菜单)，元数据文件导入的子界面(图6中的“导入”菜单)，以及，进行元数据实例查询的子界面(图6中的“请输入字关键字搜索”框)，等等。As shown in FIG. 6, an embodiment of the present invention provides a visual operation interface. In this operation interface, the left side is a list of metadata types, and the right side includes: a sub-interface for managing metadata instances (the "Metadata Information" menu in Figure 6), and a sub-interface for querying the change history of metadata instances (The "Change History" menu in Figure 6), the sub-interface for querying blood relationship (the "Blood Influence" menu in Figure 6), the sub-interface for adding tags and deleting tags to metadata instances ("Category Tags" in Figure 6) "Menu), the sub-interface for importing metadata files (the "Import" menu in Figure 6), and the sub-interface for querying metadata instances (the "Please enter a keyword to search" box in Figure 6), etc. .

上述操作界面的布局以及菜单仅为示例，可以根据需要使用其他操作界面与元数据管理装置进行交互，本申请对此不作限定。The layout and menu of the above operation interface are only examples, and other operation interfaces can be used to interact with the metadata management device as needed, which is not limited in this application.

图7为本发明另一实施例提供的元数据管理装置示意图。如图7所示，该元数据管理装置100包括元数据类型管理模块101、元数据采集模块102、元数据存储索引模块103、元数据可视化维护模块104和元数据服务模块105，元数据存储索引模块103包括大数据平台1031和图数据库引擎1032。图数据库引擎1032对元数据类型和元数据实例处理后存储到大数据平台1031。用户通过WEB(网页)客户端701与元数据管理装置100交互。FIG. 7 is a schematic diagram of a metadata management device provided by another embodiment of the present invention. As shown in Figure 7, the metadata management device 100 includes a metadata type management module 101, a metadata collection module 102, a metadata storage index module 103, a metadata visualization maintenance module 104, and a metadata service module 105. The metadata storage index The module 103 includes a big data platform 1031 and a graph database engine 1032. The graph database engine 1032 processes the metadata types and metadata instances and stores them in the big data platform 1031. The user interacts with the metadata management apparatus 100 through the WEB (web page) client 701.

用户使用此元数据管理装置的步骤如下：The steps for a user to use this metadata management device are as follows:

步骤一：在用户根据业务要求对元数据类型建模(支持离线、在线模式)后，元数据类型管理模块101加载元数据类型，元数据存储索引模块103记录元数据类型及其图谱。Step 1: After the user models the metadata type according to business requirements (supporting offline and online modes), the metadata type management module 101 loads the metadata type, and the metadata storage index module 103 records the metadata type and its map.

步骤二：元数据可视化维护模块104维护对应元数据类型的元数据实例，元数据存储索引模块103记录元数据实例及其图谱。Step 2: The metadata visualization maintenance module 104 maintains the metadata instance corresponding to the metadata type, and the metadata storage index module 103 records the metadata instance and its map.

若用户不需要可视化维护功能时，可以通过步骤三达到同样目的。If the user does not need the visual maintenance function, the same goal can be achieved through step three.

步骤三：元数据采集模块102自动采集元数据实例或者接收外部同步来的元数据实例，通过元数据存储索引模块103存储元数据实例及其图谱。Step 3: The metadata collection module 102 automatically collects metadata instances or receives externally synchronized metadata instances, and stores the metadata instances and their maps through the metadata storage index module 103.

步骤四：通过元数据服务模块105开放多类元数据查询接口，提供给第三方***使用。Step 4: Open multiple types of metadata query interfaces through the metadata service module 105 and provide them for use by third-party systems.

本发明实施例中，可以管理任意类型的元数据，根据业务需求对不同类型，不同地理位置的元数据信息进行抽取，合并以及进行元数据数据建模的异构处理。In the embodiment of the present invention, any type of metadata can be managed, and metadata information of different types and different geographic locations can be extracted, merged, and heterogeneous processing of metadata data modeling according to business requirements.

本发明实施例提供整套完整的元数据管理功能，解决了相关技术中无法自动化、通用化对任意类型、大规模元数据进行管理的问题，增强了元数据血缘关系追溯能力。The embodiment of the present invention provides a complete set of metadata management functions, solves the problem of the inability to automate and generalize the management of any type and large-scale metadata in related technologies, and enhances the ability of metadata blood relationship traceability.

如图8所示，本发明一实施例提供一种元数据管理方法，包括：As shown in FIG. 8, an embodiment of the present invention provides a metadata management method, including:

步骤801，加载基于面向对象模式定义的元数据类型，存储所述元数据类型；Step 801: Load a metadata type defined based on the object-oriented mode, and store the metadata type;

步骤802，根据所述元数据类型获取所述元数据类型对应的元数据实例，存储所述元数据实例。Step 802: Obtain a metadata instance corresponding to the metadata type according to the metadata type, and store the metadata instance.

在一实施例中，所述方法还包括：存储所述元数据类型时，还使用图数据库存储元数据类型之间的关系，以及，在存储所述元数据实例时，还使用图数据库存储元数据实例之间的关系。In an embodiment, the method further includes: when storing the metadata type, further using a graph database to store the relationship between the metadata types, and, when storing the metadata instance, further using a graph database storage element The relationship between data instances.

在一实施例中，所述存储所述元数据类型包括：使用列式存储数据库存储所述元数据类型；所述存储所述元数据实例包括：使用列式存储数据库存储所述元数据实例。In an embodiment, the storing the metadata type includes: using a columnar storage database to store the metadata type; and the storing the metadata instance includes: using a columnar storage database to store the metadata instance.

在一实施例中，所述根据所述元数据类型获取所述元数据类型对应的元数据实例包括以下至少之一：使用所述元数据类型对应的采集器采集外部信息，并根据所述外部信息创建所述元数据类型对应的元数据实例；根据所述元数据类型接收外部同步的信息，并根据所述外部同步的信息创建所述元数据类型对应的元数据实例。In an embodiment, the obtaining the metadata instance corresponding to the metadata type according to the metadata type includes at least one of the following: using a collector corresponding to the metadata type to collect external information, and according to the external Information creates a metadata instance corresponding to the metadata type; receives externally synchronized information according to the metadata type, and creates a metadata instance corresponding to the metadata type according to the externally synchronized information.

在一实施例中，所述方法还包括：通过操作界面接收对所述元数据类型进行管理的管理指令，执行相应的对所述元数据实例的管理操作。In an embodiment, the method further includes: receiving a management instruction for managing the metadata type through an operating interface, and executing corresponding management operations on the metadata instance.

在一实施例中，所述管理指令包括：对所述元数据类型的属性进行定义的指令。对所述元数据类型的属性定义满足预设属性扩展规则。对元数据类型的管理指令还包括元数据类型的新增、删除、更新、查询等等指令。In an embodiment, the management instruction includes an instruction to define attributes of the metadata type. The attribute definition of the metadata type satisfies a preset attribute expansion rule. The management commands for metadata types also include commands for adding, deleting, updating, and querying metadata types.

在一实施例中，所述方法还包括，接收到对所述元数据实例的血缘关系的查询请求时，输出所述元数据实例的血缘关系信息。In an embodiment, the method further includes outputting the blood relationship information of the metadata instance when receiving a query request for the blood relationship of the metadata instance.

本发明实施例提供的方案，具有广泛的市场应用场景，包括：The solutions provided by the embodiments of the present invention have a wide range of market application scenarios, including:

1)数据治理领域，元数据管理是数据治理的基础支撑，应用本发明实施例中的元数据管理方案可以帮助用户建立统一的数据地图，对外提供高效、灵活的查询服务，数据集成、数据安全、数据质量等功能模块均可通过本发明实施例提供的元数据管理方案获取待治理的数据并设计数据处理任务，完成数据治理的目标。1) In the field of data governance, metadata management is the basic support for data governance. The application of the metadata management scheme in the embodiment of the present invention can help users build a unified data map, provide efficient and flexible query services, data integration, and data security. The functional modules such as data quality can obtain the data to be managed and design data processing tasks through the metadata management scheme provided by the embodiment of the present invention, so as to complete the goal of data management.

2)数据共享交换领域，政府部门在信息化的过程中积累了海量的数据，然而由于部门壁垒导致数据孤岛化，政府正在积极推进部门间数据的共享交换。建设数据共享交换***，需要收集和管理相关部门的共享资源信息，需要知道数据的使用流向，运用本发明实施例提供的方案，可以对任意类型的元数据进行建模和管理，可以轻松追溯数据的血缘关系。2) In the field of data sharing and exchange, government departments have accumulated massive amounts of data in the process of informatization. However, due to departmental barriers leading to data islanding, the government is actively promoting the sharing and exchange of data between departments. To build a data sharing and exchange system, it is necessary to collect and manage shared resource information of relevant departments, and to know the flow of data usage. Using the solution provided by the embodiment of the present invention, any type of metadata can be modeled and managed, and data can be easily traced Blood relationship.

3)数据开放领域，企业和政府对社会开放数据，共享数据资源，需要对外开放的数据服务是元数据，运用本发明实施例提供的方案可以对多种类型的数据服务建模，统一管理数据服务信息。3) In the field of data openness, enterprises and governments open data to the society and share data resources. The data service that needs to be opened to the outside world is metadata. The solution provided by the embodiments of the present invention can model various types of data services and manage data uniformly. Service Information.

以下举例说明本申请在不同场景的应用。The following examples illustrate the application of this application in different scenarios.

实施例1Example 1

一地市政府需要统一采集公安局、卫生局、工商局的数据，实现集中存储，进而实现部门间数据的共享交换。以该政府使用本装置为例对本公开进行描述：The government of a city needs to collect data from the Public Security Bureau, the Health Bureau, and the Industrial and Commercial Bureau in a unified manner to realize centralized storage, and then realize the sharing and exchange of data between departments. This disclosure is described by taking the government using this device as an example:

实施环境说明：本实施例中提供一数据共享交换***。每个部门的数据存在于各自的业务***中，每个部门的业务***与所述数据共享交换***通过专网连接，所述业务***开放的接口有关系数据库、FTP文件服务器、WEB服务器多种形式。所述数据共享交换***需要采集多个部门的数据统一存储，提供部门间数据共享交换能力。所述数据共享交换***部署本发明实施例所述的元数据管理装置实现共享资源数据信息的管理，能够获知共享资源对象的产生、处理、使用的过程。Implementation environment description: In this embodiment, a data sharing exchange system is provided. The data of each department exists in its own business system, and the business system of each department is connected to the data sharing and exchange system through a private network. The open interfaces of the business system include relational database, FTP file server, and WEB server. form. The data sharing and exchange system needs to collect data from multiple departments for unified storage, and provide data sharing and exchange capabilities between departments. The data sharing and exchange system deploys the metadata management device described in the embodiment of the present invention to realize the management of shared resource data information, and can learn the process of generating, processing, and using shared resource objects.

图9是本发明一实施例提供的数据共享领域实现流程图，如图9所示，包括：Fig. 9 is a flow chart for realizing the data sharing field provided by an embodiment of the present invention, as shown in Fig. 9, including:

步骤901，加载元数据类型。Step 901: Load the metadata type.

所述元数据类型由客户定义，所述元数据类型基于面向对象模式定义。The metadata type is defined by the customer, and the metadata type is defined based on the object-oriented model.

本实施例中，所述元数据类型包括：部门业务***开放的资源接口，部门业务***开放的资源对象，数据共享交换***的数据仓库，数据共享交换***的数据集成作业，数据共享交换***的数据订阅。In this embodiment, the metadata types include: resource interfaces opened by the department business system, resource objects opened by the department business system, data warehouse of the data sharing exchange system, data integration operations of the data sharing exchange system, and data sharing exchange system Data subscription.

步骤902，根据配置的部门的资源接口信息，通过元数据采集模块102自动采集部门的资源对象的元数据实例。Step 902: According to the resource interface information of the configured department, the metadata collection module 102 automatically collects metadata instances of the resource objects of the department.

步骤903，根据客户配置的数据共享交换***信息，通过元数据采集模块102自动采集数据集成作业的元数据实例、数据订阅申请的元数据实例。Step 903: According to the data sharing and exchange system information configured by the customer, the metadata collection module 102 automatically collects the metadata instance of the data integration job and the metadata instance of the data subscription application.

数据集成作业由客户在数据共享交换***查询元数据后创建，数据订阅申请由客户在数据共享交换***查询元数据后提交。The data integration job is created by the customer after querying the metadata in the data sharing exchange system, and the data subscription application is submitted by the customer after querying the metadata in the data sharing exchange system.

步骤904，接收客户查看元数据血缘关系的指令，输出所查询的元数据实例的血缘关系。Step 904: Receive an instruction from the client to view the blood relationship of the metadata, and output the blood relationship of the metadata instance inquired.

步骤905，接收客户的管理元数据的指令，进行元数据实例的管理，所述管理包括以下至少之一：元数据实例的新增、修改、删除和查看。Step 905: Receive an instruction for managing metadata from the client, and perform management of metadata instances. The management includes at least one of the following: adding, modifying, deleting, and viewing metadata instances.

实施例2Example 2

以一政府部门使用本发明实施例中的元数据管理装置管理多类数据标准规范为例进行描述。Take a government department using the metadata management device in the embodiment of the present invention to manage multiple types of data standard specifications as an example for description.

实施环境说明：一政府部门需要管理国家、部、省、地方发布的数据标准规范。该政府部门借助本发明实施例所述的元数据管理装置管理多类规范，本实施例中，元数据管理装置包括元数据类型管理模块、元数据采集模块、元数据存储索引模块和元数据可视化维护模块。Implementation environment description: A government agency needs to manage data standards and specifications issued by the state, ministry, province, and locality. The government department manages multiple specifications by means of the metadata management device described in the embodiment of the present invention. In this embodiment, the metadata management device includes a metadata type management module, a metadata collection module, a metadata storage index module, and a metadata visualization Maintenance module.

图10是本发明一实施例提供的应用元数据管理装置进行数据标准管理的流程图，如图10所示，包括：Fig. 10 is a flowchart of data standard management performed by an application metadata management apparatus provided by an embodiment of the present invention, as shown in Fig. 10, including:

步骤1001，加载定义好的元数据类型。Step 1001: Load the defined metadata type.

所述元数据类型由客户基于面向对象模式定义。客户梳理多类数据标准规范文档，抽取公共数据标准，定义、抽象元数据类型；客户按照对象继承关系、包容关系、依赖关系，定义多种数据标准规范的元数据类型。The metadata type is defined by the client based on the object-oriented model. Clients sort out multiple data standard specification documents, extract common data standards, define and abstract metadata types; clients define metadata types for multiple data standards and specifications according to object inheritance, tolerance, and dependency relationships.

步骤1002，根据客户在元数据管理装置中导入的对应规范的数据标准文件，使用元数据采集模块102的文件采集器，自动采集多类数据标准规范的元数据实例(本实施例中，元数据实例为数据标准)；Step 1002: Use the file collector of the metadata collection module 102 to automatically collect metadata instances of multiple types of data standards and specifications (in this embodiment, metadata Examples are data standards);

步骤1003，根据客户指令，管理多类规范的数据标准。Step 1003: Manage multiple types of standardized data standards according to customer instructions.

所述管理包括以下至少之一：进行数据标准(即元数据实例)的新增、修改、删除和查看。The management includes at least one of the following: adding, modifying, deleting, and viewing data standards (ie, metadata instances).

实施例3Example 3

一企业内部多部门有多个异构业务***，需要集中管理多个业务***的数据，进行数据治理，提升数据质量，保证数据安全。以该企业使用本发明实施例提供的元数据管理装置为例对本公开进行描述。An enterprise has multiple heterogeneous business systems in multiple departments, and needs to centrally manage the data of multiple business systems, conduct data governance, improve data quality, and ensure data security. The present disclosure is described by taking the enterprise using the metadata management device provided by the embodiment of the present invention as an example.

实施环境说明：企业建立一个数据治理***，所述数据治理***与异构业务***之间专网互连，所述异构业务***中的部分业务***直接开放数据库或者文件服务器，部分业务***开放WEB服务接口，部分业务***主动同步数据到所述数据治理***。所述数据治理***部署本发明实施例所述的元数据管理装置，基于统一的元数据管理实现数据治理多项功能。Implementation environment description: The enterprise establishes a data management system, the data management system and the heterogeneous business system are interconnected by a private network, and some business systems in the heterogeneous business system directly open the database or file server, and some business systems are open WEB service interface, some business systems actively synchronize data to the data management system. The data management system deploys the metadata management device described in the embodiment of the present invention, and realizes multiple functions of data management based on unified metadata management.

以该企业使用元数据管理装置为例对本公开进行描述。图11是本发明一实施例提供的应用元数据管理装置进行数据治理的流程图，如图11所示，包括：The present disclosure will be described by taking the enterprise using a metadata management device as an example. FIG. 11 is a flowchart of data management performed by an application metadata management apparatus provided by an embodiment of the present invention, as shown in FIG. 11, including:

步骤1101，加载定义好的元数据类型。Step 1101: Load the defined metadata type.

所述元数据类型由客户基于面向对象模式定义，本实施例中的元数据类型，包括：The metadata type is defined by the client based on the object-oriented model. The metadata type in this embodiment includes:

1.业务***开放的资源接口；1. The open resource interface of the business system;

2.业务***开放的资源对象；2. Resource objects open to the business system;

3.数据治理***的数据仓库；3. Data warehouse of data management system;

4.数据治理***的数据集成作业；4. Data integration operations of the data management system;

5.数据治理***的数据质量作业；5. Data quality operations of the data governance system;

6.数据治理***的数据安全作业。6. Data security operations of the data management system.

步骤1102，根据客户配置的业务***的资源接口信息获取资源对象的元数据实例。Step 1102: Obtain the metadata instance of the resource object according to the resource interface information of the business system configured by the customer.

当资源接口为直连接口时，使用元数据采集模块自动采集业务***的资源对象的元数据实例；当资源接口为同步接口时，使用元数据采集模块接收外部同步的信息创建资源对象的元数据实例。When the resource interface is a direct connection port, use the metadata collection module to automatically collect the metadata instance of the resource object of the business system; when the resource interface is a synchronous interface, use the metadata collection module to receive externally synchronized information to create the metadata of the resource object Instance.

步骤1103，根据客户配置的数据治理***地址信息，使用元数据采集模块102自动采集数据治理***的数据集成作业的元数据实例，自动采集数据治理***的数据质量作业的元数据实例，以及，自动采集数据安全作业的元数据实例。 Step 1103, according to the address information of the data management system configured by the customer, use the metadata collection module 102 to automatically collect metadata instances of data integration operations of the data management system, automatically collect metadata instances of data quality operations of the data management system, and, automatically Collect metadata examples of data security operations.

数据集成作业、数据质量作业、数据安全作业由客户在数据治理***查询元数据后创建。Data integration jobs, data quality jobs, and data security jobs are created by customers after querying metadata in the data management system.

采集元数据实例时，基于所定义的数据治理***的数据集成作业的元数据类型采集其相应的元数据实例，基于所定义的数据治理***的数据质量作业的元数据类型采集其相应的元数据实例，基于所定义的数据治理***的数据安全作业的元数据类型采集其相应的元数据实例。When collecting metadata instances, collect the corresponding metadata instances based on the defined metadata types of the data integration tasks of the data management system, and collect corresponding metadata based on the metadata types of the defined data quality tasks of the data management system Instance, collect the corresponding metadata instance based on the defined metadata type of the data security operation of the data management system.

步骤1104，接收客户的查看元数据血缘关系的指令，输出所查询的元数据实例的血缘关系。Step 1104: Receive an instruction from the client to view the blood relationship of the metadata, and output the blood relationship of the metadata instance inquired.

步骤1105，接收客户的元数据管理指令，进行元数据实例的管理操作。Step 1105: Receive a metadata management instruction from the client, and perform management operations of metadata instances.

所述元数据实例的管理操作包括以下至少之一：元数据实例的新增、修改、删除和查看。The management operation of the metadata instance includes at least one of the following: adding, modifying, deleting and viewing the metadata instance.

实施例4Example 4

一政府部门对公众发布数据服务，共享数据资源，发挥数据社会价值。以该政府使用本发明实施例提供的元数据管理装置为例对本公开进行描述：A government department releases data services to the public, shares data resources, and exerts the social value of data. This disclosure is described by taking the government using the metadata management device provided by the embodiment of the present invention as an example:

实施环境说明：政府部门建立一个数据服务***，该数据服务***登记部门对外开放的多种数据服务，公众可以浏览并且访问。政府部门部署本发明实施例中的元数据管理装置管理多种数据服务，本实施例中，元数据管理装置包括元数据类型管理模块、元数据采集模块、元数据存储索引模块和元数据服务模块。Implementation environment description: The government department establishes a data service system, and the data service system registers various data services opened to the outside world by the public to browse and access. The government department deploys the metadata management device in the embodiment of the present invention to manage multiple data services. In this embodiment, the metadata management device includes a metadata type management module, a metadata collection module, a metadata storage index module, and a metadata service module .

图12是本发明一实施例提供的应用元数据管理装置提供数据服务的流程图，如图12所示，包括：FIG. 12 is a flowchart of data services provided by an application metadata management apparatus according to an embodiment of the present invention, as shown in FIG. 12, including:

步骤1201，加载定义好的元数据类型。Step 1201: Load the defined metadata type.

所述元数据类型由客户根据部门对外开放的数据服务设计，所述元数据类型基于面向对象模式定义。The metadata type is designed by the customer according to the data service opened to the outside world by the department, and the metadata type is defined based on the object-oriented model.

步骤1202，根据客户提交的数据服务定义文件，使用元数据采集模块102分析数据服务定义文件，例如SOAP服务的Web服务描述语言(Web Services Description Language，WSDL)，REST服务的YAML(YAML Ain't a Markup Language，YAML不是一种标记语言)，根据所定义的元数据类型创建元数据实例。Step 1202: Use the metadata collection module 102 to analyze the data service definition file according to the data service definition file submitted by the customer, such as the Web Services Description Language (WSDL) of SOAP service and YAML (YAML Ain't) of REST service. a Markup Language, YAML is not a markup language), create metadata instances according to the defined metadata types.

步骤1203，接收客户提交的数据服务信息查询请求，输出相应的数据服务信息。Step 1203: Receive the data service information query request submitted by the client, and output corresponding data service information.

综上，本发明实施例提供的方案具有广阔的市场应用场景，能够带来较大的研究和经济价值。In summary, the solutions provided by the embodiments of the present invention have broad market application scenarios and can bring greater research and economic value.

如图13所示，本发明一实施例提供一种元数据管理设备130，包括存储器1310和处理器1320，所述存储器1310存储有程序，所述程序在被所述处理器1320读取执行时，实现任一实施例所述的元数据管理方法。As shown in FIG. 13, an embodiment of the present invention provides a metadata management device 130, including a memory 1310 and a processor 1320, the memory 1310 stores a program, and the program is read and executed by the processor 1320. , To implement the metadata management method described in any embodiment.

如图14所示，本发明一实施例提供一种计算机可读存储介质140，所述计算机可读存储介质140存储有一个或者多个程序141，所述一个或者多个程序141可被一个或者多个处理器执行，以实现任一实施例所述的元数据管理方法。As shown in FIG. 14, an embodiment of the present invention provides a computer-readable storage medium 140. The computer-readable storage medium 140 stores one or more programs 141, and the one or more programs 141 can be used by one or A plurality of processors execute to implement the metadata management method described in any embodiment.

本领域普通技术人员可以理解，上文中所公开方法中的全部或一些步骤、***、装置中的功能模块/单元可以被实施为软件、固件、硬件及其适当的组合。在硬件实施方式中，在以上描述中提及的功能模块/单元之间的划分不一定对应于物理组件的划分；例如，一个物理组件可以具有多个功能，或者一个功能或步骤可以由多个物理组件合作执行。一些组件或所有组件可以被实施为由处理器，如数字信号处理器或微处理器执行的软件，或者被实施为硬件，或者被实施为集成电路，如专用集成电路。这样的软件可以分布在计算机可读介质上，计算机可读介质可以包括计算机存储介质(或非暂时性介质)和通信介质(或暂时性介质)。如本领域普通技术人员公知的，术语计算机存储介质包括在用于存储信息(诸如计算机可读指令、数据结构、程序模块或其他数据)的任何方法或技术中实施的易失性和非易失性、可移除和不可移除介质。计算机存储介质包括但不限于随机存取存储器(Random Access Memory，RAM)、只读存储器(Read-Only Memory，ROM)、电可擦除可编程只读存储器(Electrically Erasable Programmable Read-Only Memory，EEPROM、闪存或其他存储器技术、光盘只读存储器(Compact Disc Read-Only Memory，CD-ROM)、数字多功能盘(Digital Versatile Disc，DVD)或其他光盘存储、磁盒、磁带、磁盘存储或其他磁存储装置、或者可以用于存储期望的信息并且可以被计算机访问的任何其他的介质。此外，本领域普通技术人员公知的是，通信介质通常包含计算机可读指令、数据结构、程序模块或者诸如载波或其他传输机制之类的调制数据信号中的其他数据，并且可包括任何信息递送介质。A person of ordinary skill in the art can understand that all or some of the steps, functional modules/units in the system, and apparatus in the methods disclosed above can be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between functional modules/units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may consist of multiple The physical components cooperate to execute. Some or all of the components may be implemented as software executed by a processor, such as a digital signal processor or a microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on a computer-readable medium, and the computer-readable medium may include a computer storage medium (or non-transitory medium) and a communication medium (or transitory medium). As is well known to those of ordinary skill in the art, the term computer storage medium includes volatile and non-volatile memory implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Flexible, removable and non-removable media. Computer storage media include but are not limited to Random Access Memory (RAM), Read-Only Memory (ROM), Electrically Erasable Programmable Read-Only Memory, EEPROM , Flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (Digital Versatile Disc, DVD) or other optical disc storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic A storage device, or any other medium that can be used to store desired information and that can be accessed by a computer. In addition, as is well known to those of ordinary skill in the art, communication media usually include computer-readable instructions, data structures, program modules, or such Or other data in the modulated data signal such as other transmission mechanisms, and may include any information delivery medium.

Claims

一种元数据管理装置，包括元数据类型管理模块、元数据采集模块和元数据存储索引模块，其中：A metadata management device includes a metadata type management module, a metadata collection module, and a metadata storage index module, wherein:

所述元数据类型管理模块设置为，加载基于面向对象模式定义的元数据类型，将所述元数据类型记录在所述元数据存储索引模块；The metadata type management module is configured to load the metadata type defined based on the object-oriented mode, and record the metadata type in the metadata storage index module;

所述元数据采集模块设置为，根据所述元数据类型获取所述元数据类型对应的元数据实例，将所述元数据实例存储在所述元数据存储索引模块；The metadata collection module is configured to obtain a metadata instance corresponding to the metadata type according to the metadata type, and store the metadata instance in the metadata storage index module;

所述元数据存储索引模块设置为，存储所述元数据类型和所述元数据实例。The metadata storage index module is configured to store the metadata type and the metadata instance.
根据权利要求1所述的装置，所述元数据存储索引模块还设置为，在存储所述元数据类型时，使用图数据库存储所述元数据类型之间的关系，以及，在存储所述元数据实例时，使用图数据库存储所述元数据实例之间的关系。The apparatus according to claim 1, wherein the metadata storage index module is further configured to use a graph database to store the relationship between the metadata types when storing the metadata types, and to store the metadata When data instances are used, a graph database is used to store the relationships between the metadata instances.
根据权利要求1所述的装置，其中，所述元数据存储索引模块是设置为：使用列式存储数据库存储所述元数据类型和所述元数据实例。The apparatus according to claim 1, wherein the metadata storage index module is configured to store the metadata type and the metadata instance using a columnar storage database.
根据权利要求1所述的装置，其中，所述元数据采集模块是设置为以下至少之一：The device according to claim 1, wherein the metadata collection module is set to at least one of the following:

使用所述元数据类型对应的采集器采集外部信息，并根据所述外部信息创建所述元数据类型对应的元数据实例；Use the collector corresponding to the metadata type to collect external information, and create a metadata instance corresponding to the metadata type according to the external information;

根据所述元数据类型接收外部同步的信息，并根据所述外部同步的信息创建所述元数据类型对应的元数据实例。Receive externally synchronized information according to the metadata type, and create a metadata instance corresponding to the metadata type according to the externally synchronized information.
根据权利要求1至4任一项所述的装置，还包括元数据可视化维护模块，其中：The device according to any one of claims 1 to 4, further comprising a metadata visualization maintenance module, wherein:

所述元数据可视化维护模块设置为，提供与所述元数据管理装置交互的操作界面，所述操作界面上包括对所述元数据实例进行管理的子界面。The metadata visualization maintenance module is configured to provide an operation interface for interacting with the metadata management device, and the operation interface includes a sub-interface for managing the metadata instance.
根据权利要求1至4任一项所述的装置，还包括元数据服务模块，所述元数据服务模块设置为提供对所述元数据实例的查询服务，所述查询服务包括对所述元数据实例的血缘关系的查询。The device according to any one of claims 1 to 4, further comprising a metadata service module configured to provide a query service for the metadata instance, and the query service includes a query service for the metadata Query the blood relationship of the instance.
一种元数据管理方法，包括：A metadata management method, including:

加载基于面向对象模式定义的元数据类型，存储所述元数据类型；Load the metadata type defined based on the object-oriented mode, and store the metadata type;

根据所述元数据类型获取所述元数据类型对应的元数据实例，存储所述元数据实例。Obtain the metadata instance corresponding to the metadata type according to the metadata type, and store the metadata instance.
根据权利要求7所述的方法，还包括：在存储所述元数据类型时，使用图数据库存储所述元数据类型之间的关系，以及，在存储所述元数据实例时，使用图数据库存储所述元数据实例之间的关系。The method according to claim 7, further comprising: when storing the metadata type, using a graph database to store the relationship between the metadata types, and when storing the metadata instance, using a graph database to store The relationship between the metadata instances.
根据权利要求7所述的方法，其中，The method according to claim 7, wherein:

所述存储所述元数据类型包括：使用列式存储数据库存储所述元数据类型；The storing the metadata type includes: using a columnar storage database to store the metadata type;

所述存储所述元数据实例包括：使用列式存储数据库存储所述元数据实例。The storing the metadata instance includes: using a columnar storage database to store the metadata instance.
根据权利要求7所述的方法，其中，所述根据所述元数据类型获取所述元数据类型对应的元数据实例包括以下至少之一：8. The method according to claim 7, wherein the obtaining the metadata instance corresponding to the metadata type according to the metadata type comprises at least one of the following:

使用所述元数据类型对应的采集器采集外部信息，并根据所述外部信息创建所述元数据类型对应的元数据实例；Use the collector corresponding to the metadata type to collect external information, and create a metadata instance corresponding to the metadata type according to the external information;

根据所述元数据类型接收外部同步的信息，并根据所述外部同步的信息创建所述元数据类型对应的元数据实例。Receive externally synchronized information according to the metadata type, and create a metadata instance corresponding to the metadata type according to the externally synchronized information.
根据权利要求7至10任一项所述的方法，还包括，接收到对所述元数据实例的血缘关系的查询请求时，输出所述元数据实例的血缘关系信息。The method according to any one of claims 7 to 10, further comprising, when receiving a query request for the blood relationship of the metadata instance, outputting the blood relationship information of the metadata instance.
一种元数据管理设备，包括存储器和处理器，所述存储器存储有程序，所述程序在被所述处理器读取执行时，实现如权利要求7至11任一项所述的元数据管理方法。A metadata management device, comprising a memory and a processor, the memory stores a program, and when the program is read and executed by the processor, the metadata management according to any one of claims 7 to 11 is realized method.
一种计算机可读存储介质，存储有至少一个程序，所述至少一个程序可被至少一个处理器执行，以实现如权利要求7至11任一项所述的元数据管理方法。A computer-readable storage medium storing at least one program, and the at least one program can be executed by at least one processor to implement the metadata management method according to any one of claims 7 to 11.