CN115129716A - Data management method, equipment and storage medium for industrial big data - Google Patents
Data management method, equipment and storage medium for industrial big data Download PDFInfo
- Publication number
- CN115129716A CN115129716A CN202210735928.9A CN202210735928A CN115129716A CN 115129716 A CN115129716 A CN 115129716A CN 202210735928 A CN202210735928 A CN 202210735928A CN 115129716 A CN115129716 A CN 115129716A
- Authority
- CN
- China
- Prior art keywords
- data
- standard
- target
- source
- data table
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 50
- 238000013523 data management Methods 0.000 title claims abstract description 29
- 238000006243 chemical reaction Methods 0.000 claims abstract description 35
- 238000012545 processing Methods 0.000 claims abstract description 17
- 238000007726 management method Methods 0.000 claims abstract description 15
- 238000013499 data model Methods 0.000 claims description 32
- 238000013075 data extraction Methods 0.000 claims description 25
- 238000001514 detection method Methods 0.000 claims description 21
- 238000000586 desensitisation Methods 0.000 claims description 10
- 238000012795 verification Methods 0.000 claims description 6
- 230000001960 triggered effect Effects 0.000 claims description 4
- 238000013507 mapping Methods 0.000 claims description 3
- 230000010354 integration Effects 0.000 abstract description 3
- 238000010586 diagram Methods 0.000 description 10
- 238000004590 computer program Methods 0.000 description 7
- 230000008569 process Effects 0.000 description 5
- 230000006870 function Effects 0.000 description 4
- 230000004048 modification Effects 0.000 description 4
- 238000012986 modification Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 101100217298 Mus musculus Aspm gene Proteins 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000000873 masking effect Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24564—Applying rules; Deductive queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/258—Data format conversion from or to a database
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/602—Providing cryptographic facilities or services
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/62—Protecting access to data via a platform, e.g. using keys or access control rules
- G06F21/6218—Protecting access to data via a platform, e.g. using keys or access control rules to a system of files or objects, e.g. local or distributed file system or database
- G06F21/6227—Protecting access to data via a platform, e.g. using keys or access control rules to a system of files or objects, e.g. local or distributed file system or database where protection concerns the structure of data, e.g. records, types, queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2211/00—Indexing scheme relating to details of data-processing equipment not covered by groups G06F3/00 - G06F13/00
- G06F2211/007—Encryption, En-/decode, En-/decipher, En-/decypher, Scramble, (De-)compress
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2221/00—Indexing scheme relating to security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F2221/21—Indexing scheme relating to G06F21/00 and subgroups addressing additional information or applications relating to security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F2221/2107—File encryption
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Computer Hardware Design (AREA)
- Computer Security & Cryptography (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Bioethics (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a data management method, equipment and a storage medium for industrial big data. The method comprises the following steps: determining a plurality of service system data sources, and analyzing data dictionary tables and model fields in the service system data sources to determine data conversion rules corresponding to source data tables in the service system data sources; performing format conversion on the source data in each source data table according to a data conversion rule to generate initial standard data, and processing the initial standard data based on a preset data security management algorithm to generate standard data; extracting the standard data into a target data table corresponding to the source data table; and determining metadata and main data information corresponding to the target data table, and issuing the target data table based on the metadata and main data information corresponding to the target data table. The method realizes the integration and treatment of the industrial data of each service system according to the unified standard.
Description
Technical Field
The present application relates to the field of data management and application technology for industrial big data in an industrial internet, and in particular, to a data management method, device and storage medium for industrial big data.
Background
With the advent of the industrial information age, enterprises are continuously perfecting and optimizing the construction of business information systems, and because the service systems are different in providers, different in data storage modes and different in data standards, it is very difficult to connect the data of the business systems in series according to a unified standard to become valuable data assets.
For the integrated data, different data users have different understandings on the index data, the same index has a non-uniform caliber, and the accuracy of the data cannot be distinguished. Therefore, how to integrate and manage the industrial data of each business system according to the unified standard becomes a technical problem to be solved urgently.
Disclosure of Invention
The embodiment of the application provides a data management method, equipment and a storage medium for industrial big data, which are used for solving the following technical problems: how to integrate and manage the industrial data of each service system according to a unified standard.
In a first aspect, an embodiment of the present application provides a data management method for industrial big data, where the method includes: determining a plurality of service system data sources, and analyzing data dictionary tables and model fields in the service system data sources to determine data conversion rules corresponding to source data tables in the service system data sources; the data dictionary table comprises data types stored in the source data tables in the corresponding service system data sources, and the model field is used for describing data representation forms of the corresponding data types; performing format conversion on the source data in each source data table according to a data conversion rule to generate initial standard data, and processing the initial standard data based on a preset data security management algorithm to generate standard data; extracting the standard data into a target data table corresponding to the source data table; determining metadata and main data information corresponding to the target data table, and issuing the target data table based on the metadata and the main data information corresponding to the target data table; and the main data information corresponding to the target data table is used for describing the data type of the standard data contained in the target data table.
In an implementation manner of the present application, processing the initial standard data based on a preset data security management algorithm specifically includes: determining whether the initial standard data is sensitive data according to a preset sensitive data judgment rule, and carrying out desensitization processing on the initial standard data through a desensitization algorithm under the condition that the initial standard data is determined to be sensitive data; and encrypting the initial standard data based on the data reading authority and the data encryption algorithm corresponding to each service system data source.
In an implementation manner of the present application, extracting standard data into a target data table corresponding to a source data table specifically includes: constructing data extraction logic of the target data table to the source data table; the data extraction logic comprises the data type of standard data to be extracted and the mapping relation between a source data table and a target data table; determining a data extraction mode corresponding to the standard data based on the table type of the source data table; wherein the table types include: the modifiable data table and the non-modifiable data table; the data extraction method comprises the following steps: extracting full data and incremental data; and under the condition that a timing data extraction task corresponding to the standard data is triggered, extracting the standard data into a target data table corresponding to the source data table based on a data extraction mode corresponding to the standard data.
In an implementation manner of the present application, after the standard data is extracted into the target data table corresponding to the source data table, the method further includes: constructing a data quality detection rule corresponding to the target data table; under the condition of triggering a timing data detection task corresponding to a target data table, detecting standard data in the target data table based on a data quality detection rule, and generating a data quality detection report to determine whether the standard data in the target data table has a quality problem; the data quality detection report comprises judgment results of each standard data based on the data quality detection rule; in the case of quality problems of the standard data in the target data table, the data extraction logic and/or the data conversion rules are modified based on the data quality detection report to ensure the quality of the standard data re-extracted into the target data table.
In one implementation of the present application, after target data table publishing is performed based on the metadata and main data information pair, the method further includes: determining metadata and main data information corresponding to a to-be-generated demand data model based on preset business demands; and extracting main data in the target data table based on metadata and main data information corresponding to the to-be-generated demand data model to construct a demand data model.
In one implementation of the present application, after building the demand data model, the method further includes: determining a storage address of a demand data model; based on the storage address of the demand data model, creating a data sharing service corresponding to the demand data model, which specifically comprises the following steps: defining an access path, an access protocol, an access parameter, a return parameter and a required data model summarizing logic of the data sharing service; and issuing the data sharing service, and setting the data sharing service to be called in an API mode.
In an implementation manner of the present application, after issuing the data sharing service and setting that the data sharing service can be called by an API manner, the method further includes: receiving and analyzing a call request corresponding to the data sharing service to determine a request address and user permission contained in the call request; based on the user authority, determining displayable data of corresponding authority in the demand data model, and sending key verification information; and returning exposable data requiring corresponding rights in the data model under the condition that a correct key is received based on the key verification information.
In one implementation manner of the present application, the types of the service system data source include: JDBC type database, big data type database, DB type database.
In a second aspect, an embodiment of the present application further provides a data management device for industrial big data, where the device includes: a processor; and a memory having executable code stored thereon, the executable code, when executed, causing the processor to perform a method of data management for industrial big data as claimed in any of claims 1-8.
In a third aspect, an embodiment of the present application further provides a non-volatile computer storage medium for data management of industrial big data, where computer-executable instructions are stored, and the computer-executable instructions are configured to: determining a plurality of service system data sources, and analyzing data dictionary tables and model fields in the service system data sources to determine data conversion rules corresponding to source data tables in the service system data sources; the data dictionary table comprises data types stored in the source data tables in the corresponding service system data sources, and the model field is used for describing data representation forms of the corresponding data types; performing format conversion on source data in each source data table according to a data conversion rule to generate initial standard data, and processing the initial standard data based on a preset data security management algorithm to generate standard data; extracting the standard data into a target data table corresponding to the source data table; determining metadata and main data information corresponding to the target data table, and issuing the target data table based on the metadata and main data information corresponding to the target data table; and the main data information corresponding to the target data table is used for describing the data type of the standard data contained in the target data table.
According to the data management method, the equipment and the storage medium for the industrial big data, the industrial data of each business system are integrated and managed according to a unified standard through data integration, conversion of a unified data format, management of data quality, management of data safety and data sharing of data sources of different business systems, and data assets are formed after integration and management, so that a data user can directly use standardized data indexes.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the application and together with the description serve to explain the application and not to limit the application. In the drawings:
fig. 1 is a flowchart of a data management method for industrial big data according to an embodiment of the present application;
fig. 2 is a schematic diagram of an internal structure of a data management device for industrial big data according to an embodiment of the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the technical solutions of the present application will be described in detail and completely with reference to the following specific embodiments of the present application and the accompanying drawings. It should be apparent that the described embodiments are only a few embodiments of the present application, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
The embodiment of the application provides a data management method, equipment and a storage medium for industrial big data, which are used for solving the following technical problems: how to integrate and manage the industrial data of each service system according to a uniform standard.
The technical solutions proposed in the embodiments of the present application are described in detail below with reference to the accompanying drawings.
Fig. 1 is a flowchart of a data management method for industrial big data according to an embodiment of the present application. As shown in fig. 1, a data management method for industrial big data provided in an embodiment of the present application specifically includes the following steps:
It should be noted that, in this embodiment of the present application, before managing multiple service system data sources, a connection needs to be established with multiple service system data sources to be managed first, so that it can be determined that service data in the multiple service system data sources can be read.
Further, after determining a plurality of service system data sources, determining a data conversion rule corresponding to each source data table in each service system data source based on a data dictionary table pre-stored in each service system data source and a model field set in the data dictionary table.
It should be noted that the source data table is used to store service data, the data dictionary table includes data types stored in each source data table in the data source of the corresponding service system, and the model field is used to describe data representation forms of the corresponding data types.
It can be understood that the data conversion rule corresponding to the source data table is not only one rule, each data type in the corresponding source data table corresponds to one set of rule, and each rule corresponds to a conversion rule of one data type in the source data table and a standard data type.
For example: the standard data type specifies that the sex expression is a male field and a female field, and the sex expression in the source data table is a 0 field and a1 field respectively expressing the sex of the male and the female. After determining that the difference expression in the source data table is the fields of 0 and 1 through analysis, establishing a data conversion rule corresponding to the data type in the source data table.
In one embodiment of the present application, management of multiple data sources is supported, including: JDBC type databases (DB2, Oracle, sqlserver, Mysql and Postgresql, etc.), big data type databases (Hbase, Hive, Hadoop, Kafka, etc.), DB type databases (Dameng, Shentong, etc.).
And 102, performing format conversion on the source data in each source data table according to a data conversion rule to generate initial standard data, and processing the initial standard data based on a preset data security management algorithm to generate standard data.
In an embodiment of the application, after determining a data conversion rule corresponding to each source data table in a plurality of service system data sources, firstly, format conversion is performed on source data in each source data table according to the data conversion rule to generate initial standard data, then, whether the initial standard data are sensitive data or not is determined according to a preset sensitive data judgment rule, and in the case that the initial standard data are sensitive data, desensitization processing is performed on the initial standard data through a desensitization algorithm; and encrypting the initial standard data based on the data reading authority and the data encryption algorithm corresponding to each service system data source.
It should be noted that, in order to ensure the security of data in the transmission and use processes, the embodiment of the present application supports data desensitization management (including data dynamic desensitization and data static desensitization), and may flexibly support a plurality of desensitization algorithms (including Hash, encryption, masking, emulation, transformation, replacement, and the like). The encryption and decryption of data are supported, various types of encryption algorithms are built in, public encryption algorithms DES, AES, RSA and the like are supported, digest algorithms such as MD5 and SHA1 are supported, national encryption algorithms approved by countries such as SM2 and SM3 are supported, and the encryption requirement of data developers on industrial data related to information such as national secrets and enterprise secrets is met.
And 103, extracting the standard data into a target data table corresponding to the source data table.
In one embodiment of the application, after the source data is converted into the standard data, data extraction logic of the target data table to the source data table is constructed; the data extraction logic comprises the data type of standard data to be extracted and the mapping relation between a source data table and a target data table; determining a data extraction mode corresponding to the standard data based on the table type of the source data table; wherein the table types include: the modifiable data table and the non-modifiable data table; the data extraction method comprises the following steps: extracting full data and incremental data; and under the condition that a timing data extraction task corresponding to the standard data is triggered, extracting the standard data into a target data table corresponding to the source data table based on a data extraction mode corresponding to the standard data.
It is understood that the modifiable data table is a data table in which there is a possibility of modification of data in the source data table, and the non-modifiable data table is a data table in which only data is added, but previous data is not modified. Performing full data extraction, namely deleting data in the previous data table by the target data table during each extraction, and then converting all the data in the source data table again and then extracting; incremental data extraction is to extract only the added data, and the data extracted before is not extracted any more. It will be appreciated that the full data extraction corresponds to a modifiable data table and the incremental data extraction corresponds to a non-modifiable data table.
In one embodiment of the application, after the standard data is extracted into the target data table corresponding to the source data table, a data quality detection rule corresponding to the target data table is constructed; under the condition of triggering a timing data detection task corresponding to a target data table, detecting standard data in the target data table based on a data quality detection rule, and generating a data quality detection report to determine whether the standard data in the target data table has a quality problem; the data quality detection report comprises judgment results of each standard data based on the data quality detection rule; in the case of quality problem of the standard data in the target data table, modifying the data extraction logic and/or the data conversion rule based on the data quality detection report, and then converting and extracting the data in the source data table again to ensure the quality of the standard data extracted into the target data table again.
And 104, determining metadata and main data information corresponding to the target data table, and issuing the target data table based on the metadata and the main data information corresponding to the target data table.
It can be understood that the data in the target data table can be formally used only after being published, and the data publication is based on the metadata and the main data information of the target data table. Therefore, in the embodiment of the present application, after the standard data is extracted into the target data table corresponding to the source data table, the metadata and the main data information corresponding to the target data table need to be determined. It can be understood that: metadata corresponding to the target data table is mainly information describing attributes of the target data table and is determined based on information such as an indication storage position and a modification log of the target data table; and the main data information corresponding to the target data table is determined by the data type of the standard data contained in the target data table.
Further, after target data table release is carried out on the basis of metadata and main data information, metadata and main data information corresponding to a demand data model to be generated are determined on the basis of preset service demands; and extracting the main data in the target data table based on the metadata and the main data information corresponding to the to-be-generated demand data model to construct the demand data model.
Further, after the demand data model is constructed, determining a storage address of the demand data model; and creating a data sharing service corresponding to the demand data model based on the storage address of the demand data model. Then, the data sharing service is released, and the data sharing service can be called in an API mode, so that the data model can be called by a user and used.
In an embodiment of the present application, creating a data sharing service corresponding to a demand data model includes: defining an access path, an access protocol, an access parameter, a return parameter, a required data model summary logic and the like of the data sharing service.
Further, after the data sharing service is issued and the data sharing service can be called in an API mode, under the condition that a call request corresponding to the data sharing service is received, the call request is analyzed to determine a request address and user permission contained in the call request; based on the user authority, determining displayable data of corresponding authority in the demand data model, and sending key verification information; and returning exposable data requiring corresponding rights in the data model under the condition of receiving a correct key based on the key verification information.
Based on the same inventive concept, the embodiment of the present application further provides a data management device for industrial big data, and the internal structure of the data management device is shown in fig. 2.
Fig. 2 is a schematic diagram of an internal structure of a data management device for industrial big data according to an embodiment of the present application. As shown in fig. 2, the apparatus includes: a processor 201; the memory 202 has executable instructions stored thereon, which when executed cause the processor 201 to perform a data management method for industrial big data as described above.
In an embodiment of the present application, the processor 201 is configured to determine a plurality of service system data sources, and analyze data dictionary tables and model fields in the plurality of service system data sources to determine a data conversion rule corresponding to each source data table in the plurality of service system data sources; the data dictionary table comprises data types stored in the source data tables in the data source of the corresponding service system, and the model field is used for describing data representation forms of the corresponding data types; performing format conversion on source data in each source data table according to a data conversion rule to generate initial standard data, and processing the initial standard data based on a preset data security management algorithm to generate standard data; extracting standard data into a target data table corresponding to the source data table; determining metadata and main data information corresponding to the target data table, and issuing the target data table based on the metadata and main data information corresponding to the target data table; and the main data information corresponding to the target data table is used for describing the data type of the standard data contained in the target data table.
Some embodiments of the present application provide a non-transitory computer storage medium corresponding to one of fig. 1, storing computer-executable instructions configured to:
determining a plurality of service system data sources, and analyzing data dictionary tables and model fields in the service system data sources to determine data conversion rules corresponding to source data tables in the service system data sources; the data dictionary table comprises data types stored in the source data tables in the corresponding service system data sources, and the model field is used for describing data representation forms of the corresponding data types;
performing format conversion on source data in each source data table according to a data conversion rule to generate initial standard data, and processing the initial standard data based on a preset data security management algorithm to generate standard data;
extracting standard data into a target data table corresponding to the source data table;
determining metadata and main data information corresponding to the target data table, and issuing the target data table based on the metadata and main data information corresponding to the target data table; the main data information corresponding to the target data table is used for describing the data type of the standard data contained in the target data table.
The embodiments in the present application are described in a progressive manner, and the same and similar parts among the embodiments can be referred to each other, and each embodiment focuses on differences from other embodiments. Especially, for the internet of things device and medium embodiments, since they are substantially similar to the method embodiments, the description is simple, and the relevant points can be referred to the partial description of the method embodiments.
The system and the medium provided by the embodiment of the application correspond to the method one to one, so the system and the medium also have the beneficial technical effects similar to the corresponding method.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium.
Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in the process, method, article, or apparatus that comprises the element.
The above description is only an example of the present application and is not intended to limit the present application. Various modifications and changes may occur to those skilled in the art. Any modification, equivalent replacement, improvement or the like made within the spirit and principle of the present application shall be included in the scope of the claims of the present application.
Claims (10)
1. A data management method for industrial big data is characterized by comprising the following steps:
determining a plurality of service system data sources, and analyzing data dictionary tables and model fields in the service system data sources to determine data conversion rules corresponding to source data tables in the service system data sources; the source data table is used for storing service data, the data dictionary table comprises data types stored in the source data tables in the data source of the corresponding service system, and the model field is used for describing data representation forms of the corresponding data types;
performing format conversion on the source data in each source data table according to the data conversion rule to generate initial standard data, and processing the initial standard data based on a preset data security management algorithm to generate standard data;
extracting the standard data into a target data table corresponding to the source data table;
determining metadata and main data information corresponding to the target data table, and issuing the target data table based on the metadata and the main data information corresponding to the target data table; and the main data information corresponding to the target data table is used for describing the data type of the standard data contained in the target data table.
2. The data management method for the industrial big data according to claim 1, wherein the processing of the initial standard data based on a preset data security management algorithm specifically comprises:
determining whether the initial standard data is sensitive data according to a preset sensitive data judgment rule, and performing desensitization processing on the initial standard data through a desensitization algorithm under the condition that the initial standard data is determined to be sensitive data; and the number of the first and second groups,
and encrypting the initial standard data based on the data reading authority and the data encryption algorithm corresponding to each service system data source.
3. The method for managing data of industrial big data according to claim 1, wherein the extracting the standard data into the target data table corresponding to the source data table specifically includes:
constructing data extraction logic of the target data table to the source data table; the data extraction logic comprises a data type of standard data to be extracted and a mapping relation between the source data table and the target data table;
determining a data extraction mode corresponding to the standard data based on the table type of the source data table; wherein the table types include: the modifiable data table and the non-modifiable data table; the data extraction mode comprises the following steps: extracting full data and incremental data;
and under the condition that a timing data extraction task corresponding to the standard data is triggered, extracting the standard data into a target data table corresponding to the source data table based on a data extraction mode corresponding to the standard data.
4. The method for managing data of industrial big data according to claim 3, wherein after the standard data is extracted into the target data table corresponding to the source data table, the method further comprises:
constructing a data quality detection rule corresponding to the target data table;
under the condition that a timing data detection task corresponding to the target data table is triggered, detecting standard data in the target data table based on the data quality detection rule, and generating a data quality detection report to determine whether the standard data in the target data table has a quality problem; wherein the data quality detection report includes a judgment result of each standard data based on the data quality detection rule;
and in the case of quality problems of the standard data in the target data table, modifying the data extraction logic and/or the data conversion rule based on the data quality detection report to ensure the quality of the standard data re-extracted into the target data table.
5. The data management method for industrial big data according to claim 1, wherein after the target data table publication is performed based on the metadata and the main data information, the method further comprises:
determining metadata and main data information corresponding to a to-be-generated demand data model based on a preset business demand;
and extracting main data in the target data table based on the metadata and the main data information corresponding to the to-be-generated demand data model to construct a demand data model.
6. The data management method for the industrial big data as claimed in claim 5, wherein after the demand data model is built, the method further comprises:
determining a storage address of the demand data model;
creating a data sharing service corresponding to the demand data model based on the storage address of the demand data model, specifically including:
defining an access path, an access protocol, an access parameter, a return parameter and a required data model summarizing logic of the data sharing service;
and issuing the data sharing service, and setting that the data sharing service can be called in an API mode.
7. The data management method for the industrial big data, according to claim 6, after the data sharing service is released and the data sharing service is set to be able to be called by an API manner, the method further comprises:
receiving and analyzing a call request corresponding to the data sharing service to determine a request address and user permission contained in the call request;
based on the user authority, determining displayable data of corresponding authority in the demand data model, and sending key verification information;
and returning displayable data of corresponding authority in the demand data model under the condition that a correct key is received based on the key verification information.
8. The data management method for the industrial big data according to claim 1, wherein the types of the business system data sources comprise: JDBC type database, big data type database, DB type database.
9. A data management device for industrial big data, characterized in that the device comprises:
a processor;
and a memory having executable code stored thereon, which when executed, causes the processor to perform a method of data management for industrial big data according to any of claims 1-8.
10. A non-transitory computer storage medium for data management of industrial big data, storing computer-executable instructions, the computer-executable instructions configured to:
determining a plurality of service system data sources, and analyzing data dictionary tables and model fields in the service system data sources to determine data conversion rules corresponding to source data tables in the service system data sources; the source data table is used for storing service data, the data dictionary table comprises data types stored in the source data tables in the data source of the corresponding service system, and the model field is used for describing data representation forms of the corresponding data types;
performing format conversion on the source data in each source data table according to the data conversion rule to generate initial standard data, and processing the initial standard data based on a preset data security management algorithm to generate standard data;
extracting the standard data into a target data table corresponding to the source data table;
determining metadata and main data information corresponding to the target data table, and issuing the target data table based on the metadata and the main data information corresponding to the target data table; and the main data information corresponding to the target data table is used for describing the data type of the standard data contained in the target data table.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210735928.9A CN115129716A (en) | 2022-06-27 | 2022-06-27 | Data management method, equipment and storage medium for industrial big data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210735928.9A CN115129716A (en) | 2022-06-27 | 2022-06-27 | Data management method, equipment and storage medium for industrial big data |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115129716A true CN115129716A (en) | 2022-09-30 |
Family
ID=83379412
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210735928.9A Pending CN115129716A (en) | 2022-06-27 | 2022-06-27 | Data management method, equipment and storage medium for industrial big data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115129716A (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115935421A (en) * | 2022-12-23 | 2023-04-07 | 星环信息科技(上海)股份有限公司 | Data product publishing method, system and storage medium |
| CN116069981A (en) * | 2023-01-17 | 2023-05-05 | 深圳银兴智能数据有限公司 | Enterprise data storage method, enterprise data calling method and enterprise data calling system |
| CN116108021A (en) * | 2023-04-11 | 2023-05-12 | 北方健康医疗大数据科技有限公司 | Method, device and equipment for performing multi-dimensional data quality control based on standard data set |
| CN116226786A (en) * | 2023-03-22 | 2023-06-06 | 中国人民解放军军事科学院***工程研究院 | Data processing method and device for information system data fusion |
| CN116431863A (en) * | 2023-06-13 | 2023-07-14 | 北京大数据先进技术研究院 | Digital object generation method and device, electronic equipment and readable storage medium |
| CN116629805A (en) * | 2023-06-07 | 2023-08-22 | 浪潮智慧科技有限公司 | Water conservancy index service method, equipment and medium for distributed flow batch integration |
| CN117635371A (en) * | 2023-11-25 | 2024-03-01 | 中国交通信息科技集团有限公司 | Enterprise data quality management method and device, electronic equipment and storage medium |
-
2022
- 2022-06-27 CN CN202210735928.9A patent/CN115129716A/en active Pending
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115935421A (en) * | 2022-12-23 | 2023-04-07 | 星环信息科技(上海)股份有限公司 | Data product publishing method, system and storage medium |
| CN115935421B (en) * | 2022-12-23 | 2024-01-30 | 星环信息科技(上海)股份有限公司 | Data product release method, system and storage medium |
| CN116069981A (en) * | 2023-01-17 | 2023-05-05 | 深圳银兴智能数据有限公司 | Enterprise data storage method, enterprise data calling method and enterprise data calling system |
| CN116226786A (en) * | 2023-03-22 | 2023-06-06 | 中国人民解放军军事科学院***工程研究院 | Data processing method and device for information system data fusion |
| CN116226786B (en) * | 2023-03-22 | 2023-08-22 | 中国人民解放军军事科学院***工程研究院 | Data processing method and device for information system data fusion |
| CN116108021A (en) * | 2023-04-11 | 2023-05-12 | 北方健康医疗大数据科技有限公司 | Method, device and equipment for performing multi-dimensional data quality control based on standard data set |
| CN116108021B (en) * | 2023-04-11 | 2023-09-08 | 北方健康医疗大数据科技有限公司 | Method, device and equipment for performing multi-dimensional data quality control based on standard data set |
| CN116629805A (en) * | 2023-06-07 | 2023-08-22 | 浪潮智慧科技有限公司 | Water conservancy index service method, equipment and medium for distributed flow batch integration |
| CN116629805B (en) * | 2023-06-07 | 2023-12-01 | 浪潮智慧科技有限公司 | Water conservancy index service method, equipment and medium for distributed flow batch integration |
| CN116431863A (en) * | 2023-06-13 | 2023-07-14 | 北京大数据先进技术研究院 | Digital object generation method and device, electronic equipment and readable storage medium |
| CN116431863B (en) * | 2023-06-13 | 2023-09-12 | 北京大数据先进技术研究院 | Digital object generation method and device, electronic equipment and readable storage medium |
| CN117635371A (en) * | 2023-11-25 | 2024-03-01 | 中国交通信息科技集团有限公司 | Enterprise data quality management method and device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN115129716A (en) | Data management method, equipment and storage medium for industrial big data | |
| WO2018201895A1 (en) | Interface code generation method, apparatus, terminal device and medium | |
| US20190251292A1 (en) | Runtime control of automation accuracy using adjustable thresholds | |
| CN111782668B (en) | Data structure reading and updating method and device and electronic equipment | |
| EP3814929B1 (en) | Blockchain-based content management method, apparatus, and electronic device | |
| US11544758B2 (en) | Distributed database structures for anonymous information exchange | |
| CN113079200A (en) | Data processing method, device and system | |
| CN108958959B (en) | Method and device for detecting hive data table | |
| US11153071B2 (en) | Citation and attribution management methods and systems | |
| CN110032568B (en) | Data structure reading and updating method and device, and electronic equipment | |
| CN111885153B (en) | Block chain-based data acquisition method, device, computer equipment and storage medium | |
| CN112000992B (en) | Data leakage prevention protection method and device, computer readable medium and electronic equipment | |
| CN114091099A (en) | Authority hierarchical control method, equipment and storage medium for business system | |
| CN115758459A (en) | Data authority management method and device | |
| CN111159192B (en) | Big data based data warehousing method and device, storage medium and processor | |
| CN115730012A (en) | Database desensitization method and system | |
| CN111427871B (en) | Data processing method, device and equipment | |
| CN115827589A (en) | Authority verification method and device, electronic equipment and storage medium | |
| CN113673844A (en) | Information feedback method, device and equipment | |
| CN110647766A (en) | Method and system for ensuring file downloading safety of data warehouse | |
| CN113687973B (en) | Control method, equipment and medium for log dynamic output | |
| CN112818367B (en) | File encryption method, device, storage medium and processor | |
| CN110912851B (en) | Method, device and equipment for monitoring flow data | |
| CN116662373A (en) | Data access control method, device, equipment and medium | |
| CN116186726A (en) | Database operation processing method, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |