WO2021051612A1 - 数据授权脱敏自动化方法、***、装置及存储介质 - Google Patents

数据授权脱敏自动化方法、***、装置及存储介质 Download PDF

Info

Publication number
WO2021051612A1
WO2021051612A1 PCT/CN2019/118433 CN2019118433W WO2021051612A1 WO 2021051612 A1 WO2021051612 A1 WO 2021051612A1 CN 2019118433 W CN2019118433 W CN 2019118433W WO 2021051612 A1 WO2021051612 A1 WO 2021051612A1
Authority
WO
WIPO (PCT)
Prior art keywords
desensitization
data
source
fields
target
Prior art date
Application number
PCT/CN2019/118433
Other languages
English (en)
French (fr)
Inventor
王兴川
Original Assignee
平安科技（深圳）有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 平安科技（深圳）有限公司 filed Critical 平安科技（深圳）有限公司
Publication of WO2021051612A1 publication Critical patent/WO2021051612A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F21/60—Protecting data
    • G06F21/62—Protecting access to data via a platform, e.g. using keys or access control rules
    • G06F21/6218—Protecting access to data via a platform, e.g. using keys or access control rules to a system of files or objects, e.g. local or distributed file system or database
    • G06F21/6245—Protecting personal data, e.g. for financial or medical purposes
    • G06F21/6254—Protecting personal data, e.g. for financial or medical purposes by anonymising data, e.g. decorrelating personal data from the owner's identification
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F21/60—Protecting data
    • G06F21/62—Protecting access to data via a platform, e.g. using keys or access control rules
    • G06F21/6218—Protecting access to data via a platform, e.g. using keys or access control rules to a system of files or objects, e.g. local or distributed file system or database
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Definitions

  • This application relates to the field of data processing, and in particular to an automated method, system, device and storage medium for data authorization and desensitization.
  • the existing data desensitization method is only to use esProc to write SPL scripts for data report query, and to identify sensitive information fields (such as: name, certificate number, bank account, address, telephone number, business name, industrial and commercial registration number, and taxpayer identification). No.), etc., perform data desensitization and deformation through predefined desensitization rules to realize the protection of sensitive privacy data.
  • sensitive information fields such as: name, certificate number, bank account, address, telephone number, business name, industrial and commercial registration number, and taxpayer identification. No.), etc., perform data desensitization and deformation through predefined desensitization rules to realize the protection of sensitive privacy data.
  • the applicant realizes that in the existing data desensitization method, the data is not authorized first, and then the data is automatically desensitized.
  • This application provides an automated method, electronic device, and computer-readable storage medium for data authorization and desensitization. Its main purpose is to determine table access permissions and data access scopes by creating desensitization requirements, and then use DMP according to the access permissions and data access scopes. Initiate an application for permission to access the business table. DPM is used to approve the application. DMP performs desensitization configuration on the approved permission application, compulsory configuration of sensitive fields and specifies desensitization rules to form a desensitization task, and then execute the desensitization task To ensure the smooth completion of desensitization tasks, improve desensitization efficiency, and reduce storage space.
  • the data authorization desensitization automated method provided in this application is applied to an electronic device, and the method includes:
  • S110 Create a desensitization requirement according to a preset rule, and determine table access authority and data access scope according to the desensitization requirement;
  • S120 Initiate an application for permission to access the service table in the DMP according to the table access permission and data access scope, and the DMP is used to approve the permission application;
  • S130 Perform desensitization configuration on the approved permission application through the DMP, where the desensitization configuration includes mandatory configuration designated desensitization rules for sensitive fields;
  • S140 Synchronize the specified desensitization rule and the data access range to ETL, the ETL generates a target field according to the data access range, and automatically generates a desensitization task for the desensitization field according to the desensitization rule ;
  • S150 Check whether the desensitization source and target fields are the same. If they are the same, continue to keep the desensitization source and target fields and sequence consistent; if they are inconsistent, make the desensitization source and target fields consistent. After the check is completed, execute ETL and complete the description Desensitization task.
  • the desensitization requirement is a desensitization requirement created according to a preset rule, and the desensitization requirement is restricted by a data provider;
  • the specific desensitization field of the desensitization requirement includes business sensitive fields;
  • the data access range includes at least a time range and a span range.
  • the DMP initiates an application for permission to access the business table according to the table access permission and data access scope, and the step of using the DMP to approve the permission application includes:
  • the data user initiates an application for permission to access the business table
  • the criteria for determining that the desensitization requirements of the data user meet the data use requirements include at least not damaging the desensitization source and not containing out-of-range instructions.
  • the corresponding desensitization method in the desensitization rule includes direct desensitization and associated desensitization; wherein,
  • the direct desensitization is direct desensitization by applying a preset rule to the desensitization source, and has nothing to do with other fields;
  • the association desensitization is to apply the same desensitization rule to the associated fields between desensitization sources, so that the desensitized fields can still be associated.
  • the preset rule includes at least one of the following rules:
  • the amount category randomly floats 1% to 5%, and the address category intercepts the first 15 characters or displays only up to the level of the district.
  • the process of automatically generating the desensitization task of the target field according to the desensitization rule includes:
  • the desensitization table is established according to the fields of the desensitization source; if a view and a materialized view are created, a new view is generated according to the desensitization rule and the previous view is replaced.
  • the process of determining whether to establish a desensitization table or a view according to the desensitization rule includes:
  • the view is created; if it is in the interval, the materialized view is created; if it is greater than the interval, the desensitization table is created.
  • the method further includes:
  • the desensitization source has a new field relative to the target field, add the same field based on the target field and replace the previous desensitization table, materialized view or view.
  • the desensitization source has a deleted field relative to the target field, then The same field is deleted on the basis of the target field, and the previous desensitized table, materialized view or view is replaced.
  • the present application also provides an automatic electronic device for data authorization and desensitization.
  • the electronic device includes a memory, a processor, and a computer program stored in the memory and running on the processor.
  • the processor implements the steps in the automatic monitoring method for data authorization desensitization according to any one of claims 1-8 when the computer program is executed by the processor;
  • the present application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a data authorization desensitization automated analysis program, and the data authorization desensitization automation program is When the processor is executed, the steps of the aforementioned automatic method for data authorization and desensitization are realized.
  • the data authorization and desensitization automation method, electronic device, and computer readable storage medium proposed in this application in which the table access authority and data access scope are determined according to the desensitization requirement, and then the data user is in DMP according to the table access authority and data access scope Initiate an application for permission to access the business table, and DMP will approve the permission application. After the approval is passed, the desensitization configuration is performed in DMP, and the designated desensitization rules are mandatory for sensitive fields, and then the designated desensitization rules and data access scope are synchronized To ETL, ETL automatically generates corresponding desensitization tasks according to corresponding information, generates target fields, and checks whether the desensitization source and target fields are consistent.
  • FIG. 1 is a schematic diagram of an application environment of an automated method for data authorization and desensitization according to an embodiment of the present application
  • Fig. 2 is a flowchart of an automated method for data authorization and desensitization according to an embodiment of the present application
  • Fig. 3 is a system framework diagram in an automated electronic device for data authorization and desensitization according to an embodiment of the present application
  • Fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
  • the DMP (Data Management Platform) data management platform integrates scattered multi-party data into a unified technology platform, and standardizes and subdivides these data, so that users can push these subdivided results to the existing interactive marketing environment In the platform.
  • Extract-Transform-Load the abbreviation of Extract-Transform-Load in English, is used to describe the process of extracting, transforming, and loading data from the source to the destination.
  • an automated method for data authorization and desensitization is provided, which is applied to the electronic device 40.
  • Fig. 1 is a schematic diagram of an application environment of an automated method for data authorization and desensitization according to an embodiment of the present application. As shown in FIG. 1, in the implementation environment of this embodiment, it is a computer device 110.
  • the computer device 110 is a computer device, such as a terminal device such as a computer.
  • the computer terminal device 110 may be a tablet computer, a notebook computer, a desktop computer, etc., which is a cenOS (linux) system, but is not limited to this.
  • the terminal device 110 such as a computer device can be connected via Bluetooth, USB (Universal Serial Bus) or other communication connection methods, which is not limited in this application.
  • Fig. 2 is a flowchart of an automated method for data authorization and desensitization according to an embodiment of the present application. As shown in Figure 2, in this embodiment, the automatic data authorization desensitization method includes the following steps:
  • S110 Create desensitization requirements according to preset rules, and determine table access permissions and data access scopes according to the desensitization requirements;
  • the created desensitization requirement refers to the desensitization requirement created by the data provider according to preset rules, which is restricted by the data provider;
  • the specific desensitization fields included in the desensitization requirement are business sensitive fields, such as amount, sales, Telephone, address, ID number, etc.;
  • the data access range generally includes time range, span range, etc., such as only allowing access to the records of the most recent month, only allowing access to the records of a certain department of the data user, or all records of the data user.
  • S120 Initiate an application for permission to access the business table in DMP according to the access permission and data access scope of the table, and the DMP is used to approve the permission application;
  • the data user is the person to be approved
  • the data provider is the approver
  • the data provider determines that the desensitization requirement meets the data use requirements, it will pass the approval
  • the criteria for judging that the desensitization requirement meets the data use requirements are: no damage to the desensitization source, no over-range instructions (for example, there is no "phone” column in the desensitization source, and the desensitization requirement is selected as "phone"), etc., here
  • the desensitization source is the data that is initially screened by the data user and waiting to be desensitized.
  • S130 Perform desensitization configuration on approved permission applications through DMP, and compulsorily configure designated desensitization rules for sensitive fields;
  • the desensitization configuration is performed in the DMP.
  • the specified desensitization rules are mandatory to ensure that the data can be used correctly. Undermine the ambiguity of data and maintain the business integrity of the data.
  • users can choose between direct desensitization or associated desensitization; different specialized business sensitive rules may be inconsistent, and the system provides a basis It is not necessary to formulate different and unified desensitization rules in order to achieve "individualized" customized desensitization rules.
  • Direct desensitization means applying preset rules to desensitize the fields of a certain table, and has nothing to do with the fields of other tables.
  • the preset rule includes at least one of the following rules:
  • the amount category randomly floats 1% to 5%, and the address category intercepts the first 15 characters or displays only up to the level of the district.
  • Association desensitization is to apply the same desensitization rule to the associated fields between desensitization sources, so that the desensitized fields can still be associated, that is, the associated fields between tables apply the same desensitization rule to achieve desensitization.
  • the sensitive fields can still be associated without destroying the consistency of the data.
  • sensitive fields are fields that must be desensitized identified by the data provider.
  • DMP must choose to specify one of several desensitization rules before providing it, otherwise it cannot be submitted.
  • Such fields are directly desensitized.
  • S140 Synchronize the specified desensitization rules and data access scope to ETL, and ETL generates target fields according to the data access scope, and automatically generates desensitization tasks for the target fields according to the desensitization rules;
  • the process of the ETL automatically generating the corresponding desensitization task according to the corresponding information includes:
  • the corresponding rules and data access scope are synchronized to ETL, and ETL automatically generates corresponding desensitization tasks according to the received desensitization rules and data access scope, including automatically creating target fields , That is, a desensitized table, materialized view, or view to prepare for desensitizing data.
  • the process of determining whether to establish a desensitized table, a materialized view or a view according to the desensitization rule includes:
  • the view is created, if it is in the interval, the materialized view is created, and if it is greater than the interval, the desensitization table is created. For example, when the original Hive table occupies less than 100G of HDFS space, the view is created, and if it is greater than 100G and less than 500G, then a materialized view is created. If a single table is greater than 500G, a desensitized table is created, and the desensitized data is directly inserted into the desensitized table. , The specific rules are based on business needs.
  • S150 Check whether the desensitization source and target fields are the same. If they are the same, continue to keep the desensitization source and target fields and sequence consistent; if they are inconsistent, make the desensitization source and target fields consistent, execute ETL after checking and complete desensitization Task; the desensitization source is the data that is initially filtered by the data consumer and waiting to be desensitized.
  • the desensitization source has a new field relative to the target field, add the same field based on the target field and replace the previous desensitization table, materialized view or view. If the desensitization source has a deleted field relative to the target field, then Delete the same field on the basis of the target field, and replace the previous desensitized table, materialized view, or view.
  • the data after applying the desensitization rule is also stored in Hive;
  • the desensitization source is the data that is initially filtered by the data user and is waiting to be desensitized
  • the target field is the accurate desensitization data that is immediately desensitized.
  • the data is stored in Hive in the form of a table.
  • the target field table is a Hive table, which cannot be processed like a relational database. Therefore, the order of the fields is particularly important. Otherwise, the desensitized fields may not be desensitized, and the fields that should not be desensitized are desensitized. Make sure that the desensitization source table and the target field table have the same meaning in a specific order.
  • the field that uses the desensitization rule is the field that the user really needs to desensitize, and it needs to be judged according to the order of the fields. Therefore, before the desensitization task runs, it is necessary to keep the desensitization source table and the target field table and sequence consistent. If they are inconsistent, the task will fail or the data will be misplaced. Therefore, check whether the desensitization source table and the target field table are Before running the desensitization task, the desensitization can be carried out smoothly after the two sides are consistent. Otherwise, the task will fail and human intervention will be required, which will increase the workload of task operation and maintenance.
  • the desensitization task is to create a view, if there is a field change, that is, add or delete, it will be judged according to the order of the fields. If the desensitization source adds a new field at the end, it can be added on the original basis. Field, replace the previous view, otherwise you need to delete and create a new hive table. After completion, perform subsequent ETL processes such as view replacement or materialized view rebuild and desensitization table data reloading.
  • the automatic data authorization desensitization method in this embodiment judges the table access authority and data access scope according to the desensitization requirement, and then the data user initiates an application for permission to access the business table in the DMP according to the table access authority and data access scope, and the DMP applies for the authority
  • the desensitization configuration is performed in the DMP, and the designated desensitization rules are mandatory for sensitive fields, and then the designated desensitization rules and data access scope are synchronized to the ETL, and the ETL automatically generates the corresponding information according to the corresponding information.
  • For the desensitization task generate the target field, check whether the desensitization source and target fields are consistent, if they are the same, continue to keep the fields and sequence of the desensitization source table and the target table consistent, if they are inconsistent, make the source and target fields consistent, check After completion, the ETL is executed and the desensitization task is completed, which greatly ensures the automatic synchronization of changes to the field, reduces human participation, saves manpower, improves the desensitization efficiency, and reduces storage space.
  • FIG. 3 is a framework diagram of a data authorization and desensitization automation system according to an embodiment of the application.
  • the system corresponds to the data authorization and desensitization automation method and can be set in the data Authorized desensitization in automated electronic devices.
  • the data authorization desensitization automation system includes a DMP module 310, an ETL module 320, an original library module 330, and a desensitization library module 340.
  • the DMP module 310 is used to accept the permission application for accessing the business table initiated by the data user, and the permission application includes the table access permission and the data access scope; then the DMP module 310 uses the data according to the data user’s access permission and access scope.
  • the authority application of the party is approved. If approved, the DMP module 310 starts to perform desensitization configuration, that is, compulsorily configure the designated desensitization rule for the desensitization field, and synchronize the desensitization rule to the ETL module 320.
  • the table access authority and data access scope are determined according to the desensitization requirements, which are created by the data user according to preset rules.
  • the specific desensitization fields included are some business sensitive fields, such as amount, sales, phone number, and address. , ID number, etc.
  • the scope of data access generally includes time range, span range, etc., such as only allowing access to the records of the most recent month, only allowing access to the records of a certain department of the data user, or all the records of the data user.
  • the data user applies for desensitization and desensitization rules in the DMP module 310, the approval of the data provider is required; specifically, the data user is the person to be approved , The data provider is the approver, and the data provider determines that the desensitization requirement meets the data use requirements, it will pass the approval; whether it is passed is determined by the data provider, and the data user approved by the DMP module 310 can Apply for desensitization; the criteria for determining that the desensitization requirement meets the data use requirements are: no damage to the desensitization source and no over-range instructions, such as: there is no "phone" column in the desensitization source, and the desensitization requirement is selected as "phone”.
  • the desensitization source here is the data that is initially screened by the data user and waiting to be desensitized.
  • the ETL module 320 generates a target field according to the data access range, automatically generates a desensitization task for the target field according to a desensitization rule, and executes the desensitization task; and, before performing the desensitization task, the ETL module needs to check the source of the desensitization Whether it is consistent with the target field, if they are consistent, continue to keep the desensitization source and target fields and sequence consistent, if they are inconsistent, make the desensitization source and target fields consistent, and perform the desensitization task after the inspection is completed.
  • the ETL module 320 automatically generates and executes desensitization tasks according to the desensitization rules synchronized by the DMP module 310; wherein, the process of the ETL module 320 automatically generates corresponding desensitization tasks according to corresponding information includes:
  • the desensitization methods corresponding to the desensitization rules include direct desensitization and associated desensitization; among them, direct desensitization refers to direct desensitization by applying preset rules to the desensitization source, and has nothing to do with other fields; and associated desensitization refers to desensitization.
  • the associated fields between sensitive sources apply the same desensitization rule, so that the desensitized fields can still be associated.
  • the preset rules include at least one of the following rules:
  • the amount category randomly floats 1% to 5%, and the address category intercepts the first 15 characters or displays only up to the level of the district.
  • the ETL module presets the interval of the HDFS space storage occupied by the desensitization source according to the desensitization rule; if the HDFS space storage occupied by the desensitization source is less than the interval, Create a view; if it is within the interval, create a materialized view; if it is greater than the interval, create a desensitization table.
  • the ETL module checks whether the desensitization source and target fields are consistent, if the desensitization source and target fields are inconsistent, it will determine whether to add or delete fields according to the order of the target fields; if the desensitization source is newly added relative to the target field Field, the same field is added on the basis of the target field, and the previous desensitization table, materialized view, or view is replaced. If the desensitization source has a deleted field relative to the target field, it will be deleted on the basis of the target field The same field, and replace the previous desensitized table, materialized view or view.
  • the original library module 330 is used to store desensitization source data
  • the desensitization library module 340 is used to store data such as desensitization tasks and target fields, combined with the DMP module 310 and the ETL module 320 to complete the desensitization tasks.
  • FIG. 4 is a schematic diagram of the electronic device of this application.
  • the electronic device 40 may be a terminal device with arithmetic functions such as a server, a tablet computer, a portable computer, a desktop computer, and the like.
  • the electronic device 40 includes a processor 41, a memory 42, a computer program 43, a network interface, and a communication bus.
  • the electronic device 40 may be a tablet computer, a desktop computer, or a smart phone, but is not limited thereto.
  • the memory 42 includes at least one type of readable storage medium.
  • the at least one type of readable storage medium may be a non-volatile storage medium such as flash memory, hard disk, multimedia card, card-type memory, and the like.
  • the readable storage medium may be an internal storage unit of the electronic device 40, such as a hard disk of the electronic device 40.
  • the readable storage medium may also be an external memory of the electronic device 40, such as a plug-in hard disk equipped on the electronic device 40, a smart memory card (Smart Media Card, SMC), and a secure digital (Secure Digital, SD) card, flash card (Flash Card), etc.
  • the readable storage medium of the memory 42 is generally used to store the computer program 43 installed in the electronic device 40, the key generation unit 310, the key management unit 320, the transmission unit 330, the alarm unit 340, and so on.
  • the processor 41 may be a central processing unit (CPU), microprocessor or other data processing chip in some embodiments, and is used to run the program code stored in the memory 42 or process data, such as data authorization desensitization Automation program 43 and so on.
  • CPU central processing unit
  • microprocessor or other data processing chip in some embodiments, and is used to run the program code stored in the memory 42 or process data, such as data authorization desensitization Automation program 43 and so on.
  • the network interface may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface), and is generally used to establish a communication connection between the electronic device 40 and other electronic devices.
  • a standard wired interface such as a WI-FI interface
  • WI-FI interface wireless interface
  • the communication bus is used to realize the connection and communication between these components.
  • FIG. 4 only shows the electronic device 40 with the components 41-43, but it should be understood that it is not required to implement all the illustrated components, and more or fewer components may be implemented instead.
  • the memory 42 as a computer storage medium can store an operating system and a data authorization desensitization automation program 43; the processor 41 executes the data authorization desensitization automation program stored in the memory 42 At 43 o'clock, the following steps are implemented:
  • S110 Create a desensitization requirement according to a preset rule, and determine table access authority and data access scope according to the desensitization requirement;
  • S120 Initiate an application for permission to access the service table in the DMP according to the table access permission and data access scope, and the DMP is used to approve the permission application;
  • S130 Perform desensitization configuration on the approved permission application through the DMP, where the desensitization configuration includes mandatory configuration designated desensitization rules for sensitive fields;
  • S140 Synchronize the specified desensitization rule and the data access range to ETL, the ETL generates a target field according to the data access range, and automatically generates a desensitization task for the target field according to the desensitization rule;
  • S150 Check whether the desensitization source and target fields are the same. If they are the same, continue to keep the desensitization source and target fields and sequence consistent; if they are inconsistent, make the desensitization source and target fields consistent. After the check is completed, execute ETL and complete the description Desensitization task.
  • the embodiment of the present application also proposes a computer-readable storage medium.
  • the computer-readable storage medium includes a data authorization and desensitization automation program, and when the data authorization and desensitization automation program is executed by a processor, the following operations are implemented:
  • S110 Create a desensitization requirement according to a preset rule, and determine table access authority and data access scope according to the desensitization requirement;
  • S120 Initiate an application for permission to access the service table in the DMP according to the table access permission and data access scope, and the DMP is used to approve the permission application;
  • S130 Perform desensitization configuration on the approved permission application through the DMP, where the desensitization configuration includes mandatory configuration designated desensitization rules for sensitive fields;
  • S140 Synchronize the specified desensitization rule and the data access range to ETL, the ETL generates a target field according to the data access range, and automatically generates a desensitization task for the target field according to the desensitization rule;
  • S150 Check whether the desensitization source and target fields are the same. If they are the same, continue to keep the desensitization source and target fields and sequence consistent; if they are inconsistent, make the desensitization source and target fields consistent. After the check is completed, execute ETL and complete the description Desensitization task.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Bioethics (AREA)
  • General Health & Medical Sciences (AREA)
  • Computer Hardware Design (AREA)
  • Databases & Information Systems (AREA)
  • Computer Security & Cryptography (AREA)
  • Software Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Medical Informatics (AREA)
  • Storage Device Security (AREA)

Abstract

一种数据授权脱敏自动化方法，该方法包括：根据预设规则创建脱敏需求，根据脱敏需求确定表访问权限和数据访问范围（S110）；根据表访问权限和数据访问范围在DMP发起访问业务表的权限申请，DMP用于对权限申请进行审批（S120）；通过DMP对通过审批的权限申请进行脱敏配置，脱敏配置包括针对敏感字段强制性配置指定的脱敏规则（S130）；将指定的脱敏规则及数据访问范围同步到ETL，ETL根据数据访问范围产生目标字段，根据脱敏规则自动生成所述目标字段的脱敏任务（S140）；检查脱敏源和目标字段是否一致，若一致，则继续保持脱敏源和目标字段及顺序一致，若不一致，则使脱敏源和目标字段一致，检查完成后执行ETL并完成所述脱敏任务（S150）。

Description

数据授权脱敏自动化方法、***、装置及存储介质
本申请要求申请号为201910885652.0，申请日为2019年9月19日，发明创造名称为“数据授权脱敏自动化方法、装置及存储介质”的专利申请的优先权。
技术领域
本申请涉及数据处理领域，尤其涉及一种数据授权脱敏自动化方法、***、装置及存储介质。
背景技术
随着大数据时代的到来,大数据商业价值的挖掘,用户的精准定位,大数据中蕴藏的巨大商业价值被逐步挖掘出来,但是同时也带来了巨大的挑战--个人隐私信息的保护。个人信息与个人行为(比如位置信息、消费行为、网络访问行为)等，这些都是人的隐私，也是我们所关注的一类敏感信息，在大数据价值挖掘的基础上如何保护人的隐私信息，也将是数据脱敏必须解决的难题。
现有的数据脱敏方法只是将数据报表查询利用集算器编写SPL脚本，对敏感信息字段(如:姓名、证件号、银行账户、住址、电话号码、企业名称、工商注册号、纳税人识别号)等通过预定义的脱敏规则进行数据脱敏、变形，实现敏感隐私数据的保护。但是，申请人意识到，在现有的数据脱敏方法中，并不是先将数据授权，然后再自动化的对数据进行脱敏，存在极大地安全隐患，而且一旦新增或者删除字段，脱敏任务就不能顺序进行，需要人为参与，并且数据脱敏前后的存储相当于是存储了2份数据，浪费了存储。
因此，亟需一种节省存储空间，且能提升脱敏效率的自动脱敏方法。
发明内容
本申请提供一种数据授权脱敏自动化方法、电子装置及计算机可读存储介质，其主要目的在于通过创建脱敏需求，判断表访问权限和数据访问范围，然后根据访问权限和数据访问范围在DMP发起访问业务表的权限申请，DPM用于对该申请进行审批，DMP对通过审批的权限申请进行脱敏配置，针对敏 感字段强制性配置指定脱敏规则，形成脱敏任务，进而执行脱敏任务以确保顺利完成脱敏任务，且提高脱敏效率，减少存储空间。
为实现上述目的，本申请提供的数据授权脱敏自动化方法，应用于电子装置，所述方法包括：
S110：根据预设规则创建脱敏需求，根据所述脱敏需求确定表访问权限和数据访问范围；
S120：根据所述表访问权限和数据访问范围在DMP发起访问业务表的权限申请，所述DMP用于对所述权限申请进行审批；
S130：通过所述DMP对通过审批的权限申请进行脱敏配置，所述脱敏配置包括针对敏感字段强制性配置指定的脱敏规则；
S140：将所述指定的脱敏规则及所述数据访问范围同步到ETL，所述ETL根据所述数据访问范围产生目标字段，根据所述脱敏规则自动生成所述脱敏字段的脱敏任务；
S150：检查脱敏源和目标字段是否一致，若一致，则继续保持脱敏源和目标字段及顺序一致，若不一致，则使脱敏源和目标字段一致，检查完成后执行ETL并完成所述脱敏任务。
优选地，所述脱敏需求为根据预设规则创建的脱敏需求，所述脱敏需求受数据提供方的限制；所述脱敏需求的具体脱敏字段包括业务敏感字段；
所述数据访问范围至少包括时间范围、跨度范围。
优选地，在根据所述表访问权限和数据访问范围在DMP发起访问业务表的权限申请，所述DMP用于所述权限申请进行审批的步骤包括：
由数据使用方发起访问业务表的权限申请；
判定所述数据使用方的脱敏需求符合数据使用要求的标准至少包括不破坏脱敏源、不含有超范围指令。
优选地，所述脱敏规则时对应的脱敏方式包括直接脱敏和关联脱敏；其中，
所述直接脱敏为对脱敏源应用预设规则进行直接脱敏，与其他字段并无关系；
所述关联脱敏为针对脱敏源之间的关联字段应用同一个脱敏规则，以使脱敏后的字段仍能相关联。
优选地，所述预设规则，包括以下规则中的至少一个：
对任意字符串提供HASH加密名称统一改为“数据使用方的名称”；
电话后4位改为1234或屏蔽中间4位；
邮箱全部屏蔽为所述数据使用方的预设邮箱；
金额类随机浮动1％～5％地址类截取前15个字符或只显示到区的级别。
优选地，根据所述脱敏规则自动生成所述目标字段的脱敏任务的过程包括：
根据所述脱敏规则确定要创建脱敏表、视图还是物化视图；
若创建脱敏表，则根据所述脱敏源的字段建立脱敏表；若创建视图和物化视图，则根据所述脱敏规则生成新的视图，并替换之前的视图。
优选地，在根据所述脱敏规则判断建立脱敏表还是视图的过程中，包括：
根据所述脱敏规则预先设定所述脱敏源占用HDFS空间存储量的区间；
若所述脱敏源占用HDFS的空间存储量小于所述区间，则创建视图；若在所述区间内，则创建物化视图；若大于所述区间，则创建脱敏表。
优选地，在所述检查脱敏源和目标字段是否一致的步骤之后，所述方法还包括：
若脱敏源和目标字段不一致，则根据所述目标字段的顺序判断是新增字段还是删除字段；
若脱敏源相对目标字段有新增字段，则在所述目标字段基础上新增同样的字段，并替换之前的脱敏表、物化视图或视图，脱敏源相对目标字段有删除字段，则在所述目标字段基础上删减同样的字段，并替换之前的脱敏表、物化视图或视图。
为实现上述目的，本申请还提供一种数据授权脱敏自动化电子装置，该电子装置包括：存储器、处理器以及存储在所述存储器中并可在所述处理器上运行的计算机程序，所述处理器执行所述计算机程序时实现权利要求1-8中任一项所述的数据授权脱敏自动化监控方法中的步骤；
此外，为实现上述目的，本申请还提供一种计算机可读存储介质，其特征在于，所述计算机可读存储介质中存储有数据授权脱敏自动化分析程序，所述数据授权脱敏自动化程序被处理器执行时，实现前述的数据授权脱敏自动化方法的步骤。
本申请提出的数据授权脱敏自动化方法、电子装置及计算机可读存储介 质，其中根据脱敏需求判断表访问权限和数据访问范围，然后数据使用方根据所述表访问权限和数据访问范围在DMP发起访问业务表的权限申请，DMP对权限申请进行审批，审批通过后，在DMP进行脱敏配置，针对敏感字段强制性配置指定的脱敏规则，进而将指定的脱敏规则及数据访问范围同步到ETL，ETL根据相应的信息自动的生成相应的脱敏任务，产生目标字段，检查脱敏源和目标字段是否一致，若一致，则继续保持脱敏源表和目标表字段及顺序一致，若不一致，则使源和目标的字段一致，检查完成后执行ETL并完成所述脱敏任务，极大地确保了自动对字段的变更进行同步更改，减少了人为参与，节省了人力，提高了脱敏效率，减少了存储空间。
附图说明
图1为根据本申请实施例的数据授权脱敏自动化方法应用环境示意图；
图2为根据本申请实施例的数据授权脱敏自动化方法的流程图；
图3为根据本申请实施例的数据授权脱敏自动化电子装置中的***框架图；
图4为根据本申请实施例的电子装置的结构示意图。
本申请目的的实现、功能特点及优点将结合实施例，参照附图做进一步说明。
具体实施方式
应当理解，此处所描述的具体实施例仅用以解释本申请，并不用于限定本申请。
为了解决前述现有的数据脱敏方法只是将数据报表查询利用集算器编写SPL脚本，对敏感信息字段通过预定义的脱敏规则进行数据脱敏、变形，实现敏感隐私数据的保护，然后再自动化的对数据进行脱敏，存在极大的安全隐患，而且一旦新增或者删除字段，脱敏任务就不能顺序进行，需要人为参与，浪费存储的问题，本申请从数据授权入手，首先根据脱敏需求判断表访问权限和数据访问范围，然后数据使用方根据表访问权限和数据访问范围在DMP发起访问业务表的权限申请，DMP对权限申请进行审批，通过审批后，DMP进行脱敏配置，针对敏感字段配置脱敏规则，形成脱敏任务，进而完成脱敏任务。
DMP(Data Management Platform)数据管理平台，是把分散的多方数据进行整合纳入统一的技术平台，并对这些数据进行标准化和细分，让用户可以把这些细分结果推向现有的互动营销环境里的平台。
ETL，是英文Extract-Transform-Load的缩写，用来描述将数据从来源端经过抽取(extract)、转换(transform)、加载(load)至目的端的过程。
具体的，根据本申请的一个实施例，提供一种数据授权脱敏自动化方法，应用于电子装置40。
图1为根据本申请实施例的数据授权脱敏自动化方法应用环境示意图。如图1所示，在本实施例的实施环境中为计算机设备110。
其中的计算机设备110为计算机设备，例如电脑等终端设备。
需要说明的是，计算机终端设备110可为平板电脑、笔记本电脑、台式计算机等，其为cenOS(linux)***，但并不局限于此。计算机设备等终端设备110可以通过蓝牙、USB(Universal Serial Bus，通用串行总线)或者其他通讯连接方式进行连接，本申请在此不做限制。
图2为根据本申请实施例的数据授权脱敏自动化方法的流程图。如图2所示，在本实施例中，数据授权脱敏自动化方法包括如下步骤：
S110：根据预设规则创建脱敏需求，根据脱敏需求确定表访问权限和数据访问范围；
该创建的脱敏需求是指数据提供方根据预设规则创建的脱敏需求，受数据提供方的限制；该脱敏需求包括的具体脱敏字段是一些业务敏感字段，比如金额、销售额、电话、地址、证件号等；
该数据访问范围一般包括时间范围、跨度范围等，比如只允许访问最近一个月的记录、只允许访问数据使用方某一部门的记录或者数据使用方所有的记录。
S120：根据该表访问权限和数据访问范围在DMP发起访问业务表的权限申请，DMP用于对该权限申请进行审批；
其中，数据提供方在DMP平台上能看到的所有专业公司数据，数据使用方在DMP平台申请脱敏及脱敏规则，需要数据提供方的审批；
具体的，该数据使用方为待审批者，该数据提供方为审批者，该数据提供方判定该脱敏需求符合数据使用要求，则会通过审批；
是否通过都由数据提供方决定，通过该DMP审批后的数据使用方都可以申请提交脱敏；
判定脱敏需求符合数据使用要求的标准为：不破坏脱敏源、不含有超范围指令(如：脱敏源中无“电话”栏，选择脱敏需求为“电话”)等，此处的脱敏源为数据使用方初始筛选出的等待被脱敏的数据。
S130：通过DMP对通过审批的权限申请进行脱敏配置，针对敏感字段强制性配置指定的脱敏规则；
在审批通过后，在DMP进行脱敏配置，针对“特殊”敏感的字段(如身份证号、电话、金额等)，强制性配置指定的脱敏规则，确保数据能被正确的使用，为不破坏数据的二义性，保持数据的业务完整性，在脱敏规则对应的脱敏方式中，可供用户选择有直接脱敏还是关联脱敏；不同专业务敏规则可能不一致，***提供了根据不能需求制定不同而又统一的脱敏规则，以达到“个性化”定制脱敏规则。
直接脱敏的意思是对某个表的字段应用预设规则进行脱敏，与其他表的字段没有任何关系。该预设规则，包括以下规则中的至少一个：
对任意字符串提供HASH加密名称统一改为“数据使用方的名称”；
电话后4位改为1234或屏蔽中间4位；
邮箱全部屏蔽为所述数据使用方的预设邮箱；
金额类随机浮动1％～5％地址类截取前15个字符或只显示到区的级别。
关联脱敏为针对脱敏源之间的关联字段应用同一个脱敏规则，以使脱敏后的字段仍能相关联，即表之间的关联字段，应用同一个脱敏规则，以达到脱敏后的字段依然能关联上，不破坏数据的一致性。
其中，敏感字段是由数据提供方标识哪些字段是必须脱敏的字段，针对这类字段，DMP在提供之前必须选择指定几个脱敏规则中的一个，否则无法提交。这类字段为直接脱敏。
S140：将指定的脱敏规则及数据访问范围同步到ETL，ETL根据数据访问范围产生目标字段，根据脱敏规则自动生成目标字段的脱敏任务；
其中，该ETL根据相应的信息自动的生成相应的脱敏任务的过程包括：
根据脱敏规则判断创建脱敏表、视图还是物化视图；
若创建脱敏表，则根据脱敏源的字段建立脱敏表；若创建视图和物化视图， 则根据脱敏规则生成新的视图，并替换之前的视图。
具体的，配置完脱敏规则后，则把相应的规则及数据访问的范围同步给ETL，ETL根据接收到的脱敏规则和数据访问范围自动化的生成相应的脱敏任务，包括自动化创建目标字段，即脱敏表、物化视图或者视图，以备对数据进行脱敏操作。
为提高后续使用的效率且减少存储空间，根据脱敏规则判断建立脱敏表、物化视图还是视图的过程中包括：
根据脱敏规则预设脱敏源占用HDFS空间存储量的区间；
若脱敏源占用HDFS的空间存储量小于区间，则创建视图，若在区间内，则创建物化视图，若大于区间，则创建脱敏表。比如，当原始Hive表占用HDFS空间大小小于100G，则创建视图，大于100G小于500G，则创建物化视图，单表大于500G，则创建脱敏表，并将脱敏数据直接Insert到该脱敏表，具体的规则根据业务需求而定。
S150：检查脱敏源和目标字段是否一致，若一致，则继续保持脱敏源和目标字段及顺序一致，若不一致，则使脱敏源和目标字段一致，检查完成后执行ETL并完成脱敏任务；脱敏源为数据使用方初始筛选出的等待被脱敏的数据。
检查脱敏源和目标字段是否一致的过程中，包括：
若脱敏源字段有变更，则根据该目标字段的顺序判断是新增字段还是删除字段；
若脱敏源相对目标字段有新增字段，则在该目标字段基础上新增同样的字段，并替换之前的脱敏表、物化视图或视图，若脱敏源相对目标字段有删除字段，则在该目标字段基础上删减同样的字段，并替换之前的脱敏表、物化视图或视图。
具体的，应用了脱敏规则后的数据也储存在Hive里；
其中，脱敏源为数据使用方初始筛选出的等待被脱敏的数据，目标字段为立即被脱敏的准确脱敏数据，数据均以表格形式存储在Hive里，由于脱敏源的表和目标字段的表是Hive表，不能像关系型数据库那样来处理，因此字段的顺序就格外重要，否则可能会出现该脱敏的字段未脱敏，不该脱敏的字段却脱敏了，为了确保脱敏源的表和目标字段的表在特定顺序位置的字段代表的含义是同一个，确保运用脱敏规则的字段是用户真正需要脱敏的字段，需要按 字段顺序来判断。因此，在脱敏任务运行之前，需要保持脱敏源的表和目标字段的表及顺序一致，若不一致则会导致任务失败或者数据错位，因此，检查脱敏源的表和目标字段的表是在运行脱敏任务之前，两边一致之后，才能顺利进行脱敏，否则任务失败，需要人为介入，会增加任务运维工作量。
若脱敏任务为创建视图，如果有字段的变更，即新增或删除，则根据字段的顺序来判断，如果脱敏源是在最后面新增了一个字段，则可在原基础上进行新增字段，替换之前的视图，否则需要删除再新建hive表，完成后，则进行视图的替换或物化视图的rebuild及脱敏表数据的重新装载等后续ETL过程。
本实施例中数据授权脱敏自动化方法通过根据脱敏需求判断表访问权限和数据访问范围，然后数据使用方根据表访问权限和数据访问范围在DMP发起访问业务表的权限申请，DMP对权限申请进行审批，审批通过后，在DMP进行脱敏配置，针对敏感字段强制性配置指定的脱敏规则，进而将指定的脱敏规则及数据访问范围同步到ETL，ETL根据相应的信息自动的生成相应的脱敏任务，产生目标字段，检查脱敏源和目标字段是否一致，若一致，则继续保持脱敏源表和目标表字段及顺序一致，若不一致，则使源和目标的字段一致，检查完成后执行ETL并完成脱敏任务，极大地确保了自动对字段的变更进行同步更改，减少了人为参与，节省了人力，提高了脱敏效率，减少了存储空间。
另一方面，本申请还提供一种数据授权脱敏自动化***，图3为根据本申请实施例的数据授权脱敏自动化***框架图，该***对应于数据授权脱敏自动化方法，可以设置于数据授权脱敏自动化电子装置中。
如图3所示，该数据授权脱敏自动化***包括DMP模块310、ETL模块320、原始库模块330和脱敏库模块340。
其中的DMP模块310用于接受数据使用方发起的访问业务表的权限申请，该权限申请包括表访问权限和数据访问范围；然后DMP模块310根据该数据使用方的访问权限和访问范围对数据使用方的权限申请进行审批，若通过审批，则该DMP模块310开始进行脱敏配置，即针对脱敏字段强制性配置指定的脱敏规则，并将该脱敏规则同步到ETL模块320。
其中，表访问权限和数据访问范围根据脱敏需求确定，该脱敏需求由数据使用方根据预设规则创建，包括的具体脱敏字段是一些业务敏感字段，比如金额、销售额、电话、地址、证件号等。数据访问范围一般包括时间范围、跨度 范围等，比如只允许访问最近一个月的记录、只允许访问数据使用方某一部门的记录或者数据使用方所有的记录。
数据提供方在DMP模块310上能看到的所有专业公司数据，数据使用方在DMP模块310申请脱敏及脱敏规则，需要数据提供方的审批；具体的，该数据使用方为待审批者，该数据提供方为审批者，该数据提供方判定该脱敏需求符合数据使用要求，则会通过审批；是否通过都由数据提供方决定，通过该DMP模块310审批后的数据使用方都可以申请提交脱敏；判定脱敏需求符合数据使用要求的标准为：不破坏脱敏源、不含有超范围指令，如：脱敏源中无“电话”栏，选择脱敏需求为“电话”。此处的脱敏源为数据使用方初始筛选出的等待被脱敏的数据。
该ETL模块320根据数据访问范围产生目标字段，根据脱敏规则自动生成该目标字段的脱敏任务并执行该脱敏任务；并且，ETL模块在执行所述脱敏任务之前，需要检查脱敏源和目标字段是否一致，若一致，则继续保持脱敏源和目标字段及顺序一致，若不一致，则使脱敏源和目标字段一致，检查完成后执行所述脱敏任务。
简言之，ETL模块320根据DMP模块310同步来的脱敏规则自动生成脱敏任务并执行；其中，该ETL模块320根据相应的信息自动的生成相应的脱敏任务的过程包括：
根据脱敏规则判断创建脱敏表、视图还是物化视图；若创建脱敏表，则根据脱敏源的字段建立脱敏表；若创建视图和物化视图，则根据脱敏规则生成新的视图，并替换之前的视图。
其中，脱敏规则对应的脱敏方式包括直接脱敏和关联脱敏；其中直接脱敏为对脱敏源应用预设规则进行直接脱敏，与其他字段并无关系；关联脱敏为针对脱敏源之间的关联字段应用同一个脱敏规则，以使脱敏后的字段仍能相关联。
其中的预设规则，包括以下规则中的至少一个：
对任意字符串提供HASH加密名称统一改为“数据使用方的名称”；
电话后4位改为1234或屏蔽中间4位；
邮箱全部屏蔽为所述数据使用方的预设邮箱；
金额类随机浮动1％～5％地址类截取前15个字符或只显示到区的级别。
在本实施例的一个具体实施方式中，ETL模块根据脱敏规则预先设定所述脱敏源占用HDFS空间存储量的区间；若所述脱敏源占用HDFS的空间存储量小于所述区间，则创建视图；若在所述区间内，则创建物化视图；若大于所述区间，则创建脱敏表。
ETL模块在检查脱敏源和目标字段是否一致之后，若脱敏源和目标字段不一致，则根据所述目标字段的顺序判断是新增字段还是删除字段；若脱敏源相对目标字段有新增字段，则在所述目标字段基础上新增同样的字段，并替换之前的脱敏表、物化视图或视图，若脱敏源相对目标字段有删除字段，则在所述目标字段基础上删减同样的字段，并替换之前的脱敏表、物化视图或视图。
原始库模块330用于存放脱敏源数据，该脱敏库模块340用于存放脱敏任务和目标字段等数据，结合DMP模块310、ETL模块320完成脱敏任务。
图4为本申请电子装置示意图，在本实施例中，电子装置40可以是服务器、平板计算机、便携计算机、桌上型计算机等具有运算功能的终端设备。
该电子装置40包括：处理器41、存储器42、计算机程序43、网络接口及通信总线。
电子装置40可以是平板电脑、台式电脑、智能手机，但不限于此。
存储器42包括至少一种类型的可读存储介质。至少一种类型的可读存储介质可为如闪存、硬盘、多媒体卡、卡型存储器等的非易失性存储介质。在一些实施例中，可读存储介质可以是电子装置40的内部存储单元，例如该电子装置40的硬盘。在另一些实施例中，可读存储介质也可以是电子装置40的外部存储器，例如电子装置40上配备的插接式硬盘，智能存储卡(Smart Media Card,SMC)，安全数字(Secure Digital,SD)卡，闪存卡(Flash Card)等。
在本实施例中，存储器42的可读存储介质通常用于存储安装于电子装置40的计算机程序43，密钥生成单元310、密钥管理单元320、传输单元330和告警单元340等。
处理器41在一些实施例中可以是一中央处理器(Central Processing Unit,CPU)，微处理器或其他数据处理芯片，用于运行存储器42中存储的程序代码或处理数据，例如数据授权脱敏自动化程序43等。
网络接口可选地可以包括标准的有线接口、无线接口(如WI-FI接口)，通常用于在该电子装置40与其他电子设备之间建立通信连接。
通信总线用于实现这些组件之间的连接通信。
图4仅示出了具有组件41-43的电子装置40，但是应理解的是，并不要求实施所有示出的组件，可以替代的实施更多或者更少的组件。
在图4所示的电子装置实施例中，作为一种计算机存储介质的存储器42中可以存储操作***以及数据授权脱敏自动化程序43；处理器41执行存储器42中存储的数据授权脱敏自动化程序43时实现如下步骤：
S110：根据预设规则创建脱敏需求，根据所述脱敏需求确定表访问权限和数据访问范围；
S120：根据所述表访问权限和数据访问范围在DMP发起访问业务表的权限申请，所述DMP用于对所述权限申请进行审批；
S130：通过所述DMP对通过审批的权限申请进行脱敏配置，所述脱敏配置包括针对敏感字段强制性配置指定的脱敏规则；
S140：将所述指定的脱敏规则及所述数据访问范围同步到ETL，所述ETL根据所述数据访问范围产生目标字段，根据所述脱敏规则自动生成所述目标字段的脱敏任务；
S150：检查脱敏源和目标字段是否一致，若一致，则继续保持脱敏源和目标字段及顺序一致，若不一致，则使脱敏源和目标字段一致，检查完成后执行ETL并完成所述脱敏任务。
此外，本申请实施例还提出一种计算机可读存储介质，计算机可读存储介质中包括数据授权脱敏自动化程序，该数据授权脱敏自动化程序被处理器执行时实现如下操作：
S110：根据预设规则创建脱敏需求，根据所述脱敏需求确定表访问权限和数据访问范围；
S120：根据所述表访问权限和数据访问范围在DMP发起访问业务表的权限申请，所述DMP用于对所述权限申请进行审批；
S130：通过所述DMP对通过审批的权限申请进行脱敏配置，所述脱敏配置包括针对敏感字段强制性配置指定的脱敏规则；
S140：将所述指定的脱敏规则及所述数据访问范围同步到ETL，所述ETL根据所述数据访问范围产生目标字段，根据所述脱敏规则自动生成所述目标 字段的脱敏任务；
S150：检查脱敏源和目标字段是否一致，若一致，则继续保持脱敏源和目标字段及顺序一致，若不一致，则使脱敏源和目标字段一致，检查完成后执行ETL并完成所述脱敏任务。
本申请之计算机可读存储介质的具体实施方式与上述数据授权脱敏自动化方法、电子装置的具体实施方式大致相同，在此不再赘述。
需要说明的是，在本文中，术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的包含，从而使得包括一系列要素的过程、装置、物品或者方法不仅包括那些要素，而且还包括没有明确列出的其他要素，或者是还包括为这种过程、装置、物品或者方法所固有的要素。在没有更多限制的情况下，由语句“包括一个……”限定的要素，并不排除在包括该要素的过程、装置、物品或者方法中还存在另外的相同要素。
上述本申请实施例序号仅仅为了描述，不代表实施例的优劣。通过以上的实施方式的描述，本领域的技术人员可以清楚地了解到上述实施例方法可借助软件加必需的通用硬件平台的方式来实现，当然也可以通过硬件，但很多情况下前者是更佳的实施方式。基于这样的理解，本申请的技术方案本质上或者说对现有技术做出贡献的部分可以以软件产品的形式体现出来，该计算机软件产品存储在如上所述的一个存储介质(如ROM/RAM、磁碟、光盘)中，包括若干指令用以使得一台终端设备(可以是计算机，服务器，或者网络设备等)执行本申请各个实施例所述的方法。
以上仅为本申请的优选实施例，并非因此限制本申请的专利范围，凡是利用本申请说明书及附图内容所作的等效结构或等效流程变换，或直接或间接运用在其他相关的技术领域，均同理包括在本申请的专利保护范围内。

Claims (20)

  1. 一种数据授权脱敏自动化方法，应用于电子装置，其特征在于，所述方法包括：
    S110：根据预设规则创建脱敏需求，根据所述脱敏需求确定表访问权限和数据访问范围；
    S120：根据所述表访问权限和数据访问范围在DMP发起访问业务表的权限申请，所述DMP用于对所述权限申请进行审批；
    S130：通过所述DMP对通过审批的权限申请进行脱敏配置，所述脱敏配置包括针对敏感字段强制性配置指定的脱敏规则；
    S140：将所述指定的脱敏规则及所述数据访问范围同步到ETL，所述ETL根据所述数据访问范围产生目标字段，根据所述脱敏规则自动生成所述目标字段的脱敏任务；
    S150：检查脱敏源和目标字段是否一致，若一致，则继续保持脱敏源和目标字段及顺序一致，若不一致，则使脱敏源和目标字段一致，检查完成后执行ETL并完成所述脱敏任务。
  2. 根据权利要求1所述的数据授权脱敏自动化方法，其特征在于，所述脱敏需求由数据提供方根据所述预设规则创建；所述脱敏需求的具体脱敏字段包括业务敏感字段；
    所述数据访问范围至少包括时间范围、跨度范围。
  3. 根据权利要求1所述的数据授权脱敏自动化方法，其特征在于，在根据所述表访问权限和数据访问范围在DMP发起访问业务表的权限申请，所述DMP用于对所述权限申请进行审批的步骤包括：
    由数据使用方发起访问业务表的权限申请；
    判定所述数据使用方的脱敏需求符合数据使用要求的标准至少包括不破坏脱敏源、不含有超范围指令。
  4. 根据权利要求3所述的数据授权脱敏自动化方法，其特征在于，
    所述数据脱敏源为所述数据使用方初始筛选出的等待被脱敏的数据；
    所述目标字段为立即被脱敏的准确脱敏数据。
  5. 根据权利要求4所述的数据授权脱敏自动化方法，其特征在于，
    所述数据脱敏源和所述目标字段均以表格形式存储在Hive中。
  6. 根据权利要求1所述的数据授权脱敏自动化方法，其特征在于，所述脱敏规则对应的脱敏方式包括直接脱敏和关联脱敏；其中，
    所述直接脱敏为对脱敏源应用预设规则进行直接脱敏，与其他字段并无关系；
    所述关联脱敏为针对脱敏源之间的关联字段应用同一个脱敏规则，以使脱敏后的字段仍能相关联。
  7. 根据权利要求6所述的数据授权脱敏自动化方法，其特征在于，所述预设规则，包括以下规则中的至少一个：
    对任意字符串提供HASH加密名称统一改为“数据使用方的名称”；
    电话后4位改为1234或屏蔽中间4位；
    邮箱全部屏蔽为所述数据使用方的预设邮箱；
    金额类随机浮动1％～5％地址类截取前15个字符或只显示到区的级别。
  8. 根据权利要求1所述的数据授权脱敏自动化方法，其特征在于，根据所述脱敏规则自动生成所述目标字段的脱敏任务的过程包括：
    根据所述脱敏规则确定要创建脱敏表、视图还是物化视图；
    若创建脱敏表，则根据所述脱敏源的字段建立脱敏表；若创建视图和物化视图，则根据所述脱敏规则生成新的视图，并替换之前的视图。
  9. 根据权利要求8所述的数据授权脱敏自动化方法，其特征在于，在根据所述脱敏规则判断建立脱敏表、物化视图还是视图的过程中，包括：
    根据所述脱敏规则预先设定所述脱敏源占用HDFS空间存储量的区间；
    若所述脱敏源占用HDFS的空间存储量小于所述区间，则创建视图；若在所述区间内，则创建物化视图；若大于所述区间，则创建脱敏表。
  10. 根据权利要求8所述的数据授权脱敏自动化方法，其特征在于，所述检查脱敏源和目标字段是否一致的步骤之后，所述方法还包括：
    若脱敏源和目标字段不一致，则根据所述目标字段的顺序判断是新增字段还是删除字段；
    若脱敏源相对目标字段有新增字段，则在所述目标字段基础上新增同样的字段，并替换之前的脱敏表、物化视图或视图，若脱敏源相对目标字段有删除字段，则在所述目标字段基础上删减同样的字段，并替换之前的脱敏表、 物化视图或视图。
  11. 一种数据授权脱敏自动化***，其特征在于，包括DMP模块、ETL模块、原始库模块和脱敏库模块，其中，
    所述DMP模块，用于接受数据使用方根据表访问权限和数据访问范围发起的访问业务表的权限访问申请，并根据所述权限申请进行审批，若通过审批，则所述DMP模块开始进行脱敏配置，所述脱敏配置包括针对敏感字段强制性配置指定的脱敏规则，并将所述脱敏规则同步到所述ETL模块；
    所述ETL模块，用于根据所述数据访问范围产生目标字段，根据所述脱敏规则自动生成所述目标字段的脱敏任务并执行所述脱敏任务；并且，所述ETL模块在执行所述脱敏任务之前，检查脱敏源和目标字段是否一致，若一致，则继续保持脱敏源和目标字段及顺序一致，若不一致，则使脱敏源和目标字段一致，检查完成后执行所述脱敏任务；
    所述原始库模块，用于存放脱敏源；
    所述脱敏库模块，用于存放目标字段及脱敏任务。
  12. 根据权利要求11所述的数据授权脱敏自动化***，其特征在于，
    所述数据访问范围至少包括时间范围、跨度范围。
  13. 根据权利要求11所述的数据授权脱敏自动化***，其特征在于，
    所述DMP模块对所述权限申请进行审批的标准包括：不破坏脱敏源、不含有超范围指令。
  14. 根据权利要求11所述的数据授权脱敏自动化***，其特征在于，
    所述脱敏规则对应的脱敏方式包括直接脱敏和关联脱敏；其中，
    所述直接脱敏为对脱敏源应用预设规则进行直接脱敏，与其他字段并无关系；
    所述关联脱敏为针对脱敏源之间的关联字段应用同一个脱敏规则，以使脱敏后的字段仍能相关联。
  15. 根据权利要求14所述的数据授权脱敏自动化***，其特征在于，
    所述预设规则，包括以下规则中的至少一个：
    对任意字符串提供HASH加密名称统一改为“数据使用方的名称”；
    电话后4位改为1234或屏蔽中间4位；
    邮箱全部屏蔽为所述数据使用方的预设邮箱；
    金额类随机浮动1％～5％地址类截取前15个字符或只显示到区的级别。
  16. 根据权利要求11所述的数据授权脱敏自动化***，其特征在于，所述ETL模块在根据所述脱敏规则自动生成所述目标字段的脱敏任务的过程中，
    根据所述脱敏规则确定要创建脱敏表、视图还是物化视图；
    若创建脱敏表，则根据所述脱敏源的字段建立脱敏表；若创建视图和物化视图，则根据所述脱敏规则生成新的视图，并替换之前的视图。
  17. 根据权利要求16所述的数据授权脱敏自动化***，其特征在于，所述ETL模块根据所述脱敏规则预先设定所述脱敏源占用HDFS空间存储量的区间；
    若所述脱敏源占用HDFS的空间存储量小于所述区间，则创建视图；若在所述区间内，则创建物化视图；若大于所述区间，则创建脱敏表。
  18. 根据权利要求11所述的数据授权脱敏自动化***，其特征在于，所述ETL模块在检查脱敏源和目标字段是否一致之后，
    若脱敏源和目标字段不一致，则根据所述目标字段的顺序判断是新增字段还是删除字段；
    若脱敏源相对目标字段有新增字段，则在所述目标字段基础上新增同样的字段，并替换之前的脱敏表、物化视图或视图，若脱敏源相对目标字段有删除字段，则在所述目标字段基础上删减同样的字段，并替换之前的脱敏表、物化视图或视图。
  19. 一种电子装置，其特征在于，该电子装置包括：存储器、处理器以及存储在所述存储器中并可在所述处理器上运行的计算机程序，所述处理器执行所述计算机程序时实现权利要求1-10中任一项所述的数据授权脱敏自动化方法中的步骤。
  20. 一种计算机可读存储介质，其特征在于，所述计算机可读存储介质中存储有数据授权脱敏自动化分析程序，所述基于数据授权脱敏自动化分析程序被处理器执行时，实现如权利要求1至10中任一项所述的数据授权脱敏自动化方法的步骤。
PCT/CN2019/118433 2019-09-19 2019-11-14 数据授权脱敏自动化方法、***、装置及存储介质 WO2021051612A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910885652.0 2019-09-19
CN201910885652.0A CN110727954B (zh) 2019-09-19 2019-09-19 数据授权脱敏自动化方法、装置及存储介质

Publications (1)

Publication Number Publication Date
WO2021051612A1 true WO2021051612A1 (zh) 2021-03-25

Family

ID=69219193

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/118433 WO2021051612A1 (zh) 2019-09-19 2019-11-14 数据授权脱敏自动化方法、***、装置及存储介质

Country Status (2)

Country Link
CN (1) CN110727954B (zh)
WO (1) WO2021051612A1 (zh)

Cited By (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113626847A (zh) * 2021-08-24 2021-11-09 北京京东乾石科技有限公司 一种数据处理的方法和装置
CN113642039A (zh) * 2021-08-09 2021-11-12 平安科技（深圳）有限公司 单证模板的配置方法、装置、计算机设备和存储介质
CN113868697A (zh) * 2021-08-25 2021-12-31 中通服公众信息产业股份有限公司 一种基于电信数据仓库实时分析数据脱敏的方法
CN114491585A (zh) * 2021-12-31 2022-05-13 上海柯林布瑞信息技术有限公司 医疗数据的动态脱敏加密方法、装置、设备和介质
CN115080827A (zh) * 2022-07-01 2022-09-20 中银金融科技有限公司 一种敏感数据处理方法及装置
CN116205236A (zh) * 2023-05-06 2023-06-02 四川三合力通科技发展集团有限公司 基于实体命名识别的数据快速脱敏***及方法
CN116933287A (zh) * 2023-07-28 2023-10-24 南京维拓科技股份有限公司 一种工业软件中敏感信息脱敏和加密的方法
CN117390659A (zh) * 2023-12-13 2024-01-12 江苏量界数据科技有限公司 一种基于分布式数据计算的权限控制方法
CN117932676A (zh) * 2024-01-26 2024-04-26 湖北消费金融股份有限公司 基于网络接口访问控制的数据脱敏方法及***

Families Citing this family (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113268517B (zh) * 2020-02-14 2024-04-02 中电长城网际***应用有限公司 数据分析方法和装置、电子设备、可读介质
CN113157902A (zh) * 2020-12-24 2021-07-23 中国能源建设股份有限公司 一种利用多种信息手段完成信息脱敏审批的查询方式
CN112818383A (zh) * 2021-01-14 2021-05-18 内蒙古蒙商消费金融股份有限公司 一种表注册方法及装置
CN113158233B (zh) * 2021-03-29 2023-06-27 重庆首亨软件股份有限公司 数据预处理方法、装置及计算机存储介质
CN113420330A (zh) * 2021-06-28 2021-09-21 国网湖南省电力有限公司 大数据***的可视化脱敏数据生成方法
CN113360946B (zh) * 2021-06-29 2024-01-30 招商局金融科技有限公司 新闻脱敏处理方法、装置、电子设备及可读存储介质

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107958158A (zh) * 2017-10-27 2018-04-24 国网辽宁省电力有限公司 一种大数据平台的动态数据脱敏方法及***
CN108228830A (zh) * 2018-01-03 2018-06-29 广东工业大学 一种数据处理***
CN108268558A (zh) * 2017-01-03 2018-07-10 中移(苏州)软件技术有限公司 一种数据分析的方法和装置
US20190156024A1 (en) * 2017-11-20 2019-05-23 Somansa Co., Ltd. Method and apparatus for automatically classifying malignant code on basis of malignant behavior information
CN110084053A (zh) * 2019-05-07 2019-08-02 江苏满运软件科技有限公司 数据脱敏方法、装置、电子设备及存储介质
CN110110543A (zh) * 2019-03-14 2019-08-09 深圳壹账通智能科技有限公司 数据处理方法、装置、服务器及存储介质

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20060048224A1 (en) * 2004-08-30 2006-03-02 Encryptx Corporation Method and apparatus for automatically detecting sensitive information, applying policies based on a structured taxonomy and dynamically enforcing and reporting on the protection of sensitive data through a software permission wrapper
CN107403111A (zh) * 2017-08-10 2017-11-28 中国民航信息网络股份有限公司 Hive数据脱敏方法及装置
US10796013B2 (en) * 2017-11-13 2020-10-06 Veeva Systems Inc. User programmatic interface for supporting data access control in a database system
CN108171069A (zh) * 2018-01-03 2018-06-15 平安科技（深圳）有限公司 脱敏方法、应用服务器及计算机可读存储介质
CN109729076B (zh) * 2018-12-19 2022-06-24 上海晶赞融宣科技有限公司 数据脱敏、逆脱敏方法及装置、存储介质、终端
CN110232291A (zh) * 2019-04-25 2019-09-13 深圳壹账通智能科技有限公司 智能数据脱敏方法、装置、计算机设备和存储介质

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108268558A (zh) * 2017-01-03 2018-07-10 中移(苏州)软件技术有限公司 一种数据分析的方法和装置
CN107958158A (zh) * 2017-10-27 2018-04-24 国网辽宁省电力有限公司 一种大数据平台的动态数据脱敏方法及***
US20190156024A1 (en) * 2017-11-20 2019-05-23 Somansa Co., Ltd. Method and apparatus for automatically classifying malignant code on basis of malignant behavior information
CN108228830A (zh) * 2018-01-03 2018-06-29 广东工业大学 一种数据处理***
CN110110543A (zh) * 2019-03-14 2019-08-09 深圳壹账通智能科技有限公司 数据处理方法、装置、服务器及存储介质
CN110084053A (zh) * 2019-05-07 2019-08-02 江苏满运软件科技有限公司 数据脱敏方法、装置、电子设备及存储介质

Cited By (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113642039A (zh) * 2021-08-09 2021-11-12 平安科技（深圳）有限公司 单证模板的配置方法、装置、计算机设备和存储介质
CN113642039B (zh) * 2021-08-09 2024-05-28 平安科技（深圳）有限公司 单证模板的配置方法、装置、计算机设备和存储介质
CN113626847A (zh) * 2021-08-24 2021-11-09 北京京东乾石科技有限公司 一种数据处理的方法和装置
CN113868697B (zh) * 2021-08-25 2024-04-19 中通服公众信息产业股份有限公司 一种基于电信数据仓库实时分析数据脱敏的方法
CN113868697A (zh) * 2021-08-25 2021-12-31 中通服公众信息产业股份有限公司 一种基于电信数据仓库实时分析数据脱敏的方法
CN114491585A (zh) * 2021-12-31 2022-05-13 上海柯林布瑞信息技术有限公司 医疗数据的动态脱敏加密方法、装置、设备和介质
CN115080827A (zh) * 2022-07-01 2022-09-20 中银金融科技有限公司 一种敏感数据处理方法及装置
CN115080827B (zh) * 2022-07-01 2024-05-24 中银金融科技有限公司 一种敏感数据处理方法及装置
CN116205236B (zh) * 2023-05-06 2023-08-18 四川三合力通科技发展集团有限公司 基于实体命名识别的数据快速脱敏***及方法
CN116205236A (zh) * 2023-05-06 2023-06-02 四川三合力通科技发展集团有限公司 基于实体命名识别的数据快速脱敏***及方法
CN116933287A (zh) * 2023-07-28 2023-10-24 南京维拓科技股份有限公司 一种工业软件中敏感信息脱敏和加密的方法
CN117390659B (zh) * 2023-12-13 2024-04-02 江苏量界数据科技有限公司 一种基于分布式数据计算的权限控制方法
CN117390659A (zh) * 2023-12-13 2024-01-12 江苏量界数据科技有限公司 一种基于分布式数据计算的权限控制方法
CN117932676A (zh) * 2024-01-26 2024-04-26 湖北消费金融股份有限公司 基于网络接口访问控制的数据脱敏方法及***

Also Published As

Publication number Publication date
CN110727954A (zh) 2020-01-24
CN110727954B (zh) 2023-08-29

Similar Documents

Publication Publication Date Title
WO2021051612A1 (zh) 数据授权脱敏自动化方法、***、装置及存储介质
EP3356964B1 (en) Policy enforcement system
US11328081B2 (en) Consent-based data privacy management system
US9904798B2 (en) Focused personal identifying information redaction
US8949209B2 (en) Method and system for anonymizing data during export
US9699193B2 (en) Enterprise-specific functionality watermarking and management
JP5707250B2 (ja) データベースアクセス管理システム、方法、及びプログラム
JP2020053091A (ja) 個人番号管理装置、個人番号管理方法、および個人番号管理プログラム
US20150254577A1 (en) System and methods for location based management of cloud platform data
US9584437B2 (en) Resource watermarking and management
CN111931140A (zh) 权限管理方法、资源访问控制方法、装置和电子设备
US20170149568A1 (en) Generating Enhanced Digital Signatures for Artifacts
CN114244598A (zh) 一种内网数据访问控制方法、装置、设备及存储介质
US20220398337A1 (en) Data governance systems and methods
JP2006244177A (ja) データベース装置
US9195811B2 (en) Functionality watermarking and management
CN115878655A (zh) 数据操作方法、装置、计算机设备和存储介质
JP2012063896A (ja) データアクセス制御システム、データアクセス制御方法及びデータアクセス制御プログラム
US8806217B2 (en) Functionality watermarking and management
CN115935421B (zh) 一种数据产品发布方法、***及存储介质
KR102235775B1 (ko) 개인정보처리 대행 및 관리 방법 및 프로그램
US12032642B1 (en) Systems and methods for sharing user data
US20230185938A1 (en) Managing purpose-based processing of data using a purpose agent
CN113935046A (zh) 一种数据处理方法、装置、设备及存储介质
CN118296622A (zh) 一种数据库账号管理方法、设备、介质和产品

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19945742

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19945742

Country of ref document: EP

Kind code of ref document: A1