CN105515963A - Data gateway device and big data system - Google Patents

Data gateway device and big data system Download PDF

Info

Publication number
CN105515963A
CN105515963A CN201510881918.6A CN201510881918A CN105515963A CN 105515963 A CN105515963 A CN 105515963A CN 201510881918 A CN201510881918 A CN 201510881918A CN 105515963 A CN105515963 A CN 105515963A
Authority
CN
China
Prior art keywords
data
service
peek
gateway device
user
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201510881918.6A
Other languages
Chinese (zh)
Inventor
李卫
魏进武
张基恒
张呈宇
霍玉嵩
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
China United Network Communications Group Co Ltd
Original Assignee
China United Network Communications Group Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by China United Network Communications Group Co Ltd filed Critical China United Network Communications Group Co Ltd
Priority to CN201510881918.6A priority Critical patent/CN105515963A/en
Publication of CN105515963A publication Critical patent/CN105515963A/en
Pending legal-status Critical Current

Links

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L12/00Data switching networks
    • H04L12/66Arrangements for connecting between networks having differing types of switching systems, e.g. gateways
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/01Protocols
    • H04L67/02Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/01Protocols
    • H04L67/06Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]

Abstract

The invention discloses a data gateway device and a big data system. The data gateway device comprises a user management module for managing user information; a service management module for managing the information of a data service opened to a user; a rule management module for setting different compliance checking rules aiming at different users and different data services; an access management module for receiving the service access request sent by the user, sending a corresponding access request to a data center and receiving the source data returned by the data center; and a data desensitization processing module for carrying out compliance checking to the source data returned by the data center according to a rule set by the rule management module so as to generate service data; the service data are data satisfying the compliance checking; the access management module also is used for providing the service data generated by the data desensitization processing module for the user.

Description

Data gateway device and large data system
Technical field
The invention belongs to technical field of data processing, be specifically related to a kind of data gateway (Gateway) device and a kind of large data system comprising this data gateway device, it can ensure the safe operation of large data.
Background technology
At large data age, vitalize data assets, opening and shares data become trend of the times.Open data, services, the operation realizing large data and realization are the hot issues of current large data development.But, in the operation or realization process of large data, the compliance how ensureing data, the leakage avoiding sensitive information, transaction data to be measured or charging and carry out auditing etc. to data and become the current problem needing solution badly.
Summary of the invention
The object of this invention is to provide a kind of data gateway device and a kind of large data system comprising this data gateway device, it can solve at least one problem above-mentioned existed in prior art.
For realizing object of the present invention, according to an aspect of the present invention, provide a kind of data gateway device, it comprises:
User management module, it is for managing user information;
Service management module, it is for managing the information of the data, services opened to user;
Rules administration module, it advises inspection rule for arranging different conjunction for different user and different data, services;
Peek administration module, it, for the peek request of service receiving user and send, sends to data center the request of peeking accordingly, and accepts the source data that data center returns; And
Data desensitization processing module, its rule for arranging according to rules administration module, the source data returned data center is carried out conjunctions rule and is checked, thus generates service data, and described service data is meet the data of closing and advising inspection, wherein
The service data that described peek administration module also generates for the processing module that data desensitized is supplied to user.
Alternatively, described rules administration module can also be used for arranging different data filtering desensitization rules for different user and different data, services, and described data desensitize, processing module may be used for carrying out filtration desensitization to the source data that data center returns, and then carries out the inspection of conjunction rule to the data of filtering desensitization.
Alternatively, the service data that described peek module can also be used for exporting is measured and charging.
Alternatively, described rules administration module can also be used for the output line number of the service data arranging user, when satisfied total line number of closing the data that rule check is greater than the output line number of the service data of described setting, described data desensitization processing module can also be used for the output line number of the service data arranged according to described rules administration module, to the satisfied data of closing rule and checking, from first trip, order intercepts the data of certain line number, generate described service data, the output line number of wherein said line number=service data.
Alternatively, described peek administration module can also be used for generating peek task for the peek request of user, and provides the look facility of peek task.
Alternatively, described user management module can also be used for being configured the authority of dissimilar user, and described authority comprises function menu access rights.
Described conjunction rule check that rule can comprise: data dictionary coupling, number range inspection and field length inspection.
Described data filtering desensitization rule can comprise: the condition filter of the screening of service output field, field and field contents process.
Alternatively, described data gateway device can also comprise memory cell, the service data that its source data returned for data cached center and data desensitization processing module generate, wherein said peek administration module is configured to: first judge whether to exist in described memory cell the data that client asks, if buffer memory has been in the memory unit for the data of asking, then the buffer memory data of asking in the memory unit are supplied to client as service data, otherwise send to data center the request of peeking accordingly.
Alternatively, described data gateway device can also comprise security audit module, and it is for recording Operation Log and peek daily record, and by Operation Log and peek log archive.
Alternatively, described data gateway device can also comprise workform management module, and it is for the treatment of system work order task, and described work order task comprises work order establishment and work order is checked.
Alternatively, described data gateway device can also comprise system management module, it is for the basic data of configuration-system, described basic data comprises data cycle and timer, and the described data cycle is the caching period of service data in data gateway device that the source data that returns of data center and described data desensitization processing module generate.
According to a further aspect in the invention, additionally provide the addressable large data system of a kind of client, it comprises data center and data gateway device, described data gateway device is connected with described data center by interface, wherein said data gateway device is any one data gateway device above-mentioned, and described data gateway device be configured to can by interface and client's side link.
Described data gateway device and described data center can by data service information synchronized update interface, peek and apply for that interface, service data generate notification interface and serve Data synchronization interface and realize data cube computation, wherein,
By described information on services synchronized update interface, the data, services that the information of the data, services in the service management module of described data gateway device and described data center open is consistent;
By described peek application interface, the peek administration module of described data gateway device sends the peek request of service to described data center, and, when described service is real time service, by described peek, described data center applies for that interface returns source data to the peek administration module of described data gateway device, when described service is cycle service and the account phase of described service is the history account phase, described data center returns the storage information of institute's request source data to the peek administration module of described data gateway device by described peek application interface, when described service is cycle service and the account phase of described service is the non-vostro account phase, described data center returns the account phase by described peek application interface to the peek administration module of described gateway apparatus and does not arrive message,
Generate notification interface by described service data, described data center notifies the peek administration module institute request source data ready of described data gateway device and returns the storage information of institute's request source data; And
By described service data sync cap, the peek administration module of described data gateway device is according to the storage acquisition of information data of institute's request source data.Described data service information synchronized update interface, peek application interface, service data are generated notification interface and can be realized by web Service interface; Described service data sync cap can pass through FTP Interface realization.
Data gateway device is also configured to be connected with client data with service data-pushing interface by peek application interface, wherein,
By peek application interface, the peek administration module of data gateway device accepts the peek request of the service that client sends, and the peek administration module of data gateway device provides asked service data to user;
Push interface by service data, asked service data is pushed to assigned address by the peek administration module of data gateway device.
Described peek application interface can be realized by web Service interface; Described service data pushes interface can pass through FTP Interface realization.
In data gateway device provided by the invention and large data system, for different user and different data types, utilize different conjunction rule to check that rule is audited the source data that data center provides, ensure that the normalization of the service data being supplied to user; For the sensitive information in the data of definition, carry out filtration desensitization according to the rule arranged, ensure in the data (that is, service data) that transaction exports without sensitive content; For the output data after the desensitization of conjunction rule, the data volume size exported according to different user, transaction and data value attribute carry out measuring and charging; In addition, data export or in process of exchange, carry out permanent file, so that the operation compliance audit in future to the time of data object output (that is, trade user), output data and the form etc. of output data.
Accompanying drawing explanation
The schematic diagram of the data gateway device that Fig. 1 provides for one embodiment of the invention;
The schematic diagram of the data gateway device that Fig. 2 provides for another embodiment of the present invention;
The workflow diagram of the data gateway device that Fig. 3 provides for the embodiment of the present invention; And
The schematic diagram of the large data system that Fig. 4 provides for the embodiment of the present invention.
Embodiment
For making those skilled in the art understand technical scheme of the present invention better, below in conjunction with the drawings and specific embodiments, the present invention is described in further detail.
One embodiment of the invention provides a kind of for ensureing the data gateway device that large data security is runed.As shown in Figure 1, this data gateway device can comprise user management module, service management module, rules administration module, peek administration module and data desensitization processing module.User management module is used for managing user information.Service management module is for managing the information of the data, services opened to user; Rules administration module is used for arranging different conjunction for different user and different data, services and advises inspection rule; Peek administration module, for the peek request of service receiving user and send, sends to data center the request of peeking accordingly, and accepts the source data that data center returns; Data desensitization processing module is used for the rule arranged according to rules administration module, and carry out the inspection of conjunction rule to the source data that data center returns, thus generate service data, described service data is meet the data of closing rule inspection; Wherein, the service data that described peek administration module also generates for the processing module that data desensitized is supplied to user.
According to other embodiments of the invention, provided by the invention for ensureing that the data gateway device of large data security operation can also comprise security audit module, rules administration module, workform management module and/or system management module.Such as, Fig. 2 shows the schematic diagram of the data gateway device that another embodiment of the present invention provides.
Below, with reference to Fig. 1 and Fig. 2, each functional module of data gateway apparatus is described in detail.
User management module
User management module is used for managing user information, and can be configured the function menu access rights of dissimilar user, thus can support polytype user management, realizes many tenants management.
User can comprise data user and organization user.Data user is demand user or the Third party system/application of data, services, and each data user must associate a mechanism, and data user can apply for obtaining data, services, and obtains the service data of closing rule.Organization user is as the person liable of data user, and enterprise or institutional person liable, only have an organization user, but can have multiple data user under a mechanism.Being appreciated that organization user and data user all can apply for voluntarily, by auditing the application of organization user and data user and amendment, can following the trail of and record the flow direction of service data better.
User can also comprise Batch Processing administrative staff, such as, can comprise data safety management person and auditor.Data safety management person's configuration data filters, desensitize and close rule checks rule.Auditor audits the registration/amendment/cancellation of user, the level of confidentiality of distributing user, data, services are checked authority, peek authority, peek priority authority and examination & verification data filtering, desensitization and closed rule and check rule etc.
User can also comprise background system administrative staff user, such as, can comprise: operation maintenance personnel and system manager.Operation maintenance personnel operation and maintenance system.The systematic whole authority of system manager's tool.Be appreciated that system manager can newly-built auditor, data safety management person and operation maintenance personnel.
User management module can also for dissimilar user, as data user, organization user, data safety management person, auditor, operation maintenance personnel and system manager, and the function menu access rights of configure user.Generally speaking, different users has the function menu access rights of different stage.
Service management module
Service management module, for managing the information of the data, services opened to user, such as, creates the bookkeepings such as the data, services that new data, services, amendment and deletion opened.The data, services that data service information in data management module and data network operator open to user is consistent.
Data service information can comprise: service essential information, and it can comprise Services Code, service name, COS (real time service or cycle service), seeervice cycle (year, season, the moon, week or day; For cycle service), the cycle data ready date (for cycle service), service export collection coding, service export set name claim; Service output field information, it comprises service and exports collection coding, code field, field name, field data types, field description etc.; And service charging information, such as, every elements field or the price etc. of often going.
Peek administration module
Peek administration module, for receiving the peek request of user, sends to data center the request of peeking accordingly, and accepts the source data that data center returns.In the present invention, the source data that data center returns comprises: for real time service, the source data that data center returns, and for cycle service, the source data of the storage acquisition of information of the source data that peek administration module returns according to data center.Process the source data that data center returns in data desensitization processing module (aftermentioned), after producing service data (that is, closing the data of rule), service data is supplied to user by peek administration module.
Peek administration module can also be used for generating peek task for the peek request of user.The establishment mode of peek task has four kinds, and one is for real time service, initiates by user is manual task of peeking in real time; Two is for cycle service, initiates peek task by hand first by user; Three is for cycle service, after user initiates peek task first by hand, periodically automatically generates peek task by peek administration module; Four be data safety management person revise close rule check rule exist side by side namely come into force after regenerate peek task.
Peek administration module can also provide the look facility of service peek task.Service peek task can take list mode to show.
The information of service peek contained by task list can comprise: data user's title, Services Code, service name, COS, seeervice cycle, task account phase, data ready, task status, job start time, job end time.
Wherein, COS comprises the cycle or real-time.
The task account phase, this field just had value only when COS is the cycle, if task is by the manual real-time peek task initiated, then the display of this field in real time.According to bill cycle type, display format can be: year-yyyy, the moon-yyyymm, season-yyyy*Q, week-yyyy**, day-yyyy***.
Task status can comprise effectively and lose efficacy.After task creation, task status is defaulted as effectively, data safety management person's alteration ruler (comprise data filtering desensitization rule and close rule check rule) and select come into force after, by the current all relevant peek tasks do not completed of termination, re-create peek task, perform new safety regulation, the task status of the peek task stopped is set to lose efficacy.
The value of data ready can be as the criterion and be on the waiting list number, start peek, peek, desensitize in, close rule check in, data ready, illegally end of service, serve Delivered or service is sent in failure, retry, every is described as follows:
Service role list can also support query function, such as, according to the inquiry that data user's title, Services Code, service name, time started, end time carry out.
Service peek task list can also provide check result to check for administrative staff, serve the feature operations such as the sampling of source data and service data.
Check result checks data desensitization and the check result of the service source data providing this peek task, comprises this peek data user's login name of task, whether data user's name, institutional affiliation user login name, institutional affiliation address name, Services Code, service name, COS, seeervice cycle, task account phase, job start time, service source data close rule, closes rule and check total line number, data desensitization/close rule to check regular.If service source data check result is not for conforming to rule, then by with list mode display all examined out do not conform to rule data, list exhibition information comprises: do not conform to the sequence number of rule data line, serving the line number in source data, do not conform to all data that rule data are expert at, rule data are not conformed to red font mark, eject with mouse triggering mode the conjunction rule that these data should defer to and check rule, and the export function of closing rule check result is provided, support to derive with txt, excel file format.
Service source data and the sampling of service data, support the sampling of continuous nominated bank data content (specify walk to XXX from XXX capable), continuous line number maximum number is 100, terminates total line number that line number must be less than or equal to source data/service data.
As needed the service data to user obtains to add up, peek administration module can also measure the service data exporting this user to and charging.For the service data exported, according to different users, carry out measuring and charging according to data volume size and data value attribute.Service data amount and toll amount is increased in service peek task list exhibition information.Such as, the toll amount of calculated data amount and this data, services can be carried out according to following formula:
Data volume=Field Count × line number;
Price × data volume × user's Scaling Standards coefficient of data, services toll amount=every elements field/row.
Here, user's Scaling Standards coefficient of user can be pre-determined in user management module.
Generally speaking, data user can only check the information of the data, services of having authorized, and proposes peek request.
Be appreciated that data gateway device can also comprise memory cell, the service data that its source data returned for data cached center and data desensitization processing module generate.Now, peek administration module is configured to first to confirm whether to exist in memory cell the data that client asks, if buffer memory has been in the memory unit for the data of asking, then the buffer memory data of asking in the memory unit are supplied to user as service data, otherwise send to data center the request of peeking accordingly.
It should be noted that, memory cell can as independently unit existence, also can as a part for peek administration module, the present invention does not limit this, as long as the source data that data center returns or peek administration module obtain source data can by data desensitize module read, and data desensitization CMOS macro cell service data can by peek administration module read.
Rules administration module
Rules administration module is used for carrying out separate configurations to the authority of each organization user and/or data user, generate different checking authority, peek application mandate, peek priority authority, data filtering desensitization rule and close to advise for different users and different data, services and check rule, thus the resource isolation between different pieces of information user can be realized.By rules administration module, safety regulation unified management function can be provided, be appreciated that any rule configuration can only be operated by data safety management person.Data, services check authority, refer to that can organization user and/or data user see this service and details thereof in data, services list.The service list that different users sees can be different.
The peek application mandate of data, services, refers to that can data user obtain the data of this service.Data user at peek administration module for there being the service checking authority, can submit peek application to, after examination & verification, configuration peek application are authorized, initiates peek request by peek administration module.
The peek priority authority of data, services, when referring to that multiple data user initiates peek request simultaneously, peek administration module creates the Sort Priority rule of peek task.
Because the level of confidentiality of different institutions user or data user requires general different, therefore, in order to avoid multiple data user by sharing the service data of acquisition and carrying out recombinating and obtain sensitive information as far as possible, preferably, for different user and different pieces of information service, formulate different data filtering desensitization rules and check rule with closing to advise.Data filtering desensitization rule and close rule and check that rule can arrange at random or pre-set, as long as the service data that the data user under making to belong to same organization user is difficult to by obtaining recovers sensitive information as far as possible.
Close rule and check that rule can comprise: data dictionary coupling, number range inspection and field length inspection.Close rule and check that rule can be imported by excel file format.
Filter desensitization rule can comprise: the condition filter of the screening of service output field, field and field contents process.The condition filter of field can comprise: the dictionary according to field filters, and for the field of value type, filters according to numerical value comparison expression.Field contents process can comprise: the process of replacing spcial character, and carries out continuous figure place intercepting the process of replacing.Below, desensitization rule will be filtered to these to be described in detail.
(1) output field screening is served
The row that the screening of service output field is tables of data filter.Such as, source data of certain service comprises 9 output fields, and field name is respectively province, subscriber-coded, name, date of birth, identification card number, cell-phone number, set meal type, type, monthly telephone expenses.According to " telecommunications and Internet user's personal information protection regulation ", do not allow to export the fields such as address name, date of birth, identification card number, address, telephone number, account and password.Therefore, for this service request of data user, rules administration module arranges field filter rule, make this service only export province, subscriber-coded, set meal type, type and monthly telephone expenses 5 fields, then serve the name in source data, date of birth, identification card number and these 4 fields of cell-phone number and all will be filtered.
(2) condition filter of field
The condition filter of field is the row filtration of tables of data.As mentioned above, the condition filter of field can be divided into again two kinds of filtrations.
The first is the filtration carried out according to the dictionary of field.Such as, the province dictionary of output field " province " correspondence in upper example service comprises " Beijing ", " Hebei ", " Tianjin ", " Henan ", " Shandong " 5 values, rules administration module can arrange the value filter condition of the output field " province " of this service for only including in " Beijing " for data user, then for this data user, service source data in province field value be " Hebei ", " Tianjin ", " Henan ", " Shandong " data provisional capital be filtered.
The second is the field for value type, according to the filtration that numerical value comparison expression carries out.Such as, the data type of the output field " monthly telephone expenses " in upper example service is double, rules administration module arranges the output field " monthly telephone expenses " of this service value filter condition for data user is: field span between 50 ~ 100, then the value of the monthly telephone expenses field of serving in source data be less than 50 or the data provisional capital that is greater than 100 be filtered.
(3) field contents process
As implied above, field contents process comprises two kinds of process.
The first is the process of replacing spcial character.Such as, in upper example service, the output field " subscriber-coded " that rules administration module is arranged this service for data user carries out spcial character replacement, replaced by spcial character (123) and be processed into * * *, then " 123 " served in the field " subscriber-coded " in source data all will be replaced by " * * * ".Such as, " subscriber-coded " in service source data is " 1235678 ", is shown as " * * * 5678 " after replacing process.
The second carries out intercepting the process of replacing to the character of continuous figure place.Intercepting replacement comprises: N position, front end character is replaced to designated character, and middle N continuous position character is replaced to designated character, or by rear end from which character replace to designated character.
When require by " subscriber-coded " front 4 change 6666 into time, if service source data in " subscriber-coded " be " 1235678 ", then through replacement process after be shown as: 6666678;
When requiring to change the 2 to 5 in " subscriber-coded " into 6666, if " subscriber-coded " in service source data is " 1235678 ", be then shown as after replacing process: 1666678;
When require by " subscriber-coded " from the 4th after character change 6666 into time, if service source data in " subscriber-coded " be " 1235678 ", then through replacement process after be shown as: 1236666.
It should be noted that, in rules administration module, the conjunction rule of necessary setting data check rule, thus guarantee the compliance of the service data being supplied to user.When the heart has been configured with and has filtered desensitization function in the data, data gateway device is without the need to the filtration desensitization rule of configuration data.
Data desensitization module
Data desensitization processing module is used for the configuration according to rules administration module, filtration desensitization process is carried out to the source data that data center returns, and the inspection of conjunction rule is carried out to the data after filtering desensitization, if these data close rule, then the data of this conjunction rule are supplied to user as service data by peek administration module, otherwise, stop exporting these data.
If when rules administration module also arranges the output line number of service data for different users and different data, services, when satisfied total line number of closing the data that rule check is greater than the output line number of the service data of described setting, the output line number of the service data of user that data desensitization processing module will be arranged according to rules administration module, to the satisfied data of closing rule and checking, from first trip, order intercepts the data of certain line number, generate described service data, wherein the output line number of line number=service data.
Be appreciated that, under rules administration module filters the feelings situation of desensitization rule without the need to configuration data (namely, when data center has been configured with and has filtered desensitization function), the source data that desensitization desensitization module directly returns data center or the source data that peek application interface obtains are carried out conjunctions and are advised inspection.
Security audit module
Operation Log, peek daily record for recording Operation Log, peek daily record, and are filed by security audit module.
Operation Log refers to the operation information of all users in plant running, comprise: operate user account, by operation user account, action type, operating function, content of operation (as: inquiring about oneself Pending tasks list, data volume: 25), the operating time etc.
Peek daily record refers to the acquisition operation information of the service data of all users in plant running, comprise: obtain time of data, data volume, service essential information, service output field information (that is, the field information of service data), the service data desensitization/close rule to check rule etc. that performs.The acquisition operation of service data comprises two kinds of situations, and one is administrative staff's initiatively download or sample download service source data and service datas, and two is that service data is successfully returned to data user by data gateway device.
Security audit module can periodically be audited to Operation Log, peek daily record, and support the field level audit of the service data obtained for organization user or data user, thus the data flow of responsive field can be followed the trail of more accurately, make large data operation meet operation compliance audit better.
Workform management module
Workform management module is for the treatment of the work order task of data gateway device, and work order task comprises work order establishment and work order is checked.For the task of needing approval distributing, data gateway device can create work order automatically.For different business, in work flow Node configuration work order task trigger condition.System manager can the user type of each work order task of configuration process, when work order task is triggered, automatically generates pending work order to such user.Such as, business is registered for new user, the submission of log-on message can be set to trigger condition, and the user type that this new user of process registers work order is set to auditor.New for triggering user is registered work order by user after submission log-on message, and automatically generates a pending work order to auditor on duty process.
In addition, can check work order by workform management module, administrative staff can check pending/work order of doing/recalling.
Checking to process user of pending work order provides work order to inquire about and processing capacity.User can carry out fuzzy query according to pending type, pending title, pending content.System will carry out descending sort displaying according to the pending rise time, and user directly clicks process and enters the corresponding worksheet page.After worksheet, meeting automatic flow is to doing work order.
Do checking of work order and will show processed work order history process information, having comprised: the information such as previous step worksheet people, processing time, handling suggestion, processing time.
That recalls work order checks the details will shown and recall work order, comprises work order revocation reason.By workform management module, data gateway system can realize the running to the user profile of all types of user and the auditing flow of authority.
System management module
System management module is used for the basic data of configuration-system, and this basic data can comprise data cycle and timer.
The data cycle is service source data and the caching period of service data in data gateway device, and this data cycle can pre-set or specified by operation maintenance personnel or revise.
The type of the caching period of data can be year, season, the moon, week or day, and the cycle, unit was sky, namely cleared up these data at this week after date.
Whether system management module scanography every day data expired, for expired data, generate stale data and clear up pending work order, notice operation maintenance personnel, through operation maintenance personnel examination & approval by after automatically perform the cleaning of stale data.
Timer can comprise: the task of cycle service creates supervision timer automatically, service source data trace interval, data desensitization/normative review time interval, FTP push the time interval, the FTP propelling movement repeating transmission frequency of failure etc. that again push unsuccessfully.
The workflow diagram for ensureing the data gateway device that large data security is runed that Fig. 3 provides for the embodiment of the present invention.As shown in Figure 3, this workflow specifically comprises the following steps.
Step 1: user carries out user's registration by the user management module of data gateway device.
Step 2: the service management module of data gateway device is from the information of the synchronous various data, services of data center.
Step 3: the rules administration module of data gateway device carries out the mandate of data, services search access right to user.
Step 4: user submits the peek application of service to by the peek administration module of gateway apparatus.
Step 5: the rules administration module of data gateway device carries out the configuration of data, services peek authority to user.
The data filtering desensitization rule of the peek application mandate that rules administration module can carry out serving for user, the priority grant of service data peek, service, the conjunction rule of service data check the output line number etc. of rule and service data.Same its data filtering of service orientation different user separate configurations desensitization rule, conjunction rule check the output line number of rule, service data.
Step 6: the peek application of subscriber authorisation service is submitted in the peek management of data gateway device to data center.
Step 7: data center generates service source data.Especially, if user's submission is real time service peek application, then data center generates real time service source data.If what user submitted to is the peek application of cycle service, then data center is by the generating period according to cycle service, generating period service source data.
Step 8: data center will serve source data (real time service source data or cycle service source data store information) and return to data gateway device.
Step 9: the data desensitization processing module of data gateway device, according to the desensitization of the data filtering for this user rule arranged in rules administration module, carries out data filtering desensitization to service source data.Concrete data filtering desensitization can refer to aforementioned, repeats no more herein.
Step 10: the data desensitization processing module of data gateway device checks rule according to the conjunction rule arranged for this user in rules administration module, data are carried out to the service source data after data filtering desensitization and closes rule inspection, if exist and do not conform to rule data, then interrupt this service data peek flow process.
As previously mentioned, close rule inspection and can comprise data dictionary coupling, number range inspection and field length inspection.
Step 11: the data desensitization processing module of data gateway device is according to the output line number of the service data arranged for this user in rules administration module, to meet data close rule check rule, filter desensitization after service source data, from first trip, order intercepts the data of regulation line number, generates service data.
Step 12: service data is returned to user by the peek administration module of data gateway device, and carry out measuring, charging.
It should be noted that, the service data of real time service can return to user by web Service interface.The service data of cycle service can adopt data file mode to be pushed to the ftp server of user by FTP interface.
Step 13: the security audit module of data gateway device is regularly audited to Operation Log, peek daily record.
According to a further aspect in the invention, additionally provide the addressable large data communication system of a kind of client, as shown in Figure 4, it comprises data center and data gateway device, this data gateway device is by interface and data center's data cube computation, and this data gateway device is configured to by interface and client data communications.The data service request that client (representative data demand for services person) sends is sent to data center by data gateway device, the service source data that data center returns carries out allergy desensitization process and compliance inspection by data gateway device, then the service data meeting compliance inspection after desensitization is supplied to client, prevent leakage that is crucial and sensitive data, meet compliance.In addition, data gateway device provided by the invention can also measure or charging sending to the service data of client, and audits to these service datas.This data gateway device is above-mentioned data gateway device, therefore, no longer repeats it.
According to the flow direction of the data in large data system, barrier in data center's composition data of operator, data gateway device provided by the invention forms examination & verification district, it can ensure the fail safe of the open operation of data, services, the client of request msg service forms outer net Cooperative District, client can send various data service request, as shown in Figure 4.
Data gateway device (examination & verification district) can be connected by interface with between data center (in data barrier).In one example, data gateway device and data center by data service information synchronized update interface, peek and apply for that interface, service data generate notification interface and serve Data synchronization interface and realize data cube computation.
Data service information synchronized update interface can be realized by web Service interface.When the data service information that data center is open changes (such as, newdata service or Update Table service), data service information is initiatively pushed to data gateway device by web Service interface by data center.By this data service information synchronized update interface, the data, services that the information in the service management module of data gateway device and described data center open is consistent
Peek application interface also can pass through web Service interface implementation.After receiving the service data extraction application from client, data gateway device checks whether local service cache table exists asked data, if there is no then initiates the data retrieval request of this service to data center.If the COS of this service is real time service, then data center generates source data in real time and returns to data gateway device; If the COS of this service is cycle service, and the service account phase of asking is the history account phase, then the storage information (comprising the FTP address of such as file server, access port, login username, password, source data file store path, source data file name etc.) of this service source data is returned to data gateway device by data center; If the COS of this service is cycle service, and the service account phase of asking is the following account phase, then data center will return message that an account phase do not arrive to data gateway device, is meeting account phase condition and source data generates notification interface notification data gateway apparatus by service data after generating again.
Service data is generated notification interface and also can be realized by web Service interface.As aforementioned, after data gateway device to initiate the peek application of certain cycle service to data center, because the asked service account phase is the following account phase, data center meet account phase condition and source data generate after again by this interface notification data gateway device data ready, and the storage information (comprising the FTP address of such as file server, access port, login username, password, source data file store path, source data file name) of this service source data is supplied to data gateway device.
Service data sync cap can be realized by FTP interface, and data gateway device, according to the storage information of service source data, obtains service source data by FTP mode.
Data gateway device (examination & verification district) can be connected by interface with client (outer net Cooperative District).In one example, data gateway device and client apply for interface and serve data-pushing interface realizing data cube computation by peeking.
Peek application interface can be realized by web Service interface.Client submits to service data to extract application by web Service interface to data gateway device.If the COS of this service is real time service, then data gateway device is after source data being carried out to desensitization process, closing the examination & verification process such as rule check, service data is returned to client.
Service data is pushed interface and can be realized by FTP interface.After at data gateway device the source data of cycle service being carried out to the examination & verification process such as desensitization process and compliance inspection, push by service data the FTP that service data adopts data file mode to be pushed to data, services demander to specify by interface.Data, services demander can specify FTP when registering, that is, specify the propelling movement FTP join dependency information for receiving service data, it can comprise: FTP address, access port, login username, password, file store path etc.
Although be appreciated that and above-mentionedly describe the present invention for web Service interface and FTP interface, the present invention is not limited to this, and any interface that can realize Real Data Exchangs and cycle data read-write may be used to the present invention.
Be understandable that, the illustrative embodiments that above execution mode is only used to principle of the present invention is described and adopts, but the present invention is not limited thereto.For those skilled in the art, without departing from the spirit and substance in the present invention, can make various modification and improvement, these modification and improvement are also considered as protection scope of the present invention.

Claims (13)

1. a data gateway device, comprising:
User management module, it is for managing user information;
Service management module, it is for managing the information of the data, services opened to user;
Rules administration module, it advises inspection rule for arranging different conjunction for different user and different data, services;
Peek administration module, it, for the peek request of service receiving user and send, sends to data center the request of peeking accordingly, and accepts the source data that data center returns; And
Data desensitization processing module, its rule for arranging according to rules administration module, the source data returned data center is carried out conjunctions rule and is checked, thus generates service data, and described service data is meet the data of closing and advising inspection, wherein
The service data that described peek administration module also generates for the processing module that data desensitized is supplied to user.
2. data gateway device according to claim 1, described rules administration module is also for arranging different data filtering desensitization rules for different user and different data, services, and described data desensitization processing module carries out filtration desensitization for the source data returned data center, and then carries out the inspection of conjunction rule to the data of filtering desensitization.
3. data gateway device according to claim 1, wherein, described peek module is also for measuring and charging the service data exported.
4. data gateway device according to claim 3, wherein, described rules administration module is also for arranging the output line number of the service data of user, when satisfied total line number of closing the data that rule check is greater than the output line number of the service data of described setting, the output line number of service data of described data desensitization processing module also for arranging according to described rules administration module, to the satisfied data of closing rule and checking, from first trip, order intercepts the data of certain line number, generate described service data, the output line number of wherein said line number=service data.
5. data gateway device according to claim 1, wherein, described conjunction rule check that rule comprises: data dictionary coupling, number range inspection and field length inspection.
6. data gateway device according to claim 1, wherein, described desensitization rule of filtering comprises: the condition filter of the screening of service output field, field and field contents process.
7. the data gateway device according to any one of claim 1-6, also comprises:
Memory cell, the service data that its source data returned for data cached center and data desensitization processing module generate, wherein said peek administration module is configured to: first judge whether to exist in described memory cell the data that client asks, if buffer memory has been in the memory unit for the data of asking, then the buffer memory data of asking in the memory unit are supplied to client as service data, otherwise send to data center the request of peeking accordingly.
8. the data gateway device according to any one of claim 1-6, also comprises:
Security audit module, it is for recording Operation Log and peek daily record, and by Operation Log and peek log archive.
9. the data gateway device according to any one of claim 1-6, also comprises:
System management module, it is for the basic data of configuration-system, described basic data comprises data cycle and timer, and the described data cycle is the caching period of service data in data gateway device that the source data that returns of data center and described data desensitization processing module generate.
10. the addressable large data system of client, it comprises data center and data gateway device, described data gateway device is connected with described data center by interface, wherein said data gateway device is the data gateway device according to any one of claim 1-8, and described data gateway device be configured to can by interface and client's side link.
11. large data systems according to claim 10, wherein, described data gateway device and described data center by data service information synchronized update interface, peek and apply for that interface, service data generate notification interface and serve Data synchronization interface and realize data cube computation, wherein
By described information on services synchronized update interface, the data, services that the information of the data, services in the service management module of described data gateway device and described data center open is consistent;
By described peek application interface, the peek administration module of described data gateway device sends the peek request of service to described data center, and, when described service is real time service, by described peek, described data center applies for that interface returns source data to the peek administration module of described data gateway device, when described service is cycle service and the account phase of described service is the history account phase, described data center returns the storage information of institute's request source data to the peek administration module of described data gateway device by described peek application interface, when described service is cycle service and the account phase of described service is the non-vostro account phase, described data center returns the account phase by described peek application interface to the peek administration module of described gateway apparatus and does not arrive message,
Generate notification interface by described service data, described data center notifies the peek administration module institute request source data ready of described data gateway device and returns the storage information of institute's request source data; And
By described service data sync cap, the peek administration module of described data gateway device is according to the storage acquisition of information source data of institute's request source data.
12. large data systems according to claim 10 or 11, wherein, data gateway device is also configured to pass peek application interface and service data-pushing interface is connected with client data, wherein,
By peek application interface, the peek administration module of data gateway device accepts the peek request of the service that client sends, and the peek administration module of data gateway device provides asked service data to user;
Push interface by service data, asked service data is pushed to assigned address by the peek administration module of data gateway device.
13. large data systems according to claim 12, wherein, described data service information synchronized update interface, peek application interface, service data are generated notification interface and are realized by web Service interface, and described service data sync cap is by FTP Interface realization; Described peek application interface is realized by web Service interface, and described service data pushes interface by FTP Interface realization.
CN201510881918.6A 2015-12-03 2015-12-03 Data gateway device and big data system Pending CN105515963A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201510881918.6A CN105515963A (en) 2015-12-03 2015-12-03 Data gateway device and big data system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201510881918.6A CN105515963A (en) 2015-12-03 2015-12-03 Data gateway device and big data system

Publications (1)

Publication Number Publication Date
CN105515963A true CN105515963A (en) 2016-04-20

Family

ID=55723621

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201510881918.6A Pending CN105515963A (en) 2015-12-03 2015-12-03 Data gateway device and big data system

Country Status (1)

Country Link
CN (1) CN105515963A (en)

Cited By (22)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2016188325A1 (en) * 2015-11-27 2016-12-01 中兴通讯股份有限公司 Data charging method and apparatus
CN106371975A (en) * 2016-08-31 2017-02-01 国信优易数据有限公司 Automatic operation and maintenance early-warning method and system
CN106529329A (en) * 2016-10-11 2017-03-22 中国电子科技网络信息安全有限公司 Desensitization system and desensitization method used for big data
CN107194660A (en) * 2017-04-27 2017-09-22 合肥视尔信息科技有限公司 A kind of with no paper measures and procedures for the examination and approval immediately of vital document
CN107194661A (en) * 2017-04-27 2017-09-22 合肥视尔信息科技有限公司 A kind of with no paper document approvals system based on information network
CN108009435A (en) * 2017-12-18 2018-05-08 网智天元科技集团股份有限公司 Data desensitization method, device and storage medium
CN108154047A (en) * 2017-12-25 2018-06-12 网智天元科技集团股份有限公司 A kind of data desensitization method and device
CN108156195A (en) * 2016-12-02 2018-06-12 航天星图科技(北京)有限公司 A kind of business datum inspection method and system
CN108984625A (en) * 2018-06-19 2018-12-11 平安科技(深圳)有限公司 Information filtering method, device, computer equipment and storage medium
CN109417576A (en) * 2016-07-26 2019-03-01 思科技术公司 For providing the system and method for closing the transmission that rule require for cloud application
CN109583987A (en) * 2018-10-09 2019-04-05 阿里巴巴集团控股有限公司 A kind of processing method of data, device and equipment
CN109634836A (en) * 2018-10-23 2019-04-16 平安科技(深圳)有限公司 Test data packaging method, device, equipment and storage medium
CN109697368A (en) * 2017-10-20 2019-04-30 北京比邻共赢信息技术有限公司 Method, equipment and system that user information data safety uses, storage medium
WO2019114766A1 (en) * 2017-12-14 2019-06-20 中兴通讯股份有限公司 Data desensitising method, server, terminal, and computer-readable storage medium
CN110096625A (en) * 2019-05-14 2019-08-06 中国联合网络通信集团有限公司 Data close rule inspection method and device
CN110197083A (en) * 2019-06-05 2019-09-03 深圳市优网科技有限公司 Sensitive data desensitization system and processing method
CN110457330A (en) * 2019-08-21 2019-11-15 北京远舢智能科技有限公司 A kind of time series data management platform
CN111143880A (en) * 2019-12-27 2020-05-12 中电长城网际***应用有限公司 Data processing method and device, electronic equipment and readable medium
CN111506905A (en) * 2019-01-31 2020-08-07 百度在线网络技术(北京)有限公司 Data processing method, device, server and storage medium
CN112181957A (en) * 2020-09-08 2021-01-05 支付宝(杭州)信息技术有限公司 Archive data supervision processing method and device and electronic equipment
CN112732489A (en) * 2021-01-11 2021-04-30 上海上讯信息技术股份有限公司 Data desensitization method and device based on database virtualization
CN114285616A (en) * 2021-12-16 2022-04-05 上海商汤科技开发有限公司 Data transmission method and device, electronic equipment and storage medium

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7398311B2 (en) * 2000-07-10 2008-07-08 Oracle International Corporation Selective cache flushing in identity and access management systems
CN101986599A (en) * 2010-12-09 2011-03-16 北京交通大学 Network security control method based on cloud service and cloud security gateway
US20120259877A1 (en) * 2011-04-07 2012-10-11 Infosys Technologies Limited Methods and systems for runtime data anonymization
US20140047551A1 (en) * 2012-08-10 2014-02-13 Sekhar Nagasundaram Privacy firewall
US20140164405A1 (en) * 2012-12-12 2014-06-12 Institute For Information Industry Dynamic data masking method and database system
US20140337614A1 (en) * 2013-05-07 2014-11-13 Imperva, Inc. Selective modification of encrypted application layer data in a transparent security gateway
CN104699777A (en) * 2015-03-10 2015-06-10 中国联合网络通信集团有限公司 Association method and system of management plane and service plane of big data analysis and mining
CN105119956A (en) * 2015-07-09 2015-12-02 传成文化传媒(上海)有限公司 Network application system and disposition method

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7398311B2 (en) * 2000-07-10 2008-07-08 Oracle International Corporation Selective cache flushing in identity and access management systems
CN101986599A (en) * 2010-12-09 2011-03-16 北京交通大学 Network security control method based on cloud service and cloud security gateway
US20120259877A1 (en) * 2011-04-07 2012-10-11 Infosys Technologies Limited Methods and systems for runtime data anonymization
US20140047551A1 (en) * 2012-08-10 2014-02-13 Sekhar Nagasundaram Privacy firewall
US20140164405A1 (en) * 2012-12-12 2014-06-12 Institute For Information Industry Dynamic data masking method and database system
US20140337614A1 (en) * 2013-05-07 2014-11-13 Imperva, Inc. Selective modification of encrypted application layer data in a transparent security gateway
CN104699777A (en) * 2015-03-10 2015-06-10 中国联合网络通信集团有限公司 Association method and system of management plane and service plane of big data analysis and mining
CN105119956A (en) * 2015-07-09 2015-12-02 传成文化传媒(上海)有限公司 Network application system and disposition method

Cited By (31)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2016188325A1 (en) * 2015-11-27 2016-12-01 中兴通讯股份有限公司 Data charging method and apparatus
CN106817228A (en) * 2015-11-27 2017-06-09 中兴通讯股份有限公司 Data charging method and device
CN109417576A (en) * 2016-07-26 2019-03-01 思科技术公司 For providing the system and method for closing the transmission that rule require for cloud application
CN106371975A (en) * 2016-08-31 2017-02-01 国信优易数据有限公司 Automatic operation and maintenance early-warning method and system
CN106371975B (en) * 2016-08-31 2019-03-01 国信优易数据有限公司 A kind of O&M automation method for early warning and system
CN106529329A (en) * 2016-10-11 2017-03-22 中国电子科技网络信息安全有限公司 Desensitization system and desensitization method used for big data
CN108156195B (en) * 2016-12-02 2021-08-20 中科星图股份有限公司 Service data checking method and system
CN108156195A (en) * 2016-12-02 2018-06-12 航天星图科技(北京)有限公司 A kind of business datum inspection method and system
CN107194661A (en) * 2017-04-27 2017-09-22 合肥视尔信息科技有限公司 A kind of with no paper document approvals system based on information network
CN107194660A (en) * 2017-04-27 2017-09-22 合肥视尔信息科技有限公司 A kind of with no paper measures and procedures for the examination and approval immediately of vital document
CN109697368B (en) * 2017-10-20 2021-02-02 北京比邻共赢信息技术有限公司 Method, device and system for safe use of user information data and storage medium
CN109697368A (en) * 2017-10-20 2019-04-30 北京比邻共赢信息技术有限公司 Method, equipment and system that user information data safety uses, storage medium
WO2019114766A1 (en) * 2017-12-14 2019-06-20 中兴通讯股份有限公司 Data desensitising method, server, terminal, and computer-readable storage medium
CN109960944A (en) * 2017-12-14 2019-07-02 中兴通讯股份有限公司 A kind of data desensitization method, server, terminal and computer readable storage medium
CN108009435A (en) * 2017-12-18 2018-05-08 网智天元科技集团股份有限公司 Data desensitization method, device and storage medium
CN108154047A (en) * 2017-12-25 2018-06-12 网智天元科技集团股份有限公司 A kind of data desensitization method and device
CN108984625A (en) * 2018-06-19 2018-12-11 平安科技(深圳)有限公司 Information filtering method, device, computer equipment and storage medium
CN109583987A (en) * 2018-10-09 2019-04-05 阿里巴巴集团控股有限公司 A kind of processing method of data, device and equipment
CN109634836A (en) * 2018-10-23 2019-04-16 平安科技(深圳)有限公司 Test data packaging method, device, equipment and storage medium
CN111506905A (en) * 2019-01-31 2020-08-07 百度在线网络技术(北京)有限公司 Data processing method, device, server and storage medium
CN110096625A (en) * 2019-05-14 2019-08-06 中国联合网络通信集团有限公司 Data close rule inspection method and device
CN110197083A (en) * 2019-06-05 2019-09-03 深圳市优网科技有限公司 Sensitive data desensitization system and processing method
CN110457330A (en) * 2019-08-21 2019-11-15 北京远舢智能科技有限公司 A kind of time series data management platform
CN110457330B (en) * 2019-08-21 2022-09-13 北京远舢智能科技有限公司 Time sequence data management platform
CN111143880A (en) * 2019-12-27 2020-05-12 中电长城网际***应用有限公司 Data processing method and device, electronic equipment and readable medium
CN111143880B (en) * 2019-12-27 2022-06-07 中电长城网际***应用有限公司 Data processing method and device, electronic equipment and readable medium
CN112181957A (en) * 2020-09-08 2021-01-05 支付宝(杭州)信息技术有限公司 Archive data supervision processing method and device and electronic equipment
CN112181957B (en) * 2020-09-08 2024-04-12 支付宝(杭州)信息技术有限公司 File data supervision processing method and device and electronic equipment
CN112732489A (en) * 2021-01-11 2021-04-30 上海上讯信息技术股份有限公司 Data desensitization method and device based on database virtualization
CN112732489B (en) * 2021-01-11 2023-05-09 上海上讯信息技术股份有限公司 Data desensitization method and device based on database virtualization
CN114285616A (en) * 2021-12-16 2022-04-05 上海商汤科技开发有限公司 Data transmission method and device, electronic equipment and storage medium

Similar Documents

Publication Publication Date Title
CN105515963A (en) Data gateway device and big data system
US20190005210A1 (en) Centralized consent management
CN104809634B (en) Tourism data is investigated and monitoring system
CN110100429A (en) Real-time detection is simultaneously prevented from cheating and be abused
CN108701182A (en) The data management of multi-tenant identity cloud service
US7810145B2 (en) Distributed data consolidation network
CN109871690A (en) The management method and device of equipment permission, storage medium, electronic device
CN104573832A (en) Time-phased appointment registration system based on multi-hospital registration source sharing pool
CN109472605A (en) A kind of mathematic for business capitalization management method and system based on block chain
CN107465692A (en) Unification user identity identifying method, system and storage medium
CN107730430A (en) Production, teaching & research public service system based on big data
CN107798037A (en) The acquisition methods and server of user characteristic data
CN109189844A (en) A kind of inspection examining report business management system
CN104580446A (en) Cloud service real-time charging and managing system oriented to OpenStack open source cloud platform
CN108647357A (en) The method and device of data query
CN110751992A (en) Health card management platform
CN114329290A (en) Capability opening platform and authorized access method thereof
KR100494975B1 (en) Customer finance management method and system using screen scrapping
CN202033796U (en) Credit investigation assistance system for credit cards
CN114254881A (en) Data processing method and device, electronic equipment, readable storage medium and product
CN112732812A (en) Personal credit analysis method based on big data portrait
Moses et al. Information security management in German local government.
CN105869023A (en) System based on Internet-of-things M2M system and realizing method thereof
Oyeniran et al. A Robust National Centralized Database System for Identity Management and Security Control in Nigeria
KR20010091377A (en) Network-based Enterprise Resource Planning System and method

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
C10 Entry into substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication
RJ01 Rejection of invention patent application after publication

Application publication date: 20160420