CN116319068A - Method and system for improving penetrating data processing efficiency in strong isolation environment - Google Patents
Method and system for improving penetrating data processing efficiency in strong isolation environment Download PDFInfo
- Publication number
- CN116319068A CN116319068A CN202310523686.1A CN202310523686A CN116319068A CN 116319068 A CN116319068 A CN 116319068A CN 202310523686 A CN202310523686 A CN 202310523686A CN 116319068 A CN116319068 A CN 116319068A
- Authority
- CN
- China
- Prior art keywords
- data
- penetrated
- primary key
- penetrating
- intranet
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 230000000149 penetrating effect Effects 0.000 title claims abstract description 99
- 238000012545 processing Methods 0.000 title claims abstract description 54
- 238000000034 method Methods 0.000 title claims abstract description 47
- 238000002955 isolation Methods 0.000 title claims abstract description 44
- 238000004891 communication Methods 0.000 claims abstract description 31
- 230000003139 buffering effect Effects 0.000 claims abstract description 9
- 230000035515 penetration Effects 0.000 claims description 114
- 238000005192 partition Methods 0.000 claims description 5
- 230000005540 biological transmission Effects 0.000 claims 3
- 230000010365 information processing Effects 0.000 abstract description 2
- 238000005516 engineering process Methods 0.000 description 4
- 230000007547 defect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 230000007704 transition Effects 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/02—Network architectures or network communication protocols for network security for separating internal from external traffic, e.g. firewalls
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/14—Session management
- H04L67/141—Setup of application sessions
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computer Hardware Design (AREA)
- Computer Security & Cryptography (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a method and a system for improving the processing efficiency of penetrating data in a strong isolation environment, which relate to the technical field of information processing, wherein an intranet buffer queue is used for buffering data to be penetrated of an intranet service system, an intranet penetrating module is used for obtaining a plurality of pieces of data to be penetrated in the intranet buffer queue in batches, obtaining batch data and storing the batch data in a penetrating library, and an extranet penetrating module is used for penetrating a strong isolation device through a database communication protocol, obtaining the batch data from the penetrating library and transmitting the batch data to an extranet service system.
Description
Technical Field
The present invention relates to the field of information processing technologies, and in particular, to a method and a system for improving data processing efficiency under a strong isolation environment.
Background
Under the big environment that the information technology increasingly pays attention to internet security, more and more enterprises adopt a strong isolation device to logically and strongly isolate the internet (external network) from an enterprise internal information system (internal network), and only a database communication protocol is allowed to penetrate through the strong isolation device to access the enterprise internal information system, and any other protocol is forbidden to penetrate through the strong isolation device to access the enterprise internal information system. At present, when service data of an enterprise internal information system needs to be transmitted to the internet for further processing, such as sending a short message, sending a mail and the like, a penetrating module needs to penetrate through a strong isolation device to take out corresponding service data, then the penetrating strong isolation device marks the state of the corresponding service data as completed, and therefore the penetrating module needs to read one service data of the enterprise internal information system to the internet through a penetrating process for two times, and the processing efficiency of the service data to be penetrated is quite low. Under traditional solution, in order to realize high concurrency penetration of business data of the information system in the enterprise, the use of business needs to be supported by continuously expanding penetration resources, so that the traditional solution needs to increase investment in terms of penetration resources, and the problem of performance bottleneck of low penetration data processing efficiency cannot be solved with high efficiency.
Based on this, a technology for greatly improving the processing efficiency of the penetration data without expanding the penetration resources is needed.
Disclosure of Invention
The invention aims to provide a method and a system for improving the processing efficiency of penetrating data under a strong isolation environment, which can improve the processing efficiency of penetrating data on the premise of not expanding penetrating resources.
In order to achieve the above object, the present invention provides the following solutions:
a system for improving the efficiency of penetrating data processing in a strongly isolated environment, the system comprising:
an intranet buffer queue, which is in communication connection with an intranet service system and is used for buffering data to be penetrated of the intranet service system; the data to be penetrated is service data which is generated by the intranet service system and is required to be transmitted to the Internet for processing;
the intranet penetration module is in communication connection with the intranet cache queue and is used for obtaining a plurality of data to be penetrated in the intranet cache queue in batches to obtain batch data;
the penetration library is in communication connection with the intranet penetration module and is used for receiving and storing the batch data;
and the external network penetration module is used for penetrating the strong isolation device through a database communication protocol, acquiring the batch data from the penetration library and transmitting the batch data to an external network service system.
In some embodiments, the intranet penetration module is further configured to obtain, in batch, a plurality of data to be penetrated in the intranet cache queue when a preset time interval is set or the number of data to be penetrated in the intranet cache queue reaches a preset number, so as to obtain batch data.
In some embodiments, the penetration library is further configured to set a globally unique primary key for each data to be penetrated in the batch of data after receiving the batch of data, and store each data to be penetrated in the batch of data and the globally unique primary key corresponding to the data to be penetrated in a data table to be penetrated; the external network penetrating module is further configured to obtain the batch data from the data table to be penetrated according to the global unique primary key after penetrating the strong isolation device.
In some embodiments, the penetration library is further configured to divide the globally unique primary keys corresponding to all data to be penetrated in the batch of data into primary key value strings by a partition Fu Pinjie, store the primary key value strings in a primary key lock table, and set the state of the primary key value strings as unprocessed; the external network penetrating module is further configured to obtain the batch data from the data table to be penetrated according to the primary key value string in the primary key lock table, which is in an unprocessed state, after penetrating the strong isolation device.
In some embodiments, the penetration library is further configured to set an update trigger for the primary key table, create a temporary table when the state of the primary key string in the primary key table is updated to processed, and store the primary key string whose state is updated to processed in the temporary table; the external network penetrating module is further configured to update the state of the primary key value string in the primary key lock table, which is not processed, to processed after penetrating the strong isolation device, and obtain the batch data from the data table to be penetrated according to the primary key value string in the temporary table.
In some embodiments, the system further comprises an external network cache queue, the external network cache queue being communicatively connected to the external network penetration module and the external network service system, respectively; the external network buffer queue is used for buffering the data to be penetrated in the batch data acquired by the external network penetrating module; the external network service system is used for acquiring the data to be penetrated in the external network cache queue.
A method for improving the efficiency of penetrating data processing in a strongly isolated environment, the method comprising:
the intranet penetration module acquires a plurality of pieces of data to be penetrated in an intranet cache queue in batches, acquires batch data, and transmits the batch data to a penetration library for storage; the data to be penetrated is service data which is generated by an intranet service system and is required to be transmitted to the Internet for processing;
the external network penetrating module penetrates through the strong isolation device through a database communication protocol, acquires the batch data from the penetrating library, and transmits the batch data to an external network service system.
In some embodiments, the intranet penetration module obtains a plurality of data to be penetrated in the intranet cache queue in batches, and the obtaining the batch data specifically includes:
and when the interval preset time or the number of the data to be penetrated in the intranet cache queue reaches the preset number, the intranet penetration module acquires a plurality of data to be penetrated in the intranet cache queue in batches to obtain batch data.
In some embodiments, before the foreign network penetration module penetrates the strong isolation device via the database communication protocol, the method further comprises:
after the penetration library receives the batch data, setting a global unique primary key for each data to be penetrated in the batch data, and storing each data to be penetrated in the batch data and the global unique primary key corresponding to the data to be penetrated in a data table to be penetrated;
the penetration library forms a primary key value character string from global unique primary keys corresponding to all data to be penetrated in the batch of data through a partition Fu Pinjie, stores the primary key value character string into a primary key lock table provided with an update trigger, and sets the state of the primary key value character string as unprocessed; when the state of the primary key value character string in the primary key lock table is updated to be processed, the penetration library creates a temporary table, and stores the primary key value character string whose state is updated to be processed in the temporary table.
In some embodiments, the external network penetration module penetrates the strong isolation device through a database communication protocol, and the obtaining the batch data from the penetration library specifically includes:
the external network penetrating module penetrates through the strong isolation device through a database communication protocol, updates the state of the main key value character string which is in an unprocessed state in the main key lock table to be processed, and acquires the batch data from the data table to be penetrated according to the main key value character string in the temporary table.
According to the specific embodiment provided by the invention, the invention discloses the following technical effects:
the invention provides a method and a system for improving the processing efficiency of penetrating data in a strong isolation environment, an intranet buffer queue is used for buffering data to be penetrated of an intranet service system, an intranet penetrating module is used for obtaining a plurality of pieces of data to be penetrated in the intranet buffer queue in batches, obtaining batch data and storing the batch data in a penetrating library, and an extranet penetrating module is used for penetrating a strong isolation device through a database communication protocol, obtaining the batch data from the penetrating library and transmitting the batch data to an extranet service system.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are needed in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic diagram of a system for improving the efficiency of penetrating data processing in a strongly isolated environment according to embodiment 1 of the present invention;
FIG. 2 is a flow chart of a method for improving the efficiency of penetrating data processing in a strongly isolated environment according to embodiment 2 of the present invention;
FIG. 3 is a detailed flowchart of a method for improving the efficiency of penetrating data processing in a strongly isolated environment according to embodiment 2 of the present invention.
Symbol description:
1-an intranet service system; 2-an intranet cache queue; 3-an intranet penetration module; 4-penetrating libraries; 5-an extranet penetration module; 6-an external network cache queue; 7-extranet business system.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The invention aims to provide a method and a system for improving the processing efficiency of penetrating data under a strong isolation environment, which can improve the processing efficiency of penetrating data on the premise of not expanding penetrating resources.
In order that the above-recited objects, features and advantages of the present invention will become more readily apparent, a more particular description of the invention will be rendered by reference to the appended drawings and appended detailed description.
Example 1:
the present embodiment is used to provide a system for improving the processing efficiency of penetrating data in a strongly isolated environment, as shown in fig. 1, where the system includes:
the intranet buffer queue 2 is in communication connection with the intranet service system 1 and is used for buffering data to be penetrated of the intranet service system 1, wherein the data to be penetrated is service data which are generated by the intranet service system 1 and are required to be transmitted to the Internet for processing.
The intranet penetration module 3 is in communication connection with the intranet cache queue 2, and is used for obtaining a plurality of data to be penetrated in the intranet cache queue 2 in batches to obtain batch data.
The penetration library 4 is in communication connection with the intranet penetration module 3 and is used for receiving and storing batch data.
The external network penetration module 5 is configured to penetrate the strong isolation device through the database communication protocol, obtain batch data from the penetration library 4, and transmit the batch data to the external network service system 7.
The intranet service system 1, the intranet buffer queue 2, the intranet penetration module 3 and the penetration library 4 of the embodiment are all arranged in the information intranet, namely belong to a part of the enterprise internal information system. If the data to be penetrated is to be transmitted to the internet for further processing, the data to be penetrated needs to be transmitted from the intranet service system 1 to the penetrating library 4 for storage, in the prior art, the intranet service system 1 and the penetrating library 4 are directly connected in communication, and the data to be penetrated can be directly transmitted from the intranet service system 1 to the penetrating library 4 for storage, but the problem that the intranet service system 1 excessively depends on the performance of the penetrating library 4 exists in the way, so that the intranet service system 1 is influenced by the penetrating library 4, and the complexity and the instability of the system are increased. In this embodiment, the intranet buffer queue 2 and the intranet penetration module 3 are additionally disposed between the intranet service system 1 and the penetration library 4, and the intranet buffer queue 2 and the intranet penetration module 3 may be mutually matched to transmit data to be penetrated in the intranet service system 1 to the penetration library 4 for storage, so that the data buffer function of peak clipping and valley filling can be realized, that is, no matter how much data to be penetrated is generated by the intranet service system 1, the data to be penetrated can be uniformly formed into batch data through the intranet buffer queue 2 and the intranet penetration module 3 and stored in the penetration library 4, and the function of isolating the intranet service system 1 from the penetration library 4 can be also achieved, the influence of the penetration library 4 on the intranet service system 1 is avoided, and the system stability is improved.
According to the embodiment, the intranet buffer queue 2 and the intranet penetration module 3 are arranged and can be matched with each other to process to-be-penetrated data of the intranet service system 1, a plurality of to-be-penetrated data are formed into batch data and are stored in the penetration library 4 at the same time, and when the external network penetration module 5 obtains data from the penetration library 4 through the database communication protocol penetration strong isolation device, the batch data can be directly obtained, so that the number of to-be-penetrated data obtained at one time is greatly increased, and the processing efficiency of the penetration data is remarkably improved.
In this embodiment, an intranet penetration module 3 is disposed between the intranet buffer queue 2 and the penetration library 4, the intranet penetration module 3 can read a plurality of data to be penetrated in the intranet buffer queue 2 in batches, all the data to be penetrated read in batches form a batch of data, and the intranet penetration module 3 stores the batch of data in the penetration library 4. Specifically, the working mechanism of the intranet penetration module 3 may refer to the consumption mode of the queue (various large-batch data are often processed and put in storage in the actual development process, the queue is used at this time, the data are written into the queue first, then a plurality of consumption processes are started to slowly consume and put in storage, and the consumption of the data from the queue is two ways, namely, single consumption and batch consumption), the intranet penetration module 3 in this embodiment includes a plurality of consumption processes, and the batch consumption mode is adopted to form batch data from the data to be penetrated in the intranet cache queue 2 and store the batch data in the penetration library 4. More specifically, the principle that the intranet penetration module 3 gathers the data to be penetrated is that the time expires or the quantity reaches a batch limit, that is, the intranet penetration module 3 is configured to obtain a plurality of data to be penetrated in the intranet cache queue 2 in batches when the interval preset time or the quantity of the data to be penetrated in the intranet cache queue 2 reaches the preset quantity, so as to obtain batch data. The preset time may be 1s, the preset number may be 100, that is, when the number of data to be penetrated in the intranet buffer queue 2 reaches 100, the intranet penetration module 3 may obtain the 100 data to be penetrated in the intranet buffer queue 2 in batches, the 100 data to be penetrated form batch data, or the number of data to be penetrated in the intranet buffer queue 2 is 10, which is smaller than the preset number (100), but at this time, the time from the last batch obtaining of the intranet penetration module 3 has been 1s, then the intranet penetration module 3 may also obtain the 10 data to be penetrated in the intranet buffer queue 2 in batches, and the 10 data to be penetrated form batch data.
The penetration library 4 in this embodiment is additionally provided with a data table to be penetrated, where the data table to be penetrated includes two fields, namely a data field to be penetrated and a global unique primary key field, and by additionally adding a global unique primary key field outside the data field to be penetrated, the service support capability of the external network penetration module 5 when the external network penetration module obtains the data to be penetrated in multiple processes can be ensured, and the external network penetration module 5 in this embodiment can include multiple instance modules, each instance module has multiple processing processes, and the multiple instance modules and the multiple processing processes can work simultaneously, that is, each processing process of each instance module can work independently to obtain batch data. At this time, the intranet penetration module 3 transmits the batch data to the penetration library 4, and specifically may be inserted into a data table to be penetrated in the penetration library 4 for storage, when inserting a batch data, the penetration library 4 sets a global unique primary key for each data to be penetrated in the batch data, specifically, the penetration library 4 is configured to set a global unique primary key for each data to be penetrated in the batch data after receiving the batch data, and store each data to be penetrated in the batch data and the global unique primary key corresponding to the data to be penetrated in the data table to be penetrated, where a row of data in the data table to be penetrated corresponds to one piece of data to be penetrated, including the data to be penetrated and the global unique primary key corresponding to the data to be penetrated. By setting the data table to be penetrated and setting the global unique main key for each data to be penetrated, after the external network penetrating module 5 penetrates through the strong isolation device, batch data can be obtained from the data table to be penetrated according to the global unique main key, so that batch data can be quickly obtained, and the processing efficiency of penetrating data is further improved.
Furthermore, a primary key lock table may be additionally provided in the penetrating library 4 in this embodiment, where the primary key lock table mainly includes two fields, namely a primary key value string field and a status field, and the length of the primary key value string field may at least accommodate a string formed by splicing a preset number of globally unique primary keys. At this time, after setting a global unique primary key for each data to be penetrated in the batch of data and inserting each data to be penetrated and the corresponding global unique primary key in the batch of data into the data table to be penetrated, the penetration library 4 will collect the global unique primary keys of all the data to be penetrated in the batch of data once, splice the global unique primary keys corresponding to all the data to be penetrated in the collected batch of data into a primary key value string through separators (such as commas, semicolons, etc.), store the primary key value string into the primary key value string field of the primary key lock table, and set the state of the primary key value string as unprocessed, i.e. set the state field corresponding to the primary key value string as unprocessed (or to be processed). By setting the primary key lock table and setting a state for each primary key value character string in the primary key lock table, after the external network penetrating module 5 penetrates through the strong isolation device, it can be determined according to the states of the primary key value character strings in the primary key lock table which are not yet processed, or which primary key value character strings corresponding batch data are not yet obtained, further batch data can be obtained from the data table to be penetrated according to the primary key value character strings in the primary key lock table, specifically, the primary key value character strings in the primary key lock table which are not processed are analyzed, all global unique primary keys included in the primary key value character strings are determined, and batch data are obtained from the data table to be penetrated according to all global unique primary keys, so that whether the batch data are processed can be rapidly distinguished, only the unprocessed batch data are obtained, and the processing efficiency of the penetrating data is further improved. After the batch data is obtained, the state of the primary key value character string corresponding to the batch data is updated to be processed, so that the same batch data is prevented from being repeatedly processed.
After receiving a batch of data, the penetration library 4 of this embodiment sets a global unique primary key for each data to be penetrated in the batch of data, stores all the data to be penetrated in the batch of data and the corresponding global unique primary key into the data table to be penetrated, at this time, the data table to be penetrated in the penetration library 4 will increase a batch of data, and at the same time, the primary key value strings are assembled based on the global unique primary keys of all the data to be penetrated in the batch of data, and store the primary key value strings into the primary key lock table, and set the state corresponding to the primary key value strings as unprocessed, at this time, the primary key lock table in the penetration library 4 will increase a record. After the external network penetrating module 5 obtains a batch of data, the state of the primary key value character string corresponding to the batch of data in the primary key lock table is updated to be processed, so that repeated processing is avoided.
Of course, the primary key table of the present embodiment may further include other penetration auxiliary fields, such as a process number field, when the external network penetration module 5 obtains a batch of data, the penetration library 4 not only updates the status of the primary key string corresponding to the batch of data in the primary key table to processed, but also fills the process number of the external network penetration module 5 obtaining the batch of data in the process number field corresponding to the primary key string updated to processed in the primary key table, so as to determine which process of the external network penetration module 5 obtains the batch of data.
If the batch data is obtained according to the primary key value string, and then the state of the primary key value string is updated to be processed, the problem that when the processing process A obtains the batch data according to the primary key value string B, the processing process C finds that the state of the primary key value string B is unprocessed, and then obtains the batch data according to the primary key value string B, and unread reading is generated when the processing process A obtains the data according to the primary key value string B. To avoid this problem, the present embodiment may further add an update trigger to the primary key table, creating a temporary table when data is updated, and inserting the updated data into the temporary table. Specifically, the penetrating library 4 of the present embodiment is further configured to set an update trigger for the primary key table, and when the state of the primary key string in the primary key table is updated to be processed, create a temporary table, and store the primary key string whose state is updated to be processed in the temporary table. The external network penetrating module 5 is configured to update the state of the primary key value string in the primary key lock table, which is not processed, to processed after penetrating the strong isolation device, and then generate a temporary table based on the update trigger, where the temporary table stores the primary key value string whose state is updated to processed, and at this time, the external network penetrating module 5 obtains batch data from the data table to be penetrated according to the primary key value string in the temporary table. By setting the update trigger, the external network penetrating module 5 will first try to update the status field of the primary key lock table in the penetrating library 4 to the status of the primary key value character string to be processed, after the update is successful, the update trigger of the primary key lock table will transfer the primary key value character string data of the updated status to the temporary table, at this time, the external network penetrating module 5 will read the batch data from the data table to be penetrated based on the temporary table, thus completing the status update process before reading the batch data, avoiding the problem that two processing processes process one batch data at the same time, and solving the problem of unread reading when multiple connections acquire data.
Specifically, the process of the external network penetration module 5 for reading batch data from the data table to be penetrated based on the temporary table may include: the external network penetration module 5 obtains a group of globally unique primary keys of the data to be penetrated which can be penetrated by reading the primary key value character strings in the temporary table and analyzing the primary key value character strings according to the separator, namely, dividing the primary key value character strings according to the separator in a character string dividing mode, and further obtains a batch of data to be penetrated from the data to be penetrated in the penetration library 4 according to the globally unique primary keys.
According to the embodiment, by arranging the module, the single piece of data to be penetrated can be successfully combined into a batch of data, meanwhile, the strong isolation device is penetrated, the data is transmitted from the inner network to the outer network for further processing, the processing efficiency of the penetrated data is greatly improved, the problems of unread reading, misreading and the like of the data can be avoided, and the accuracy of data penetration is ensured.
The penetration library 4 of the present embodiment may be a relational database, such as Mysql, under which an update trigger needs to be created, and when the primary key table data is updated, the updated data in the primary key table is triggered to enter the temporary table, and then the temporary table is acquired. The penetrating library 4 of this embodiment may also be Oracle, where Oracle's advanced feature (advanced queue) AQ may be directly used to replace the data table to be penetrated, the master key lock table and the update trigger, where the obtained batch data are directly stored in the queue according to the sequence, the external network penetrating module 5 reads the batch data in the queue according to the sequence, and the advanced queue of Oracle can ensure the service consistency of data obtaining.
Furthermore, in this embodiment, an external network buffer queue 6 may be additionally disposed between the external network service system 7 and the external network penetrating module 5, the external network buffer queue 6 and the external network service system 7 all belong to a part of the information external network. Namely, the system of this embodiment further includes an external network buffer queue 6, the external network buffer queue 6 is respectively in communication connection with the external network penetrating module 5 and the external network service system 7, after the external network penetrating module 5 obtains bulk data, the bulk data is transmitted to the external network buffer queue 6 for buffering, that is, the external network buffer queue 6 is used for buffering the data to be penetrated in the bulk data obtained by the external network penetrating module 5, so as to achieve the purpose of placing the data to be penetrated into the external network buffer queue 6 for further processing by the external network service system 7, and the external network service system 7 is used for obtaining the data to be penetrated in the external network buffer queue 6 for further processing the data to be penetrated. For example, when the data to be penetrated is a short message, the short message is sent to an operator for sending the short message.
When processing high concurrency, short transaction and consumer data, as only one piece of data to be penetrated can be penetrated at a time, the data penetrating efficiency is low, namely the penetrating efficiency of the data to be penetrated is extremely low, so that the data to be penetrated is not timely processed, and in the utilization of the existing strong isolation technology, the problem that unstructured and consumption queue type data (batch data) cannot be obtained through a database communication protocol by internet requests exists. According to the method, the external network penetrating module 5 can obtain the data to be penetrated in the penetrating library 4 simultaneously (namely, a plurality of processing processes of the external network penetrating module 5 can work simultaneously), and one batch of data can be obtained by one penetrating, so that the data penetrating efficiency is improved. In this embodiment, under the condition that the strong isolation penetration security model is not changed (i.e., the strong isolation device is not cancelled), and the system architecture of the database protocol penetration mode is not changed (i.e., the database communication protocol is still adopted for penetration), the penetration resource utilization efficiency is greatly improved, and the resource investment under the equal concurrent support is reduced.
Compared with the prior art, the advantages of this embodiment are as follows: the access control method for relieving the transition dependence on the strong isolation performance is provided, and by setting the intranet buffer queue 2 and the intranet penetration module 3, the method avoids the influence of the penetration library 4 on the intranet service system 1, overcomes the defects of the penetration library 4 serving as a key data dump scheme, and has the characteristics of high performance, safety, better service continuity experience and the like. Meanwhile, a method for improving the utilization efficiency of the strong isolation device is provided, the data penetration concurrency capacity is improved under the condition of not increasing resources, the data penetration speed is accelerated, and the service supporting capacity of a related system is improved.
Example 2:
the present embodiment is used to provide a method for improving the processing efficiency of penetrating data in a strongly isolated environment, as shown in fig. 2 and fig. 3, where the method includes:
s1: the intranet penetration module acquires a plurality of pieces of data to be penetrated in an intranet cache queue in batches, acquires batch data, and transmits the batch data to a penetration library for storage; the data to be penetrated is service data which is generated by an intranet service system and is required to be transmitted to the Internet for processing;
in S1, the obtaining, by the intranet penetration module, the plurality of data to be penetrated in the intranet cache queue in batches, where obtaining the batch data may include: and when the interval preset time or the number of the data to be penetrated in the intranet cache queue reaches the preset number, the intranet penetration module acquires a plurality of data to be penetrated in the intranet cache queue in batches to obtain batch data.
S2: the external network penetrating module penetrates through the strong isolation device through a database communication protocol, acquires the batch data from the penetrating library, and transmits the batch data to an external network service system.
Prior to S2, the method of this embodiment further includes:
after the penetration library receives the batch data, setting a global unique primary key for each data to be penetrated in the batch data, and storing each data to be penetrated in the batch data and the global unique primary key corresponding to the data to be penetrated in a data table to be penetrated;
the penetration library forms a primary key value character string from global unique primary keys corresponding to all data to be penetrated in the batch of data through a partition Fu Pinjie, stores the primary key value character string into a primary key lock table provided with an update trigger, and sets the state of the primary key value character string as unprocessed; when the state of the primary key value character string in the primary key lock table is updated to be processed, the penetration library creates a temporary table, and stores the primary key value character string whose state is updated to be processed in the temporary table.
Based on the above means, in S2, the external network penetration module penetrates the strong isolation device through the database communication protocol, and the obtaining the batch data from the penetration library may include: the external network penetrating module penetrates through the strong isolation device through a database communication protocol, updates the state of the main key value character string which is in an unprocessed state in the main key lock table to be processed, and acquires the batch data from the data table to be penetrated according to the main key value character string in the temporary table.
In this specification, each embodiment is mainly described in the specification as a difference from other embodiments, and the same similar parts between the embodiments are referred to each other.
The principles and embodiments of the present invention have been described herein with reference to specific examples, the description of which is intended only to assist in understanding the methods of the present invention and the core ideas thereof; also, it is within the scope of the present invention to be modified by those of ordinary skill in the art in light of the present teachings. In view of the foregoing, this description should not be construed as limiting the invention.
Claims (10)
1. A system for improving the efficiency of penetrating data processing in a strongly isolated environment, the system comprising:
an intranet buffer queue, which is in communication connection with an intranet service system and is used for buffering data to be penetrated of the intranet service system; the data to be penetrated is service data which is generated by the intranet service system and is required to be transmitted to the Internet for processing;
the intranet penetration module is in communication connection with the intranet cache queue and is used for obtaining a plurality of data to be penetrated in the intranet cache queue in batches to obtain batch data;
the penetration library is in communication connection with the intranet penetration module and is used for receiving and storing the batch data;
and the external network penetration module is used for penetrating the strong isolation device through a database communication protocol, acquiring the batch data from the penetration library and transmitting the batch data to an external network service system.
2. The system for improving the processing efficiency of penetrating data in a strong isolation environment according to claim 1, wherein the intranet penetrating module is further configured to obtain a plurality of data to be penetrated in the intranet cache queue in batches when a preset time is spaced or the number of data to be penetrated in the intranet cache queue reaches a preset number, so as to obtain batch data.
3. The system for improving the processing efficiency of penetrating data in a strongly isolated environment according to claim 1, wherein the penetrating library is further configured to set a globally unique primary key for each data to be penetrated in the batch of data after receiving the batch of data, and store each data to be penetrated in the batch of data and the globally unique primary key corresponding to the data to be penetrated into a data table to be penetrated; the external network penetrating module is further configured to obtain the batch data from the data table to be penetrated according to the global unique primary key after penetrating the strong isolation device.
4. The system for improving the processing efficiency of penetrating data in a strongly isolated environment according to claim 3, wherein the penetrating library is further configured to divide a globally unique primary key corresponding to all data to be penetrated in the batch of data into a primary key value string by a partition Fu Pinjie, store the primary key value string into a primary key lock table, and set a state of the primary key value string as unprocessed; the external network penetrating module is further configured to obtain the batch data from the data table to be penetrated according to the primary key value string in the primary key lock table, which is in an unprocessed state, after penetrating the strong isolation device.
5. The system for improving efficiency of pass-through data processing in a strongly isolated environment of claim 4, wherein the pass-through library is further configured to set an update trigger for the primary key table, create a temporary table when the status of the primary key string in the primary key table is updated to processed, and store the primary key string whose status is updated to processed in the temporary table; the external network penetrating module is further configured to update the state of the primary key value string in the primary key lock table, which is not processed, to processed after penetrating the strong isolation device, and obtain the batch data from the data table to be penetrated according to the primary key value string in the temporary table.
6. The system for improving the processing efficiency of penetrating data in a strongly isolated environment according to claim 1, further comprising an external network cache queue, wherein the external network cache queue is communicatively connected to the external network penetrating module and the external network service system, respectively; the external network buffer queue is used for buffering the data to be penetrated in the batch data acquired by the external network penetrating module; the external network service system is used for acquiring the data to be penetrated in the external network cache queue.
7. A method for improving the efficiency of penetrating data processing in a strongly isolated environment, the method comprising:
the intranet penetration module acquires a plurality of pieces of data to be penetrated in an intranet cache queue in batches, acquires batch data, and transmits the batch data to a penetration library for storage; the data to be penetrated is service data which is generated by an intranet service system and is required to be transmitted to the Internet for processing;
the external network penetrating module penetrates through the strong isolation device through a database communication protocol, acquires the batch data from the penetrating library, and transmits the batch data to an external network service system.
8. The method for improving the processing efficiency of penetrating data in a strong isolation environment according to claim 7, wherein the intranet penetrating module obtains a plurality of data to be penetrated in the intranet buffer queue in batches, and the obtaining of the batch data specifically includes:
and when the interval preset time or the number of the data to be penetrated in the intranet cache queue reaches the preset number, the intranet penetration module acquires a plurality of data to be penetrated in the intranet cache queue in batches to obtain batch data.
9. The method for improving the efficiency of pass-through data processing in a strongly isolated environment of claim 7, wherein before the outside network pass-through module passes through the strongly isolated device via the database communication protocol, the method further comprises:
after the penetration library receives the batch data, setting a global unique primary key for each data to be penetrated in the batch data, and storing each data to be penetrated in the batch data and the global unique primary key corresponding to the data to be penetrated in a data table to be penetrated;
the penetration library forms a primary key value character string from global unique primary keys corresponding to all data to be penetrated in the batch of data through a partition Fu Pinjie, stores the primary key value character string into a primary key lock table provided with an update trigger, and sets the state of the primary key value character string as unprocessed; when the state of the primary key value character string in the primary key lock table is updated to be processed, the penetration library creates a temporary table, and stores the primary key value character string whose state is updated to be processed in the temporary table.
10. The method for improving the processing efficiency of the transmission data in the strongly isolated environment according to claim 9, wherein the external network transmission module transmits the strongly isolated device through a database communication protocol, and the obtaining the batch data from the transmission library specifically comprises:
the external network penetrating module penetrates through the strong isolation device through a database communication protocol, updates the state of the main key value character string which is in an unprocessed state in the main key lock table to be processed, and acquires the batch data from the data table to be penetrated according to the main key value character string in the temporary table.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310523686.1A CN116319068B (en) | 2023-05-11 | 2023-05-11 | Method and system for improving penetrating data processing efficiency in strong isolation environment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310523686.1A CN116319068B (en) | 2023-05-11 | 2023-05-11 | Method and system for improving penetrating data processing efficiency in strong isolation environment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116319068A true CN116319068A (en) | 2023-06-23 |
| CN116319068B CN116319068B (en) | 2023-08-08 |
Family
ID=86826072
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310523686.1A Active CN116319068B (en) | 2023-05-11 | 2023-05-11 | Method and system for improving penetrating data processing efficiency in strong isolation environment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116319068B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101697536A (en) * | 2009-10-16 | 2010-04-21 | 深圳市科陆电子科技股份有限公司 | Method for transmitting mass data in real time by penetrating positive physical isolation device |
| US20100280991A1 (en) * | 2009-05-01 | 2010-11-04 | International Business Machines Corporation | Method and system for versioning data warehouses |
| US20140351924A1 (en) * | 2013-05-21 | 2014-11-27 | Verizon Patent And Licensing Inc. | Method and system for providing limited secure access to sensitive data |
| CN110555041A (en) * | 2018-03-30 | 2019-12-10 | 腾讯科技（深圳）有限公司 | Data processing method, data processing device, computer equipment and storage medium |
| US20210334239A1 (en) * | 2020-04-27 | 2021-10-28 | Richard Banister | System and Method for Re-Synchronizing a Portion of or an Entire Source Database and a Target Database |
| CN115460281A (en) * | 2022-09-16 | 2022-12-09 | 福建亿榕信息技术有限公司 | To-do service data penetration and push method and system based on network interaction |
-
2023
- 2023-05-11 CN CN202310523686.1A patent/CN116319068B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100280991A1 (en) * | 2009-05-01 | 2010-11-04 | International Business Machines Corporation | Method and system for versioning data warehouses |
| CN101697536A (en) * | 2009-10-16 | 2010-04-21 | 深圳市科陆电子科技股份有限公司 | Method for transmitting mass data in real time by penetrating positive physical isolation device |
| US20140351924A1 (en) * | 2013-05-21 | 2014-11-27 | Verizon Patent And Licensing Inc. | Method and system for providing limited secure access to sensitive data |
| CN110555041A (en) * | 2018-03-30 | 2019-12-10 | 腾讯科技（深圳）有限公司 | Data processing method, data processing device, computer equipment and storage medium |
| US20210334239A1 (en) * | 2020-04-27 | 2021-10-28 | Richard Banister | System and Method for Re-Synchronizing a Portion of or an Entire Source Database and a Target Database |
| CN115460281A (en) * | 2022-09-16 | 2022-12-09 | 福建亿榕信息技术有限公司 | To-do service data penetration and push method and system based on network interaction |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116319068B (en) | 2023-08-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108076098B (en) | Service processing method and system | |
| EP3120261B1 (en) | Dependency-aware transaction batching for data replication | |
| US5864851A (en) | Method and system for managing replicated data with enhanced consistency and concurrency | |
| US20080059469A1 (en) | Replication Token Based Synchronization | |
| US5812773A (en) | System and method for the distribution of hierarchically structured data | |
| US20070226269A1 (en) | Method and system for an update synchronization of a domain information file | |
| CN110908906B (en) | Regression testing method and system | |
| KR20060079078A (en) | Server queuing system and method | |
| CN114116665B (en) | Method for writing transaction log in parallel in database to promote processing efficiency | |
| CN111563095A (en) | Data retrieval device based on HBase | |
| CN110737710A (en) | Distributed data automatic structured warehousing method and system | |
| CN115562676B (en) | Triggering method of graph calculation engine | |
| US20050262432A1 (en) | Systems and methods for providing simultaneous access to documents | |
| CN111159020B (en) | Method and device applied to synchronous software test | |
| CN116319068B (en) | Method and system for improving penetrating data processing efficiency in strong isolation environment | |
| US20040181510A1 (en) | System and method for cooperative database acceleration | |
| CN116150263B (en) | Distributed graph calculation engine | |
| US20220365914A1 (en) | Moving Window Data Deduplication in Distributed Storage | |
| CN112395294A (en) | Database data management method and system and database | |
| CN112269820A (en) | Method and platform for realizing data batch synchronization based on CMSP | |
| CN107102901A (en) | A kind of task processing method and device | |
| CN112559628A (en) | Multi-cluster message synchronization method, device, medium and electronic equipment | |
| US5961609A (en) | Extensible test method and apparatus useful for rendered controls | |
| CN113590695B (en) | Data transmission method, system and terminal based on trigger and primary key | |
| CN117331513B (en) | Data reduction method and system based on Hadoop architecture |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |