CN117271571A - Database uniqueness constraint processing method, device, equipment and storage medium - Google Patents
Database uniqueness constraint processing method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN117271571A CN117271571A CN202311255229.5A CN202311255229A CN117271571A CN 117271571 A CN117271571 A CN 117271571A CN 202311255229 A CN202311255229 A CN 202311255229A CN 117271571 A CN117271571 A CN 117271571A
- Authority
- CN
- China
- Prior art keywords
- target
- data
- database
- bloom filter
- value
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003860 storage Methods 0.000 title claims abstract description 33
- 238000003672 processing method Methods 0.000 title claims description 10
- 238000012545 processing Methods 0.000 claims abstract description 57
- 238000000034 method Methods 0.000 claims abstract description 46
- 238000012795 verification Methods 0.000 claims abstract description 43
- 230000015654 memory Effects 0.000 claims abstract description 35
- 230000004044 response Effects 0.000 claims abstract description 16
- 230000006870 function Effects 0.000 claims description 56
- 238000010586 diagram Methods 0.000 description 16
- 230000008569 process Effects 0.000 description 10
- 238000004590 computer program Methods 0.000 description 9
- 238000003780 insertion Methods 0.000 description 8
- 230000037431 insertion Effects 0.000 description 8
- 230000008676 import Effects 0.000 description 6
- 238000013524 data verification Methods 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 4
- 238000004364 calculation method Methods 0.000 description 3
- 230000001360 synchronised effect Effects 0.000 description 3
- 230000006978 adaptation Effects 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 238000007726 management method Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 101150096185 PAAS gene Proteins 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000013475 authorization Methods 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 230000002688 persistence Effects 0.000 description 1
- 238000004904 shortening Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2453—Query optimisation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
- G06F16/2255—Hash tables
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application provides a method, a device, equipment and a storage medium for processing database uniqueness constraint, wherein the method comprises the following steps: acquiring data to be inserted, and determining a target model corresponding to the data to be inserted and a target bloom filter corresponding to the target model; determining a target constraint field corresponding to the target model, and determining target data of the data to be inserted in the target constraint field; verifying the target data based on the target bloom filter; in the case that the target data passes the verification, the data to be inserted including the target data is stored in the database. According to the method and the device, the target data to be inserted into the data target constraint field is verified through the target bloom filter, a large amount of data of the unique constraint field is not required to be stored in the memory or the disk, occupation of storage space can be reduced, meanwhile, the query speed of the target bloom filter is higher, query efficiency can be effectively improved, and response time of a service is shortened.
Description
Technical Field
The present disclosure relates to the field of data processing, and in particular, to a method, an apparatus, a device, and a storage medium for processing a database uniqueness constraint.
Background
With the continuous development of internet technology, online services are increasingly widely applied, and data generated by corresponding online services are also increasingly more. When data is stored in a database, the data value under part of fields must be unique due to different requirements of services, so that the data needs to be subjected to unique constraint processing, and the data is prevented from being repeated.
However, since the database itself does not have the capability of unique constraint, in related art, the unique constraint field is usually stored in a memory or a disk, and query verification is performed when data is stored, and this way occupies a larger storage space when the data volume is larger, and the query speed is slower, so that the response time of online service is too long.
Disclosure of Invention
The application provides a method, a device, equipment and a storage medium for processing database uniqueness constraint, which are used for reducing the occupation of storage space for processing the database uniqueness constraint, improving the query speed and shortening the response time of a service.
In a first aspect, the present application provides a method for processing a database uniqueness constraint, including:
acquiring data to be inserted, and determining a target model corresponding to the data to be inserted and a target bloom filter corresponding to the target model;
Determining a target constraint field corresponding to the target model, and determining target data of the data to be inserted in the target constraint field;
verifying the target data based on the target bloom filter;
and storing the data to be inserted comprising the target data into a database under the condition that the target data passes verification.
In one possible implementation, the verifying the target data based on the target bloom filter includes:
determining a first value of the target data on a preset number of hash functions;
determining a second value corresponding to the first value in a binary number group based on the first value;
and under the condition that the second value comprises a target value, determining that the target data passes verification.
In one possible embodiment, the method further comprises:
under the condition that the second value does not comprise the target value, inquiring and verifying the target data in a database based on a preset batch inquiring function;
and determining that the target data passes verification under the condition that the target data is not included in the database.
In a possible implementation manner, after the storing the data to be inserted including the target data in a database, the method further includes:
Updating the target bloom filter based on the target data.
In one possible implementation, the updating the target bloom filter based on the target data includes:
acquiring first values of the target data on a preset number of hash functions;
and updating the second value corresponding to the first value into a non-target value in the binary number group of the target bloom filter.
In a possible implementation manner, before the acquiring the data to be inserted, the method further includes:
in response to a configuration operation of a user, a target model corresponding to the configuration operation is created, a target constraint field corresponding to the target model is determined, and a target bloom filter corresponding to the target model is created.
In one possible embodiment, the method further comprises:
in an initial state, determining each target model corresponding to the database and a target constraint field corresponding to the target model;
aiming at each target model, acquiring the existing data of the target model on the target constraint field in the database;
and adding the existing data into a target bloom filter corresponding to the target model.
In a second aspect, the present application provides a database uniqueness constraint processing apparatus, including:
the acquisition module is used for acquiring data to be inserted, and determining a target model corresponding to the data to be inserted and a target bloom filter corresponding to the target model;
the determining module is used for determining a target constraint field corresponding to the target model and determining target data of the data to be inserted in the target constraint field;
the verification module is used for verifying the target data based on the target bloom filter;
and the storage module is used for storing the data to be inserted comprising the target data into a database under the condition that the target data passes verification.
In a possible implementation manner, the verification module is specifically configured to:
determining a first value of the target data on a preset number of hash functions;
determining a second value corresponding to the first value in a binary number group based on the first value;
and under the condition that the second value comprises a target value, determining that the target data passes verification.
In a possible implementation manner, the verification module is further specifically configured to:
Under the condition that the second value does not comprise the target value, inquiring and verifying the target data in a database based on a preset batch inquiring function;
and determining that the target data passes verification under the condition that the target data is not included in the database.
In one possible embodiment, the apparatus is further for:
updating the target bloom filter based on the target data.
In one possible embodiment, the apparatus is further for:
acquiring first values of the target data on a preset number of hash functions;
and updating the second value corresponding to the first value into a non-target value in the binary number group of the target bloom filter.
In one possible embodiment, the apparatus is further for:
in response to a configuration operation of a user, a target model corresponding to the configuration operation is created, a target constraint field corresponding to the target model is determined, and a target bloom filter corresponding to the target model is created.
In one possible embodiment, the apparatus is further for:
in an initial state, determining each target model corresponding to the database and a target constraint field corresponding to the target model;
Aiming at each target model, acquiring the existing data of the target model on the target constraint field in the database;
and adding the existing data into a target bloom filter corresponding to the target model.
In a third aspect, the present application provides a database uniqueness constraint processing apparatus, including: a processor, a memory;
the memory stores computer-executable instructions;
the processor executes computer-executable instructions stored in the memory to implement the database uniqueness constraint processing method of any one of the first aspect.
In a fourth aspect, the present application provides a computer-readable storage medium having stored therein computer-executable instructions for implementing the database uniqueness constraint processing method of any one of the first aspects when the computer-executable instructions are executed.
In a fifth aspect, the present application further provides a computer program product comprising a computer program which, when executed, implements the method for processing a database uniqueness constraint according to any one of the first aspects.
The method, the device, the equipment and the storage medium for processing the database uniqueness constraint acquire data to be inserted, and determine a target model corresponding to the data to be inserted and a target bloom filter corresponding to the target model; determining a target constraint field corresponding to the target model, and determining target data of the data to be inserted in the target constraint field; verifying the target data based on the target bloom filter; in the case that the target data passes the verification, the data to be inserted including the target data is stored in the database. According to the method and the device, the target data to be inserted into the data target constraint field is verified through the target bloom filter, a large amount of data of the unique constraint field is not required to be stored in the memory or the disk, occupation of storage space can be reduced, meanwhile, the query speed of the target bloom filter is higher, query efficiency can be effectively improved, and response time of a service is shortened.
Drawings
Fig. 1 is a schematic view of an application scenario provided in an embodiment of the present application;
FIG. 2 is a schematic flow chart of a method for processing database uniqueness constraint according to an embodiment of the present application;
FIG. 3 is a flowchart of another method for processing database uniqueness constraint according to an embodiment of the present application;
FIG. 4 is a schematic diagram of data import of a bloom filter according to an embodiment of the present application;
FIG. 5 is a schematic diagram of target data verification of a target bloom filter according to an embodiment of the present application;
FIG. 6 is a schematic diagram of a database uniqueness constraint process according to an embodiment of the present application;
fig. 7 is a schematic structural diagram of a database uniqueness constraint processing device according to an embodiment of the present application;
fig. 8 is a schematic structural diagram of a database uniqueness constraint processing device according to an embodiment of the present application.
Detailed Description
In order to better understand the technical solutions of the present application, the present application is further described in detail below with reference to the drawings and examples. It is to be understood that the specific embodiments and figures described herein are for purposes of illustration only and are not intended to be limiting.
It should be noted that, the user information (including but not limited to user equipment information, user personal information, etc.) and the data (including but not limited to data for analysis, stored data, presented data, etc.) referred to in the present application are information and data authorized by the user or fully authorized by each party, and the collection, use and processing of the related data need to comply with the related laws and regulations and standards, and provide corresponding operation entries for the user to select authorization or rejection.
It should be noted that the method, the device, the equipment and the storage medium for processing the unique constraint of the database can be used in the technical field of data processing, and can also be used in any field except the technical field of data processing, and the specific application field of the method, the device, the equipment and the storage medium for processing the unique constraint of the database is not limited.
With the continuous development of internet technology, the scale of online service is continuously enlarged, and the corresponding data volume is also continuously increased. The configuration management database (Configuration Management Database, CMDB) manages a large amount of data related to hardware infrastructure, infrastructure as a service (Infrastructure as a Service, IAAS) infrastructure, platform as a service (Platform as a Service, PAAS) component resources, middleware, business systems, applications, and the like. In some scenarios, in order to meet the full text retrieval requirements, the CMDB needs to incorporate an Elastic Search analysis Engine (ES) database to provide efficient full text retrieval capabilities, where the core business data of the CMDB needs to be stored in the ES database. The core service data of the CMDB is of two types: model data and instance data. However, the partial model requires that the underlying instance data must be unique in some fields, but the ES database itself does not have the capability of unique constraints and needs to be done by other means.
To implement the uniqueness constraint on the database, in the related art, the uniqueness constraint field is usually stored in a memory by using a specific data structure (for example, set or list, etc.), or the uniqueness constraint field is stored on a disk by using a relational database (for example, mySQL, etc.), non-relational data (for example, remote dictionary service (Remote Dictionary Server, dis) database, etc.), or a text file, and then the uniqueness constraint on the database is implemented by adopting a data query and verification manner.
Illustratively, the server may store the unique constraint field on a disk, perform storage and query verification based on MySQL, and specifically, may first define an ES index structure storing service data; secondly, defining a MySQL library table for storing the uniqueness constraint field, and providing uniqueness checking capability by using a unique index of MySQL; when data insertion is performed, firstly, the insertion of the unique constraint field into the MySQL database table is attempted, if the insertion can be successfully performed, the data can be inserted into the ES without violating the unique constraint, otherwise, the insertion is not allowed.
However, in the above-mentioned unique constraint processing manner in the related art, with the rapid growth of service data, the data in MySQL database is also rapidly grown, whether stored in a memory or on a disk, when the data volume reaches more than ten millions, the data stored in the database occupies a large amount of storage space, and the query performance starts to gradually decrease, so that the requirement of service on response time is less and less satisfied. Also, if data is saved to memory, there is a need to provide a means of persistence so that the application can be reloaded into memory upon restart. Therefore, the unique constraint processing mode in the related technology has the defects of larger occupied storage space and slower query speed, so that the response time of the online service is overlong.
In order to solve the problems, the method, the device, the equipment and the storage medium for processing the uniqueness constraint of the database in the application are characterized in that the electronic equipment performs the uniqueness constraint processing on the data to be inserted through the target bloom filter, the target bloom filter consists of a hash function and a binary (bit) array, the storage space occupation is small, the time complexity and the space complexity of the query are low, the data storage space and the insertion query time cannot be increased along with the increase of the data volume, the storage space can be saved, the query speed is improved, and the response time of the online service is shortened.
Fig. 1 is a schematic view of an application scenario provided in an embodiment of the present application. Referring to fig. 1, an electronic device 101 is included. The electronic device 101 may be a computer, a server, or the like, and the embodiment of the present application is not limited to a specific type of the electronic device 101. As shown in fig. 1, in the related art, in order to implement a unique constraint process on a database, an electronic device 101 generally stores data under a unique constraint field in the database in a memory or a disk, and performs a query check when the data is stored in the database. This approach takes up more memory and the query speed is slower.
In the embodiment of the present application, the electronic device 101 performs unique constraint processing on the data to be inserted of the target model based on the bloom filter, and stores the data to be inserted into the database after the bloom filter passes the verification, so that the storage space occupation can be effectively reduced, and the query speed can be improved.
The following describes the embodiments shown in the present application in detail by way of specific examples. It should be noted that the following embodiments may exist independently or may be combined with each other, and for the same or similar content, the description will not be repeated in different embodiments.
Fig. 2 is a flow chart of a method for processing database uniqueness constraint according to an embodiment of the present application. Referring to fig. 2, the method for processing the database uniqueness constraint may include:
s201, obtaining data to be inserted, and determining a target model corresponding to the data to be inserted and a target bloom filter corresponding to the target model.
The execution body of the embodiment of the application may be an electronic device, specifically may refer to a server, or the like, or may be a database uniqueness constraint processing device disposed in the electronic device. The database uniqueness constraint processing device can be realized by software or a combination of software and hardware. For ease of understanding, hereinafter, an execution body will be described as an example of an electronic device.
In the embodiment of the present application, the data to be inserted may refer to example data that needs to be stored in a database. The database may refer to an ES database, etc., and the embodiment of the present application does not limit the specific type of the database. The target model may refer to a model predefined in the CMDB. The target bloom filter may refer to a pre-created bloom filter that may be used to uniquely constrain the data to be inserted. Bloom filters are a data structure that includes a binary array (or binary vector) and a plurality of random mapping functions (or hash functions) that can be used to retrieve whether an element is present in a collection.
Specifically, in the CMDB, a user may define various target models based on actual requirements, the electronic device may create various target models accordingly, and then may import corresponding instance data in the target models, and store the instance data in the database. However, since the database requires a certain field or the numerical value of certain fields must be unique, the electronic device needs to perform unique constraint processing of the data. For data to be inserted of the database, the electronic device may first acquire a target model and a target bloom filter corresponding to the data to be inserted, and may subsequently perform unique constraint processing on the data to be inserted based on the target bloom filter.
S202, determining a target constraint field corresponding to the target model, and determining target data of the data to be inserted in the target constraint field.
In the embodiment of the present application, the target constraint field may refer to a unique constraint field corresponding to the target model, for example, when the target model is a server, the target constraint field may be an internet protocol (Internet Protocol, IP) address or the like. The target data may refer to the value of the data to be inserted in the target constraint field.
In this step, after determining the target model corresponding to the data to be inserted, the electronic device may further determine a target constraint field corresponding to the target model, then may read a value of the data to be inserted on the target constraint field, may record as a target data key, and then may verify the target data based on a target bloom filter to determine whether a repeated value exists in the database for the target data.
S203, verifying the target data based on the target bloom filter.
In this embodiment of the present application, after reading the target data corresponding to the data to be inserted, the electronic device may verify the target data based on the target bloom filter. In the initial state, the electronic device can import the existing data in the database under the target constraint field into the bloom filter to obtain the target bloom filter, so that the target bloom filter can perform unique constraint processing on the data to be inserted based on the existing data in the database. Specifically, the electronic device may input the target data into the hash function of the target bloom filter to obtain a value of the hash function, and then determine whether the value of the corresponding position exists 0 in the binary number set based on the value of the hash function, if so, it is indicated that the target data does not exist in the database, and the target data passes verification.
S204, storing the data to be inserted including the target data into a database under the condition that the target data passes verification.
In the embodiment of the application, when the target data verification passes, the electronic device can determine that the target data does not exist in the database, and can store the data to be inserted including the target data into the database, so that the unique constraint processing during the data insertion of the database is realized.
According to the database uniqueness constraint processing method, electronic equipment acquires data to be inserted, and determines a target model corresponding to the data to be inserted and a target bloom filter corresponding to the target model; determining a target constraint field corresponding to the target model, and determining target data of the data to be inserted in the target constraint field; verifying the target data based on the target bloom filter; in the case that the target data passes the verification, the data to be inserted including the target data is stored in the database. According to the method and the device, the target data to be inserted into the data target constraint field is verified through the target bloom filter, a large amount of data of the unique constraint field does not need to be stored in a memory or a disk, occupation of storage space can be reduced, meanwhile, the query speed of the target bloom filter is higher, query efficiency can be effectively improved, and response time of a service is shortened.
Based on the above embodiments, fig. 3 is a schematic flow chart of another method for processing database uniqueness constraint according to an embodiment of the present application. Referring to fig. 3, the method for processing the database uniqueness constraint may include:
s301, responding to configuration operation of a user, creating a target model corresponding to the configuration operation, determining a target constraint field corresponding to the target model, and creating a target bloom filter corresponding to the target model.
In the embodiment of the application, the user may refer to an operation and maintenance person and the like. The configuration operation may refer to a predefined operation by a user, may include a model configuration operation, a configuration operation of model unique constraint rules, and the like. Specifically, the electronic device may create a target model predefined by the user and determine a target constraint field corresponding to the target model in response to a configuration operation of the user, and may create a target bloom filter corresponding to the target model, where the target bloom filter may include a binary array and a plurality of hash functions.
S302, in an initial state, determining each target model corresponding to the database and a target constraint field corresponding to the target model.
S303, aiming at each target model, acquiring the existing data of the target model in the database on a target constraint field; and adding the existing data into the target bloom filter corresponding to the target model.
In this embodiment of the present application, the initial state may refer to an initial state of initial start or restart of the service application, where existing data in the database may not be stored in the target bloom filter, for example, all values in the binary number set of the target bloom filter may be 0 in the initial state. In an initial state, the electronic device may query the respective object models in the database. Then, for each target model, the electronic device may determine a target constraint field corresponding to the target model, then may traverse the instance data under the target model in the database, determine existing data of the target model under the target constraint field, and then add the existing data to a target bloom filter corresponding to the target model.
Specifically, when adding, the electronic device may input the existing data into the hash function of the target bloom filter to obtain the value of the hash function, and then, based on the value of the hash function, set the value of the position to be 1 at the corresponding position in the binary number group, so as to implement the importing of the data in the target bloom filter.
Exemplary, fig. 4 is a schematic diagram of data import of a bloom filter according to an embodiment of the present application. As shown in fig. 4, for existing data keys (e.g., key1 and key 2) of the object model under the object constraint field in the database, the electronic device may first input the existing data keys into a plurality of hash (hash) functions 401, which may specifically refer to hash1, hash2, hash3, hash4, and so on. Then, the electronic device may set the binary value to 1 at the corresponding position in the binary group 402 according to the hash function value corresponding to the existing data key. It should be noted that different existing data may occupy 1's in the same position of the binary array 402. Therefore, the electronic equipment can realize conversion and storage of the existing data by importing the existing data into the binary number group of the target bloom filter, can reduce the occupation of storage space and can also improve the follow-up query speed.
S304, obtaining data to be inserted, and determining a target model corresponding to the data to be inserted and a target bloom filter corresponding to the target model.
S305, determining a target constraint field corresponding to the target model, and determining target data of the data to be inserted in the target constraint field.
S306, determining first values of target data on a preset number of hash functions; determining a second value corresponding to the first value in the binary number group based on the first value; and in the case that the second value comprises the target value, determining that the target data passes verification.
In this embodiment of the present application, the preset number may refer to a preset number of hash functions, where the number may be recorded as k, specifically may be 3 or 4, etc., and may be flexibly configured based on actual requirements, which is not limited in this embodiment of the present application. The first value may refer to a result of calculation of the target data on the hash function. The hash function has various types and can be flexibly configured based on actual requirements. The hash function may be, for example, "convert a string into an integer and then leave the integer pair 10, or" flip a string and then convert the flipped string into an integer and then leave the integer pair 10 ", etc.
The second value may refer to a specific value of the first value at a corresponding position in the binary number set, and may be 0 or 1. The target value may refer to 0. Specifically, the electronic device may first input the target data into k hash functions to obtain k first values, then determine a second value at a corresponding position in the binary number set according to the k first values, and if the second value includes 0, it indicates that the target data does not exist in the target bloom filter, that is, the target data does not exist in the database, and the target data passes verification.
Illustratively, fig. 5 is a schematic diagram of target data verification of a target bloom filter according to an embodiment of the present application. As shown in fig. 5, when the electronic device imports existing data, for existing data gid1234567 and gid23456789, the electronic device first inputs the existing data into hash functions (including hash1, hash2 and hash 3) to obtain hash function values, where each hash function value represents a subscript position of a binary group in the target bloom filter, and the hash function value corresponding to gid1234567 is 2, 3 and 6; the hash function corresponding to gid23456789 has values of 1, 5 and 7. Correspondingly, the binary number group in the target bloom filter is a 9-bit array, the subscript positions are sequentially 0 to 8, and the binary number value in each subscript position in the initial state is 0. When the existing data is imported, the electronic device can set the binary number value of the target bloom filter binary number group at the position corresponding to the subscript position of the hash function value to be 1 based on the hash function value of the existing data, and specifically can set the binary number values of the subscript positions of 2, 3, 6, 1, 5 and 7 to be 1, so that the existing data importing process is realized.
In the verification process for the target data, the electronic device may input the target data to the same hash function (including hash1, hash2, and hash 3) to obtain a calculation result of the hash function, i.e. a first value, where the first value corresponding to the gid1234567 is 2, 3, and 6; the first values corresponding to gid43456789 are 1, 4 and 7. Then, the electronic device may determine a second value corresponding to the first value in the binary array based on the first value, where the second value corresponding to gid1234567 is 1, and the second value corresponding to gid43456789 is 1, 0, 1. At this time, the electronic device may determine whether the second value corresponding to the target data includes the target value 0, and if the second value corresponding to the target data gid43456789 includes the target value 0, it is indicated that the target data does not exist in the database, and the target data passes the verification; if the second value corresponding to the target data does not include the target value 0, for example, the second value corresponding to the target data gid1234567 does not include the target value 0, which indicates that the target data may exist in the database, the electronic device may perform the second verification, so as to further improve accuracy of the data constraint processing.
In the embodiment of the application, after the electronic device reads the target data of the data to be inserted on the target constraint field, first, a first value corresponding to the target data is calculated through a hash function, then, a second value corresponding to the first value is determined in the binary number group, and then, under the condition that the second value does not include the target value 0, the verification of the target data is determined to pass. Therefore, the electronic equipment can quickly determine whether the target data does not exist in the database based on the target bloom filter, so that the query speed can be improved, and the response time of the service can be shortened.
S307, under the condition that the second value does not comprise the target value, inquiring and verifying the target data in the database based on a preset batch inquiring function; in the case that the target data is not included in the database, it is determined that the target data is verified.
In this embodiment of the present application, the preset batch query function may refer to a preset batch query function of a database, for example, may refer to a batch query function msearch in an ES database, etc. Since the bloom filter can only guarantee that certain data is not necessarily present, secondary judgment and verification are required when the bloom filter prompts that the data may be present. Specifically, when the second value corresponding to the binary number group of the target data in the target bloom filter does not include the target numerical value, that is, the second values are all 1, the electronic device can determine that the weight determination result of the target bloom filter is possible, and at this time, the electronic device can perform secondary verification in the database based on a preset batch query function. If the target data exists in the database, the target data verification is not passed, and if the target data does not exist in the database, the target data verification is passed. Therefore, the electronic equipment performs secondary verification through the preset batch query function, the accuracy of the data uniqueness constraint processing can be improved, and meanwhile, the batch query efficiency is higher than that of the ordinary query search of the database in the related technology, and the query speed of the data can be improved.
And S308, storing the data to be inserted including the target data into a database and updating the target bloom filter based on the target data under the condition that the target data passes verification.
In this embodiment of the present application, in the case that verification of the target data is passed, the electronic device may determine that the target data does not exist in the database, and may store the data to be inserted including the target data in the database. The electronic device may then update the target bloom filter based on the target data, and may specifically store the target data into the target bloom filter to ensure real-time and validity of the target bloom filter.
In one possible implementation, the updating process of the target bloom filter in step S308 may be implemented as follows:
acquiring first values of target data on a preset number of hash functions; and updating the second value corresponding to the first value into a non-target value in the binary number group of the target bloom filter.
In the embodiment of the present application, the non-target value may refer to 1. After the electronic device stores the data to be inserted into the database, the target data can be stored into the target bloom filter, so that the existing data stored in the target bloom filter is consistent with the database, and the validity of the unique constraint processing is ensured. Specifically, the electronic device may obtain the hash function calculation result of the target data, that is, the first value, and then may set the second value of the position where the first value is located to be the non-target value 1 in the binary number group of the target bloom filter, so as to realize synchronous update of the target bloom filter, and ensure accuracy of the subsequent data weight judgment.
Illustratively, fig. 6 is a logic schematic diagram of a database uniqueness constraint process provided in an embodiment of the present application. As shown in fig. 6, in an initial state (e.g., application restart, etc.), the electronic device may query all model data in the database to determine each target model and the target constraint field corresponding to each target model. Then, for each target model, the electronic device can traverse the instance data under the target model, and import the existing data under the target constraint field of the target model into the target bloom filter, so as to reconstruct the target bloom filter for judging the uniqueness constraint for each target model.
When a user needs to insert instance data into a database, the electronic equipment can acquire data to be inserted input by the user, determine a target model corresponding to the data to be inserted, and determine a target bloom filter corresponding to the target model; and then the electronic equipment can read out the value of the data to be inserted in the target constraint field, namely the target data key, according to the target constraint field corresponding to the target model. The electronic device inputs the target data into k hash functions of the target bloom filter to obtain k first values, and then determines second values corresponding to the k first values in a binary number group of the target bloom filter. If the second value includes a target value 0, the electronic device determines that the target data passes verification. If the second value does not include the target value 0, the electronic device may further perform a secondary verification in the database based on a preset batch query function. If the database includes target data (for example, the target constraint field in the ES database index takes the value of key and is valid), the electronic device determines that the target data is not verified; if the database does not contain the target data, the electronic equipment determines that the target data passes the verification.
Under the condition that the target data passes verification, the electronic device can store the data to be inserted comprising the target data into a database, and then can synchronously update the target bloom filter based on the target data, so that the instantaneity and the effectiveness of the target bloom filter are ensured. If the target data is not verified, the electronic device can output prompt information of failure in inserting the data to be inserted to prompt a user, and meanwhile, the unique constraint processing is continuously carried out on the next data to be inserted.
In the related art, for an application system using an ES database as an underlying storage, if the capability of uniquely restricting a similar relational database is used, a large amount of data of a uniquely restricting field needs to be saved in a memory or a disk by an electronic device to make a determination. However, whether the memory or the disk is used to store the unique constraint field, when the data volume reaches a certain level, the storage space is occupied greatly, and the inquiry performance such as the inquiry speed and the like is reduced, so that the response time of the service is overlong.
In the embodiment of the application, the electronic device realizes the unique constraint processing of the database through the target bloom filter. Assuming that the binary number group length of the bloom filter is m and the hash function is k, the space complexity of the bloom filter is O (m), the time complexity of insertion and query is O (k), and the storage space and the insertion and query time are not increased along with the increase of elements, so that the performance of the database duplication judgment operation of the data quantity of billions can be ensured, the faster query speed can be realized by adopting fewer storage spaces, and the response time of the service can be ensured.
Fig. 7 is a schematic structural diagram of a database uniqueness constraint processing device according to an embodiment of the present application. Referring to fig. 7, the database uniqueness constraint processing apparatus 70 may include:
an obtaining module 71, configured to obtain data to be inserted, and determine a target model corresponding to the data to be inserted and a target bloom filter corresponding to the target model;
a determining module 72, configured to determine a target constraint field corresponding to the target model, and determine target data of the data to be inserted in the target constraint field;
a verification module 73 for verifying the target data based on the target bloom filter;
and a storage module 74 for storing the data to be inserted including the target data into the database in case that the target data passes the verification.
In one possible implementation, the verification module 73 is specifically configured to:
determining first values of target data on a preset number of hash functions;
determining a second value corresponding to the first value in the binary number group based on the first value;
and in the case that the second value comprises the target value, determining that the target data passes verification.
In one possible implementation, the verification module 73 is further specifically configured to:
Under the condition that the second value does not comprise the target value, inquiring and verifying the target data in the database based on a preset batch inquiring function;
in the case that the target data is not included in the database, it is determined that the target data is verified.
In one possible embodiment, the apparatus 70 is further configured to:
updating the target bloom filter based on the target data.
In one possible embodiment, the apparatus 70 is further configured to:
acquiring first values of target data on a preset number of hash functions;
and updating the second value corresponding to the first value into a non-target value in the binary number group of the target bloom filter.
In one possible embodiment, the apparatus 70 is further configured to:
in response to a configuration operation of a user, creating a target model corresponding to the configuration operation, determining a target constraint field corresponding to the target model, and creating a target bloom filter corresponding to the target model.
In one possible embodiment, the apparatus 70 is further configured to:
in an initial state, determining each target model corresponding to a database and a target constraint field corresponding to the target model;
aiming at each target model, acquiring the existing data of the target model in the database on a target constraint field;
And adding the existing data into the target bloom filter corresponding to the target model.
The database uniqueness constraint processing device 70 provided in the embodiment of the present application may execute the technical scheme shown in the foregoing method embodiment, and its implementation principle and beneficial effects are similar, and will not be described herein again.
Fig. 8 is a schematic structural diagram of a database uniqueness constraint processing device according to an embodiment of the present application. Referring to fig. 8, the database uniqueness constraint processing apparatus 80 may include: a memory 81, and a processor 82. The memory 81 and the processor 82 are illustratively interconnected by a bus 83.
The memory 81 is used for storing program instructions;
the processor 82 is configured to execute the program instructions stored in the memory to implement the database uniqueness constraint processing method shown in the above embodiment.
The database uniqueness constraint processing device 80 shown in the embodiment of fig. 8 may execute the technical solution shown in the embodiment of the method described above, and its implementation principle and beneficial effects are similar, and will not be described herein again.
Embodiments of the present application provide a computer readable storage medium having stored therein computer executable instructions for implementing the above-described database uniqueness constraint processing method when the computer executable instructions are executed by a processor.
Embodiments of the present application may also provide a computer program product, including a computer program, which when executed by a processor may implement the database uniqueness constraint processing method described above.
It should be noted that the processor mentioned in the embodiments of the present application may be a central processing unit (Central Processing Unit, CPU), but may also be other general purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), off-the-shelf programmable gate arrays (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
It should be understood that the memories mentioned in the embodiments of the present application may be volatile memories or nonvolatile memories, or may include both volatile and nonvolatile memories. The nonvolatile Memory may be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically Erasable EPROM (EEPROM), or a flash Memory. The volatile memory may be random access memory (Random Access Memory, RAM) which acts as an external cache. By way of example, and not limitation, many forms of RAM are available, such as Static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double Data Rate SDRAM (Double Data Rate SDRAM), enhanced SDRAM (ESDRAM), synchronous Link DRAM (SLDRAM), and Direct RAM (DR RAM). Note that when the processor is a general-purpose processor, DSP, ASIC, FPGA or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, the memory (storage module) is integrated into the processor. It should be noted that the memory described herein is intended to comprise, without being limited to, these and any other suitable types of memory.
It should be understood that, in various embodiments of the present application, the sequence numbers of the foregoing processes do not mean the order of execution, and the order of execution of the processes should be determined by the functions and internal logic thereof, and should not constitute any limitation on the implementation process of the embodiments of the present application.
Embodiments of the present application are described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processing unit of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processing unit of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
With respect to each of the apparatuses and each of the modules/units included in the products described in the above embodiments, it may be a software module/unit, a hardware module/unit, or a software module/unit, and a hardware module/unit. The individual devices, products may be applied to or integrated in a chip, a chip module or a terminal device. For example, for each device or product applied to or integrated on a chip, each module/chip included in the device or product may be implemented by hardware such as a circuit, or at least part of modules/units may be implemented by software programs, where the software programs are running on a processor integrated inside the chip, and the rest of modules/units may be implemented by hardware such as a circuit.
In the present application, the term "include" and variations thereof may refer to non-limiting inclusion; the term "or" and variations thereof may refer to "and/or". The terms "first," "second," and the like in this application are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. In the present application, "plurality" means two or more. "and/or", describes an association relationship of an association object, and indicates that there may be three relationships, for example, a and/or B, and may indicate: a exists alone, A and B exist together, and B exists alone. The character "/" generally indicates that the context-dependent object is an "or" relationship.
The foregoing is only a partial embodiment of the present application and it should be noted that it will be apparent to those skilled in the art that numerous modifications and adaptations can be made without departing from the principles of the present application and such modifications and adaptations are intended to be comprehended within the scope of the present application.
Claims (10)
1. A method for processing a database uniqueness constraint, comprising:
acquiring data to be inserted, and determining a target model corresponding to the data to be inserted and a target bloom filter corresponding to the target model;
Determining a target constraint field corresponding to the target model, and determining target data of the data to be inserted in the target constraint field;
verifying the target data based on the target bloom filter;
and storing the data to be inserted comprising the target data into a database under the condition that the target data passes verification.
2. The method of claim 1, wherein the validating the target data based on the target bloom filter comprises:
determining a first value of the target data on a preset number of hash functions;
determining a second value corresponding to the first value in a binary number group based on the first value;
and under the condition that the second value comprises a target value, determining that the target data passes verification.
3. The method according to claim 2, wherein the method further comprises:
under the condition that the second value does not comprise the target value, inquiring and verifying the target data in a database based on a preset batch inquiring function;
and determining that the target data passes verification under the condition that the target data is not included in the database.
4. The method of claim 1, wherein after the storing the data to be inserted including the target data into a database, the method further comprises:
updating the target bloom filter based on the target data.
5. The method of claim 4, wherein the updating the target bloom filter based on the target data comprises:
acquiring first values of the target data on a preset number of hash functions;
and updating the second value corresponding to the first value into a non-target value in the binary number group of the target bloom filter.
6. The method according to any one of claims 1 to 5, wherein prior to said obtaining the data to be inserted, the method further comprises:
in response to a configuration operation of a user, a target model corresponding to the configuration operation is created, a target constraint field corresponding to the target model is determined, and a target bloom filter corresponding to the target model is created.
7. The method according to any one of claims 1 to 5, further comprising:
in an initial state, determining each target model corresponding to the database and a target constraint field corresponding to the target model;
Aiming at each target model, acquiring the existing data of the target model on the target constraint field in the database;
and adding the existing data into a target bloom filter corresponding to the target model.
8. A database uniqueness constraint processing apparatus, comprising:
the acquisition module is used for acquiring data to be inserted, and determining a target model corresponding to the data to be inserted and a target bloom filter corresponding to the target model;
the determining module is used for determining a target constraint field corresponding to the target model and determining target data of the data to be inserted in the target constraint field;
the verification module is used for verifying the target data based on the target bloom filter;
and the storage module is used for storing the data to be inserted comprising the target data into a database under the condition that the target data passes verification.
9. A database uniqueness constraint processing apparatus, comprising: a processor, a memory;
the memory stores computer-executable instructions;
the processor executes computer-executable instructions stored in the memory to implement the database uniqueness constraint processing method of any one of claims 1 to 7.
10. A computer-readable storage medium having stored therein computer-executable instructions for implementing the database uniqueness constraint processing method of any one of claims 1 to 7 when the computer-executable instructions are executed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311255229.5A CN117271571A (en) | 2023-09-26 | 2023-09-26 | Database uniqueness constraint processing method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311255229.5A CN117271571A (en) | 2023-09-26 | 2023-09-26 | Database uniqueness constraint processing method, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117271571A true CN117271571A (en) | 2023-12-22 |
Family
ID=89205789
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311255229.5A Pending CN117271571A (en) | 2023-09-26 | 2023-09-26 | Database uniqueness constraint processing method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117271571A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117743472A (en) * | 2024-02-06 | 2024-03-22 | 之江实验室 | Storage task breakpoint synchronization method, device, medium and equipment |
-
2023
- 2023-09-26 CN CN202311255229.5A patent/CN117271571A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117743472A (en) * | 2024-02-06 | 2024-03-22 | 之江实验室 | Storage task breakpoint synchronization method, device, medium and equipment |
| CN117743472B (en) * | 2024-02-06 | 2024-05-07 | 之江实验室 | Storage task breakpoint synchronization method, device, medium and equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106649346B (en) | Data repeatability checking method and device | |
| US20200036755A1 (en) | Data Access Policies | |
| CN109992285B (en) | Independent updating method and device for block chain code block and electronic equipment | |
| CN112487083B (en) | Data verification method and device | |
| CN117271571A (en) | Database uniqueness constraint processing method, device, equipment and storage medium | |
| CN108427736B (en) | Method for querying data | |
| CN108616361B (en) | Method and device for identifying uniqueness of equipment | |
| CN112948473A (en) | Data processing method, device and system of data warehouse and storage medium | |
| CN110659019A (en) | Parameter checking method and device and server | |
| CN109145621B (en) | Document management method and device | |
| CN115988001A (en) | Consensus voting processing method, device, equipment and medium for block chain system | |
| CN116094849B (en) | Application access authentication method, device, computer equipment and storage medium | |
| CN113885834A (en) | Random number generation method based on block chain, voting method and equipment | |
| CN107133163A (en) | A kind of method and apparatus for verifying description class API | |
| CN115391188A (en) | Scene test case generation method, device, equipment and storage medium | |
| CN115795563A (en) | State data checking method and device | |
| CN111382179B (en) | Data processing method and device and electronic equipment | |
| US20160196331A1 (en) | Reconstitution order of entity evaluations | |
| CN113641708A (en) | Rule engine optimization method, data matching method and device, storage medium and terminal | |
| CN113468037A (en) | Data quality evaluation method, device, medium and electronic equipment | |
| CN110334098A (en) | A kind of database combining method and system based on script | |
| US11748656B1 (en) | Decision tree learning with missing data | |
| CN115062044A (en) | Data query method, device, equipment and storage medium | |
| CN116881164A (en) | Verification and correction method, device and equipment for test data in financial information system | |
| CN113536245A (en) | Data processing method, device, related equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |