CN112631884A - Pressure measurement method and device based on data synchronization, computer equipment and storage medium - Google Patents
Pressure measurement method and device based on data synchronization, computer equipment and storage medium Download PDFInfo
- Publication number
- CN112631884A CN112631884A CN202011504827.8A CN202011504827A CN112631884A CN 112631884 A CN112631884 A CN 112631884A CN 202011504827 A CN202011504827 A CN 202011504827A CN 112631884 A CN112631884 A CN 112631884A
- Authority
- CN
- China
- Prior art keywords
- data
- test
- source database
- template
- pressure
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 41
- 238000009530 blood pressure measurement Methods 0.000 title claims description 26
- 238000012360 testing method Methods 0.000 claims abstract description 346
- 238000004519 manufacturing process Methods 0.000 claims description 35
- 238000010998 test method Methods 0.000 claims description 12
- 238000006243 chemical reaction Methods 0.000 claims description 5
- 230000008676 import Effects 0.000 claims description 4
- 238000012545 processing Methods 0.000 claims description 4
- 238000005516 engineering process Methods 0.000 abstract description 2
- 230000005540 biological transmission Effects 0.000 description 11
- 230000008569 process Effects 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- 230000001360 synchronised effect Effects 0.000 description 5
- 238000004891 communication Methods 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 230000006835 compression Effects 0.000 description 2
- 238000007906 compression Methods 0.000 description 2
- 238000010276 construction Methods 0.000 description 2
- 230000008520 organization Effects 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3409—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment for performance assessment
- G06F11/3433—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment for performance assessment for load management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F16/275—Synchronous replication
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Computer Hardware Design (AREA)
- Quality & Reliability (AREA)
- Computing Systems (AREA)
- Data Mining & Analysis (AREA)
- Debugging And Monitoring (AREA)
Abstract
The method comprises the steps of receiving a pressure test request input by a user, wherein the pressure test request comprises template data, a test data generation condition and a test instruction, sending the template data to a source database, judging whether historical data matched with the template data exists in the source database, if so, sending the template data and the test data generation condition to an artifact platform, indicating the artifact platform to generate test data based on the template data and the test data generation condition, sending the test instruction and the test data to a kafka cluster, indicating the kafka cluster to perform pressure test, and generating a test result. In addition, the application also relates to a block chain technology, and the pressure test request can be stored in the block chain. According to the method and the device, batch number making is not needed in the source database, so that a large amount of dirty data cannot be made in the source database, and the stability of the whole system is guaranteed.
Description
Technical Field
The application belongs to the technical field of cloud, and particularly relates to a pressure measurement method and device based on data synchronization, computer equipment and a storage medium.
Background
At present, in the IT industry, due to the popularity of big data and the popularity of IOEs, more and more companies are expecting to place company business data into a big data environment. Among other things, the OGG tool of ORACLE corporation can synchronize data in the ORACLE database to the big data environment, such as the kafka cluster through which data in the ORACLE database can be synchronized.
In the process of synchronizing data in the oracle database to kafka, pressure testing needs to be performed on kafka from time to time, and the existing pressure testing mode mainly includes performing batch modeling on a source database to obtain a large amount of test data, and then synchronizing the test data to a template database through an OGG link, that is, synchronizing the test data to kafka through the OGG link. However, on one hand, such an operation consumes a lot of time for transmission of the test data, and simultaneously, a lot of dirty data exists in the source database, and after the test data is completely synchronized to kafka, the dirty data in the source database needs to be manually deleted by a tester, otherwise, the performance of the database is affected, and because the test data has a large amount of test data, a link of a test environment is easily interrupted during transmission of the test data, so that the test is terminated.
Disclosure of Invention
The embodiment of the application aims to provide a pressure test method, a pressure test device, computer equipment and a storage medium based on data synchronization, so as to solve the technical problem that the existing kafka cluster pressure test mode needs to perform batch data making on a source database, so that a large amount of dirty data exists in the source database, and the performance of the database is affected.
In order to solve the above technical problem, an embodiment of the present application provides a pressure measurement method based on data synchronization, which adopts the following technical scheme:
a pressure test method based on data synchronization is applied to a pressure test system of a kafka cluster, the pressure test system of the kafka cluster comprises a source database, the kafka cluster and an OGG link, the OGG link is used for connecting the source database and the kafka cluster, the pressure test method further comprises a modeling platform, and the pressure test method based on data synchronization specifically comprises the following steps:
receiving a pressure test request input by a user, wherein the pressure test request comprises template data, a test data generation condition and a test instruction;
sending the template data to a source database, traversing historical data stored in the source database, and judging whether historical data matched with the template data exists in the source database;
if the test data exists, sending the template data and the test data generation condition to the manufacture platform, and indicating the manufacture platform to generate the test data based on the template data and the test data generation condition;
and sending the test instruction and the test data to the kafka cluster, and indicating the kafka cluster to perform pressure test according to the test instruction and the test data to generate a test result.
Further, the steps of sending the template data to the source database, traversing the historical data stored in the source database, and determining whether the historical data matched with the template data exists in the source database specifically include:
analyzing the template data to obtain format information of the template data;
traversing the historical data stored in the source database, and comparing the format of the template data with all the historical data stored in the source database based on the format information of the template data;
and judging whether historical data matched with the template data exists in the source database according to the comparison result.
Further, after judging whether the source database has historical data matched with the template data according to the comparison result, the method further includes:
if the historical data which is completely consistent with the format of the template data does not exist in the source database, calculating the similarity between the template data and all the historical data stored in the source database;
and sorting the calculated similarity, and outputting historical data corresponding to the maximum value of the similarity sorting result.
Further, the step of sending the template data and the test data generation condition to the manufacture platform and instructing the manufacture platform to generate the test data based on the template data and the test data generation condition specifically includes:
sending the template data and the test data generation conditions to the number making platform;
instructing the number making platform to acquire field identifications in the test data generation condition and instructing the number making platform to determine variable fields of the template data based on the field identifications;
and instructing the number making platform to randomly fill each byte in the variable field to obtain test data.
Further, the step of sending the test instruction and the test data to the kafka cluster specifically includes:
carrying out format conversion on the test data to obtain byte type test data;
the kafka cluster's own message producer is invoked and instructed to write bytes type test data to the kafka cluster's consumption topic.
Further, instructing the kafka cluster to perform a stress test according to the test instruction and the test data, and generating a test result, specifically including:
instructing the kafka cluster to call a preset pressure test script based on the test instruction;
and instructing the kafka cluster to import the test data into a preset pressure test script, and acquiring the output of the pressure test script to obtain a pressure test result of the kafka cluster.
Further, the pressure measurement method based on data synchronization further comprises:
and storing the pressure test request input by the user into the block chain.
In order to solve the above technical problem, an embodiment of the present application further provides a pressure measurement device based on data synchronization, which adopts the following technical scheme:
the utility model provides a pressure measurement device based on data synchronization, wherein, kafka clustered pressure test system deploys at pressure measurement device based on data synchronization, and kafka clustered pressure test system includes source database, kafka cluster and OGG link, and the OGG link is used for connecting source database and kafka cluster, still includes the making number platform, and pressure measurement device based on data synchronization specifically includes:
the device comprises a request receiving module, a pressure test module and a data processing module, wherein the request receiving module is used for receiving a pressure test request input by a user, and the pressure test request comprises template data, test data generation conditions and a test instruction;
the template checking module is used for sending template data to the source database, traversing historical data stored in the source database and judging whether historical data matched with the template data exists in the source database;
the data generation module is used for sending the template data and the test data generation condition to the manufacture platform and instructing the manufacture platform to generate the test data based on the template data and the test data generation condition when historical data matched with the template data exists in the source database;
and the pressure testing module is used for sending the testing instruction and the testing data to the kafka cluster and indicating the kafka cluster to carry out pressure testing according to the testing instruction and the testing data to generate a testing result.
In order to solve the above technical problem, an embodiment of the present application further provides a computer device, which adopts the following technical solutions:
a computer device comprising a memory having computer readable instructions stored therein, and a processor that when executed implements the steps of the data synchronization-based pressure measurement method of any one of the above.
In order to solve the above technical problem, an embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solutions:
a computer readable storage medium having computer readable instructions stored thereon which, when executed by a processor, implement the steps of the data synchronization-based pressure measurement method according to any one of the preceding claims.
Compared with the prior art, the embodiment of the application mainly has the following beneficial effects:
the method generates test data matched with template data input by a user on a modeling platform, then directly guides the test data generated on the modeling platform into a kafka cluster for pressure test, and performs pressure test on the kafka cluster in a manner of batch modeling on a source database and synchronizing the test data to the kafka cluster through an OGG link. Meanwhile, test data are directly written into the kafka cluster in a mode that the manufacturing platform simulates OGG link synchronization data, transmission through the OGG link is not needed, and the kafka cluster pressure test speed is improved.
Drawings
In order to more clearly illustrate the solution of the present application, the drawings needed for describing the embodiments of the present application will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present application, and that other drawings can be obtained by those skilled in the art without inventive effort.
FIG. 1 illustrates an exemplary system architecture diagram in which the present application may be applied;
FIG. 2 illustrates a flow diagram of one embodiment of a data synchronization based pressure measurement method according to the present application;
FIG. 3 illustrates a schematic structural diagram of one embodiment of a data synchronization based pressure measurement device according to the present application;
FIG. 4 shows a schematic block diagram of one embodiment of a computer device according to the present application.
Detailed Description
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs; the terminology used in the description of the application herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "including" and "having," and any variations thereof, in the description and claims of this application and the description of the above figures are intended to cover non-exclusive inclusions. The terms "first," "second," and the like in the description and claims of this application or in the above-described drawings are used for distinguishing between different objects and not for describing a particular order.
Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. It is explicitly and implicitly understood by one skilled in the art that the embodiments described herein can be combined with other embodiments.
In order to make the technical solutions better understood by those skilled in the art, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.
As shown in fig. 1, the system architecture 100 may include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 serves as a medium for providing communication links between the terminal devices 101, 102, 103 and the server 105. Network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few.
The user may use the terminal devices 101, 102, 103 to interact with the server 105 via the network 104 to receive or send messages or the like. The terminal devices 101, 102, 103 may have various communication client applications installed thereon, such as a web browser application, a shopping application, a search application, an instant messaging tool, a mailbox client, social platform software, and the like.
The terminal devices 101, 102, 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture experts Group Audio Layer III, mpeg compression standard Audio Layer 3), MP4 players (Moving Picture experts Group Audio Layer IV, mpeg compression standard Audio Layer 4), laptop portable computers, desktop computers, and the like.
The server 105 may be a server providing various services, such as a background server providing support for pages displayed on the terminal devices 101, 102, 103.
It should be noted that, the data synchronization-based pressure measurement method provided in the embodiment of the present application is generally executed by a server/terminal device, and accordingly, a data synchronization-based pressure measurement apparatus is generally disposed in the server/terminal device.
It should be understood that the number of terminal devices, networks, and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
In the process of synchronizing data in the oracle database to kafka, pressure testing needs to be performed on kafka from time to time, and in the existing pressure testing mode, batch modeling is performed on a source database to obtain a large amount of test data, and then the test data is synchronized to a template database through an OGG link, that is, the test data is synchronized to kafka through the OGG link. However, on one hand, such an operation consumes a lot of time for transmission of test data, and simultaneously, a lot of dirty data exist in the source relational database, and after the test data are completely synchronized to kafka, the dirty data in the source relational database also needs to be deleted manually by a tester, otherwise, the performance of the database is affected, and because the amount of the test data used during the stress test is large, during the transmission of the test data, the OGG link is easily interrupted, and the test is terminated.
The ogg (oracle golden gate) is a tool for real-time data synchronization, and is mainly used for copying and integrating transaction data between various enterprise systems at a sub-second level, and can flexibly implement data synchronization between different types of databases. The OGG reads the source database log in real time, captures the change content of the source database, generates a compressed and encrypted queue file, transmits the queue file to the template database, and performs corresponding queue file analysis and obtains the change content of the source database at the target end, so as to complete the update of the template database.
Based on the problems existing in the existing pressure testing mode, the application provides a kafka pressure testing scheme for simulating OGG data synchronization, and the scheme of the application specifically comprises the following steps: and constructing a build platform, generating test data with the same format as the data in the source database through the build platform, and then directly importing the test data generated on the build platform into the kafka cluster for pressure test. Compared with the existing scheme of conducting pressure test by conducting batch construction on the source database and then synchronizing to the kafka cluster through the OGG link, batch construction on the source database is not needed, so that pollution of the source database cannot be caused, decoupling between the source database and the kafka cluster environment is achieved by isolating the source database and the kafka cluster environment, and stability of the whole system is guaranteed. Meanwhile, test data are directly written into the kafka cluster in a mode that the manufacturing platform simulates OGG link synchronization data, transmission through the OGG link is not needed, and the kafka cluster pressure test speed is improved. The technical solution of the present application will be described in detail below.
With continuing reference to fig. 2, a data synchronization-based pressure testing method according to the present application is illustrated, where the data synchronization-based pressure testing method is applied to a kafka cluster pressure testing system, the kafka cluster pressure testing system includes a source database, a kafka cluster, and an OGG link, the OGG link is used to connect the source database and the kafka cluster, and further includes a build platform, and the data synchronization-based pressure testing method specifically includes:
s201, receiving a pressure test request input by a user, wherein the pressure test request comprises template data, test data generation conditions and test instructions.
The source database is generally a relational database, such as an oracle database, a mysql database, a pg database, and the like, and the relational database is generally a database used by an enterprise or an organization to store business data of the enterprise or the organization.
Specifically, when a stress test requirement exists, the server receives a stress test request input by a user, wherein the stress test request comprises template data, a test data generation condition and a test instruction. The template data are templates of the test data, the template data and the test data generating conditions are compiled by testers according to requirements and input from clients, and the modeling platform automatically generates the test data for the kafka cluster pressure test according to the template data and the test data generating conditions meeting the requirements.
In this embodiment, an electronic device (for example, the server/terminal device shown in fig. 1) on which the data synchronization-based pressure test method operates may receive a pressure test request input by a user through a wired connection manner or a wireless connection manner. It should be noted that the wireless connection means may include, but is not limited to, a 3G/4G connection, a WiFi connection, a bluetooth connection, a WiMAX connection, a Zigbee connection, a uwb (ultra wideband) connection, and other wireless connection means now known or developed in the future.
S202, sending the template data to the source database, traversing the historical data stored in the source database, and judging whether the historical data matched with the template data exists in the source database.
Wherein, before the test data is manufactured in batch, the template data input by the user is required to be verified whether the template data meets the requirement of the kafka cluster pressure test. Specifically, the server sends the template data to the source database, traverses the historical data stored in the source database, and determines whether historical data matched with the template data exists in the source database, if any one or more pieces of historical data matched with the template data exist in the source database, the template data input by the user meets the requirement of the kafka cluster pressure test, otherwise, the template data input by the user does not meet the requirement of the kafka cluster pressure test. And if any historical data matched with the template data does not exist in the source database, outputting prompt information and sending the prompt information to the user terminal.
S203, if the test data exists, sending the template data and the test data generation condition to the manufacture platform, and instructing the manufacture platform to generate the test data based on the template data and the test data generation condition.
The test data generation condition comprises the number of test data required to be generated by the pressure test and field identification, and the field identification is used for determining a variable field in the template data. The test data is constructed through the modeling platform, and under the condition that the format of the template data input by a user is correct, the number of the test data required to be generated and variable fields in the template data also need to be acquired.
And S204, sending the test instruction and the test data to the kafka cluster, and indicating the kafka cluster to perform pressure test according to the test instruction and the test data to generate a test result.
Wherein the test instruction "bin/kafka-producer-perf-test
- -record-size100- -num-records10000- -through cpu 100 ". Wherein topic is the consumption theme, topic specifies the location where the test data is stored, record-size is the size of the test data (in bytes), num-records is the total test data amount, and throughput is the amount of test data written per second.
Specifically, after the test data are generated, the test instruction and the test data are sent to the kafka cluster, the kafka cluster is instructed to call a preset pressure test script to perform pressure test according to the test instruction, the test data are written and read through the preset pressure test script, and feedback data of the pressure test script in the process of writing and reading the test data are obtained to obtain a pressure test result.
The embodiment discloses a pressure test method based on data synchronization, which belongs to the technical field of cloud, and is characterized in that test data matched with template data input by a user are generated on a modeling platform, then the test data generated on the modeling platform are directly guided into a kafka cluster for pressure test, and compared with the existing scheme of carrying out pressure test by carrying out batch modeling on a source database and synchronizing the kafka cluster through an OGG link, the method does not need to carry out batch modeling on the source database, so that a large amount of dirty data cannot be manufactured in the source database, decoupling between the source database and the kafka cluster environment is realized by isolating the source database and the kafka cluster environment, and the stability of the whole system is ensured. Meanwhile, test data are directly written into the kafka cluster in a mode that the manufacturing platform simulates OGG link synchronization data, transmission through the OGG link is not needed, and the kafka cluster pressure test speed is improved.
Further, the steps of sending the template data to the source database, traversing the historical data stored in the source database, and determining whether the historical data matched with the template data exists in the source database specifically include:
analyzing the template data to obtain format information of the template data;
traversing the historical data stored in the source database, and comparing the format of the template data with all the historical data stored in the source database based on the format information of the template data;
and judging whether historical data matched with the template data exists in the source database according to the comparison result.
Specifically, template data input by a user is obtained, the template data is analyzed to obtain format information of the template data, historical data stored in a source database are traversed, the format of the template data is compared with the format of each historical data stored in the source database one by one, if any one or more historical data in the source database are completely consistent with the format of the template data, the template data input by the user is considered to be valid data, otherwise, the template data input by the user is considered to be invalid, and a judgment result is pushed to the user. For example, the template data input by the user is { "a":1111 "," b ": 0000", "c":1100}, and if any one or more pieces of history data in accordance with the format of the template data exist in the source database, that is, one or more pieces of history data in the form of three fields a, b and c exist in the source database, the template data input by the user is considered to be valid data, and the test data can be manufactured through the template data.
In the above embodiment, by obtaining the format information of the template data and comparing the format of the template data with all the historical data stored in the source database, it can be quickly determined whether the template data input by the user meets the requirement of the kafka cluster pressure test.
Further, after judging whether the source database has historical data matched with the template data according to the comparison result, the method further includes:
if the historical data which is completely consistent with the format of the template data does not exist in the source database, calculating the similarity between the template data and all the historical data stored in the source database;
and sorting the calculated similarity, and outputting historical data corresponding to the maximum value of the similarity sorting result.
Specifically, if there is no history data in the source database that is completely consistent with the format of the template data, it indicates that the template data input by the user does not meet the requirement of the kafka cluster pressure test, and the user is required to modify the template data accordingly. The similarity of the template data and all historical data stored in the source database is calculated, the calculated similarities are sorted to obtain similarity sorting results, the historical data corresponding to the maximum value in the similarity sorting results are output to the user terminal based on the similarity sorting results, and the user can modify the format of the template data according to the output historical data.
In the above embodiment, if there is no history data in the source database that is completely consistent with the format of the template data, the template data input by the user meets the requirement of the kafka cluster pressure test by calculating the similarity between the template data and all history data stored in the source database, sorting the calculated similarity, and outputting the history data corresponding to the maximum value of the similarity sorting result, so that the user modifies the format of the template data with reference to the history data.
Further, the step of sending the template data and the test data generation condition to the manufacture platform and instructing the manufacture platform to generate the test data based on the template data and the test data generation condition specifically includes:
sending the template data and the test data generation conditions to the number making platform;
instructing the number making platform to acquire field identifications in the test data generation condition and instructing the number making platform to determine variable fields of the template data based on the field identifications;
and instructing the number making platform to randomly fill each byte in the variable field to obtain test data.
In the embodiment, for example, the number of test data to be generated is 1000 in the test data generation condition specified by the user, and the variable field in the template data is "a", that is, the "a" field needs to be changed (a is the primary key field) in the manufacturing process, so that the server randomly constructs 1000 pieces of test data with the same format as the template data by changing the data of the variable field of the template data according to the test data demand input by the user and the variable field of the template data, and supplies the kafka cluster to perform the stress test. In the specific embodiment of the application, a test data form is as { "a":0001, "b":0000, "c":1100 }.
In the above embodiment, the template data and the test data generation condition are sent to the manufacture platform, the manufacture platform is instructed to obtain the field identifier in the test data generation condition, the manufacture platform is instructed to determine the variable field of the template data based on the field identifier, and the manufacture platform is instructed to randomly fill each byte in the variable field, so that the test data can be manufactured in batches on the manufacture platform.
Further, the step of sending the test instruction and the test data to the kafka cluster specifically includes:
carrying out format conversion on the test data to obtain byte type test data;
the kafka cluster's own message producer is invoked and instructed to write bytes type test data to the kafka cluster's consumption topic.
Specifically, before the test data are transmitted to the kafka cluster, the test data are assembled into a JSON format to obtain JSON format test data, then the JSON format test data are sent to the kafka cluster, and the JSON format test data are written into a consumption subject topic of the kafka cluster through a producer of the kafka self-contained device. The JSON format test data is converted into bytes types and sent to a specific consumption subject topic in the kafka cluster to achieve test data writing operation, and after the test data writing is completed, the test data are read and consumed in kafka to obtain kafka cluster writing and consumption pressure test results.
Further, instructing the kafka cluster to perform a stress test according to the test instruction and the test data, and generating a test result, specifically including:
instructing the kafka cluster to call a preset pressure test script based on the test instruction;
and instructing the kafka cluster to import the test data into a preset pressure test script, and acquiring the output of the pressure test script to obtain a pressure test result of the kafka cluster.
The preset pressure test script can be a pressure test script configured by the Kafka official authority or an additionally configured pressure test script. In a specific embodiment of the present application, the pressure test is performed using a producer pressure test script and a consumer pressure test script officially configured by Kafka. The Kafka authority provides Kafka-producer-perf-test.sh (producer stress test script) and Kafka-consumer-perf-test.sh (consumer stress test script) under the bin catalog of the installation package for testing producer and consumer performance, respectively.
Specifically, the server instructs the kafka cluster to call a preset pressure test script based on the test instruction, and instructs the kafka cluster to import the test data into the preset pressure test script, so as to obtain the output of the pressure test script, and obtain the pressure test result of the kafka cluster.
In the above embodiment, the pressure test result of the Kafka cluster is obtained through the pressure test script carried by the Kafka, that is, the performance bottleneck (such as CPU/memory/network bandwidth) of the Kafka cluster can be known, and the CPU and IO stability of each server in the Kafka cluster can be known, if the server downtime exists.
Further, the pressure measurement method based on data synchronization further comprises:
and storing the pressure test request input by the user into the block chain.
It is emphasized that, in order to further ensure the privacy and security of the user-entered stress test request, the user-entered stress test request may also be stored in a node of a block chain.
The block chain referred by the application is a novel application mode of computer technologies such as distributed data storage, point-to-point transmission, a consensus mechanism, an encryption algorithm and the like. A block chain (Blockchain), which is essentially a decentralized database, is a series of data blocks associated by using a cryptographic method, and each data block contains information of a batch of network transactions, so as to verify the validity (anti-counterfeiting) of the information and generate a next block. The blockchain may include a blockchain underlying platform, a platform product service layer, an application service layer, and the like.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware associated with computer readable instructions, which can be stored in a computer readable storage medium, and when executed, can include processes of the embodiments of the methods described above. The storage medium may be a non-volatile storage medium such as a magnetic disk, an optical disk, a Read-Only Memory (ROM), or a Random Access Memory (RAM).
It should be understood that, although the steps in the flowcharts of the figures are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and may be performed in other orders unless explicitly stated herein. Moreover, at least a portion of the steps in the flow chart of the figure may include multiple sub-steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, which are not necessarily performed in sequence, but may be performed alternately or alternately with other steps or at least a portion of the sub-steps or stages of other steps.
With further reference to fig. 3, as an implementation of the method shown in fig. 2, the present application provides an embodiment of a pressure measurement apparatus based on data synchronization, where the embodiment of the apparatus corresponds to the embodiment of the method shown in fig. 2, and the apparatus may be applied to various electronic devices.
As shown in fig. 3, in the pressure testing apparatus based on data synchronization described in this embodiment, a pressure testing system of a kafka cluster is deployed in the pressure testing apparatus based on data synchronization, the pressure testing system of the kafka cluster includes a source database, the kafka cluster, and an OGG link, the OGG link is used to connect the source database and the kafka cluster, and further includes a build platform, and the pressure testing apparatus based on data synchronization specifically includes:
a request receiving module 301, configured to receive a stress test request input by a user, where the stress test request includes template data, a test data generation condition, and a test instruction;
the template checking module 302 is configured to send template data to the source database, traverse historical data stored in the source database, and determine whether historical data matched with the template data exists in the source database;
the data generating module 303 is configured to send the template data and the test data generating condition to the manufacture platform when historical data matching the template data exists in the source database, and instruct the manufacture platform to generate test data based on the template data and the test data generating condition;
and the pressure testing module 304 is used for sending the test instruction and the test data to the kafka cluster and instructing the kafka cluster to perform pressure testing according to the test instruction and the test data to generate a test result.
Further, the template checking module 302 specifically includes:
the data analysis unit is used for analyzing the template data to acquire format information of the template data;
the format comparison unit is used for traversing the historical data stored in the source database, and performing format comparison on the template data and all the historical data stored in the source database based on the format information of the template data;
and the matching judgment unit is used for judging whether historical data matched with the template data exists in the source database according to the comparison result.
Further, the template checking module 302 further includes:
the similarity calculation unit is used for calculating the similarity between the template data and all the historical data stored in the source database when the historical data which are completely consistent with the format of the template data do not exist in the source database;
and the similarity sorting unit is used for sorting the calculated similarities and outputting historical data corresponding to the maximum value of the similarity sorting result.
Further, the data generating module 303 specifically includes:
the information sending unit is used for sending the template data and the test data generation conditions to the number making platform;
the field identification unit is used for indicating the manufacture platform to acquire the field identification in the test data generation condition and indicating the manufacture platform to determine the variable field of the template data based on the field identification;
and the data generation unit is used for indicating the number making platform to randomly fill each byte in the variable field to obtain test data.
Further, the information sending unit specifically includes:
the format conversion subunit is used for carrying out format conversion on the test data to obtain byte type test data;
and the information writing subunit is used for calling the message producer of the kafka cluster and instructing the message producer to write the bytes type test data into the consumption subject of the kafka cluster.
Further, the pressure testing module 304 specifically includes:
the script calling unit is used for indicating the kafka cluster to call a preset pressure test script based on the test instruction;
and the pressure test unit is used for indicating the kafka cluster to introduce the test data into a preset pressure test script, acquiring the output of the pressure test script and obtaining the pressure test result of the kafka cluster.
Further, the pressure measurement device based on data synchronization further comprises:
and the request storage module is used for storing the stress test request input by the user into the block chain.
The embodiment discloses a pressure test device based on data synchronization, which belongs to the technical field of cloud, the method comprises the steps of generating test data matched with template data input by a user on a modeling platform, then directly leading the test data generated on the modeling platform into a kafka cluster for pressure test, and synchronizing the test data to the kafka cluster for pressure test through an OGG link. Meanwhile, test data are directly written into the kafka cluster in a mode that the manufacturing platform simulates OGG link synchronization data, transmission through the OGG link is not needed, and the kafka cluster pressure test speed is improved.
In order to solve the technical problem, an embodiment of the present application further provides a computer device. Referring to fig. 4, fig. 4 is a block diagram of a basic structure of a computer device according to the present embodiment.
The computer device 4 comprises a memory 41, a processor 42, a network interface 43 communicatively connected to each other via a system bus. It is noted that only computer device 4 having components 41-43 is shown, but it is understood that not all of the shown components are required to be implemented, and that more or fewer components may be implemented instead. As will be understood by those skilled in the art, the computer device is a device capable of automatically performing numerical calculation and/or information processing according to a preset or stored instruction, and the hardware includes, but is not limited to, a microprocessor, an Application Specific Integrated Circuit (ASIC), a Programmable Gate Array (FPGA), a Digital Signal Processor (DSP), an embedded device, and the like.
The computer device can be a desktop computer, a notebook, a palm computer, a cloud server and other computing devices. The computer equipment can carry out man-machine interaction with a user through a keyboard, a mouse, a remote controller, a touch panel or voice control equipment and the like.
The memory 41 includes at least one type of readable storage medium including a flash memory, a hard disk, a multimedia card, a card type memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a Read Only Memory (ROM), an Electrically Erasable Programmable Read Only Memory (EEPROM), a Programmable Read Only Memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 41 may be an internal storage unit of the computer device 4, such as a hard disk or a memory of the computer device 4. In other embodiments, the memory 41 may also be an external storage device of the computer device 4, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like, which are provided on the computer device 4. Of course, the memory 41 may also include both internal and external storage devices of the computer device 4. In this embodiment, the memory 41 is generally used for storing an operating system installed in the computer device 4 and various types of application software, such as computer readable instructions of a pressure measurement method based on data synchronization. Further, the memory 41 may also be used to temporarily store various types of data that have been output or are to be output.
The processor 42 may be a Central Processing Unit (CPU), controller, microcontroller, microprocessor, or other data Processing chip in some embodiments. The processor 42 is typically used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is configured to execute the computer readable instructions stored in the memory 41 or process data, for example, execute the computer readable instructions of the pressure measurement method based on data synchronization.
The network interface 43 may comprise a wireless network interface or a wired network interface, and the network interface 43 is generally used for establishing communication connection between the computer device 4 and other electronic devices.
The application discloses computer equipment, which belongs to the technical field of cloud, the method comprises the steps of generating test data matched with template data input by a user on a modeling platform, then directly leading the test data generated on the modeling platform into a kafka cluster for pressure test, and synchronizing the test data with the kafka cluster for pressure test through an OGG link in the prior art through batch modeling on a source database. Meanwhile, test data are directly written into the kafka cluster in a mode that the manufacturing platform simulates OGG link synchronization data, transmission through the OGG link is not needed, and the kafka cluster pressure test speed is improved.
The present application further provides another embodiment, which is to provide a computer-readable storage medium storing computer-readable instructions executable by at least one processor to cause the at least one processor to perform the steps of the data synchronization-based pressure measurement method as described above.
The application discloses a storage medium, which belongs to the technical field of cloud, the method comprises the steps of generating test data matched with template data input by a user on a modeling platform, then directly leading the test data generated on the modeling platform into a kafka cluster for pressure test, and synchronizing the test data with the kafka cluster for pressure test through an OGG link in the prior art through batch modeling on a source database. Meanwhile, test data are directly written into the kafka cluster in a mode that the manufacturing platform simulates OGG link synchronization data, transmission through the OGG link is not needed, and the kafka cluster pressure test speed is improved.
Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solutions of the present application may be embodied in the form of a software product, which is stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal device (such as a mobile phone, a computer, a server, an air conditioner, or a network device) to execute the method according to the embodiments of the present application.
It is to be understood that the above-described embodiments are merely illustrative of some, but not restrictive, of the broad invention, and that the appended drawings illustrate preferred embodiments of the invention and do not limit the scope of the invention. This application is capable of embodiments in many different forms and is provided for the purpose of enabling a thorough understanding of the disclosure of the application. Although the present application has been described in detail with reference to the foregoing embodiments, it will be apparent to one skilled in the art that the present application may be practiced without modification or with equivalents of some of the features described in the foregoing embodiments. All equivalent structures made by using the contents of the specification and the drawings of the present application are directly or indirectly applied to other related technical fields and are within the protection scope of the present application.
Claims (10)
1. A data synchronization-based pressure test method is applied to a kafka cluster pressure test system, wherein the kafka cluster pressure test system comprises a source database, a kafka cluster and an OGG link, the OGG link is used for connecting the source database and the kafka cluster, the data synchronization-based pressure test method further comprises a modeling platform, and the data synchronization-based pressure test method specifically comprises the following steps:
receiving a pressure test request input by a user, wherein the pressure test request comprises template data, a test data generation condition and a test instruction;
sending the template data to the source database, traversing historical data stored in the source database, and judging whether historical data matched with the template data exists in the source database;
if the template data and the test data are existed, sending the template data and the test data generation condition to the manufacture platform, and instructing the manufacture platform to generate the test data based on the template data and the test data generation condition;
and sending the test instruction and the test data to the kafka cluster, and instructing the kafka cluster to perform a pressure test according to the test instruction and the test data to generate a test result.
2. The data synchronization-based pressure measurement method according to claim 1, wherein the step of sending the template data to the source database, traversing the historical data stored in the source database, and determining whether there is historical data in the source database that matches the template data specifically includes:
analyzing the template data to acquire format information of the template data;
traversing historical data stored in the source database, and comparing the format of the template data with all the historical data stored in the source database based on the format information of the template data;
and judging whether historical data matched with the template data exists in the source database according to the comparison result.
3. The pressure measurement method based on data synchronization of claim 2, wherein after the determining whether the historical data matching the template data exists in the source database according to the comparison result, the method further comprises:
if the historical data with the format completely consistent with that of the template data does not exist in the source database, calculating the similarity between the template data and all the historical data stored in the source database;
and sorting the calculated similarity, and outputting historical data corresponding to the maximum value of the similarity sorting result.
4. The data synchronization-based pressure test method of claim 1, wherein the step of sending the template data and the test data generation condition to the manufacture platform and instructing the manufacture platform to generate the test data based on the template data and the test data generation condition specifically comprises:
sending the template data and the test data generation condition to the number making platform;
instructing the manufacture platform to acquire a field identifier in the test data generation condition and instructing the manufacture platform to determine a variable field of the template data based on the field identifier;
and instructing the number making platform to randomly fill each byte in the variable field to obtain test data.
5. The data synchronization-based pressure test method according to any one of claims 1 to 4, wherein the step of sending the test command and the test data to the kafka cluster specifically includes:
carrying out format conversion on the test data to obtain byte type test data;
invoking a message producer of the kafka cluster itself and instructing the message producer to write the bytes type of test data into the consumption topic of the kafka cluster.
6. The pressure test method based on data synchronization of claim 1, wherein the step of instructing the kafka cluster to perform a pressure test according to the test instruction and the test data to generate a test result specifically comprises:
instructing the kafka cluster to call a preset pressure test script based on the test instruction;
and instructing the kafka cluster to import the test data into the preset pressure test script, and acquiring the output of the pressure test script to obtain the pressure test result of the kafka cluster.
7. The data synchronization-based pressure measurement method of claim 1, further comprising:
and storing the pressure test request input by the user into a block chain.
8. The pressure testing device based on data synchronization is characterized by further comprising a manufacturing platform, and the pressure testing device based on data synchronization specifically comprises:
the device comprises a request receiving module, a pressure test module and a data processing module, wherein the request receiving module is used for receiving a pressure test request input by a user, and the pressure test request comprises template data, test data generation conditions and a test instruction;
the template checking module is used for sending the template data to the source database, traversing historical data stored in the source database and judging whether historical data matched with the template data exists in the source database;
the data generation module is used for sending the template data and the test data generation condition to the manufacture platform and indicating the manufacture platform to generate test data based on the template data and the test data generation condition when historical data matched with the template data exists in the source database;
and the pressure testing module is used for sending the testing instruction and the testing data to the kafka cluster and instructing the kafka cluster to perform pressure testing according to the testing instruction and the testing data to generate a testing result.
9. A computer device comprising a memory having computer readable instructions stored therein and a processor that when executed performs the steps of the data synchronization-based pressure measurement method of any one of claims 1 to 7.
10. A computer-readable storage medium, having computer-readable instructions stored thereon, which, when executed by a processor, implement the steps of the data synchronization-based pressure measurement method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011504827.8A CN112631884A (en) | 2020-12-18 | 2020-12-18 | Pressure measurement method and device based on data synchronization, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011504827.8A CN112631884A (en) | 2020-12-18 | 2020-12-18 | Pressure measurement method and device based on data synchronization, computer equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112631884A true CN112631884A (en) | 2021-04-09 |
Family
ID=75317154
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011504827.8A Pending CN112631884A (en) | 2020-12-18 | 2020-12-18 | Pressure measurement method and device based on data synchronization, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112631884A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113495845A (en) * | 2021-07-27 | 2021-10-12 | 拉扎斯网络科技（上海）有限公司 | Data testing method and device, electronic equipment and storage medium |
| CN114860617A (en) * | 2022-07-06 | 2022-08-05 | 上海金仕达软件科技有限公司 | Intelligent pressure testing method and system |
| CN115827452A (en) * | 2022-11-29 | 2023-03-21 | 广发银行股份有限公司 | Data processing type test system, method, storage medium and equipment |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108763080A (en) * | 2018-05-29 | 2018-11-06 | 平安普惠企业管理有限公司 | Test data method for uploading, device, computer equipment and storage medium |
| CN110188030A (en) * | 2019-04-08 | 2019-08-30 | 平安科技（深圳）有限公司 | A kind of test data generating method, device and computer equipment, storage medium |
| CN110245089A (en) * | 2019-06-21 | 2019-09-17 | 深圳前海微众银行股份有限公司 | Method for testing pressure, device, equipment and computer readable storage medium |
| CN110597714A (en) * | 2019-08-28 | 2019-12-20 | 深圳市彬讯科技有限公司 | Kafka message testing method and device, computer equipment and storage medium |
| CN111240961A (en) * | 2019-12-31 | 2020-06-05 | 中国电力科学研究院有限公司 | Database performance test system and method based on power grid big data platform |
-
2020
- 2020-12-18 CN CN202011504827.8A patent/CN112631884A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108763080A (en) * | 2018-05-29 | 2018-11-06 | 平安普惠企业管理有限公司 | Test data method for uploading, device, computer equipment and storage medium |
| CN110188030A (en) * | 2019-04-08 | 2019-08-30 | 平安科技（深圳）有限公司 | A kind of test data generating method, device and computer equipment, storage medium |
| CN110245089A (en) * | 2019-06-21 | 2019-09-17 | 深圳前海微众银行股份有限公司 | Method for testing pressure, device, equipment and computer readable storage medium |
| CN110597714A (en) * | 2019-08-28 | 2019-12-20 | 深圳市彬讯科技有限公司 | Kafka message testing method and device, computer equipment and storage medium |
| CN111240961A (en) * | 2019-12-31 | 2020-06-05 | 中国电力科学研究院有限公司 | Database performance test system and method based on power grid big data platform |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113495845A (en) * | 2021-07-27 | 2021-10-12 | 拉扎斯网络科技（上海）有限公司 | Data testing method and device, electronic equipment and storage medium |
| CN114860617A (en) * | 2022-07-06 | 2022-08-05 | 上海金仕达软件科技有限公司 | Intelligent pressure testing method and system |
| CN115827452A (en) * | 2022-11-29 | 2023-03-21 | 广发银行股份有限公司 | Data processing type test system, method, storage medium and equipment |
| CN115827452B (en) * | 2022-11-29 | 2023-11-10 | 广发银行股份有限公司 | Data processing type test system, method, storage medium and equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112631884A (en) | Pressure measurement method and device based on data synchronization, computer equipment and storage medium | |
| CN111414407A (en) | Data query method and device of database, computer equipment and storage medium | |
| CN112631924A (en) | Automatic testing method and device, computer equipment and storage medium | |
| CN112671734A (en) | Message processing method facing multiple data sources and related equipment thereof | |
| CN112328486A (en) | Interface automation test method and device, computer equipment and storage medium | |
| CN112860662A (en) | Data blood relationship establishing method and device, computer equipment and storage medium | |
| CN113377372A (en) | Business rule analysis method and device, computer equipment and storage medium | |
| CN114358775A (en) | Internet of things source tracing method based on Fabric and IPFS and related equipment thereof | |
| CN113010542A (en) | Service data processing method and device, computer equipment and storage medium | |
| CN112685397A (en) | Method, device, equipment and storage medium for verifying data cleaning result | |
| CN112256760A (en) | Data prediction method and device, computer equipment and storage medium | |
| CN111552663A (en) | File consistency verification method and device, computer equipment and storage medium | |
| CN116956326A (en) | Authority data processing method and device, computer equipment and storage medium | |
| CN117094729A (en) | Request processing method, device, computer equipment and storage medium | |
| CN110765610A (en) | PDM (product data management) integration method and device, computer equipment and storage medium | |
| CN116028446A (en) | Time sequence data file management method, device, equipment and storage medium thereof | |
| CN114968822A (en) | Interface testing method and device, computer equipment and storage medium | |
| CN114626352A (en) | Report automatic generation method and device, computer equipment and storage medium | |
| CN115061916A (en) | Method for automatically generating interface test case and related equipment thereof | |
| CN114374737A (en) | Message pushing method and device, computer equipment and storage medium | |
| CN114615325A (en) | Message pushing method and device, computer equipment and storage medium | |
| CN115017149A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN114143308A (en) | File uploading information processing method and device, computer equipment and storage medium | |
| CN114637672A (en) | Automatic data testing method and device, computer equipment and storage medium | |
| CN112416875A (en) | Log management method and device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20210409 |