CN116401238A - Deviation monitoring method, apparatus, device, storage medium and program product - Google Patents
Deviation monitoring method, apparatus, device, storage medium and program product Download PDFInfo
- Publication number
- CN116401238A CN116401238A CN202310372224.4A CN202310372224A CN116401238A CN 116401238 A CN116401238 A CN 116401238A CN 202310372224 A CN202310372224 A CN 202310372224A CN 116401238 A CN116401238 A CN 116401238A
- Authority
- CN
- China
- Prior art keywords
- data
- key value
- external data
- target external
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 58
- 238000012544 monitoring process Methods 0.000 title claims abstract description 43
- 238000004590 computer program Methods 0.000 claims abstract description 40
- 230000008569 process Effects 0.000 claims abstract description 15
- 238000012806 monitoring device Methods 0.000 claims abstract description 10
- 238000012545 processing Methods 0.000 claims description 52
- 238000012216 screening Methods 0.000 claims description 46
- 238000012549 training Methods 0.000 claims description 13
- 238000007405 data analysis Methods 0.000 claims description 9
- 238000011156 evaluation Methods 0.000 description 13
- 230000014759 maintenance of location Effects 0.000 description 13
- 230000002688 persistence Effects 0.000 description 8
- 238000012546 transfer Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 4
- 238000004891 communication Methods 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 238000010276 construction Methods 0.000 description 2
- 238000003062 neural network model Methods 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 230000009466 transformation Effects 0.000 description 2
- OKTJSMMVPCPJKN-UHFFFAOYSA-N Carbon Chemical compound [C] OKTJSMMVPCPJKN-UHFFFAOYSA-N 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000033228 biological regulation Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 229910021389 graphene Inorganic materials 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 239000011148 porous material Substances 0.000 description 1
- 230000000717 retained effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- Computational Linguistics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The application relates to a deviation monitoring method, a deviation monitoring device, computer equipment, a computer readable storage medium and a computer program product, and relates to the technical field of big data. The method comprises the following steps: obtaining a key value pair of target external data to be monitored in a deviation degree, wherein the target external data is data generated in a business operation process, determining an initial model from a plurality of candidate models according to a preset key value rule and the key value pair, determining a target model according to attribute information of the key value pair and the initial model, and obtaining the deviation degree of the target external data based on the target external data, the key value pair and the target model. The method can effectively improve the flexibility of deviation monitoring.
Description
Technical Field
The present application relates to the field of big data technology, and in particular, to a deviation monitoring method, a deviation monitoring device, a computer readable storage medium and a computer program product.
Background
With the construction of digital transformation, data plays an increasingly important role, and in order to ensure the quality of the data, the data needs to be monitored to achieve the purpose of checking the quality of the data, wherein the deviation degree of the data is an important index for representing the quality of the data.
In the prior art, an expert rule model is generally adopted to monitor the deviation degree of data, namely, an expert specifically customized model according to rules aiming at a certain type of data or a certain use scene, and the deviation degree of the data is monitored through the model.
However, the flexibility of bias monitoring using such expert rule models is poor.
Disclosure of Invention
In view of the foregoing, it is desirable to provide a deviation monitoring method, apparatus, computer device, computer readable storage medium, and computer program product that have high flexibility.
In a first aspect, the present application provides a method of deviation monitoring. The method comprises the following steps:
acquiring a key value pair of target external data to be monitored by the deviation, wherein the target external data is data generated in the service operation process; determining an initial model from a plurality of candidate models according to a preset key value rule and the key value pair; and determining a target model according to the attribute information of the key value pair and the initial model, and acquiring the deviation degree of the target external data based on the target external data, the key value pair and the target model.
In one embodiment, the obtaining the key value pair of the target external data to be monitored by the deviation includes: determining the target external data from an external database according to a preset screening condition, wherein the screening condition is related to at least one of a deviation monitoring time period and a service type; and carrying out unified processing on the target external data to obtain the key value pair.
In one embodiment, the unifying the target external data to obtain the key value pair includes: carrying out data analysis on source data corresponding to the target external data to obtain the data content of the target external data; and carrying out unified processing on the target external data according to the source data and the data content to obtain the key value pair.
In one embodiment, the unifying the target external data according to the source data and the data content to obtain the key value pair includes: if the source data is a JSON file, based on a serialization tool, the source data and the data content perform unified processing on the target external data to obtain the key value pair; if the source data is a database source file, the source data and the data content are subjected to unified processing on the target external data based on a reflection tool, so that the key value pair is obtained.
In one embodiment, the attribute information includes a data acquisition time, and determining the target model according to the attribute information of the key value pair and the initial model includes: comparing the data acquisition time with the model update time of the initial model to determine whether the initial model fails; if the initial model fails, training the initial model by using the reserved data and the target external data to obtain the target model, wherein the data dimension of the reserved data and the data dimension of the target external data are the same.
In one embodiment, the method further comprises: if the initial model does not fail, determining the initial model as the target model.
In a second aspect, the present application also provides a deviation monitoring device. The device comprises:
the acquisition module is used for acquiring key value pairs of target external data to be monitored in a deviation degree, wherein the target external data are data generated in the service operation process;
the determining module is used for determining an initial model from a plurality of candidate models according to a preset key value rule and the key value pair;
and the first execution module is used for determining a target model according to the attribute information of the key value pair and the initial model, and acquiring the deviation degree of the target external data based on the target external data, the key value pair and the target model.
In one embodiment, the obtaining module is specifically configured to: determining the target external data from an external database according to a preset screening condition, wherein the screening condition is related to at least one of a deviation monitoring time period and a service type; and carrying out unified processing on the target external data to obtain the key value pair.
In one embodiment, the obtaining module is specifically configured to: carrying out data analysis on source data corresponding to the target external data to obtain the data content of the target external data; and carrying out unified processing on the target external data according to the source data and the data content to obtain the key value pair.
In one embodiment, the obtaining module is specifically configured to: if the source data is a JSON file, based on a serialization tool, the source data and the data content perform unified processing on the target external data to obtain the key value pair; if the source data is a database source file, the source data and the data content are subjected to unified processing on the target external data based on a reflection tool, so that the key value pair is obtained.
In one embodiment, the first execution module is specifically configured to: comparing the data acquisition time with the model update time of the initial model to determine whether the initial model fails; if the initial model fails, training the initial model by using the reserved data and the target external data to obtain the target model, wherein the data dimension of the reserved data and the data dimension of the target external data are the same.
In one embodiment, the deviation monitoring device further includes a second execution module for: if the initial model does not fail, determining the initial model as the target model.
In a third aspect, the present application also provides a computer device. The computer device comprises a memory storing a computer program and a processor implementing the steps of any of the above first aspects when the computer program is executed.
In a fourth aspect, the present application also provides a computer-readable storage medium. A computer readable storage medium having stored thereon a computer program which when executed by a processor performs the steps of any of the first aspects described above.
In a fifth aspect, the present application also provides a computer program product. The computer program product comprising a computer program which, when executed by a processor, implements the steps of any of the first aspects described above.
The deviation monitoring method, the device, the computer equipment, the computer readable storage medium and the computer program product firstly acquire a key value pair of target external data to be monitored, wherein the target external data is data generated in a service operation process, then determine an initial model from a plurality of candidate models according to a preset key value rule and the key value pair, finally determine a target model according to attribute information of the key value pair and the initial model, and acquire the deviation degree of the target external data based on the target external data, the key value pair and the target model. The initial model in the deviation monitoring method is determined from a plurality of candidate models based on a preset key value rule and key value pairs, different target external data can correspond to different initial models, namely, the initial model can be flexibly selected for different target external data, after the initial model is determined, the initial model is further adjusted based on attribute information of the key value pairs to determine the target model, and finally, the deviation of the target external data is acquired based on the target model, so that the accuracy of the deviation of the obtained target external data is higher.
Drawings
FIG. 1 is a flow chart of a method for monitoring deviation in one embodiment;
FIG. 2 is a flow chart of obtaining key value pairs of target external data to be monitored for deviation in one embodiment;
FIG. 3 is a flowchart of a unified process for obtaining key value pairs for target external data in one embodiment;
FIG. 4 is a flow diagram of determining a target model in one embodiment;
FIG. 5 is a flow chart of a method for detecting deviation according to another embodiment;
FIG. 6 is a block diagram of a deviation monitoring device in one embodiment;
FIG. 7 is a block diagram of a deviation monitoring device in one embodiment;
fig. 8 is an internal structural diagram of a computer device in one embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application will be further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the present application.
With the construction of digital transformation, data plays an increasingly important role, and in order to ensure the quality of the data, the data needs to be monitored to achieve the purpose of checking the quality of the data, wherein the deviation degree of the data is an important index for representing the quality of the data.
In the prior art, an expert rule model is generally adopted to monitor the deviation degree of data, namely, an expert specifically customized model according to rules aiming at a certain type of data or a certain use scene, and the deviation degree of the data is monitored through the model.
However, such expert rules models are less flexible.
In view of this, the present application provides a deviation monitoring method, which can effectively improve flexibility.
According to the deviation monitoring method provided by the embodiment of the application, the execution main body can be a computer device, and the computer device can be a server.
In one embodiment, as shown in fig. 1, a deviation monitoring method is provided, which includes the following steps:
and 101, acquiring a key value pair of target external data to be monitored for the deviation.
The target external data is data generated in the service operation process.
The degree of deviation is an index for characterizing the quality of data, and refers to the proportion of the predicted data that is occupied by the absolute value of the difference between the real data and the predicted data.
The key-value pair refers to a basic data representation mode and can comprise keys and values, wherein the keys are numbers of stored values, the values are data to be stored, and the key-value pair can comprise a data source database, a data source data table, data acquisition time and data granularity information.
In one possible implementation, the key value pair of the target external data to be monitored for the degree of deviation may be obtained by writing a conversion tool.
In another possible implementation manner, the key value pair of the target external data to be monitored by the deviation degree may also be obtained through a dependency package, for example, a pore dependency may be introduced first, then a corresponding processing class may be introduced, finally a corresponding method is called, for example, the entity is entity, and finally the key value pair of the target external data to be monitored by the deviation degree may be obtained through map.
In another possible implementation, the key value pair of the target external data to be monitored for the degree of deviation may also be acquired by serialization tools.
In another possible implementation, the key value pair of the target external data to be monitored for the degree of deviation may also be acquired by a reflection tool.
The preset key value rule can be set in advance by a technician.
Alternatively, based on the preset key value rule, the computer device may determine the type of the target external data according to the key value pair of the target external data.
In one possible implementation manner, the service type of the target external data may be determined as deposit data through the key value pair of the target external data based on the preset key value rule.
In another possible implementation manner, the business type of the target external data can be determined to be withdrawal data through the key value pair of the target external data based on the preset key value rule.
Optionally, based on the preset key value rule, the computer device may determine the source database of the target external data according to the key value pair of the target external data.
In one possible implementation manner, the source database of the target external data may be determined to be the a database through the key value pair of the target external data based on the preset key value rule.
In another possible implementation manner, the source database of the target external data may be determined to be the B database through the key value pair of the target external data based on the preset key value rule.
Optionally, based on the preset key value rule, the computer device may determine the source data table of the target external data according to the key value pair of the target external data.
In one possible implementation manner, the source data table of the target external data may be determined to be an a source data table through the key value pair of the target external data based on the preset key value rule.
In another possible implementation manner, the source data table of the target external data may be determined to be a B source data table through the key value pair of the target external data based on the preset key value rule.
Optionally, based on the preset key value rule, the computer device may determine data granularity information of the target external data according to the key value pair of the target external data.
Alternatively, the candidate model refers to a neural network model, such as a back propagation neural network model (BPNN, back Propagation Neural Network).
In an alternative embodiment of the present application, the initial model may be determined from a plurality of candidate models.
In one possible implementation, as described above, the type, the source database, the source data table and the data granularity information of the target external data may be determined based on the preset key value rule through the key value pair of the target external data, so that there are multiple models correspondingly, where the multiple models are candidate models, and the initial model may be determined from the multiple candidate models after the type, the source database, the source data table and the data granularity information of the target external data are determined based on the preset key value rule and the key value pair.
In another possible implementation manner, the initial model may be determined from the multiple candidate models through historical retention data, a preset key value rule and the key value pair, where the historical retention data refers to model evaluation data that is retained in the historical retention, and will be described in detail below, and will not be repeated herein.
In another possible implementation manner, each of the plurality of candidate models may be determined as an initial model, so as to obtain a model evaluation of each candidate model for monitoring the deviation of the target external data, where the model evaluation may be used as historical retention data for determining a subsequent initial model, and the model evaluation is used as a basis for determining the initial model later, and will not be described in detail herein.
And step 103, determining a target model according to the attribute information of the key value pair and the initial model, and acquiring the deviation degree of the target external data based on the target external data, the key value pair and the target model.
Alternatively, the attribute information may be attribute information of the target external data, and the attribute information of the target external data may be directly read through a key value pair of the target external data.
In one possible implementation manner, the initial model may be further adjusted according to the attribute information of the key value pair to determine a target model, and the target external data and the key value pair of the target external data are input to the target model to obtain the deviation degree of the target external data.
In an optional embodiment of the present application, the target external data and the key value pair of the target external data are input to the target model, and the target model may further output a model evaluation, where the model evaluation is the historical retention data described above, and the model evaluation may be used as the historical retention data for determining a subsequent initial model, and as a basis for determining the initial model later, for example, the model evaluation may be that the model fitting degree is better, the model evaluation may also be that the model fitting degree is worse, and the model evaluation refers to whether the model is suitable for performing the deviation degree calculation with respect to the target external data.
In one possible implementation manner, after the target external data and the key value are input into the target model, the target model outputs the deviation degree of the target external data and the model evaluation of the target model, as shown in table 1, where R refers to a model evaluation index for determining whether the model fitting degree is good, R refers to e [0.6,1 ], and the closer the R is to the central region of the interval, the better the model fitting degree is.
TABLE 1
| ID | Date of day | Service name | Service index value | Traffic prediction value | Degree of deviation | R square | Model evaluation |
| ... | ... | ... | ... | ... | 0.3 | 0.86 | The fitting degree is good |
| ... | ... | ... | ... | ... | 0.1 | 0.88 | The fitting degree is good |
| ... | ... | ... | ... | ... | 0.5 | 0.65 | Poor fitting degree |
The deviation monitoring method, the device, the computer equipment, the storage medium and the computer program product are characterized in that key value pairs of target external data to be monitored for the deviation are firstly obtained, the target external data are data generated in the service operation process, then an initial model is determined from a plurality of candidate models according to a preset key value rule and the key value pairs, finally a target model is determined according to attribute information of the key value pairs and the initial model, and the deviation of the target external data is obtained based on the target external data, the key value pairs and the target model. According to the deviation monitoring method, an initial model is determined from a plurality of candidate models based on a preset key value rule and key value pairs, the initial model is not fixed, different target external data can correspond to different initial models, namely the initial model can be flexibly selected for different target external data, after the initial model is determined, the target model is determined according to attribute information of the key value pairs and the initial model, namely the initial model is further adjusted based on the attribute information of the target external data key value pairs to determine the target model, finally the deviation of the target external data is obtained based on the target model, so that the deviation accuracy of the obtained target external data is higher, and in combination, the deviation monitoring method provided by the application is higher in flexibility.
In one embodiment, as shown in fig. 2, the obtaining the key value pair of the target external data to be monitored for the deviation includes the following steps:
Alternatively, the preset screening conditions may be set in advance by a technician.
As described above, the screening conditions may be associated with a bias monitoring period.
In one possible implementation, the preset screening condition may be a deviation monitoring period, for example, the preset screening condition is one month to four months, and the data of one month to four months may be determined from the external database as the target external data according to the preset screening condition.
In another possible implementation, the preset screening condition may be a deviation monitoring period, for example, the preset screening condition is 2022 years, and the data of 2022 years may be determined from the external database as the target external data according to the preset screening condition.
As described above, the screening conditions may also be related to traffic type.
In one possible implementation manner, the preset screening condition may also be a service type, for example, the preset screening condition is a transfer income, and the data of which the service type is the transfer income may be determined from an external database according to the preset screening condition as target external data.
In another possible implementation manner, the preset screening condition may also be a service type, for example, the preset screening condition is consumption expense, and then the data of which the service type is consumption expense may be determined from the external database according to the preset screening condition to be target external data.
In another possible implementation manner, the preset screening condition may also be a service type, for example, the preset screening condition is credit card consumption, and then it may be determined from an external database that data of which the service type is credit card consumption is target external data according to the preset screening condition.
As described above, the screening conditions may also be related to both the traffic type feed and the departure monitoring period.
In one possible implementation manner, the preset screening condition may also be a service type and deviation monitoring period, for example, the preset screening condition is the transfer revenue from one month to four months, and the data of the service type from one month to four months as the transfer revenue may be determined from the external database according to the preset screening condition as the target external data.
In another possible implementation manner, the preset screening condition may also be a service type and deviation monitoring period, for example, the preset screening condition is 2022 years of transfer revenue, and then the data of 2022 years of service type as transfer revenue may be determined from an external database according to the preset screening condition to be target external data.
In one possible implementation manner, since the target external data is determined based on the external database, where the external database includes file data, database data, message queue data, and the like, the data structures in the external database are not uniform, so that the data structures of the target external data determined from the external database may not be uniform, and therefore, it is necessary to perform unified processing on the target external data to obtain a key value pair of the target external data, so as to ensure operation efficiency.
In one embodiment, as shown in fig. 3, the unifying processing is performed on the target external data to obtain the key value pair, which includes the following steps:
Optionally, the source data refers to data directly from the source file, such as data of a business system database, data of an offline file, ioT data, and the like.
Alternatively, the data content may include a data source database, a data source data table, data time, data granularity information, and the like.
In one possible implementation, the location of the source data of the target external data may be determined through an interface and then consumed to obtain the data content of the target external data.
And 302, performing unified processing on the target external data according to the source data and the data content to obtain the key value pair.
In an optional embodiment of the present application, the performing unified processing on the target external data according to the source data and the data content to obtain the key value pair includes: if the source data is a JSON file, based on a serialization tool, the source data and the data content perform unified processing on the target external data to obtain the key value pair; if the source data is a database source file, the source data and the data content are subjected to unified processing on the target external data based on a reflection tool, so that the key value pair is obtained.
In one possible implementation, if the source data is determined to be a JSON file, the target external data may be processed using a serialization tool, such as FastJson, to obtain key-value pairs for the target external data, as shown in table 2.
In another possible implementation manner, if it is determined that the source data is not a JSON file, the reflection tool may be further used to process the target external data to obtain a key value pair of the target external data, specifically, each row element of the target external data is named as an object, an attribute of the object may correspond to a combination of a column name and a value, and the key value pair of the target external data obtained by the method may also be shown in table 2.
In an optional embodiment of the present application, the unified processing is performed on the target external data according to the source data and the data content, and after the key value is obtained, the persistence processing is further performed on the target external data, that is, the key value pair of the target external data is stored in the external database, so that subsequent calling of the key value pair of the target external data is facilitated.
In one embodiment, as shown in fig. 4, the attribute information includes a data acquisition time, and the determining the target model according to the attribute information of the key value pair and the initial model includes the following steps:
Optionally, the Data acquisition time is the Data time data_data in table 1, and the Data acquisition time refers to the acquisition time of the external Data of the target.
Optionally, the model update time of the initial model refers to the last model update time.
In one possible implementation, if the difference between the data acquisition time and the model update time of the initial model is greater than or equal to a preset time threshold, the preset time threshold may be preset by a technician, for example, the acquisition time of the target external data is 2022, 3, 1, and the model update time of the initial model is 2021, 4, 1, and the preset time threshold is 6, and if the difference between the data acquisition time and the model update time of the initial model is greater than the preset time threshold, the initial model failure may be determined.
In another possible implementation, if the difference between the data acquisition time and the model update time of the initial model is less than a preset time threshold, the preset time threshold may be preset by a technician, for example, the acquisition time of the target external data is 2022, 3, 1, and the model update time of the initial model is 2022, 1, and the preset time threshold is 6 months, and if the difference between the data acquisition time and the model update time of the initial model is less than the preset time threshold, it may be determined that the initial model is not dead.
As described above, after the target external data is unified to obtain the key value pair in each operation process, the target external data is subjected to persistence processing, that is, the key value pair is stored in the target external database, the persistence data may be screened from the data subjected to persistence processing in the external database, the persistence data may have the same data dimensions except that the data acquisition time is different from the data dimensions of the target external data, for example, the data acquisition time of the target external data is 2022, 10, 1, and the persistence data screening preset time is 30 days, the persistence data should be 2022, 9, 1, and 2022, 9, 30 days, and the other data dimensions of the data may be the same as the target external data, and the persistence data screening preset time may be preset by a technician.
In one possible implementation manner, when the initial model is determined to be invalid, screening preset time according to the target external data and the retention data to determine the retention data, and performing mixing processing on the target external data and the retention data to obtain training data, training the initial model based on the training data, wherein the trained initial model is the target model.
In one embodiment, the method further comprises: if the initial model does not fail, determining the initial model as the target model.
The method for obtaining the target model by firstly determining whether the initial model fails or not and training the initial model by using the reserved data and the target external data if the initial model fails, wherein the model is not fixed but continuously updated, so that the problem of poor accuracy of deviation monitoring caused by larger fluctuation of data values in special holidays or events can be avoided.
In one embodiment, as shown in fig. 5, a deviation monitoring method is provided, the method comprising the steps of:
It should be understood that, although the steps in the flowcharts related to the embodiments described above are sequentially shown as indicated by arrows, these steps are not necessarily sequentially performed in the order indicated by the arrows. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least some of the steps in the flowcharts described in the above embodiments may include a plurality of steps or a plurality of stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of the steps or stages is not necessarily performed sequentially, but may be performed alternately or alternately with at least some of the other steps or stages.
Based on the same inventive concept, the embodiment of the application also provides a deviation monitoring device for realizing the above related deviation monitoring method. The implementation of the solution provided by the device is similar to that described in the above method, so specific limitations in the embodiments of one or more deviation monitoring devices provided below may be referred to above for limitations of the deviation monitoring method, and will not be described herein.
In one embodiment, as shown in fig. 6, there is provided a deviation monitoring apparatus including: the device comprises an acquisition module, a determination module and a first execution module, wherein:
the acquisition module is used for acquiring key value pairs of target external data to be monitored in a deviation degree, wherein the target external data are data generated in the service operation process;
the determining module is used for determining an initial model from a plurality of candidate models according to a preset key value rule and the key value pair;
and the first execution module is used for determining a target model according to the attribute information of the key value pair and the initial model, and acquiring the deviation degree of the target external data based on the target external data, the key value pair and the target model.
In one embodiment, the obtaining module is specifically configured to: determining the target external data from an external database according to a preset screening condition, wherein the screening condition is related to at least one of a deviation monitoring time period and a service type; and carrying out unified processing on the target external data to obtain the key value pair.
In one embodiment, the obtaining module is specifically configured to: carrying out data analysis on source data corresponding to the target external data to obtain the data content of the target external data; and carrying out unified processing on the target external data according to the source data and the data content to obtain the key value pair.
In one embodiment, the obtaining module is specifically configured to: if the source data is a JSON file, based on a serialization tool, the source data and the data content perform unified processing on the target external data to obtain the key value pair; if the source data is a database source file, the source data and the data content are subjected to unified processing on the target external data based on a reflection tool, so that the key value pair is obtained.
In one embodiment, the first execution module is specifically configured to: comparing the data acquisition time with the model update time of the initial model to determine whether the initial model fails; if the initial model fails, training the initial model by using the reserved data and the target external data to obtain the target model, wherein the data dimension of the reserved data and the data dimension of the target external data are the same.
In one embodiment, as shown in fig. 7, another deviation monitoring apparatus 700 is provided, where the deviation monitoring apparatus 700 includes, in addition to the modules included in the deviation monitoring apparatus 600, a second execution module 604, where the second execution module 604 is configured to: if the initial model does not fail, determining the initial model as the target model.
The modules in the deviation monitoring device can be implemented in whole or in part by software, hardware, and a combination thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.
In one embodiment, a computer device is provided, which may be a server, and the internal structure of which may be as shown in fig. 8. The computer device includes a processor, a memory, an Input/Output interface (I/O) and a communication interface. The processor, the memory and the input/output interface are connected through a system bus, and the communication interface is connected to the system bus through the input/output interface. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database of the computer device is for storing data. The input/output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a method of deviation monitoring.
In one embodiment, a computer device is provided comprising a memory and a processor, the memory having stored therein a computer program, the processor when executing the computer program performing the steps of:
acquiring a key value pair of target external data to be monitored by the deviation, wherein the target external data is data generated in the service operation process; determining an initial model from a plurality of candidate models according to a preset key value rule and the key value pair; and determining a target model according to the attribute information of the key value pair and the initial model, and acquiring the deviation degree of the target external data based on the target external data, the key value pair and the target model.
In one embodiment, the acquiring the key value pair of the target external data to be monitored for the deviation, when the processor executes the computer program, implements the following steps: determining the target external data from an external database according to a preset screening condition, wherein the screening condition is related to at least one of a deviation monitoring time period and a service type; and carrying out unified processing on the target external data to obtain the key value pair.
In one embodiment, the unified processing is performed on the target external data to obtain the key value pair, and the processor implements the following steps when executing the computer program: carrying out data analysis on source data corresponding to the target external data to obtain the data content of the target external data; and carrying out unified processing on the target external data according to the source data and the data content to obtain the key value pair.
In one embodiment, the unified processing is performed on the target external data according to the source data and the data content to obtain the key value pair, and the processor implements the following steps when executing the computer program: if the source data is a JSON file, based on a serialization tool, the source data and the data content perform unified processing on the target external data to obtain the key value pair; if the source data is a database source file, the source data and the data content are subjected to unified processing on the target external data based on a reflection tool, so that the key value pair is obtained.
In one embodiment, the attribute information includes a data acquisition time, the determining the target model according to the attribute information of the key value pair and the initial model, the processor executing the computer program to implement the steps of: comparing the data acquisition time with the model update time of the initial model to determine whether the initial model fails; if the initial model fails, training the initial model by using the reserved data and the target external data to obtain the target model, wherein the data dimension of the reserved data and the data dimension of the target external data are the same.
In one embodiment, the processor, when executing the computer program, performs the steps of: if the initial model does not fail, determining the initial model as the target model.
In one embodiment, a computer readable storage medium is provided having a computer program stored thereon, which when executed by a processor, performs the steps of:
acquiring a key value pair of target external data to be monitored by the deviation, wherein the target external data is data generated in the service operation process; determining an initial model from a plurality of candidate models according to a preset key value rule and the key value pair; and determining a target model according to the attribute information of the key value pair and the initial model, and acquiring the deviation degree of the target external data based on the target external data, the key value pair and the target model.
In one embodiment, the acquiring key value pairs of the target external data to be monitored for the degree of deviation, when executed by the processor, implements the steps of: determining the target external data from an external database according to a preset screening condition, wherein the screening condition is related to at least one of a deviation monitoring time period and a service type; and carrying out unified processing on the target external data to obtain the key value pair.
In one embodiment, the unified processing is performed on the target external data to obtain the key value pair, and the computer program when executed by the processor implements the following steps: carrying out data analysis on source data corresponding to the target external data to obtain the data content of the target external data; and carrying out unified processing on the target external data according to the source data and the data content to obtain the key value pair.
In one embodiment, the unified processing is performed on the target external data according to the source data and the data content to obtain the key value pair, and the computer program when executed by the processor implements the following steps: if the source data is a JSON file, based on a serialization tool, the source data and the data content perform unified processing on the target external data to obtain the key value pair; if the source data is a database source file, the source data and the data content are subjected to unified processing on the target external data based on a reflection tool, so that the key value pair is obtained.
In one embodiment, the attribute information includes a data acquisition time, the determining the target model based on the attribute information of the key-value pair and the initial model, the computer program when executed by the processor implementing the steps of: comparing the data acquisition time with the model update time of the initial model to determine whether the initial model fails; if the initial model fails, training the initial model by using the reserved data and the target external data to obtain the target model, wherein the data dimension of the reserved data and the data dimension of the target external data are the same.
In one embodiment, the computer program when executed by a processor performs the steps of: if the initial model does not fail, determining the initial model as the target model.
In one embodiment, a computer program product is provided comprising a computer program which, when executed by a processor, performs the steps of:
acquiring a key value pair of target external data to be monitored by the deviation, wherein the target external data is data generated in the service operation process; determining an initial model from a plurality of candidate models according to a preset key value rule and the key value pair; and determining a target model according to the attribute information of the key value pair and the initial model, and acquiring the deviation degree of the target external data based on the target external data, the key value pair and the target model.
In one embodiment, the acquiring key value pairs of the target external data to be monitored for the degree of deviation, when executed by the processor, implements the steps of: determining the target external data from an external database according to a preset screening condition, wherein the screening condition is related to at least one of a deviation monitoring time period and a service type; and carrying out unified processing on the target external data to obtain the key value pair.
In one embodiment, the unified processing is performed on the target external data to obtain the key value pair, and the computer program when executed by the processor implements the following steps: carrying out data analysis on source data corresponding to the target external data to obtain the data content of the target external data; and carrying out unified processing on the target external data according to the source data and the data content to obtain the key value pair.
In one embodiment, the unified processing is performed on the target external data according to the source data and the data content to obtain the key value pair, and the computer program when executed by the processor implements the following steps: if the source data is a JSON file, based on a serialization tool, the source data and the data content perform unified processing on the target external data to obtain the key value pair; if the source data is a database source file, the source data and the data content are subjected to unified processing on the target external data based on a reflection tool, so that the key value pair is obtained.
In one embodiment, the attribute information includes a data acquisition time, the determining the target model based on the attribute information of the key-value pair and the initial model, the computer program when executed by the processor implementing the steps of: comparing the data acquisition time with the model update time of the initial model to determine whether the initial model fails; if the initial model fails, training the initial model by using the reserved data and the target external data to obtain the target model, wherein the data dimension of the reserved data and the data dimension of the target external data are the same.
In one embodiment, the computer program when executed by a processor performs the steps of: if the initial model does not fail, determining the initial model as the target model.
It should be noted that, the user information (including, but not limited to, user equipment information, user personal information, etc.) and the data (including, but not limited to, data for analysis, stored data, presented data, etc.) referred to in the present application are information and data authorized by the user or sufficiently authorized by each party, and the collection, use and processing of the related data are required to comply with the related laws and regulations and standards of the related countries and regions.
Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed may comprise the steps of the embodiments of the methods described above. Any reference to memory, database, or other medium used in the various embodiments provided herein may include at least one of non-volatile and volatile memory. The nonvolatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical Memory, high density embedded nonvolatile Memory, resistive random access Memory (ReRAM), magnetic random access Memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric Memory (Ferroelectric Random Access Memory, FRAM), phase change Memory (Phase Change Memory, PCM), graphene Memory, and the like. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, and the like. By way of illustration, and not limitation, RAM can be in the form of a variety of forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), and the like. The databases referred to in the various embodiments provided herein may include at least one of relational databases and non-relational databases. The non-relational database may include, but is not limited to, a blockchain-based distributed database, and the like. The processors referred to in the embodiments provided herein may be general purpose processors, central processing units, graphics processors, digital signal processors, programmable logic units, quantum computing-based data processing logic units, etc., without being limited thereto.
The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.
The above examples only represent a few embodiments of the present application, which are described in more detail and are not to be construed as limiting the scope of the present application. It should be noted that it would be apparent to those skilled in the art that various modifications and improvements could be made without departing from the spirit of the present application, which would be within the scope of the present application. Accordingly, the scope of protection of the present application shall be subject to the appended claims.
Claims (10)
1. A method of bias monitoring, the method comprising:
acquiring a key value pair of target external data to be monitored by the deviation, wherein the target external data is data generated in the service operation process;
determining an initial model from a plurality of candidate models according to a preset key value rule and the key value pairs;
and determining a target model according to the attribute information of the key value pair and the initial model, and acquiring the deviation degree of the target external data based on the target external data, the key value pair and the target model.
2. The method according to claim 1, wherein the acquiring the key value pair of the target external data to be monitored for the degree of deviation includes:
determining the target external data from an external database according to preset screening conditions, wherein the screening conditions are related to at least one of a deviation monitoring time period and a service type;
and carrying out unified processing on the target external data to obtain the key value pair.
3. The method according to claim 2, wherein the unifying the target external data to obtain the key value pair includes:
carrying out data analysis on source data corresponding to the target external data to obtain the data content of the target external data;
and carrying out unified processing on the target external data according to the source data and the data content to obtain the key value pair.
4. The method according to claim 3, wherein the unifying the target external data according to the source data and the data content to obtain the key value pair includes:
if the source data is a JSON file, based on a serialization tool, the source data and the data content are subjected to unified processing on the target external data to obtain the key value pair;
And if the source data is a database source file, carrying out unified processing on the target external data based on a reflection tool and the data content to obtain the key value pair.
5. The method of claim 1, wherein the attribute information includes a data acquisition time, and wherein determining a target model from the attribute information of the key-value pairs and the initial model includes:
comparing the data acquisition time with the model update time of the initial model, and determining whether the initial model fails;
and if the initial model fails, training the initial model by using the reserved data and the target external data to obtain the target model, wherein the data dimensions of the reserved data and the target external data are the same.
6. The method of claim 5, wherein the method further comprises:
and if the initial model does not fail, determining the initial model as the target model.
7. A deviation monitoring device, the device comprising:
the acquisition module is used for acquiring key value pairs of target external data to be monitored in a deviation degree, wherein the target external data are data generated in the service operation process;
The determining module is used for determining an initial model from a plurality of candidate models according to a preset key value rule and the key value pairs;
and the first execution module is used for determining a target model according to the attribute information of the key value pair and the initial model, and acquiring the deviation degree of the target external data based on the target external data, the key value pair and the target model.
8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the steps of the method of any of claims 1 to 6 when the computer program is executed.
9. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 6.
10. A computer program product comprising a computer program, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310372224.4A CN116401238A (en) | 2023-04-10 | 2023-04-10 | Deviation monitoring method, apparatus, device, storage medium and program product |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310372224.4A CN116401238A (en) | 2023-04-10 | 2023-04-10 | Deviation monitoring method, apparatus, device, storage medium and program product |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116401238A true CN116401238A (en) | 2023-07-07 |
Family
ID=87017525
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310372224.4A Pending CN116401238A (en) | 2023-04-10 | 2023-04-10 | Deviation monitoring method, apparatus, device, storage medium and program product |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116401238A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117726053A (en) * | 2024-02-09 | 2024-03-19 | 广州市威士丹利智能科技有限公司 | Carbon emission monitoring method and system applied to digital platform system |
-
2023
- 2023-04-10 CN CN202310372224.4A patent/CN116401238A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117726053A (en) * | 2024-02-09 | 2024-03-19 | 广州市威士丹利智能科技有限公司 | Carbon emission monitoring method and system applied to digital platform system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN116401238A (en) | Deviation monitoring method, apparatus, device, storage medium and program product | |
| CN116468543A (en) | Credit risk assessment method, device, equipment and medium based on federal learning | |
| CN115759742A (en) | Enterprise risk assessment method and device, computer equipment and storage medium | |
| CN114238258B (en) | Database data processing method, device, computer equipment and storage medium | |
| CN116610664B (en) | Data monitoring method, device, computer equipment, storage medium and product | |
| CN117078112B (en) | Energy consumption detection method and data analysis system applied to enterprise abnormal electricity management | |
| CN111177132B (en) | Method, device, equipment and storage medium for cleaning label of relational data | |
| CN116521546A (en) | Interface performance adjusting method and device, computer equipment and storage medium | |
| CN117556473A (en) | Data processing method, device, computer equipment and storage medium | |
| CN117319488A (en) | Message pushing method, device, computer equipment and storage medium | |
| CN116401323A (en) | Index data processing method and device and computer equipment | |
| CN117312306A (en) | Financial business data sheet conversion method, apparatus, device, medium and program product | |
| CN117668355A (en) | Role recommendation method and device for virtual object, computer equipment and storage medium | |
| CN117522138A (en) | Method, device, equipment and medium for identifying testing risk of financial business system | |
| CN118152414A (en) | Data processing method, apparatus, device, storage medium, and program product | |
| CN115330122A (en) | Operation data processing method and device, computer equipment and storage medium | |
| CN116467521A (en) | Training method of account recommendation model, account recommendation method, device and equipment | |
| CN116452308A (en) | Risk assessment method, apparatus, computer device, storage medium, and program product | |
| CN117077917A (en) | Personnel capability assessment method and device based on low-code platform and computer equipment | |
| CN116091209A (en) | Credit service processing method, apparatus, computer device and storage medium | |
| CN116821817A (en) | Data prediction method and device based on joint tree model and computer equipment | |
| CN117407746A (en) | User classification method, apparatus, computer device and storage medium | |
| CN116307711A (en) | Subscription data processing method, device, computer equipment and storage medium | |
| CN115859156A (en) | Application resource intensive type classification method and device and computer equipment | |
| CN116089183A (en) | Graphics processor performance test method, device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |