CN117421312A - Data processing method, device, computer equipment and storage medium - Google Patents
Data processing method, device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN117421312A CN117421312A CN202311315525.XA CN202311315525A CN117421312A CN 117421312 A CN117421312 A CN 117421312A CN 202311315525 A CN202311315525 A CN 202311315525A CN 117421312 A CN117421312 A CN 117421312A
- Authority
- CN
- China
- Prior art keywords
- data
- processed
- preprocessing
- preprocessed
- cleaning
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 26
- 238000003860 storage Methods 0.000 title claims abstract description 19
- 238000004140 cleaning Methods 0.000 claims abstract description 60
- 238000007781 pre-processing Methods 0.000 claims abstract description 50
- 238000012545 processing Methods 0.000 claims abstract description 43
- 239000000284 extract Substances 0.000 claims abstract description 13
- 238000000034 method Methods 0.000 claims description 35
- 238000012216 screening Methods 0.000 claims description 14
- 238000013473 artificial intelligence Methods 0.000 abstract description 8
- 238000010586 diagram Methods 0.000 description 15
- 238000012360 testing method Methods 0.000 description 12
- 238000005516 engineering process Methods 0.000 description 10
- 230000008569 process Effects 0.000 description 7
- 238000009826 distribution Methods 0.000 description 5
- 230000002159 abnormal effect Effects 0.000 description 4
- 238000004891 communication Methods 0.000 description 4
- 238000011161 development Methods 0.000 description 4
- 230000018109 developmental process Effects 0.000 description 4
- 230000008520 organization Effects 0.000 description 4
- 230000006870 function Effects 0.000 description 3
- 230000003993 interaction Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 230000002776 aggregation Effects 0.000 description 2
- 238000004220 aggregation Methods 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 238000009960 carding Methods 0.000 description 2
- 230000006835 compression Effects 0.000 description 2
- 238000007906 compression Methods 0.000 description 2
- 238000010276 construction Methods 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 238000011156 evaluation Methods 0.000 description 2
- 239000013589 supplement Substances 0.000 description 2
- 230000001960 triggered effect Effects 0.000 description 2
- 230000032683 aging Effects 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000013523 data management Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 235000019800 disodium phosphate Nutrition 0.000 description 1
- 230000007717 exclusion Effects 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000003058 natural language processing Methods 0.000 description 1
- 238000012552 review Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/252—Integrating or interfacing systems involving database management systems between a Database Management System and a front-end application
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the application belongs to the field of artificial intelligence and big data, is applied to the field of smart cities, and relates to a data processing method, which comprises the steps of obtaining first data to be processed of interface operation and storing the first data into a data record table; performing first data cleaning on first data to be processed in a data record table to obtain first preprocessing data, and storing the first preprocessing data into a data pre-generation table; and based on the first preprocessing data of the data preprocessing table, performing second data cleaning to obtain second preprocessing data, and storing the second preprocessing data into a data result table so that a user can extract corresponding target data from the data result table according to the input condition. The application also provides a data processing device, computer equipment and a storage medium. The application solves the problem that the multiplexing degree of data is low, so that the working efficiency is reduced.
Description
Technical Field
The present disclosure relates to the field of artificial intelligence and big data technologies, and in particular, to a data processing method, apparatus, computer device, and storage medium.
Background
In the traditional waterfall iteration or agile iteration, manual test execution is a loop with relatively large test investment, wherein test data of each scene is built, the loop is a precondition link of test execution, the link is often influenced by factors such as joint debugging relatives, business familiarity and the like to influence the progress, how to provide test data with high availability for testers, and test data generated through various business operations is an important entry point for improving the test efficiency if the test data are recycled. Traditional test data mainly depend on manual experience maintenance or dead data writing, flexibility is poor, reusability for many business scenes is low, and manual counting actions of historical scenes often need to be repeated. The existing data has low multiplexing degree, so that the working efficiency is reduced.
Disclosure of Invention
An embodiment of the application aims to provide a data processing method, a data processing device, computer equipment and a storage medium, so as to solve the problem of low data multiplexing degree and reduced working efficiency.
In order to solve the above technical problems, the embodiments of the present application provide a data processing method, which adopts the following technical schemes:
Acquiring first data to be processed of interface operation and storing the first data into a data record table;
performing first data cleaning on first data to be processed in the data record table to obtain first preprocessing data, and storing the first preprocessing data into a data pre-generation table;
and based on the first preprocessing data of the data preprocessing table, performing second data cleaning to obtain second preprocessing data, and storing the second preprocessing data into a data result table so that a user can extract corresponding target data from the data result table according to the input condition.
Further, the step of obtaining the first data to be processed of the interface operation includes:
collecting log data of interface operation of a service database;
and acquiring first data to be processed of the interface operation based on the log data.
Further, the step of performing first data cleaning on the first data to be processed in the data record table to obtain first preprocessed data includes:
acquiring a business label of first preprocessing data;
based on the service tag, obtaining a characteristic value of the first data to be processed;
and performing first data cleaning on the characteristic value of the first data to be processed in the data record table to obtain first preprocessing data.
Further, the step of performing second data cleaning based on the first preprocessed data in the data preprocessing table to obtain second preprocessed data includes:
performing second data cleaning on the first preprocessed data in the data pre-generation table to obtain second data to be processed;
and performing model screening on the second data to be processed and the first data to be processed in the data record table, and assembling through an assembling model to obtain second preprocessed data.
Further, before the model screening is performed on the second data to be processed in advance of the first data to be processed in the data record table, and the second data to be processed is obtained by assembling through an assembling model, the method further includes:
acquiring expected data and standard deviation of first data to be processed in the data record table;
based on the expectations and standard deviations, an assembly model is obtained.
Further, before the step of performing data processing on the human resources of the target organization based on the human resource data processing policy of the business job, the method further includes:
performing second data cleaning on the first preprocessed data in the data pre-generation table to obtain third data to be processed;
And carrying out category refinement on the third data to be processed in advance of the first data to be processed in the data record table, and assembling through an assembling model to obtain second preprocessed data.
Further, after the second data is cleaned to obtain second preprocessed data based on the first preprocessed data in the data preprocessing table, the second preprocessed data is stored in a data result table, so that a user extracts corresponding target data in the data result table according to an input condition, the method further includes:
when the user needs data, the corresponding target data is directly extracted from the data result table according to the condition parameters transmitted by the user.
In order to solve the above technical problems, the embodiments of the present application further provide a data processing device, which adopts the following technical schemes:
the first acquisition module is used for acquiring first data to be processed of the interface operation and storing the first data into the data record table;
the first data cleaning module is used for cleaning first data of first data to be processed in the data record table to obtain first preprocessed data, and storing the first preprocessed data into the data pre-generation table;
And the second data cleaning module is used for cleaning the second data based on the first preprocessing data in the data preprocessing table to obtain second preprocessing data, and storing the second preprocessing data into a data result table so that a user can extract corresponding target data from the data result table according to the input condition.
In order to solve the above technical problems, the embodiments of the present application further provide a computer device, which adopts the following technical schemes:
the computer device includes a memory having stored therein computer readable instructions which when executed implement the steps of the data processing method of any of the embodiments of the present application.
In order to solve the above technical problems, embodiments of the present application further provide a computer readable storage medium, which adopts the following technical solutions:
the computer readable storage medium has stored thereon computer readable instructions which when executed by a processor implement the steps of the data processing method according to any of the embodiments of the present invention.
Compared with the prior art, the embodiment of the application has the following main beneficial effects: the first data to be processed of the interface operation is obtained and stored in the data record table, the first data to be processed in the data record table is subjected to first data cleaning to obtain first preprocessed data, the first preprocessed data is stored in the data pre-generation table, the first preprocessed data in the data pre-generation table is utilized to carry out second data cleaning, and therefore second preprocessed data is obtained, the second preprocessed data is stored in the data result table, so that a user can extract corresponding target data in the data result table according to the input conditions, and the problem that the working efficiency is reduced due to low multiplexing degree of the data is solved.
Drawings
For a clearer description of the solution in the present application, a brief description will be given below of the drawings that are needed in the description of the embodiments of the present application, it being obvious that the drawings in the following description are some embodiments of the present application, and that other drawings may be obtained from these drawings without inventive effort for a person of ordinary skill in the art.
FIG. 1 is an exemplary system architecture diagram in which the present application may be applied;
FIG. 2 is a flow chart of one embodiment of a data processing method according to the present application;
FIG. 3 is a flow chart of one embodiment of step S201 in FIG. 2;
FIG. 4 is a flow chart of a specific embodiment of step S202 in FIG. 2;
FIG. 5 is a flow chart of a specific embodiment of step S203 in FIG. 2;
FIG. 6 is a flow chart of a specific embodiment prior to step S2032 in FIG. 5;
FIG. 7 is a flow chart of another embodiment of step S203 in FIG. 2;
FIG. 8 is a flow chart of an embodiment following step S203 in FIG. 2;
FIG. 9 is a schematic diagram of a structure of one embodiment of a data processing apparatus according to the present application;
FIG. 10 is a schematic structural diagram of an embodiment of the first acquisition module 901 in FIG. 9;
FIG. 11 is a schematic diagram illustrating the structure of one embodiment of the first data cleansing module 902 of FIG. 9;
FIG. 12 is a schematic diagram of one embodiment of the second data cleansing module 903 of FIG. 9;
FIG. 13 is a schematic diagram of another embodiment of the data processing apparatus 900 of FIG. 9;
FIG. 14 is a schematic diagram of another embodiment of the second data cleaning module 903 of FIG. 9;
FIG. 15 is a schematic diagram of another embodiment of the data processing apparatus 900 of FIG. 9;
FIG. 16 is a schematic structural view of one embodiment of a computer device according to the present application.
Detailed Description
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs; the terminology used in the description of the applications herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having" and any variations thereof in the description and claims of the present application and in the description of the figures above are intended to cover non-exclusive inclusions. The terms first, second and the like in the description and in the claims or in the above-described figures, are used for distinguishing between different objects and not necessarily for describing a sequential or chronological order.
Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the present application. The appearances of such phrases in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Those of skill in the art will explicitly and implicitly appreciate that the embodiments described herein may be combined with other embodiments.
In order to better understand the technical solutions of the present application, the following description will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the accompanying drawings.
As shown in fig. 1, a system architecture 100 may include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used as a medium to provide communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.
The user may interact with the server 105 via the network 104 using the terminal devices 101, 102, 103 to receive or send messages or the like. Various communication client applications, such as a web browser application, a shopping class application, a search class application, an instant messaging tool, a mailbox client, social platform software, etc., may be installed on the terminal devices 101, 102, 103.
The terminal devices 101, 102, 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smartphones, tablet computers, electronic book readers, MP3 players (Moving Picture Experts Group Audio Layer III, dynamic video expert compression standard audio plane 3), MP4 (Moving Picture Experts Group Audio Layer IV, dynamic video expert compression standard audio plane 4) players, laptop and desktop computers, and the like.
The server 105 may be a server providing various services, such as a background server providing support for pages displayed on the terminal devices 101, 102, 103.
It should be noted that, the data processing method provided in the embodiments of the present application is generally executed by a server/terminal device, and accordingly, the data processing apparatus is generally disposed in the server/terminal device.
It should be understood that the number of terminal devices, networks and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
With continued reference to FIG. 2, a flow chart of one embodiment of a data processing method according to the present application is shown. The data processing method comprises the following steps:
In step S201, the first data to be processed of the interface operation is obtained and stored in the data record table.
In this embodiment, the electronic device on which the data processing method operates (e.g., as shown in FIG. 1Server/terminal End device) The data processing request of the terminal device may be received by a wired connection or a wireless connection. It should be noted that the wireless connection may include, but is not limited to, 3G/4G/5G connection, wiFi connection, bluetooth connection, wiMAX connection, zigbee connection, UWB (ultra wideband) connection, and other now known or later developed wireless connection.
The terminal device may be a terminal device equipped with a remote interaction function such as a personal assistant or an intelligent customer service.
The data processing method can be applied to services such as online data query, online data management and the like.
In particular, the electronic device may be used for different industry types such as medical data processing, insurance data processing, financial data processing, store data processing, etc.
The interface operation refers to the operations of collecting, processing, storing and the like of data through an API (Application Programming Interface ) of a database.
The first data to be processed may be data which is newly added or modified through interface operation triggered by the automation platform on the day of timing collection from the service side database and the log. And change data is recorded by log data.
The data record table may be understood as a table for storing data, and the data record table may be classified and stored according to five dimensions of a system-function module-test environment-data tag-data detail.
The interface operation can acquire first data to be processed of the interface operation through the terminal equipment, and after the first data to be processed is completed, the server receives the first data to be processed and acquires the first data to be processed of the interface operation.
Step S202, performing first data cleaning on first data to be processed in the data record table to obtain first pre-processed data, and storing the first pre-processed data into the data pre-generation table.
In this embodiment, the above data cleansing refers to a process of rechecking and checking data, and aims to delete duplicate information and correct existing errors. The first data cleaning may be data cleaning such as carding, aggregation, etc. of each dimension feature of the data.
The above-described data pre-generation table may be understood as a table storing the first pre-processed data.
The first preprocessing data described above may be understood as respective sets of test data that are initially classified.
Specifically, the first preprocessing data is obtained by combing and integrating according to the dimension characteristics of the first data to be processed in the data record table, eliminating error data and unavailable data, and is stored into the data pre-generation table.
Step S203, based on the first preprocessing data in the data preprocessing table, performing second data cleaning to obtain second preprocessing data, and storing the second preprocessing data in the data result table, so that the user can extract corresponding target data from the data result table according to the input condition.
In this embodiment, after obtaining the first pre-processed data in the data pre-generation table, the electronic device may perform second data cleaning on the first pre-processed data in the data pre-generation table to obtain second pre-processed data, and store the second pre-processed data in the data result table, so that the user extracts the corresponding target data from the data result table according to the input condition.
Further, after the second pre-processed data is obtained, the second pre-processed data may be stored in a corresponding database for review.
The data cleansing refers to the process of rechecking and checking the data, and aims to delete repeated information and correct existing errors. The second data cleansing may be data cleansing according to business defaults of some regularities, outlier exclusions, non-empty judgment matches, etc.
The data result table may be understood as a table storing the second pre-processed data, and may contain target data and information related to the target data.
The incoming condition refers to an incoming condition parameter.
Further, the second pre-processed data may be understood as cleaned available data.
According to the method, the first data to be processed of the interface operation is obtained and stored in the data record table, the first data to be processed in the data record table is subjected to first data cleaning to obtain first preprocessed data, the first preprocessed data is stored in the data pre-generation table, the first preprocessed data in the data pre-generation table is utilized to conduct second data cleaning, and therefore second preprocessed data is obtained, the second preprocessed data is stored in the data result table, so that a user can extract corresponding target data in the data result table according to the input conditions, and the problem that the data multiplexing degree is low and the working efficiency is reduced is solved.
With continued reference to FIG. 3, a flow chart of one embodiment of step S201 in FIG. 2 is shown. In step S201, the method specifically includes the following steps:
s2011, collecting log data of interface operation of a business database.
In this embodiment, the service database refers to a database system for storing and managing service data, and the service database contains data related to services.
The interface operation refers to the operations of collecting, processing, storing and the like of data through an API (Application Programming Interface ) of a database.
The log data refers to recording the process of adding and deleting data in a database.
And S2012, acquiring first data to be processed of the interface operation based on the log data.
In this embodiment, when the electronic device obtains the log data, the first data to be processed may be operated through the log data obtaining interface.
The first data to be processed may be data which is newly added or modified through interface operation triggered by the automation platform on the day of timing collection from the service side database and the log.
According to the method and the device, the log data of the interface operation of the service database are collected, and the first data to be processed of the interface operation is obtained according to the log data, so that the accuracy of the data is improved.
With continued reference to FIG. 4, a flow chart of one embodiment of step S202 of FIG. 3 is shown. In step S202, the method specifically includes the following steps:
In step S2021, a service tag of the first data to be processed is acquired.
In this embodiment, the service tag is an identifier of a service operation, and may be used to distinguish different service scenarios, service objects, and so on, for example, identifying according to five dimensions of a system-function module-test environment-data tag-data detail.
In step S2022, a feature value of the first data to be processed is obtained based on the service tag.
In this embodiment, the feature value refers to a feature value corresponding to a feature vector of a matrix.
Specifically, the characteristic value is a service label marked on various data in the data collector, so that the discrete distribution of the data is reduced.
The characteristic value may be a characteristic associated with a business label.
In step S2023, the feature value of the first data to be processed in the data record table is cleaned to obtain the first pre-processed data.
In this embodiment, the electronic device obtains the characteristic value of the first data to be processed in the data record table, and may perform the first data cleaning through the characteristic value of the first data to be processed in the data record table to obtain the first preprocessed data.
The data cleansing refers to the process of rechecking and checking the data, and aims to delete repeated information and correct existing errors. The first data cleaning may be data cleaning such as carding, aggregation, etc. of each dimension feature of the data.
According to the method and the device for processing the data, the characteristic value of the first data to be processed is obtained through the service label of the first data to be processed, and the characteristic value of the first data to be processed in the database table is subjected to first preprocessing, so that the accuracy and the working efficiency of data processing can be improved.
With continued reference to fig. 5, a flow chart of one embodiment of step S203 of fig. 2 is shown. In step S203, the method specifically includes the following steps:
s2031, performing second data cleaning on the first preprocessed data in the data pre-generation table to obtain second data to be processed.
In this embodiment, when the electronic device obtains the predicted report amount and the predicted task amount, the electronic device may determine the human resources of the job required for the business job by predicting the report amount and the predicted task amount.
The second data to be processed may be first pre-processed data after the first data is cleaned, without a tag value, without distinguishing data such as a system, a function, a data type, and the like. The second data to be processed may be available data and unavailable data which have a half of the ratio, or may be the remaining data after the first round of cleaning is performed according to some regular, abnormal removal and non-empty judgment matching default in the service, and a part of dirty data is removed.
The data cleaning refers to a process of rechecking and checking the data, and aims to delete repeated information and correct existing errors, and the second data cleaning can be data cleaning according to business defaults such as regular, abnormal removal, non-empty judgment matching and the like.
The method is characterized in that the case types are classified, different operation task flows are identified for different cases, operation task links are identified, and manpower consumption standards and aging standards of various task links are identified.
S2032, performing model screening on the second data to be processed and the first data to be processed of the data record table, and assembling through an assembling model to obtain second preprocessed data.
In this embodiment, when the electronic device obtains the second data to be processed and the first data to be processed in the data record table, the electronic device may perform model screening on the second data to be processed and the first data to be processed in the data record table, and assemble the second data to be processed through an assembly model, so as to obtain the second data to be preprocessed.
The model screening can be model screening preset by the system, and can be understood as comparing and screening the second data to be processed with the first data to be processed, and finding out the similarity with the model preset by the system so as to determine the accuracy of the data.
The assembly model can be formed by marking various labels on the data in the data record table as input parameters, and normal distribution expectation (average) and standard deviation parameters.
The second pre-processed data may be cleaned up available data.
According to the data preprocessing method and device, the second data is cleaned through the first preprocessed data in the data preprocessing table, the second data to be processed is obtained, the second data to be processed and the first data to be processed in the data recording table are subjected to model screening, and are assembled through the assembly model, so that the second preprocessed data is obtained, and the accuracy of the data is improved.
With continued reference to fig. 6, a flow chart of one particular embodiment mode prior to step S2032 in fig. 5 is shown. Before step S2032, the above data processing method further includes the steps of:
s601, acquiring expected and standard deviation of first to-be-processed data in a data record table.
In this embodiment, the above expectation can be understood as each data sample to be processed in the data record table, and the standard deviation is used to represent whether each data sample to be processed is available, where the more available data, the more concentrated the curve amplitude, and the more unavailable data, the more discrete the data distribution.
S602, obtaining an assembly model based on the expectation and the standard deviation.
In this embodiment, after obtaining the expectation and the standard deviation, the electronic device may obtain the assembly model through the expectation and the standard deviation.
Further, the assembly model takes various labels of the first data to be processed in the data record table as input parameters, and normally distributes the two parameters, namely the expected (average) mu and the standard deviation sigma.
Specifically, the model N (μ, σ≡2, Z) (Z: eigenvalue-data tag) is assembled.
The expected mu is each data sample in the data record table, the standard deviation sigma represents whether the data is available, the more the available data is, the more the curve amplitude is concentrated, the more the unavailable data is, the more the data distribution is discrete, meanwhile, the characteristic value parameter Z is newly added, the characteristic value is a service label marked on various data to be processed in the data record table, and the discrete data distribution is reduced.
According to the method and the device, the expected standard deviation and the expected standard deviation of the first data to be processed in the data record table are obtained, and the assembly model is obtained by utilizing the expected standard deviation and the expected standard deviation, so that the accuracy of the data is improved.
With continued reference to fig. 7, a flow chart of one specific embodiment of step S203 of fig. 2 is shown. In step S203, the method specifically includes the following steps:
S2033, performing second data cleaning on the first preprocessed data in the data pre-generation table to obtain third to-be-processed data.
In this embodiment, the second data cleaning may be data cleaning according to some regular, abnormal removal, non-empty judgment matching, and the like default in the service.
The third data to be processed may be first pre-processed data after the first data is cleaned, without a tag value, without distinguishing data such as a system, a function, a data type, and the like, or may be data to be processed with a half of the available data and the unavailable data, or may be residual data after the first round of cleaning is matched according to some regular, abnormal removal, and non-empty judgment defaults in the service, and a part of dirty data is removed.
S2034, performing category refinement on the third data to be processed and the first data to be processed in the data record table, and assembling through an assembling model to obtain second pre-processed data.
In this embodiment, when the electronic device obtains the human resources of the target organization job, the electronic device may determine the target organization business job capability processing capacity through the human resources of the target organization job.
The category refinement refers to classification and refinement according to the characteristic values of the data, so that the data of different systems and functional modules can be independently and discretely calculated. .
Specifically, the third data to be processed and the first data to be processed in the data record table are subjected to category refinement according to the characteristic values, the data of different systems and different functional modules are independently subjected to discrete calculation, finally available data in the result are assembled together, the available data are transmitted to be filtered, second preprocessed data are obtained, and the second preprocessed data are stored in the data result table for subsequent use.
According to the data preprocessing method and device, the first preprocessed data in the data preprocessing table is subjected to second data cleaning to obtain the third data to be processed, the first data to be processed in the third processed data preprocessing record table is utilized to conduct category refinement, and the second preprocessed data is obtained through assembling the assembling model, so that the accuracy of the data is improved, and the data multiplexing degree is improved.
With continued reference to fig. 8, a flow chart of one embodiment of the implementation of step S203 in fig. 2 is shown. After step S203, the above data processing method further includes the steps of:
s801, when a user needs data, the corresponding target data is directly extracted from the data result table according to the condition parameters transmitted by the user.
In this embodiment, when the user needs data, the condition parameters may be transferred through the page, and then, corresponding relevant data may be directly extracted from the data result table according to each module and the tag.
When the user needs data, the corresponding target data is directly extracted from the data result table according to the condition parameters transmitted by the user, so that the data multiplexing degree is improved.
The embodiment of the application can acquire and process the related data based on the artificial intelligence technology. Among these, artificial intelligence (Artificial Intelligence, AI) is the theory, method, technique and application system that uses a digital computer or a digital computer-controlled machine to simulate, extend and extend human intelligence, sense the environment, acquire knowledge and use knowledge to obtain optimal results.
Artificial intelligence infrastructure technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technologies, operation/interaction systems, mechatronics, and the like. The artificial intelligence software technology mainly comprises a computer vision technology, a robot technology, a biological recognition technology, a voice processing technology, a natural language processing technology, machine learning/deep learning and other directions. The method and the device can be applied to the fields of artificial intelligence and big data, so that the construction of smart cities is promoted.
This application belongs to the smart city field, can promote the construction in smart city through this scheme.
Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by computer readable instructions stored in a computer readable storage medium that, when executed, may comprise the steps of the embodiments of the methods described above. The storage medium may be a nonvolatile storage medium such as a magnetic disk, an optical disk, a Read-Only Memory (ROM), or a random access Memory (Random Access Memory, RAM).
It should be understood that, although the steps in the flowcharts of the figures are shown in order as indicated by the arrows, these steps are not necessarily performed in order as indicated by the arrows. The steps are not strictly limited in order and may be performed in other orders, unless explicitly stated herein. Moreover, at least some of the steps in the flowcharts of the figures may include a plurality of sub-steps or stages that are not necessarily performed at the same time, but may be performed at different times, the order of their execution not necessarily being sequential, but may be performed in turn or alternately with other steps or at least a portion of the other steps or stages.
With further reference to fig. 9, as an implementation of the method shown in fig. 2, the present application provides an embodiment of a data processing apparatus, where an embodiment of the apparatus corresponds to the embodiment of the method shown in fig. 2, and the apparatus may be specifically applied to various electronic devices.
As shown in fig. 9, the data processing apparatus 900 according to the present embodiment includes: a first acquisition module 901, a first data cleansing module 902, and a second data cleansing module 903. Wherein:
a first obtaining module 901, configured to obtain first data to be processed of an interface operation, and store the first data in a data record table;
a first data cleaning module 902, configured to perform first data cleaning on first data to be processed in the data record table, obtain first preprocessed data, and store the first preprocessed data into a data pre-generation table;
the second data cleaning module 903 is configured to perform second data cleaning based on the first preprocessed data in the data pre-generating table, obtain second preprocessed data, and store the second preprocessed data into a data result table, so that a user extracts corresponding target data from the data result table according to an input condition.
In this embodiment, the historical report amount and the historical task amount of the service job are used to determine the predicted report amount and the predicted task amount of the service job, the predicted report amount and the predicted task amount are obtained according to a preset prediction model, the preset prediction model is used to predict the predicted report amount and the predicted task amount of the service job in a future period, and according to the predicted report amount and the predicted task amount, the manpower resource data processing strategy of the service job is determined, and the manpower resource data processing strategy of the service job is utilized to perform data processing on the manpower resource of the target mechanism, so that the problems of low working efficiency of the service job, reduced customer experience and increased operation cost are solved.
Referring to fig. 10, a schematic structural diagram of an embodiment of the first acquisition module 901 in fig. 9, where the first acquisition module 901 includes an acquisition sub-module 9011 and a first acquisition sub-module 9012. Wherein:
the collecting submodule 9011 is used for collecting log data of interface operation of the service database;
a first obtaining submodule 9012, configured to obtain, based on the log data, first data to be processed of an interface operation.
In this embodiment, by collecting log data of interface operations of the service database and obtaining first data to be processed of the interface operations according to the log data, accuracy of the data is improved.
Referring to fig. 11, a schematic structural diagram of an embodiment of the first data cleaning module 902 in fig. 9, where the first data cleaning module 902 includes a second obtaining sub-module 9021, a first processing sub-module 9022, and a first data cleaning sub-module 9023. Wherein:
a second obtaining sub-module 9021, configured to obtain a service tag of the first data to be processed;
a first processing submodule 9022, configured to obtain a feature value of the first data to be processed based on the service tag;
and the first data cleaning submodule 9023 is used for cleaning the first data of the characteristic value of the first data to be processed in the data record table to obtain first preprocessing data.
In this embodiment, the feature value of the first data to be processed is obtained through the service tag of the first data to be processed, and the feature value of the first data to be processed in the database table is subjected to the first preprocessing, so that the accuracy and the working efficiency of data processing can be improved.
Referring to fig. 12, which is a schematic structural diagram of an embodiment of the second data cleaning module 903 in fig. 9, the second data cleaning module 903 includes a second data cleaning submodule 9031 and a model screening submodule 9032. Wherein:
a second data cleaning submodule 9031, configured to clean the second data from the first preprocessed data in the data pre-generation table, to obtain second data to be processed;
and a model screening submodule 9032, configured to perform model screening on the second data to be processed and the first data to be processed in the data record table, and assemble the second data to be processed through an assembly model to obtain second preprocessed data.
In this embodiment, the second data to be processed is obtained by cleaning the first preprocessed data in the data preprocessing table, and the second data to be processed and the first data to be processed in the data recording table are subjected to model screening, and are assembled through the assembly model, so that the second preprocessed data is obtained, and the accuracy of the data is improved.
Referring to fig. 13, which is a schematic structural diagram of an embodiment of the data processing apparatus 900 in fig. 9, the data processing apparatus 900 further includes a second obtaining module 904 and a processing module 905. Wherein:
a second obtaining module 904, configured to obtain expected and standard deviation of the first data to be processed in the data record table;
a processing module 905 for deriving an assembly model based on the expectations and the standard deviation.
In this embodiment, the assembly model is obtained by acquiring the expectation and the standard deviation of the first data to be processed in the data record table and using the expectation and the standard deviation, thereby improving the accuracy of the data.
Referring to fig. 14, a schematic structural diagram of another embodiment of the second data cleansing module 903 in fig. 9, where the second data cleansing module 903 includes a third data cleansing submodule 9033 and a class refinement submodule 9034. Wherein:
a third data cleaning submodule 9033, configured to clean the second data from the first preprocessed data in the data pre-generation table, to obtain third data to be processed;
and a class refinement sub-module 9034, configured to refine the class of the third data to be processed and the first data to be processed in the data record table, and assemble the third data to be processed and the first data to be processed through an assembly model to obtain second pre-processed data.
In this embodiment, the first pre-processed data in the data pre-generation table is cleaned to obtain the third data to be processed, the first data to be processed in the third data pre-processing record table is used for category refinement, and the second pre-processed data is obtained by assembling through the assembling model, so that the accuracy of the data is improved, and the data multiplexing degree is improved.
Referring to fig. 15, which is a schematic structural diagram of an embodiment of the data processing apparatus 900 in fig. 9, the data processing apparatus 900 further includes a data processing sub-module 9041. Wherein:
and the extracting module 906 is configured to directly extract the corresponding target data from the data result table according to the condition parameters input by the user when the user needs the data.
In this embodiment, when the user needs data, according to the condition parameters input by the user, the corresponding target data is directly extracted from the data result table, thereby improving the data multiplexing degree.
In order to solve the technical problems, the embodiment of the application also provides computer equipment. Referring specifically to fig. 16, fig. 16 is a basic structural block diagram of a computer device according to the present embodiment.
The computer device 16 includes a memory 161, a processor 162, and a network interface 163 communicatively coupled to each other via a system bus. It should be noted that only computer device 16 having components 161-163 is shown in the figures, but it should be understood that not all of the illustrated components need be implemented, and that more or fewer components may be implemented instead. It will be appreciated by those skilled in the art that the computer device herein is a device capable of automatically performing numerical calculations and/or information processing in accordance with predetermined or stored instructions, the hardware of which includes, but is not limited to, microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASICs), programmable gate arrays (fields-Programmable Gate Array, FPGAs), digital processors (Digital Signal Processor, DSPs), embedded devices, etc.
The computer equipment can be a desktop computer, a notebook computer, a palm computer, a cloud server and other computing equipment. The computer equipment can perform man-machine interaction with a user through a keyboard, a mouse, a remote controller, a touch pad or voice control equipment and the like.
The memory 161 includes at least one type of readable storage medium including flash memory, hard disk, multimedia card, card memory (e.g., SD or DX memory, etc.), random Access Memory (RAM), static Random Access Memory (SRAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), programmable Read Only Memory (PROM), magnetic memory, magnetic disk, optical disk, etc.
In some embodiments, the memory 161 may be an internal storage unit of the computer device 16, such as a hard disk or a memory of the computer device 16. In other embodiments, the memory 161 may also be an external storage device of the computer device 16, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash Card (Flash Card) or the like, which are provided on the computer device 16. Of course, the memory 161 may also include both internal storage units of the computer device 16 and external storage devices. In this embodiment, the memory 161 is typically used to store an operating system and various application software installed on the computer device 16, such as computer readable instructions for a data processing method. Further, the memory 161 may be used to temporarily store various types of data that have been output or are to be output. The processor 162 may be a central processing unit (Central Processing Unit, CPU), controller, microcontroller, microprocessor, or other data processing chip in some embodiments. The processor 162 is generally used to control the overall operation of the computer device 16. In this embodiment, the processor 162 is configured to execute computer readable instructions stored in the memory 161 or process data, such as computer readable instructions for executing the data processing method.
The network interface 163 may include a wireless network interface or a wired network interface, and the network interface 163 is typically used to establish communication connections between the computer device 16 and other electronic devices.
In this embodiment, the influence data of the system in the knowledge base may be used to generate the data affected by the target system to be tested for the user, so that the user may intuitively obtain the influence range of the target system to be tested, evaluate the influence of modification more accurately and faster according to the influence range of the target system to be tested, eliminate irrelevant interference points, simplify regression use cases, avoid that partial correlations are found to be not processed when the previous evaluation influence points are missed, and jointly debug the subsequent systems, so that the development time is required to be occupied to supplement logic, and serious possible scheme design is required to override the remarked questions, thereby improving the efficiency of project development.
The present application also provides another embodiment, namely, a computer-readable storage medium storing computer-readable instructions executable by at least one processor to cause the at least one processor to perform the steps of the data processing method as described above.
In this embodiment, the influence data of the system in the knowledge base may be used to generate the data affected by the target system to be tested for the user, so that the user may intuitively obtain the influence range of the target system to be tested, evaluate the influence of modification more accurately and faster according to the influence range of the target system to be tested, eliminate irrelevant interference points, simplify regression use cases, avoid that partial correlations are found to be not processed when the previous evaluation influence points are missed, and jointly debug the subsequent systems, so that the development time is required to be occupied to supplement logic, and serious possible scheme design is required to override the remarked questions, thereby improving the efficiency of project development.
From the above description of the embodiments, it will be clear to those skilled in the art that the above-described embodiment method may be implemented by means of software plus a necessary general hardware platform, but of course may also be implemented by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art in the form of a software product stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk), comprising several instructions for causing a terminal device (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to perform the method described in the embodiments of the present application.
It is apparent that the embodiments described above are only some embodiments of the present application, but not all embodiments, the preferred embodiments of the present application are given in the drawings, but not limiting the patent scope of the present application. This application may be embodied in many different forms, but rather, embodiments are provided in order to provide a more thorough understanding of the present disclosure. Although the present application has been described in detail with reference to the foregoing embodiments, it will be apparent to those skilled in the art that modifications may be made to the embodiments described in the foregoing, or equivalents may be substituted for elements thereof. All equivalent structures made by the specification and the drawings of the application are directly or indirectly applied to other related technical fields, and are also within the protection scope of the application.
Claims (10)
1. A method of data processing comprising the steps of:
acquiring first data to be processed of interface operation and storing the first data into a data record table;
performing first data cleaning on first data to be processed in the data record table to obtain first preprocessing data, and storing the first preprocessing data into a data pre-generation table;
And based on the first preprocessing data of the data preprocessing table, performing second data cleaning to obtain second preprocessing data, and storing the second preprocessing data into a data result table so that a user can extract corresponding target data from the data result table according to the input condition.
2. The data processing method according to claim 1, wherein the step of acquiring the first data to be processed of the interface operation includes:
collecting log data of interface operation of a service database;
and acquiring first data to be processed of the interface operation based on the log data.
3. The data processing method according to claim 2, wherein the step of performing first data cleansing on the first data to be processed in the data record table to obtain first preprocessed data comprises:
acquiring a business label of first preprocessing data;
based on the service tag, obtaining a characteristic value of the first data to be processed;
and performing first data cleaning on the characteristic value of the first data to be processed in the data record table to obtain first preprocessing data.
4. A data processing method according to claim 3, wherein the step of performing second data cleansing based on the first preprocessed data in the data preprocessing table to obtain second preprocessed data comprises:
Performing second data cleaning on the first preprocessed data in the data pre-generation table to obtain second data to be processed;
and performing model screening on the second data to be processed and the first data to be processed in the data record table, and assembling through an assembling model to obtain second preprocessed data.
5. The data processing method according to claim 4, wherein before the second data to be processed is subjected to model screening in advance of the first data to be processed in the data record table and assembled by an assembly model, the method further comprises:
acquiring expected data and standard deviation of first data to be processed in the data record table;
based on the expectations and standard deviations, an assembly model is obtained.
6. The method according to claim 4, wherein before the step of performing data processing on human resources of a target institution based on the human resource data processing policy of the business job, the method further comprises:
performing second data cleaning on the first preprocessed data in the data pre-generation table to obtain third data to be processed;
And carrying out category refinement on the third data to be processed in advance of the first data to be processed in the data record table, and assembling through an assembling model to obtain second preprocessed data.
7. The data processing method according to claim 6, wherein after performing second data cleansing based on the first preprocessed data in the data preprocessing table to obtain second preprocessed data, storing the second preprocessed data in a data result table so that a user extracts corresponding target data in the data result table according to an entry condition, the method further comprises:
when the user needs data, the corresponding target data is directly extracted from the data result table according to the condition parameters transmitted by the user.
8. A data processing apparatus, comprising:
the first acquisition module is used for acquiring first data to be processed of the interface operation and storing the first data into the data record table;
the first data cleaning module is used for cleaning first data of first data to be processed in the data record table to obtain first preprocessed data, and storing the first preprocessed data into the data pre-generation table;
and the second data cleaning module is used for cleaning the second data based on the first preprocessing data in the data preprocessing table to obtain second preprocessing data, and storing the second preprocessing data into a data result table so that a user can extract corresponding target data from the data result table according to the input condition.
9. A computer device comprising a memory having stored therein computer readable instructions which when executed by a processor implement the steps of the data processing method of any of claims 1 to 7.
10. A computer-readable storage medium, having stored thereon computer-readable instructions which, when executed by a processor, implement the steps of the data processing method according to any of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311315525.XA CN117421312A (en) | 2023-10-11 | 2023-10-11 | Data processing method, device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311315525.XA CN117421312A (en) | 2023-10-11 | 2023-10-11 | Data processing method, device, computer equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117421312A true CN117421312A (en) | 2024-01-19 |
Family
ID=89527585
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311315525.XA Pending CN117421312A (en) | 2023-10-11 | 2023-10-11 | Data processing method, device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117421312A (en) |
-
2023
- 2023-10-11 CN CN202311315525.XA patent/CN117421312A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20210142233A1 (en) | Systems and methods for process mining using unsupervised learning | |
| CN117234505A (en) | Interactive page generation method, device, equipment and storage medium thereof | |
| CN116841846A (en) | Real-time log abnormality detection method, device, equipment and storage medium thereof | |
| CN117034230A (en) | Data verification method, device, equipment and storage medium thereof | |
| CN115794545A (en) | Automatic processing method of operation and maintenance data and related equipment thereof | |
| CN117421312A (en) | Data processing method, device, computer equipment and storage medium | |
| CN114968725A (en) | Task dependency relationship correction method and device, computer equipment and storage medium | |
| CN113934595A (en) | Data analysis method and system, storage medium and electronic terminal | |
| CN117421207A (en) | Intelligent evaluation influence point test method, intelligent evaluation influence point test device, computer equipment and storage medium | |
| CN116796133A (en) | Data analysis method, device, computer equipment and storage medium | |
| CN116842011A (en) | Blood relationship analysis method, device, computer equipment and storage medium | |
| CN116795632A (en) | Task processing method, device, computer equipment and storage medium | |
| CN117290452A (en) | Data warehouse management method, device, equipment and storage medium thereof | |
| CN117786390A (en) | Feature data arrangement method to be maintained and related equipment thereof | |
| CN117453536A (en) | System abnormality analysis method, system abnormality analysis device, computer device and storage medium | |
| CN117197814A (en) | Data standardization method, device, equipment and storage medium thereof | |
| CN116389315A (en) | Interface monitoring method, device, computer equipment and storage medium | |
| CN118041977A (en) | Method and device for processing micro-service component, computer equipment and storage medium | |
| CN116795882A (en) | Data acquisition method, device, computer equipment and storage medium | |
| CN116757771A (en) | Scheme recommendation method, device, equipment and storage medium based on artificial intelligence | |
| CN117112415A (en) | Business process monitoring method based on EDA model and related equipment thereof | |
| CN114625442A (en) | Cold start recommendation method and device, electronic equipment and readable storage medium | |
| CN117034173A (en) | Data processing method, device, computer equipment and storage medium | |
| CN116611432A (en) | Drunk driving risk identification method and device, computer equipment and storage medium | |
| CN116630059A (en) | Loss prediction method, device, equipment and storage medium based on artificial intelligence |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |