CN115080771A - Data processing method and device based on artificial intelligence, medium and gateway equipment - Google Patents
Data processing method and device based on artificial intelligence, medium and gateway equipment Download PDFInfo
- Publication number
- CN115080771A CN115080771A CN202210715239.1A CN202210715239A CN115080771A CN 115080771 A CN115080771 A CN 115080771A CN 202210715239 A CN202210715239 A CN 202210715239A CN 115080771 A CN115080771 A CN 115080771A
- Authority
- CN
- China
- Prior art keywords
- data
- data processing
- processing terminal
- gateway
- transmission interface
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/40—Information retrieval; Database structures therefor; File system structures therefor of multimedia data, e.g. slideshows comprising image and additional audio data
- G06F16/41—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/40—Information retrieval; Database structures therefor; File system structures therefor of multimedia data, e.g. slideshows comprising image and additional audio data
- G06F16/45—Clustering; Classification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q40/00—Finance; Insurance; Tax strategies; Processing of corporate or income taxes
- G06Q40/08—Insurance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L12/00—Data switching networks
- H04L12/66—Arrangements for connecting between networks having differing types of switching systems, e.g. gateways
Abstract
The invention discloses a data processing method and device based on artificial intelligence, a medium and gateway equipment, relates to the technical field of artificial intelligence, and mainly aims to solve the problem that the existing unstructured data is low in data processing efficiency. The method comprises the following steps: configuring a gateway data transmission interface and binding at least one data processing terminal, wherein the gateway data transmission interface is used for transmitting data objects flowing through gateway equipment; receiving a data processing request through the gateway data transmission interface, classifying data objects carried in the data processing request based on a trained data classification model, and determining a target data processing terminal of the data objects, wherein training sample data of the data classification model is a classification label for determining different data objects to be classified to the target data processing terminal according to data access times; and processing the data object in the target data processing terminal through the gateway data transmission interface.
Description
Technical Field
The invention relates to the technical field of artificial intelligence, in particular to a data processing method and device based on artificial intelligence, a medium and gateway equipment.
Background
With the rapid development of big data processing technology, the storage demand for unstructured data is increasing, and especially in the field of medical health, unstructured data such as high-definition pictures, audios and videos, artificial intelligence models and the like need to be stored.
At present, when different business parties store unstructured data, the unstructured data are usually stored in a single object storage system based on each unstructured data type, however, the single object storage system cannot meet the requirements of security and high-efficiency storage performance in different business scenes, the storage and query costs are high, and the data processing efficiency of unstructured data is reduced, so that the efficiency of business processing based on the stored and queried data in different business scenes is influenced.
Disclosure of Invention
In view of the above, the present invention provides a data processing method and apparatus based on artificial intelligence, a medium, and a gateway device, and mainly aims to solve the problem of low data processing efficiency of existing unstructured data.
According to an aspect of the present invention, there is provided an artificial intelligence based data processing method, including:
configuring a gateway data transmission interface and binding at least one data processing terminal, wherein the gateway data transmission interface is used for transmitting data objects flowing through gateway equipment;
receiving a data processing request through the gateway data transmission interface, classifying data objects carried in the data processing request based on a trained data classification model, and determining a target data processing terminal of the data objects, wherein training sample data of the data classification model is a classification label for determining different data objects to be classified to the target data processing terminal according to data access times;
and processing the data object in the target data processing terminal through the gateway data transmission interface.
Further, before the classification processing is performed on the data object carried in the data processing request based on the trained data classification model and the target data processing terminal of the data object is determined, the method further includes:
determining a training data processing terminal from the data processing terminals, and acquiring training sample data in the training data processing terminal, wherein a classification label in the training sample data is determined based on the data access times of each data processing terminal;
and establishing a data classification model, and performing model training on the data classification model based on the classification label and the training sample data to obtain the trained data classification model.
Further, the determining a training data processing terminal from the data processing terminals comprises:
acquiring service demand information, wherein the service demand information comprises data classification types and data processing types corresponding to data objects in different service scenes, the data classification types comprise image data types, audio data types, video data types and model algorithm data types in unstructured data types, and the data processing types comprise storage processing types, query processing types and update processing types;
acquiring data processing record information of the data processing terminal which finishes data processing;
and if the data processing record information is matched with the service requirement information, determining the data processing terminal as a training data processing terminal so as to obtain the classification label of the marked data object and the marked terminal identity information in the training data processing terminal as training sample data.
Further, the method further comprises:
sending the service requirement information to the data processing terminal according to a preset time interval, and indicating the data processing terminal to carry out label marking processing based on the service requirement information;
and receiving label indication information of the data processing terminal for completing label labeling processing, and executing the step of determining a training data processing terminal from the data processing terminal so as to update and train the data classification model.
Further, after the data classification model is established and model training is performed on the data classification model based on the classification label and the training sample data to obtain the trained data classification model, the method further includes:
acquiring a third party application identifier for data transmission through the gateway data transmission interface, and determining a model verification index matched with the third party application identifier based on a preset model verification relationship, wherein the preset model verification relationship comprises a plurality of third party application identifiers and a plurality of corresponding relationships among the model verification indexes;
and if the model verification parameters of the trained data classification model are matched with the model verification indexes, confirming that the data classification model passes the index verification, and classifying the data objects through the data classification model.
Further, the processing the data object in the target data processing terminal through the gateway data transmission interface includes:
if the data processing request is a data storage request, outputting the data object to the target data processing terminal through the gateway data transmission interface, and indicating the target data processing terminal to store the data content of the data object;
if the data processing request is a data query request, sending a data query request of the data object to the target data processing terminal through the gateway data transmission interface, and instructing the target data processing terminal to query and feed back the data content of the data object;
and if the data processing request is a data updating request, outputting the data object to the target data processing terminal through the gateway data transmission interface, and instructing the target data processing terminal to replace and update the data content of the data object.
Further, before the outputting the data object to the target data processing terminal through the gateway data transmission interface, the method further includes:
acquiring data processing performance parameters of the target data processing terminal, wherein the data processing performance parameters comprise at least one of storage performance parameters, query performance parameters, terminal hardware operation parameters and terminal operation line number;
if the data performance parameters do not match the transmission conditions of the gateway equipment, the binding relationship with the target data processing terminal is removed;
and if the data performance parameters match the transmission conditions of the gateway equipment, indicating to execute the step of outputting the data object to the target data processing terminal through the gateway data transmission interface.
According to another aspect of the present invention, there is provided an artificial intelligence based data processing apparatus comprising:
the system comprises a configuration module, a data processing terminal and a gateway data transmission interface, wherein the configuration module is used for configuring a gateway data transmission interface and binding at least one data processing terminal, and the gateway data transmission interface is used for transmitting data objects flowing through gateway equipment;
the receiving module is used for receiving a data processing request through the gateway data transmission interface, classifying data objects carried in the data processing request based on a trained data classification model, and determining a target data processing terminal of the data objects, wherein training sample data of the data classification model is a classification label for determining different data objects to be classified to the target data processing terminal according to data access times;
and the processing module is used for processing the data object in the target data processing terminal through the gateway data transmission interface.
Further, the apparatus further comprises:
the determining module is used for determining a training data processing terminal from the data processing terminals and acquiring training sample data in the training data processing terminal, wherein the classification label in the training sample data is determined based on the data access times of the data processing terminals;
and the training module is used for establishing a data classification model, and performing model training on the data classification model based on the classification label and the training sample data to obtain the trained data classification model.
Further, the determining module includes:
the first obtaining unit is used for obtaining service requirement information, wherein the service requirement information comprises data classification types and data processing types corresponding to data objects in different service scenes, the data classification types comprise image data types, audio data types, video data types and model algorithm data types in unstructured data types, and the data processing types comprise storage processing types, query processing types and update processing types;
a second acquisition unit configured to acquire data processing record information in which data processing has been completed in the data processing terminal;
and the determining unit is used for determining the data processing terminal as a training data processing terminal if the data processing record information matches the service requirement information, so that the training data processing terminal acquires the classification label of the marked data object and the marked terminal identity information as training sample data.
Further, the determining module further comprises:
a sending unit, configured to send the service requirement information to the data processing terminal according to a preset time interval, and instruct the data processing terminal to perform label marking processing based on the service requirement information;
and the receiving unit is used for receiving the label indication information of the data processing terminal for completing label labeling processing, and executing the step of determining a training data processing terminal from the data processing terminal so as to update and train the data classification model.
Further, the determining module is specifically configured to acquire a third-party application identifier for performing data transmission through the gateway data transmission interface, and determine a model verification index matched with the third-party application identifier based on a preset model verification relationship, where the preset model verification relationship includes a correspondence relationship between a plurality of third-party application identifiers and a plurality of model verification indexes; and if the model verification parameters of the trained data classification model are matched with the model verification indexes, confirming that the data classification model passes the index verification, and classifying the data objects through the data classification model.
Further, the processing module comprises:
a first output unit, configured to output the data object to the target data processing terminal through the gateway data transmission interface if the data processing request is a data storage request, and instruct the target data processing terminal to store the data content of the data object;
a second output unit, configured to send a data query request for the data object to the target data processing terminal through the gateway data transmission interface if the data processing request is a data query request, and instruct the target data processing terminal to query and feed back data content of the data object;
and a third output unit, configured to output the data object to the target data processing terminal through the gateway data transmission interface if the data processing request is a data update request, and instruct the target data processing terminal to replace and update the data content of the data object.
Further, the apparatus further comprises:
the acquisition module is used for acquiring data processing performance parameters of the target data processing terminal, wherein the data processing performance parameters comprise at least one of storage performance parameters, query performance parameters, terminal hardware operation parameters and terminal operation line number;
the release module is used for releasing the binding relation with the target data processing terminal if the data performance parameters do not match the transmission conditions of the gateway equipment;
and the indicating module is used for indicating to execute the step of outputting the data object to the target data processing terminal through the gateway data transmission interface if the data performance parameters are matched with the transmission conditions of the gateway equipment.
According to still another aspect of the present invention, there is provided a storage medium having at least one executable instruction stored therein, the executable instruction causing a processor to perform operations corresponding to the artificial intelligence based data processing method as described above.
According to still another aspect of the present invention, there is provided a gateway apparatus including: the system comprises a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface complete mutual communication through the communication bus;
the memory is used for storing at least one executable instruction, and the executable instruction enables the processor to execute the operation corresponding to the artificial intelligence based data processing method.
By means of the technical scheme, the technical scheme provided by the embodiment of the invention at least has the following advantages:
compared with the prior art, the embodiment of the invention configures a gateway data transmission interface and binds at least one data processing terminal, wherein the gateway data transmission interface is used for transmitting data objects flowing through the gateway equipment; receiving a data processing request through the gateway data transmission interface, classifying data objects carried in the data processing request based on a trained data classification model, and determining a target data processing terminal of the data objects, wherein training sample data of the data classification model is a classification label for determining different data objects to be classified to the target data processing terminal according to data access times; the data objects are processed in the target data processing terminal through the gateway data transmission interface, diversified and safe data processing of multi-object data in different service scenes is achieved, the effectiveness of data processing such as data storage and query is improved, the data processing cost is reduced, and therefore the efficiency of service processing based on the stored and queried data in different service scenes is improved.
The foregoing description is only an overview of the technical solutions of the present invention, and the embodiments of the present invention are described below in order to make the technical means of the present invention more clearly understood and to make the above and other objects, features, and advantages of the present invention more clearly understandable.
Drawings
Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the invention. Also, like reference numerals are used to refer to like parts throughout the drawings. In the drawings:
FIG. 1 is a flow chart of a method for processing data based on artificial intelligence according to an embodiment of the present invention;
FIG. 2 is a flow chart of another artificial intelligence based data processing method provided by the embodiment of the invention;
FIG. 3 is a flow chart of another artificial intelligence based data processing method provided by the embodiment of the invention;
FIG. 4 is a flow chart of another artificial intelligence based data processing method provided by the embodiment of the invention;
fig. 5 is a schematic diagram illustrating a data processing flow performed by a gateway device according to an embodiment of the present invention;
FIG. 6 is a block diagram illustrating an artificial intelligence based data processing apparatus according to an embodiment of the present invention;
fig. 7 shows a schematic structural diagram of a gateway device according to an embodiment of the present invention.
Detailed Description
Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
When unstructured data is stored for different business parties, the unstructured data is usually stored in a single object storage system based on each unstructured data type, however, the single object storage system cannot meet the requirements of safety and high-efficiency storage performance in different business scenes, the storage and query costs are high, the data processing efficiency of the unstructured data is reduced, and therefore the efficiency of business processing based on the stored and queried data in different business scenes is influenced.
The embodiment of the invention provides a data processing method based on artificial intelligence, which comprises the following steps of:
101. configuring a gateway data transmission interface and binding at least one data processing terminal.
In the embodiment of the present invention, the current execution end is a Gateway device, and the Gateway device is also called an inter-network connector and a protocol converter, and is a computer system or device that provides data conversion service among multiple networks, including but not limited to a network switch having a three-layer switching function, a router, a firewall, a terminal device that opens a routing function through software, and the like, so that different application clients and application servers perform data transmission through the Gateway device serving as the current execution end. The present invention relates to a gateway device, and more particularly, to a gateway data transmission method and system for transmitting data objects, where the gateway data transmission method and system are configured in a current gateway device, and the gateway data transmission method and system are used to transmit data objects flowing through the gateway device, that is, after an application client, such as an APP, generates data objects corresponding to different service scenarios, the data objects are transmitted to the current gateway device through the gateway data transmission interface. Meanwhile, in order to enable the gateway equipment serving as the current execution end to be matched with the data processing terminal to realize a data processing function, after the gateway data transmission interface is configured, the gateway data transmission interface is bound to at least one data processing terminal, so that the gateway equipment serving as the current execution end respectively transmits the data objects to the corresponding data processing terminals to perform data processing in one step. The data object includes, but is not limited to, different data contents generated or requested by the application client and the application server, for example, video data recorded by the APP client, and the like, and at this time, the data object may be transmitted in a file form, and the embodiment of the present invention is not particularly limited.
It should be noted that the application client and the application server in the embodiment of the present invention are applicable to application fields including, but not limited to, medical health, insurance product transaction, and the like, so as to perform data processing on different data objects. In addition, since the gateway device serving as the current execution end may already have a data transmission interface or is provided with a plurality of data transmission interfaces, when configuring the gateway data transmission interface, an interface specifying a data transmission protocol may be selected from the gateway device to perform configuration, such as a gateway POST interface, or an interface specifying a data transmission protocol may be created, such as an upload interface. At this time, after receiving the data processing request through the configured gateway data transmission interface, the step 102 is executed, thereby implementing the gateway service function in the gateway device.
102. And receiving a data processing request through the gateway data transmission interface, classifying the data object carried in the data processing request based on the trained data classification model, and determining a target data processing terminal of the data object.
In the embodiment of the invention, after the gateway device serving as the current execution end receives the data processing request through the gateway data transmission interface, the data object carried in the data processing request, such as the user identity information in the form of a file, is extracted, so that the data object is classified and processed based on the data classification model after model training is completed, and the target data processing terminal is determined. The classification label of the training sample data of the data classification model is determined based on the access times of different data processing terminals, so that the target data processing terminal determined by the trained data classification model is matched according to the data access times and the classification label, and is suitable for the terminal for processing the data object. For example, the data object is user identity information, the target data processing terminal corresponding to the classification tag 1 is determined as terminal a based on the data classification model, and the data access times of the target data processing terminal is matched with the classification tag 1, so that the purpose of maximum data processing efficiency can be realized when the target data processing terminal processes the data object,
it should be noted that, because at least one data processing terminal bound in the embodiment of the present invention may be set in the form of a device cluster, one device cluster may include a plurality of sub-clusters, and a data processing terminal in each sub-cluster may process a data object of a data classification type, and when a target data processing terminal is determined, that is, a sub-cluster corresponding to a data object matching the data classification type is determined from the plurality of sub-clusters, so as to determine a specific target data processing terminal in the sub-cluster, so as to reduce processing pressure of other data processing terminals in one sub-cluster.
103. And processing the data object in the target data processing terminal through the gateway data transmission interface.
In the embodiment of the present invention, after the target data processing terminal is determined, because the data processing requests are different, the data object is processed differently on the basis of the target data processing terminal, for example, if the data processing request is a storage request, the data object may be sent to the target data processing terminal through the gateway data transmission interface, so that the target data processing terminal stores the data object.
It should be noted that the data processing terminal may be a terminal device with different data processing functions, including but not limited to a storage terminal (memory, storage bin, etc.), a processing server, etc., so as to implement different data processing functions.
In another embodiment of the present invention, for further limitation and explanation, as shown in fig. 2, before the step of classifying the data object carried in the data processing request based on the trained data classification model and determining the target data processing terminal of the data object, the method further includes:
201. determining a training data processing terminal from the data processing terminals, and acquiring training sample data in the training data processing terminal;
202. and establishing a data classification model, and performing model training on the data classification model based on the classification label and the training sample data to obtain the trained data classification model.
In order to classify a data object based on a data classification model to determine a target data processing terminal from a plurality of data processing terminals, specifically, at least one training data processing terminal is determined in advance from the plurality of data processing terminals, and a classification label of a labeled data object in the training data processing terminal and labeled terminal identity information are used as training sample data. The classification label in the training sample data is determined based on the data access times of each data processing terminal, the classification label is used for marking data object types corresponding to different data classification types, for example, a medical health information type corresponding to a video data classification type, and an insurance product evaluation information type corresponding to a model algorithm classification type, at this time, a terminal identifier in a marked terminal identity is used as an identifier for marking the identity, the data access times are the times of accessing the data processing terminal corresponding to the terminal identifier, and model training is performed on the classification network model by using the classification label based on the marked data object, the terminal identifier and the data access times as training sample data. The established data Classification model can be a two-Classification prediction model in a machine learning algorithm or a Multi-Classification prediction model, such as Multi-class Classification, so as to carry out model training.
It should be noted that, since the classification label based on the labeled data object is matched with the data access frequency of the data processing terminal, for example, the data access frequency of the data processing terminal a exceeds the preset threshold 3, and the labeled data object is a tag tenant, the classification label is 1, so that the data classification model is trained based on the training sample data, and when the obtained data classification model performs classification processing on the tenant 1, the data processing terminal a with the classification label of 1 is determined to perform processing, so as to construct and obtain the training sample data.
In another embodiment of the present invention, for further definition and explanation, as shown in fig. 3, the step of determining a training data processing terminal from the data processing terminals comprises:
301. and acquiring service demand information.
302. Acquiring data processing record information of the data processing terminal which finishes data processing;
303. and if the data processing record information is matched with the service requirement information, determining the data processing terminal as a training data processing terminal.
In the embodiment of the invention, in order to accurately find the training data processing terminal storing the training sample data from the plurality of data processing terminals to perform accurate model training and improve the accuracy of data processing, firstly, the service requirement information is acquired. The service requirement information comprises data classification types and data processing types corresponding to data objects in different service scenes, wherein the data classification types comprise image data types, audio data types, video data types and model algorithm data types in unstructured data types, and the data processing types comprise storage processing types, query processing types and update processing types. When the service demand information is acquired, request loading can be carried out on an application client side and also can be carried out on an application server side, so that service demand data which needs to be matched in gateway equipment serving as a current execution end is acquired, a training data processing terminal is accurately determined, and therefore a classification label of a marked data object and identity information of the marked terminal are acquired in the training data processing terminal and serve as training sample data. And meanwhile, data processing record information which is processed by data in each data processing terminal is acquired, wherein the data processing record information comprises specific information for storing, inquiring and updating historical data objects, so that matching is performed on the basis of the data processing record information and the service requirement information, the specific information for storing, inquiring and updating the historical data objects in the selected training data processing terminal is suitable for the service requirement information, and a data classification model matching the service requirement information is obtained by performing model training on the basis of training sample data in the training data processing terminal.
In another embodiment of the present invention, for further definition and explanation, the steps further comprise:
sending the service requirement information to the data processing terminal according to a preset time interval, and indicating the data processing terminal to carry out label marking processing based on the service requirement information;
and receiving label indication information of the data processing terminal for finishing label labeling processing, and executing the step of determining a training data processing terminal from the data processing terminal so as to update and train the data classification model.
In order to achieve the accuracy of classifying data objects based on the data classification model, the embodiment of the invention transmits the service requirement information to the data processing terminal, so that the data processing terminal performs label marking processing based on the service requirement information. The gateway device serving as the current execution end can request to acquire the latest service requirement information, send the latest service requirement information to the data processing terminal, and instruct the data processing terminal to perform label marking processing according to the service requirement information, so that the gateway device serving as the current execution end is updated as training sample data. At this time, in order to update the data classification model adaptively, the gateway device serving as the current execution end may send the service requirement information to the data processing terminal according to a preset time interval, that is, after obtaining the minimum service requirement information, the service requirement information is transmitted according to a preset time interval of 1 day or 2 days, for example.
After each data processing terminal completes label marking processing based on the received service demand information, label indication information for completing the label marking processing is fed back to inform the gateway equipment serving as the current execution end, so that the gateway equipment serving as the current execution end executes the step of determining the training data processing terminal in the data processing terminal, the purpose of updating and training the data classification model is achieved, the effectiveness of the data classification model is greatly improved, and the data classification capability of the gateway equipment under different service demand information updating scenes is met, so that intelligent data processing is performed.
In another embodiment of the present invention, for further limitation and description, as shown in fig. 4, after the step of establishing a data classification model, and performing model training on the data classification model based on the classification label and the training sample data to obtain a trained data classification model, the step further includes:
401. acquiring a third party application identifier for data transmission through the gateway data transmission interface, and determining a model verification index matched with the third party application identifier based on a preset model verification relationship;
402. and if the model verification parameters of the trained data classification model are matched with the model verification indexes, confirming that the data classification model passes the index verification, and classifying the data objects through the data classification model.
In order to improve the classification processing effectiveness of the data classification model, the gateway device serving as the current execution end acquires the third-party application identifier, determines a model verification index of the data classification model based on the preset model verification relation, and verifies the data classification model. Because different machine learning models have different functions and different verification indexes, the third-party application identifier is an identity identifier of an application client or an application server which is in data communication with the gateway data transmission interface so as to determine the model verification index matched with the third-party application identifier, and therefore the model verification index is adapted to different application scenes. For example, if the third-party application identifier is a health examination identifier, the model verification index matched with the health examination identifier is determined to be more than 80% of model precision based on the preset model verification relationship, so that after the model precision is determined, whether the model precision is matched with the model precision by more than 80% is judged based on the model verification parameters of the trained data classification model, if the model precision is matched with the model precision by more than 80%, the data classification model passes verification, and the data classification model can be applied to classification processing, namely, online application in gateway equipment.
It should be noted that, in the embodiment of the present invention, the model verification index is an index content that is preset and expected to be reached by the machine learning model, including, but not limited to, upper limits or lower limits of training duration, training times, model precision, model errors, and the like, so as to be compared with the model verification parameters of the data classification model, where the model verification parameters are specific values of the training duration, the training times, the model precision, the model errors, and the like that are calculated according to the data classification model, thereby verifying whether the data classification model has the capability of data classification processing, so as to improve the accuracy of data classification.
In another embodiment of the present invention, for further definition and explanation, the step of processing the data object in the target data processing terminal through the gateway data transmission interface comprises:
if the data processing request is a data storage request, outputting the data object to the target data processing terminal through the gateway data transmission interface, and indicating the target data processing terminal to store the data content of the data object;
if the data processing request is a data query request, sending a data query request of the data object to the target data processing terminal through the gateway data transmission interface, and instructing the target data processing terminal to query and feed back the data content of the data object;
and if the data processing request is a data updating request, outputting the data object to the target data processing terminal through the gateway data transmission interface, and instructing the target data processing terminal to replace and update the data content of the data object.
In the embodiment of the present invention, in order to execute corresponding data processing for different data processing requests, a gateway device serving as a current execution end executes corresponding processing operations through a gateway data transmission interface, where the data processing request includes a data storage request, a data query request, and a data update request. Specifically, if the data processing request is a data storage request, after the target data processing terminal is determined, the data object is output to the target data processing terminal through the gateway data transmission interface so as to store the data content of the data object. If the data processing request is a data query request, after the target data processing terminal is determined, the data query request is sent to the target data processing terminal so as to find out the data content of the data object in the target data processing terminal and feed back the data content to the current execution terminal. And if the data processing request is a data updating request, determining a target data processing terminal, and outputting the data object to the target data processing terminal so that the target data processing terminal replaces the data content of the updated data object.
It should be noted that, in a specific application scenario, based on a predefined unique one of the multiple data processing terminals as a memory for storing training sample data, that is, when each data processing terminal performs data processing, the number of data accesses of the target data processing terminal and the corresponding data tag are stored in the memory DB through the gateway data transmission interface, as shown in fig. 5, so as to directly retrieve the training sample data in the memory DB for model training when acquiring the training data processing terminal.
In another embodiment of the present invention, for further limitation and explanation, before the step of outputting the data object to the target data processing terminal through the gateway data transmission interface, the method further includes:
acquiring data processing performance parameters of the target data processing terminal;
if the data performance parameters do not match the transmission conditions of the gateway equipment, the binding relationship with the target data processing terminal is removed;
and if the data performance parameters match the transmission conditions of the gateway equipment, indicating to execute the step of outputting the data object to the target data processing terminal through the gateway data transmission interface.
In the embodiment of the present invention, as shown in fig. 5, in order to improve the processing accuracy of the data processing terminal, before outputting the data object to the target data processing terminal, it is determined whether the data processing performance parameter of the target data processing terminal matches the transmission condition of the gateway device, so as to ensure the transmission effectiveness. Wherein, the data processing performance parameter comprises at least one of a storage performance parameter, an inquiry performance parameter, a terminal hardware operation parameter and a terminal operation line number, the storage performance parameter is the space size of the data processing terminal for storing data, the inquiry performance parameter is the speed of the data processing terminal for inquiring data, the terminal hardware operation parameter is the operation environment, the operation code type and the like of the data processing terminal, the terminal operation line number is the number of threads for processing data in the data processing terminal, thereby the transmission condition of the gateway equipment is pre-configured aiming at the data processing performance parameters of different data processing terminals, namely when the data performance parameter does not match the transmission condition of the gateway equipment, the binding relation with the target data processing terminal is removed, so that the current gateway equipment can not output data objects through the gateway data transmission interface, when the data performance parameter matches the transmission condition of the gateway equipment, the current gateway device transmits the data object to the target data processing terminal through the gateway data transmission interface. The gateway data transmission condition is a pre-configured condition for limiting whether the data processing terminal can perform data processing on the data object, for example, the gateway data transmission condition is that the data processing terminal has a high-performance object storage capability, and if the gateway data transmission condition is determined to be the normal-performance object storage capability according to the storage performance parameter in the data performance parameters, the gateway data transmission condition is not matched, and the current gateway device releases the binding relationship between the gateway data transmission interface and the data processing terminal with the normal-performance object storage capability.
It should be noted that, because the data processing performance parameters in the data processing terminal change with the service processing scenario, before each transmission, the judgment that the current data performance parameters match the transmission conditions of the gateway device is performed, so as to improve the efficiency of outputting the data object to the data processing terminal for data processing based on the mesh device.
Compared with the prior art, the embodiment of the invention configures a gateway data transmission interface and binds at least one data processing terminal, wherein the gateway data transmission interface is used for transmitting data objects flowing through gateway equipment; receiving a data processing request through the gateway data transmission interface, classifying data objects carried in the data processing request based on a trained data classification model, and determining a target data processing terminal of the data objects, wherein training sample data of the data classification model is a classification label for determining different data objects to be classified to the target data processing terminal according to data access times; the data objects are processed in the target data processing terminal through the gateway data transmission interface, diversified and safe data processing of multi-object data in different service scenes is achieved, the effectiveness of data processing such as data storage and query is improved, the data processing cost is reduced, and therefore the efficiency of service processing based on the stored and queried data in different service scenes is improved.
Further, as an implementation of the method shown in fig. 1, an embodiment of the present invention provides an artificial intelligence-based data processing apparatus, as shown in fig. 6, where the apparatus includes:
a configuration module 51, configured to configure a gateway data transmission interface, and bind at least one data processing terminal, where the gateway data transmission interface is used to transmit a data object flowing through a gateway device;
a receiving module 52, configured to receive a data processing request through the gateway data transmission interface, perform classification processing on a data object carried in the data processing request based on a trained data classification model, and determine a target data processing terminal of the data object, where training sample data of the data classification model is a classification label for determining that different data objects are classified into the target data processing terminal according to data access times;
a processing module 53, configured to process the data object in the target data processing terminal through the gateway data transmission interface.
Further, the apparatus further comprises:
the determining module is used for determining a training data processing terminal from the data processing terminals and acquiring training sample data in the training data processing terminal, wherein the classification label in the training sample data is determined based on the data access times of the data processing terminals;
and the training module is used for establishing a data classification model, and performing model training on the data classification model based on the classification label and the training sample data to obtain a trained data classification model.
Further, the determining module includes:
the first obtaining unit is used for obtaining service requirement information, wherein the service requirement information comprises data classification types and data processing types corresponding to data objects in different service scenes, the data classification types comprise image data types, audio data types, video data types and model algorithm data types in unstructured data types, and the data processing types comprise storage processing types, query processing types and update processing types;
a second acquisition unit configured to acquire data processing record information in which data processing has been completed in the data processing terminal;
and the determining unit is used for determining the data processing terminal as a training data processing terminal if the data processing record information matches the service requirement information, so that the training data processing terminal acquires the classification label of the marked data object and the marked terminal identity information as training sample data.
Further, the determining module further comprises:
a sending unit, configured to send the service requirement information to the data processing terminal according to a preset time interval, and instruct the data processing terminal to perform label marking processing based on the service requirement information;
and the receiving unit is used for receiving the label indication information of the data processing terminal for completing label labeling processing, and executing the step of determining a training data processing terminal from the data processing terminal so as to update and train the data classification model.
Further, the determining module is specifically configured to acquire a third-party application identifier for performing data transmission through the gateway data transmission interface, and determine a model verification index matched with the third-party application identifier based on a preset model verification relationship, where the preset model verification relationship includes a correspondence relationship between a plurality of third-party application identifiers and a plurality of model verification indexes; and if the model verification parameters of the trained data classification model are matched with the model verification indexes, confirming that the data classification model passes the index verification, and classifying the data objects through the data classification model.
Further, the processing module comprises:
a first output unit, configured to output the data object to the target data processing terminal through the gateway data transmission interface if the data processing request is a data storage request, and instruct the target data processing terminal to store the data content of the data object;
a second output unit, configured to send a data query request for the data object to the target data processing terminal through the gateway data transmission interface if the data processing request is a data query request, and instruct the target data processing terminal to query and feed back data content of the data object;
and the third output unit is used for outputting the data object to the target data processing terminal through the gateway data transmission interface and instructing the target data processing terminal to replace and update the data content of the data object if the data processing request is a data updating request.
Further, the apparatus further comprises:
the acquisition module is used for acquiring data processing performance parameters of the target data processing terminal, wherein the data processing performance parameters comprise at least one of storage performance parameters, query performance parameters, terminal hardware operation parameters and terminal operation line number;
the release module is used for releasing the binding relation with the target data processing terminal if the data performance parameters do not match the transmission conditions of the gateway equipment;
and the indicating module is used for indicating to execute the step of outputting the data object to the target data processing terminal through the gateway data transmission interface if the data performance parameters are matched with the transmission conditions of the gateway equipment.
Compared with the prior art, the embodiment of the invention configures a gateway data transmission interface and binds at least one data processing terminal, wherein the gateway data transmission interface is used for transmitting data objects flowing through gateway equipment; receiving a data processing request through the gateway data transmission interface, classifying data objects carried in the data processing request based on a trained data classification model, and determining a target data processing terminal of the data objects, wherein training sample data of the data classification model is a classification label for determining different data objects to be classified to the target data processing terminal according to data access times; the data objects are processed in the target data processing terminal through the gateway data transmission interface, so that diversified and safe data processing of multi-object data in different service scenes is realized, the effectiveness of data processing such as data storage and query is improved, the data processing cost is reduced, and the efficiency of service processing based on the stored and queried data in different service scenes is improved.
According to an embodiment of the present invention, a storage medium is provided, where the storage medium stores at least one executable instruction, and the computer executable instruction can execute the artificial intelligence based data processing method in any of the method embodiments described above.
Fig. 7 is a schematic structural diagram of a gateway device according to an embodiment of the present invention, and the specific embodiment of the present invention does not limit the specific implementation of the computer device.
As shown in fig. 7, the computer apparatus may include: a processor (processor)602, a communication Interface 604, a memory 606, and a communication bus 608.
Wherein: the processor 602, communication interface 604, and memory 606 communicate with one another via a communication bus 608.
A communication interface 604 for communicating with network elements of other devices, such as clients or other servers.
The processor 602 is configured to execute the program 610, and may specifically execute relevant steps in the above-described data processing method based on artificial intelligence.
In particular, program 610 may include program code comprising computer operating instructions.
The processor 602 may be a central processing unit CPU or an application Specific Integrated circuit asic or one or more Integrated circuits configured to implement embodiments of the present invention. The computer device includes one or more processors, which may be the same type of processor, such as one or more CPUs; or may be different types of processors such as one or more CPUs and one or more ASICs.
And a memory 606 for storing a program 610. Memory 606 may comprise high-speed RAM memory, and may also include non-volatile memory (non-volatile memory), such as at least one disk memory.
The program 610 may specifically be configured to cause the processor 602 to perform the following operations:
configuring a gateway data transmission interface and binding at least one data processing terminal, wherein the gateway data transmission interface is used for transmitting data objects flowing through gateway equipment;
receiving a data processing request through the gateway data transmission interface, classifying data objects carried in the data processing request based on a trained data classification model, and determining a target data processing terminal of the data objects, wherein training sample data of the data classification model is a classification label for determining different data objects to be classified to the target data processing terminal according to data access times;
and processing the data object in the target data processing terminal through the gateway data transmission interface.
It will be apparent to those skilled in the art that the modules or steps of the present invention described above may be implemented by a general purpose computing device, they may be centralized on a single computing device or distributed across a network of multiple computing devices, and alternatively, they may be implemented by program code executable by a computing device, such that they may be stored in a storage device and executed by a computing device, and in some cases, the steps shown or described may be performed in an order different than that described herein, or they may be separately fabricated into individual integrated circuit modules, or multiple ones of them may be fabricated into a single integrated circuit module. Thus, the present invention is not limited to any specific combination of hardware and software.
The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
1. A data processing method based on artificial intelligence is characterized by comprising the following steps:
configuring a gateway data transmission interface and binding at least one data processing terminal, wherein the gateway data transmission interface is used for transmitting data objects flowing through gateway equipment;
receiving a data processing request through the gateway data transmission interface, classifying data objects carried in the data processing request based on a trained data classification model, and determining a target data processing terminal of the data objects, wherein training sample data of the data classification model is a classification label for determining different data objects to be classified to the target data processing terminal according to data access times;
and processing the data object in the target data processing terminal through the gateway data transmission interface.
2. The method according to claim 1, wherein before the classification processing of the data object carried in the data processing request based on the trained data classification model and the determination of the target data processing terminal of the data object, the method further comprises:
determining a training data processing terminal from the data processing terminals, and acquiring training sample data in the training data processing terminal, wherein a classification label in the training sample data is determined based on the data access times of each data processing terminal;
and establishing a data classification model, and performing model training on the data classification model based on the classification label and the training sample data to obtain the trained data classification model.
3. The method of claim 2, wherein said determining training data processing terminals from said data processing terminals comprises:
acquiring service demand information, wherein the service demand information comprises data classification types and data processing types corresponding to data objects in different service scenes, the data classification types comprise image data types, audio data types, video data types and model algorithm data types in unstructured data types, and the data processing types comprise storage processing types, query processing types and update processing types;
acquiring data processing record information of the data processing terminal which finishes data processing;
and if the data processing record information is matched with the service requirement information, determining the data processing terminal as a training data processing terminal so as to obtain the classification label of the marked data object and the marked terminal identity information in the training data processing terminal as training sample data.
4. The method of claim 3, further comprising:
sending the service requirement information to the data processing terminal according to a preset time interval, and indicating the data processing terminal to carry out label marking processing based on the service requirement information;
and receiving label indication information of the data processing terminal for completing label labeling processing, and executing the step of determining a training data processing terminal from the data processing terminal so as to update and train the data classification model.
5. The method of claim 2, wherein after the building of the data classification model and the model training of the data classification model based on the classification labels and the training sample data to obtain the trained data classification model, the method further comprises:
acquiring a third party application identifier for data transmission through the gateway data transmission interface, and determining a model verification index matched with the third party application identifier based on a preset model verification relationship, wherein the preset model verification relationship comprises a plurality of third party application identifiers and a plurality of corresponding relationships among the model verification indexes;
and if the model verification parameters of the trained data classification model are matched with the model verification indexes, confirming that the data classification model passes the index verification, and classifying the data objects through the data classification model.
6. The method according to any of claims 1-5, wherein said processing said data object in said target data processing terminal via said gateway data transfer interface comprises:
if the data processing request is a data storage request, outputting the data object to the target data processing terminal through the gateway data transmission interface, and indicating the target data processing terminal to store the data content of the data object;
if the data processing request is a data query request, sending a data query request of the data object to the target data processing terminal through the gateway data transmission interface, and instructing the target data processing terminal to query and feed back the data content of the data object;
and if the data processing request is a data updating request, outputting the data object to the target data processing terminal through the gateway data transmission interface, and instructing the target data processing terminal to replace and update the data content of the data object.
7. The method of claim 6, wherein prior to outputting the data object to the target data processing terminal via the gateway data transfer interface, the method further comprises:
acquiring data processing performance parameters of the target data processing terminal, wherein the data processing performance parameters comprise at least one of storage performance parameters, query performance parameters, terminal hardware operation parameters and terminal operation line number;
if the data performance parameters do not match the transmission conditions of the gateway equipment, the binding relationship with the target data processing terminal is removed;
and if the data performance parameters match the transmission conditions of the gateway equipment, indicating to execute the step of outputting the data object to the target data processing terminal through the gateway data transmission interface.
8. An artificial intelligence-based data processing apparatus, comprising:
the system comprises a configuration module, a data processing terminal and a gateway data transmission interface, wherein the configuration module is used for configuring a gateway data transmission interface and binding at least one data processing terminal, and the gateway data transmission interface is used for transmitting data objects flowing through gateway equipment;
the receiving module is used for receiving a data processing request through the gateway data transmission interface, classifying data objects carried in the data processing request based on a trained data classification model, and determining a target data processing terminal of the data objects, wherein training sample data of the data classification model is a classification label for determining different data objects to be classified to the target data processing terminal according to data access times;
and the processing module is used for processing the data object in the target data processing terminal through the gateway data transmission interface.
9. A storage medium having stored therein at least one executable instruction for causing a processor to perform operations corresponding to the artificial intelligence based data processing method of any one of claims 1-7.
10. A gateway device, comprising: the system comprises a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface complete mutual communication through the communication bus;
the memory is used for storing at least one executable instruction which causes the processor to execute the operation corresponding to the artificial intelligence based data processing method as claimed in any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210715239.1A CN115080771A (en) | 2022-06-23 | 2022-06-23 | Data processing method and device based on artificial intelligence, medium and gateway equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210715239.1A CN115080771A (en) | 2022-06-23 | 2022-06-23 | Data processing method and device based on artificial intelligence, medium and gateway equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115080771A true CN115080771A (en) | 2022-09-20 |
Family
ID=83254210
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210715239.1A Pending CN115080771A (en) | 2022-06-23 | 2022-06-23 | Data processing method and device based on artificial intelligence, medium and gateway equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115080771A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115865670A (en) * | 2023-02-27 | 2023-03-28 | 灵长智能科技(杭州)有限公司 | Method and device for adjusting concurrency performance of WEB security gateway based on kernel tuning |
-
2022
- 2022-06-23 CN CN202210715239.1A patent/CN115080771A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115865670A (en) * | 2023-02-27 | 2023-03-28 | 灵长智能科技(杭州)有限公司 | Method and device for adjusting concurrency performance of WEB security gateway based on kernel tuning |
| CN115865670B (en) * | 2023-02-27 | 2023-06-16 | 灵长智能科技(杭州)有限公司 | Method and device for adjusting concurrency performance of WEB security gateway based on kernel tuning |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN115022176B (en) | NWDAF network element selection method and device, electronic equipment and readable storage medium | |
| US11411897B2 (en) | Communication method and communication apparatus for message queue telemetry transport | |
| US11398966B2 (en) | Cluster wide packet tracing | |
| US20230042747A1 (en) | Message Processing Method and Device, Storage Medium, and Electronic Device | |
| CN111930598B (en) | Information processing method based on block chain and big data analysis and big data platform | |
| CN109327511B (en) | Data request method and server based on HTTP (hyper text transport protocol) | |
| CN110362455A (en) | A kind of data processing method and data processing equipment | |
| CN111083204B (en) | File transmission method, device and storage medium | |
| CN111225082B (en) | Identity management method and device of Internet of things intelligent equipment and Internet of things platform | |
| WO2016114750A1 (en) | Data link layer information | |
| CN115080771A (en) | Data processing method and device based on artificial intelligence, medium and gateway equipment | |
| CN105281944B (en) | Method for setting network protocol address and service management system | |
| KR101794200B1 (en) | Method for transmitting and receiving protocol packet using packet processor based LINUX in distributed packet transport network | |
| US11714692B2 (en) | Classical management of qubit requests | |
| CN110661896A (en) | Method for determining mapping address of data stream and server | |
| CN105939278B (en) | Traffic processing method and device | |
| CN111294221B (en) | Network isolation configuration method and device based on haproxy | |
| EP3562194B1 (en) | Method for identifying at least one network slice configuration of a mobile network, communication system, and automation system | |
| CN112055058A (en) | Data storage method and device and computer readable storage medium | |
| CN111240867A (en) | Information communication system and method | |
| CN117061072B (en) | Message transmission method, device, equipment and storage medium | |
| CN115051866B (en) | Message processing method and device, storage medium and electronic equipment | |
| CN110708208B (en) | Monitoring data acquisition method and device, storage medium and terminal | |
| US20230164030A1 (en) | Model processing method for cloud service system and cloud service system | |
| CN113127222B (en) | Data transmission method, device, equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |