CN113946362A - Consumption data processing method and storage medium - Google Patents
Consumption data processing method and storage medium Download PDFInfo
- Publication number
- CN113946362A CN113946362A CN202111028477.7A CN202111028477A CN113946362A CN 113946362 A CN113946362 A CN 113946362A CN 202111028477 A CN202111028477 A CN 202111028477A CN 113946362 A CN113946362 A CN 113946362A
- Authority
- CN
- China
- Prior art keywords
- consumption data
- configuration information
- consumption
- party
- service
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/70—Software maintenance or management
- G06F8/71—Version control; Configuration management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3051—Monitoring arrangements for monitoring the configuration of the computing system or of the computing system component, e.g. monitoring the presence of processing resources, peripherals, I/O links, software programs
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Security & Cryptography (AREA)
- Computing Systems (AREA)
- Quality & Reliability (AREA)
- Telephonic Communication Services (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
Abstract
The invention provides a consumption data processing method and a storage medium, wherein the method comprises the following steps: reading configuration information, including an acquisition path of consumption data on a third-party platform and a distribution mode of an interface calling mode and/or a message stream writing mode; the method comprises the steps that at least one first service party of which distribution mode configuration information is a message flow writing mode is obtained, corresponding first consumption data are obtained, and the first consumption data are sent to the corresponding first service party; and/or acquiring at least one second service party of which the distribution mode configuration information is an interface calling mode, acquiring corresponding second consumption data, receiving an interface access request sent by the second service party, and sending the second consumption data to the second service party. The consumption data processing method and the storage medium provided by the embodiment of the invention avoid the repeated development of codes in the consumption data acquisition process of each business party, simplify the consumption data acquisition process and improve the efficiency of the business party in acquiring the consumption data.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a consumption data processing method and a storage medium.
Background
The information publishing-subscribing mode is mainly two blocks, namely publishing and subscribing, and the completion of the publishing and subscribing process depends on the sufficient cooperation of each stage of information access, information current limitation, information processing and the like.
In the process of processing a large amount of information streams in the existing project, the message publishing-subscribing system is mainly used for publishing and subscribing messages, and needs to customize and develop access of information, service processing of information, current limiting of information and corresponding information processing capacity based on different consumers. However, in the whole process, besides the business processing, other processes are contents of respective repeated development of the business, and the repeated development process causes redundancy of the programming process and also causes low efficiency of the information processing process.
Disclosure of Invention
The invention provides a consumption data processing method and a storage medium aiming at the problems in the prior art.
The invention provides a consumption data processing method, which comprises the following steps: reading configuration information; the configuration information comprises consumption information source configuration information and distribution mode configuration information, the consumption information source configuration information comprises an acquisition path of consumption data on a third-party platform, and the distribution mode configuration information comprises a message flow writing mode and/or an interface calling mode; the configuration information corresponds to the service parties one by one;
acquiring the distribution mode configuration information as at least one first service party of the message flow writing mode, acquiring corresponding first consumption data according to an acquisition path of the consumption data on a third party platform, and sending the first consumption data to the corresponding first service party according to a preset consumption rule; and/or acquiring at least one second service party of which the distribution mode configuration information is the interface calling mode, acquiring corresponding second consumption data according to an acquisition path of the consumption data on a third-party platform, receiving an interface access request sent by the second service party and used for acquiring the second consumption data, and sending the corresponding second consumption data to the second service party.
According to the consumption data processing method provided by the invention, the configuration information further comprises process management configuration information, and the process management configuration information comprises at least one of process starting configuration information, process monitoring configuration information and process protection configuration information; the process starting configuration information comprises setting starting conditions for a main process and a sub process; the process protection configuration information comprises setting a protection process for the main process and/or configuring a new sub-process to be restarted after the sub-process is monitored to be abnormal; the process monitoring configuration information comprises the operation conditions of a main process and a sub-process, and process protection is triggered when the main process and/or the sub-process are monitored to be abnormal in operation; the method further comprises the following steps: and carrying out process management according to the process management configuration information.
According to the consumption data processing method provided by the invention, the sub-process corresponds to the first service party or the second service party, and the sub-process is used for acquiring the corresponding first consumption data according to the acquisition path of the consumption data on a third-party platform or acquiring the corresponding second consumption data according to the acquisition path of the consumption data on the third-party platform.
According to the consumption data processing method provided by the invention, the main process utilizes task table data as a data source for pulling up the subprocess and realizes the monitoring of the running condition of the subprocess; the task table data comprises a process number of the sub-process, a service identifier and an operation state of the sub-process, wherein the service identifier is an identifier of the first service party or the second service party.
According to the consumption data processing method provided by the invention, the configuration information further includes current limit configuration information, and the current limit configuration information is used for controlling the writing speed when the distribution mode configuration information is the message stream writing mode.
According to a consumption data processing method provided by the invention, the current limiting configuration information comprises a speed for configuring token delivery to a token bucket; the sending the first consumption data to the corresponding first service party according to a preset consumption rule specifically includes: and requesting to take the token away from the token bucket, if the token is taken, sending the first consumption data to the corresponding first business party according to a preset consumption rule, and if the token is not taken, turning to the request to take the token away from the token bucket.
According to a consumption data processing method provided by the present invention, the configuration information further includes: when the distribution mode configuration information is the interface calling mode, configuring a storage rule of the consumption data in a database; the storage rule comprises a data storage time and/or an upper limit of data storage; and/or the configuration information further includes service retry configuration information, where the service retry configuration information includes information for configuring the second service party to reinitiate the interface access request after the interface access request fails.
According to a consumption data processing method provided by the present invention, the method further comprises: when the service consumption is in a problem, re-consumption is carried out according to the time node and/or the offset of the consumption data stored in the database; and/or when the local storage data is lost, automatically and synchronously acquiring the data from the offset of the kafka data source node according to the consumption information source configuration information.
According to the consumption data processing method provided by the invention, the configuration information further comprises alarm rule configuration information; the method further comprises the following steps: and performing alarm processing according to the alarm rule configuration information, and sending alarm information to the corresponding first service party or the second service party.
The invention also provides an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the consumption data processing method as described in any one of the above when executing the program.
The invention also provides a non-transitory computer-readable storage medium having stored thereon a computer program which, when executed by a processor, carries out the steps of the consumption data processing method as described in any of the above.
According to the consumption data processing method and the storage medium, the specific path for the business party to acquire the consumption data from the third-party platform is determined through the configuration of the consumption rule, and the business party can accurately acquire the consumption data in the third-party platform; the configuration of the distribution mode enables the business side to have the capacity of actively requesting and passively receiving the consumption data, thereby enriching the data acquisition mode of the business side; based on the configuration of the consumption data processing process, repeated development of codes of all business parties in the consumption data acquisition process is avoided, the consumption data acquisition process is simplified, and the efficiency of the business parties for acquiring the consumption data is improved.
Drawings
In order to more clearly illustrate the technical solutions of the present invention or the prior art, the drawings needed for the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and those skilled in the art can also obtain other drawings according to the drawings without creative efforts.
FIG. 1 is a schematic flow chart of a method for processing consumption data according to the present invention;
FIG. 2 is a second flowchart of a consumption data processing method according to the present invention;
FIG. 3 is a schematic diagram of a consumer data processing apparatus provided by the present invention;
fig. 4 is a schematic structural diagram of an electronic device provided in the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is obvious that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Fig. 1 is a schematic flow chart of a consumption data processing method provided by the present invention, as shown in fig. 1, the method includes:
step S110, reading configuration information; the configuration information comprises consumption information source configuration information and distribution mode configuration information, the consumption information source configuration information comprises an acquisition path of consumption data on a third-party platform, and the distribution mode configuration information comprises a message flow writing mode and/or an interface calling mode; the configuration information corresponds to the service parties one by one;
step S120, acquiring the distribution mode configuration information as at least one first service party of the message flow writing mode, acquiring corresponding first consumption data according to an acquisition path of the consumption data on a third party platform, and sending the first consumption data to the corresponding first service party according to a preset consumption rule; and/or acquiring at least one second service party of which the distribution mode configuration information is the interface calling mode, acquiring corresponding second consumption data according to an acquisition path of the consumption data on a third-party platform, receiving an interface access request sent by the second service party and used for acquiring the second consumption data, and sending the corresponding second consumption data to the second service party.
The third party platform may be a publish-subscribe messaging system in which the producer of messages is referred to as the publisher and the consumer is referred to as the subscriber, typically representing two products, MQ and Kafka, in Kafka for example, messages are categorized by topics (Topic), each Topic corresponding to a message queue, i.e. similar to a table in a database, but if all the messages of the same class are stuffed into a "central" queue, there is a lack of scalability, either as the number of producers/consumers increases or as the number of messages increases, which may exhaust the performance or storage of the system. Based on this, the message queues in the Topic are partitioned, and each message queue is divided into different partitions (Partition), so as to realize the detailed management of the message queues.
A Kafka server, also known as a Broker, accepts messages sent by the producer and stores them in disk, and the Broker also serves requests from consumers to pull partition messages, returning messages that have been submitted. With specific machine hardware, a Broker can process thousands of partitions and millions of messages per second. Several Broker groups form a Cluster (Cluster), wherein a certain Broker in the Cluster becomes a Cluster Controller (Cluster Controller) which is responsible for managing the Cluster, including assigning partitions to the Broker, monitoring Broker failures, etc. Within a cluster, a partition is responsible for a Broker, also referred to as the Leader for that partition. Of course, one partition can be duplicated on multiple Broker for redundancy so that its partition can be reassigned to other Broker for responsibility when there is a Broker failure.
The consumption data processing method provided by the invention can be applied to a server. It should be noted that the consumption information source configuration information includes an acquisition path of consumption data on the third-party platform, and specifically includes: taking Kafka as an example, a business party needs to determine a corresponding Cluster (Cluster), a server (Broker) on the Cluster, a theme (Topic) and a Partition (Partition) on the server, and when the parameters are determined, a complete path for the business party to obtain consumption data from the Kafka is obtained.
The distribution mode configuration information comprises an interface calling mode and/or a message flow writing mode, and the two different distribution modes respectively correspond to specific modes of acquiring consumption data by a service party. The message flow writing mode belongs to a distribution mode that a service party passively receives consumption data, and specifically, a server sends the consumption data on Kafka to the service party according to a preset consumption data acquisition path, and in the process, the service party acquiring the consumption data by using the message flow writing mode is called a first service party; an interface calling mode belongs to a distribution mode that a service side actively requests to acquire consumption data, and specifically can be that the service side sends an interface access request to a server based on an http protocol, the server sends corresponding consumption data to the service side according to a preset consumption data acquisition path after receiving the request, and if the consumption data requested to be acquired by the service side exists in a database, the consumption data stored in the database is directly sent to the service side, and in the process, the service side acquiring the consumption data by using the interface calling mode is called as a second service side.
It should be noted that, in a project that handles a large number of information flows, there are a plurality of business parties, and only one distribution method may be adopted in the configuration of the distribution method, or different distribution methods may exist at the same time, and a business party may autonomously select which distribution method is specifically adopted.
According to the consumption data processing method provided by the invention, through configuration of the consumption rule, a specific path for the business party to acquire the consumption data from the third-party platform is determined, and the business party can accurately acquire the consumption data in the third-party platform; the configuration of the distribution mode enables the business side to have the capacity of actively requesting and passively receiving the consumption data, thereby enriching the data acquisition mode of the business side; based on the configuration of the consumption data processing process, repeated development of codes of all business parties in the consumption data acquisition process is avoided, the consumption data acquisition process is simplified, and the efficiency of the business parties for acquiring the consumption data is improved.
According to the consumption data processing method provided by the invention, in the invention, the configuration information further comprises process management configuration information, wherein the process management configuration information comprises at least one of process starting configuration information, process monitoring configuration information and process protection configuration information; the process starting configuration information comprises setting starting conditions for a main process and a sub process; the process protection configuration information comprises setting a protection process for the main process and/or configuring a new sub-process to be restarted after the sub-process is monitored to be abnormal; the process monitoring configuration information comprises the operation conditions of a main process and a sub-process, and process protection is triggered when the main process and/or the sub-process are monitored to be abnormal in operation;
the method further comprises the following steps: and carrying out process management according to the process management configuration information.
The complete process of the business party acquiring the consumption data from Kafka is called as a sub-process, and the business party can be specifically a first business party or a second business party. Parameter configuration is carried out on process management in a database, and specifically, omnibearing configuration is carried out on process starting, process monitoring and process protection; the process starting is used for starting the process according to the starting conditions set by the main process and the subprocess, the process monitoring is used for monitoring the running condition of the process in the process, and when the process running is monitored to be abnormal, the process protection is triggered and has two functions: firstly, when the main process is monitored to be abnormal, triggering a backup process, and continuing to perform unfinished processes based on the backup process; and secondly, restarting one sub-process when the sub-process is monitored to be abnormal.
According to the consumption data processing method provided by the invention, the process starting configuration information, the process monitoring configuration information and the process protection configuration information are carried out in the database in advance, so that when the process of acquiring the consumption data by a business side is abnormal, automatic processing can be carried out in time, the normal operation of the whole consumption data acquisition process is ensured, and the reliability of data acquisition is improved.
According to the consumption data processing method provided by the invention, in the invention, the sub-process corresponds to the first service party or the second service party, and the sub-process is used for acquiring the corresponding first consumption data according to the acquisition path of the consumption data on a third-party platform or acquiring the corresponding second consumption data according to the acquisition path of the consumption data on the third-party platform.
It should be noted that, the first service party obtains information in a message stream writing manner, and correspondingly, the information obtained by the first service party is referred to as first consumption data; and the second service party acquires the information in an interface calling mode, and correspondingly, the information acquired by the second service party is called second consumption data. According to the consumption data processing method provided by the invention, the consumption data processing process is divided into different sub-processes for respective processing according to different acquisition modes of consumption data, so that respective management of the sub-processes is favorably realized, the process management efficiency is improved, and the influence between the sub-processes is avoided.
According to the consumption data processing method provided by the invention, in the invention, the main process utilizes task table data as a data source for pulling up the subprocess and realizes the monitoring of the running condition of the subprocess; the task table data comprises a process number of the sub-process, a service identifier and an operation state of the sub-process, wherein the service identifier is an identifier of the first service party or the second service party.
The normal running of the main process is ensured by starting the resident main process and the protection process of the main process. The main process uses the task table data as a data source for pulling up the subtasks, thereby realizing the scheduling of the subtasks executed by the subtasks. The task table data comprises the process number of the sub-process, the service identifier and the running state of the sub-process. The sub-process corresponds to a business party, and thus, the business identifier is the identifier of the first business party or the second business party. The main process can regularly detect the task table data according to a certain time interval (such as every 5 minutes), monitor the running state of the subprocess, if the subprocess is found to be abnormally quit, acquire corresponding alarm information, and send the alarm information to a corresponding service party.
According to the consumption data processing method provided by the invention, the related information of the subprocess is stored by using the task table data, so that the quick control of starting and monitoring of the subprocess is realized.
According to the consumption data processing method provided by the invention, in the invention, the configuration information further includes current limiting configuration information, and the current limiting configuration information is used for controlling the writing speed when the distribution mode configuration information is the message stream writing mode.
It should be noted that, as described above, when the distribution mode is the interface call mode, the service party essentially acquires the consumption data in an active manner, so that the service party can control the acquisition frequency of the consumption data in this mode; however, when the distribution mode is a message stream writing mode, the service party acquires the consumption data in a passive receiving mode, and based on the passive receiving mode, the service party cannot control the acquisition frequency of the consumption data, and when the transmission frequency of the consumption data is too high or the transmission amount of the consumption data is too large, the acquisition pressure of the service party is caused, and the overload problem is caused. Based on this, when parameter configuration is performed in the database, current limiting configuration is performed for a distribution method such as writing to a message stream, and the purpose is to control the writing speed of the message stream.
According to the consumption data processing method provided by the invention, the control on the writing speed of the message stream is realized by the current limiting configuration in the database in the message stream writing mode, the acquisition pressure of a service party on the consumption data is effectively relieved, and the user experience is improved.
According to the consumption data processing method provided by the invention, the current limiting configuration information comprises the speed of delivering the tokens to the token bucket; the sending the first consumption data to the corresponding first service party according to a preset consumption rule specifically includes: and requesting to take the token away from the token bucket, if the token is taken, sending the first consumption data to the corresponding first business party according to a preset consumption rule, and if the token is not taken, turning to the request to take the token away from the token bucket.
The method can construct a token bucket based on Redis, wherein Redis is a high-performance key-value database, and the construction process comprises the steps of firstly setting a token bucket, storing tokens in the bucket, and setting the number of the tokens in the token bucket to be full (the number of the tokens in the bucket can be set according to the condition of a server); taking one token from the bucket in each access, and when the token in the bucket is 0, not allowing the access; at intervals, tokens are put in again, at most to make the bucket full of tokens. (several tokens can be put at intervals, or the token bucket can be filled directly, depending on the actual situation).
It should be noted that, in the case that the distribution mode is a message stream writing mode, a request needs to be sent to the token bucket before the message stream writing starts each time, the request is to take away the token from the constructed token bucket, and if the token acquisition is successful, the consumption data is sent to the first service party; and if the token acquisition fails, re-initiating a token acquisition request to the token bucket. In the above process, the reinitiation process of the token acquisition request essentially belongs to a waiting process between two successful acquisition processes of the consumption data, and the current limiting effect of the process of acquiring the consumption data by the first service party is realized based on the process.
According to the consumption data processing method provided by the invention, the token bucket is constructed based on Redis, and whether the token is acquired or not is used as a corresponding condition for the first business party to receive the consumption data, so that the current limiting effect on the consumption data acquisition process under the message flow writing condition is realized, and the data acquisition pressure of the first business party is effectively relieved.
According to a consumption data processing method provided by the present invention, in the present invention, the configuration information further includes: when the distribution mode configuration information is the interface calling mode, configuring a storage rule of the consumption data in a database; the storage rule comprises a data storage time and/or an upper limit of data storage;
and/or the configuration information further includes service retry configuration information, where the service retry configuration information includes information for configuring the second service party to reinitiate the interface access request after the interface access request fails.
It should be noted that, when the distribution mode is an interface call mode, the consumption data needs to be stored in the database so as to facilitate the subsequent call process of the consumption data, and when the consumption data is stored, the whole data storage needs to be in a dynamic balance, that is, the consumption data in the database has a storage call, and the storage capacity of the database needs to be considered, on the basis of which the storage time of the consumption data and the storage upper limit of the database are preconfigured.
When the distribution mode is the interface calling mode, the interface access request may fail due to problems such as short-time network condition difference or server stability difference, and the like, and the low efficiency of acquiring the consumption data is caused, so when parameter configuration is performed in the database, service retry configuration is performed at the same time, and the configuration is mainly directed to the situation under the interface calling distribution mode, specifically, when the second service party fails to perform the interface access request, the interface access request is automatically re-initiated.
Adding service retry configuration information into the configuration information for the interface calling mode, which is equivalent to adding a guarantee mechanism for the interface access request process, so as to ensure that the information of the interface access request can be reinitiated to make a request again on the premise of the failure of the initial request.
According to the consumption data processing method provided by the invention, under the condition that the distribution mode is the interface calling mode, the storage time and the storage upper limit in the storage process of the database are pre-configured, so that the whole database is in a dynamic balance, and the stable operation of the database is realized; meanwhile, the addition of the service retry configuration information can effectively improve the success rate of the interface access request and improve the reliability of the consumption data processing process.
According to the consumption data processing method provided by the invention, in the invention, the method further comprises the following steps: when the service consumption is in a problem, re-consumption is carried out according to the time node and/or the offset of the consumption data stored in the database;
and/or when the local storage data is lost, automatically and synchronously acquiring the data from the offset of the kafka data source node according to the consumption information source configuration information.
It should be noted that, when a problem occurs in service consumption, that is, when service consumption fails, a retry mechanism of service consumption is triggered, specifically: according to the time node and/or the offset of the consumption data stored in the database, the re-consumption is carried out, the retry mechanism can effectively ensure the accurate consumption of the consumption data, the problem of repeated consumption or missing consumption of the consumption data is avoided, and meanwhile, the retry mechanism can effectively ensure the stability and the reliability of the data consumption process.
When the locally stored data is lost, the data is automatically and synchronously acquired from the offset of the kafka data source node according to the consumption information source configuration information, the process is actually an automatic supplement process for the consumption data, and the 'synchronously acquiring the data from the offset of the kafka data source node' can effectively ensure that the supplemented data and the consumed data are not overlapped or missed, and ensure the accuracy and reliability of the supplemented data.
According to the consumption data processing method provided by the invention, when the service consumption has a problem, the secondary service consumption process can be carried out in time through the service consumption retry mechanism, so that the stability and reliability of the data consumption process are ensured; meanwhile, when the local storage data is lost through the automatic supplement process of the consumption data, the consumption data can be automatically supplemented in time, and the stable proceeding of the data consumption process is ensured.
According to the consumption data processing method provided by the invention, in the invention, the configuration information further comprises alarm rule configuration information; the method further comprises the following steps: and performing alarm processing according to the alarm rule configuration information, and sending alarm information to the corresponding first service party or the second service party.
When parameter configuration is carried out in a database, alarm rule configuration is carried out at the same time, when abnormity occurs in the consumption data acquisition process, alarm processing is carried out, alarm information is sent to a first service party or a second service party corresponding to the abnormity, and for the sending mode of the alarm information, an email can be specifically used for sending the alarm information to the first service party or the second service party corresponding to the abnormity.
It should be noted that the setting for the abnormal situation may be set to two cases according to different distribution manners: firstly, in a message flow writing mode, after current limiting, the writing of consumption data fails, at the moment, an alarm is triggered, and alarm information is sent to a corresponding first business party; and secondly, in the interface calling mode, in the process of storing the data of the consumption data, because of storage overload or failure of interface calling address identification, triggering alarm and sending alarm information to a corresponding second service party.
The consumption data processing method provided by the invention has the advantages that the alarm rules are configured in the database, different alarm rule settings are carried out according to different distribution modes, and the alarm information is transmitted to the business party, so that the business party can quickly and clearly know the current state when large abnormity occurs in the process of acquiring the consumption data, the interaction between the business party and the system is enhanced, and the normal operation of the whole system is ensured.
Fig. 2 is a second schematic flow chart of the consumption data processing method provided by the present invention, as shown in fig. 2, the method includes:
(1) pre-configuring a consumption rule, a business processing rule (including a distribution mode), a business storage rule (including a data storage rule) and an alarm rule;
the consumption rule comprises an acquisition path of consumption data in Kafka, specifically, a business party needs to determine a corresponding Cluster (Cluster), a server (Broker) on the Cluster, a theme (Topic) and a Partition (Partition) on the server, and when the parameters are determined, a complete path for the business party to acquire the consumption data from Kafka is obtained.
The service processing rule is mainly used for distinguishing which distribution mode is specifically used for realizing the distribution process of the consumption data, and specifically is used for utilizing a message flow writing mode or an interface calling mode, wherein the message flow writing mode refers to that a server sends the consumption data on Kafka to a service party according to a preset consumption data acquisition path, the interface calling mode can be that the service party sends a consumption data receiving request to the server based on an http protocol, and the server sends corresponding consumption data to the service party according to the preset consumption data acquisition path after receiving the request.
The service storage rule comprises the time for storing the consumption data in the database and the setting of the storage capacity of the consumption data in the database on the premise that the service processing rule is in the interface calling mode, so that the subsequent calling process of the consumption data based on the interface calling mode is realized.
The alarm rule includes: firstly, in a message flow writing mode, after current limiting, the writing of consumption data fails, at the moment, an alarm is triggered, and alarm information is sent to a corresponding first business party; in the interface calling mode, when the data of the consumption data is stored, because the storage is overloaded or the interface calling address identification fails, an alarm is triggered, and the alarm information is sent to a corresponding second service party;
(2) performing a consumption data acquisition process;
the server consumes the data by reading the kafka data stream and sends the consumption data to the business side to serve the function of serving the consumer (the business side is actually the final consumer). To implement the service for the business party, firstly, a consumption rule needs to be configured for the business party, and the acquisition path of the consumption data in Kafka can be obtained according to the consumption rule. And if a certain service party does not configure the corresponding consumption rule, directly ending the process. And if the consumption rule is configured, further acquiring a service processing rule, wherein the service processing rule comprises a message flow writing mode or an interface calling mode.
When the consumption data is acquired by adopting a message stream writing mode, determining the specific position of the consumption data in Kafka required by a first service party based on the ID of the first service party, sequentially comprising a Cluster (Cluster), a server (Broker) on the Cluster, a theme (Topic) and a Partition (Partition) on the server, constructing a consumption data acquisition path based on the information, sending a request to a token bucket before the writing of the message stream is started, requesting to take away the token from the constructed token bucket, and if the token acquisition is successful, sending the consumption data to the first service party; and if the token acquisition fails, re-initiating a token acquisition request to the token bucket, and finally realizing the information flow writing process of the consumption data.
When the consumption data is acquired by adopting an interface calling mode, the second service party initiates a consumption data receiving request to the server based on the http protocol, and the server sends corresponding consumption data to the second service party according to a consumption data acquisition path configured in advance after receiving the request. The service data (consumption data) can be obtained in advance according to the configured consumption data obtaining path and stored in the database, and the server obtains the corresponding service data from the database and outputs the service data to the second service party.
In order to ensure the stable operation of the consumption data acquisition process and improve the operation reliability of the process, service processing monitoring configuration and alarm configuration are performed in a server, and the method specifically comprises the following steps: the service processing monitoring configuration is used for monitoring the running condition of the process in the process, when the abnormal running of the process is monitored, the process protection is triggered, and the process protection has two functions: firstly, when the main process is monitored to be abnormal, triggering a backup process, and continuing to perform unfinished processes based on the backup process; and secondly, restarting one sub-process when the sub-process is monitored to be abnormal.
The alarm configuration mainly aims at giving alarm reminding when abnormal conditions occur in the consumption data acquisition process, so that a user can react in time and eliminate the abnormal conditions conveniently, and the abnormal conditions for which the alarm reminding is given mainly include two types, specifically: firstly, in a message flow writing mode, after current limiting, the writing of consumption data fails, at the moment, an alarm is triggered, and alarm information is sent to a corresponding first business party; and secondly, in the interface calling mode, in the process of storing the data of the consumption data, because of storage overload or failure of interface calling address identification, triggering alarm and sending alarm information to a corresponding second service party.
In fig. 2, solid lines are used to indicate the real-time processing. The dotted lines represent the manner of asynchronous processing, such as snooping. For example, in fig. 2, reading kafka data requires monitoring the state of the kafka cluster, monitoring the state of the database and the state of the message data stream to perform service processing process monitoring, and the service party may also obtain consumption data by monitoring the state of the message data stream.
According to the consumption data processing method provided by the invention, through configuration of the consumption rule, a specific path for the business party to acquire the consumption data from the third-party platform is determined, and the business party can accurately acquire the consumption data in the third-party platform; the configuration of the distribution mode enables the business side to have the capacity of actively requesting and passively receiving the consumption data, thereby enriching the data acquisition mode of the business side; based on the configuration of the consumption data processing process, repeated development of codes of all business parties in the consumption data acquisition process is avoided, the consumption data acquisition process is simplified, and the efficiency of the business parties for acquiring the consumption data is improved.
Fig. 3 is a schematic structural diagram of a consumption data processing apparatus provided by the present invention, and as shown in fig. 3, the consumption data processing apparatus 300 includes a configuration information reading module 310 and a consumption data distributing module 320, where:
a configuration information reading module 310, configured to read configuration information; the configuration information comprises consumption information source configuration information and distribution mode configuration information, the consumption information source configuration information comprises an acquisition path of consumption data on a third-party platform, and the distribution mode configuration information comprises a message flow writing mode and/or an interface calling mode; the configuration information corresponds to the service parties one by one;
the consumption data distribution module 320 is configured to acquire the distribution mode configuration information as at least one first service party of the message stream writing mode, acquire corresponding first consumption data according to an acquisition path of the consumption data on a third-party platform, and send the first consumption data to the corresponding first service party according to a preset consumption rule; and/or acquiring at least one second service party of which the distribution mode configuration information is the interface calling mode, acquiring corresponding second consumption data according to an acquisition path of the consumption data on a third-party platform, receiving an interface access request sent by the second service party and used for acquiring the second consumption data, and sending the corresponding second consumption data to the second service party.
According to the consumption data processing device provided by the invention, through the configuration of the consumption rule, the specific path of the business party for acquiring the consumption data from the third-party platform is determined, and the business party can accurately acquire the consumption data in the third-party platform; the configuration of the distribution mode enables the business side to have the capacity of actively requesting and passively receiving the consumption data, thereby enriching the data acquisition mode of the business side; based on the configuration of the consumption data processing process, repeated development of codes of all business parties in the consumption data acquisition process is avoided, the consumption data acquisition process is simplified, and the efficiency of the business parties for acquiring the consumption data is improved.
According to the consumption data processing device provided by the invention, the configuration information further comprises process management configuration information, and the process management configuration information comprises at least one of process starting configuration information, process monitoring configuration information and process protection configuration information; the process starting configuration information comprises setting starting conditions for a main process and a sub process; the process protection configuration information comprises setting a protection process for the main process and/or configuring a new sub-process to be restarted after the sub-process is monitored to be abnormal; the process monitoring configuration information comprises the operation conditions of a main process and a sub-process, and process protection is triggered when the main process and/or the sub-process are monitored to be abnormal in operation;
according to the consumption data processing device provided by the invention, the process starting configuration information, the process monitoring configuration information and the process protection configuration information are carried out in the database in advance, so that when an abnormality occurs in the process of acquiring the consumption data from Kafka by a business party, automatic processing can be carried out in time, the normal operation of the whole consumption data acquisition process is ensured, and the reliability of data acquisition is improved.
According to the consumption data processing device provided by the invention, the sub-process corresponds to the first service party or the second service party, and the sub-process is used for acquiring the corresponding first consumption data according to the acquisition path of the consumption data on a third-party platform or acquiring the corresponding second consumption data according to the acquisition path of the consumption data on the third-party platform.
According to the consumption data processing device provided by the invention, the consumption data processing process is divided into different sub-processes for respective processing according to different acquisition modes of consumption data, so that respective management of the sub-processes is favorably realized, the process management efficiency is improved, and the influence between the sub-processes is avoided.
According to the consumption data processing device provided by the invention, the main process utilizes task table data as a data source for pulling up the subprocess and realizes the monitoring of the running condition of the subprocess; the task table data comprises a process number of the sub-process, a service identifier and an operation state of the sub-process, wherein the service identifier is an identifier of the first service party or the second service party.
According to the consumption data processing device, the related information of the subprocess is stored by using the task table data, so that the starting and monitoring of the subprocess are quickly controlled.
According to the consumption data processing device provided by the invention, the configuration information further includes current limit configuration information, and the current limit configuration information is used for controlling the writing speed when the distribution mode configuration information is the message stream writing mode.
According to the consumption data processing device provided by the invention, the control on the writing speed of the message stream is realized by the current limiting configuration in the database in the message stream writing mode, the acquisition pressure of a business party on the consumption data is effectively relieved, and the user experience is improved.
According to a consumption data processing apparatus provided by the present invention, the current limit configuration information comprises a speed at which tokens are configured to be delivered to a token bucket; when the consumption data distribution module 320 is configured to execute the sending of the first consumption data to the corresponding first service party according to the preset consumption rule, specifically, the consumption data distribution module is configured to: and requesting to take the token away from the token bucket, if the token is taken, sending the first consumption data to the corresponding first business party according to a preset consumption rule, and if the token is not taken, turning to the request to take the token away from the token bucket.
According to the consumption data processing device, the token bucket is constructed based on Redis, and whether the token is acquired or not is used as the corresponding condition for the first business party to receive the consumption data, so that the current limiting effect on the consumption data acquisition process under the message flow writing condition is realized, and the data acquisition pressure of the first business party is effectively relieved.
According to a consumption data processing apparatus provided by the present invention, the configuration information further includes: when the distribution mode configuration information is the interface calling mode, configuring a storage rule of the consumption data in a database; the storage rule comprises a data storage time and/or an upper limit of data storage; and/or the configuration information further includes service retry configuration information, where the service retry configuration information includes information for configuring the second service party to reinitiate the interface access request after the interface access request fails.
The consumption data processing device provided by the invention pre-configures the storage time and the storage upper limit in the storage process of the database in a mode that the distribution mode is interface calling, so that the whole database is in a dynamic balance, and the stable operation of the database is realized; meanwhile, the addition of the service retry configuration information can effectively improve the success rate of the interface access request and improve the reliability of the consumption data processing process.
According to a consumption data processing apparatus provided by the present invention, the apparatus further comprises a retry module, the retry module is configured to: when the service consumption is in a problem, re-consumption is carried out according to the time node and/or the offset of the consumption data stored in the database;
and/or when the local storage data is lost, automatically and synchronously acquiring the data from the offset of the kafka data source node according to the consumption information source configuration information.
According to the consumption data processing device and the consumption data processing method provided by the invention, when the service consumption is in a problem, a secondary service consumption process can be carried out in time through a service consumption retry mechanism, so that the stability and reliability of the data consumption process are ensured; meanwhile, when the local storage data is lost through the automatic supplement process of the consumption data, the consumption data can be automatically supplemented in time, and the stable proceeding of the data consumption process is ensured.
According to the consumption data processing device provided by the invention, the configuration information further comprises alarm rule configuration information; the device also comprises an alarm module, wherein the alarm module is used for: and performing alarm processing according to the alarm rule configuration information, and sending alarm information to the corresponding first service party or the second service party.
The consumption data processing device provided by the invention has the advantages that the alarm rules are configured in the database, different alarm rule settings are carried out according to different distribution modes, and the alarm information is transmitted to the business party, so that the current state can be quickly clarified when the business party has large abnormity in the process of acquiring the consumption data, the interaction between the business party and the system is enhanced, and the normal operation of the whole system is ensured.
Fig. 4 is a schematic structural diagram of an electronic device provided in the present invention, and as shown in fig. 4, the electronic device may include: a processor (processor)410, a communication Interface 420, a memory (memory)430 and a communication bus 440, wherein the processor 410, the communication Interface 420 and the memory 430 are communicated with each other via the communication bus 440. Processor 410 may invoke logic instructions in memory 430 to perform a method of consuming data processing, the method comprising: reading configuration information; the configuration information comprises consumption information source configuration information and distribution mode configuration information, the consumption information source configuration information comprises an acquisition path of consumption data on a third-party platform, and the distribution mode configuration information comprises a message flow writing mode and/or an interface calling mode; the configuration information corresponds to the service parties one by one;
acquiring the distribution mode configuration information as at least one first service party of the message flow writing mode, acquiring corresponding first consumption data according to an acquisition path of the consumption data on a third party platform, and sending the first consumption data to the corresponding first service party according to a preset consumption rule; and/or acquiring at least one second service party of which the distribution mode configuration information is the interface calling mode, acquiring corresponding second consumption data according to an acquisition path of the consumption data on a third-party platform, receiving an interface access request sent by the second service party and used for acquiring the second consumption data, and sending the corresponding second consumption data to the second service party.
In addition, the logic instructions in the memory 430 may be implemented in the form of software functional units and stored in a computer readable storage medium when the software functional units are sold or used as independent products. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
In another aspect, the present invention also provides a computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions which, when executed by a computer, enable the computer to perform the consumption data processing method provided by the above methods, the method comprising: reading configuration information; the configuration information comprises consumption information source configuration information and distribution mode configuration information, the consumption information source configuration information comprises an acquisition path of consumption data on a third-party platform, and the distribution mode configuration information comprises a message flow writing mode and/or an interface calling mode; the configuration information corresponds to the service parties one by one;
acquiring the distribution mode configuration information as at least one first service party of the message flow writing mode, acquiring corresponding first consumption data according to an acquisition path of the consumption data on a third party platform, and sending the first consumption data to the corresponding first service party according to a preset consumption rule; and/or acquiring at least one second service party of which the distribution mode configuration information is the interface calling mode, acquiring corresponding second consumption data according to an acquisition path of the consumption data on a third-party platform, receiving an interface access request sent by the second service party and used for acquiring the second consumption data, and sending the corresponding second consumption data to the second service party.
In yet another aspect, the present invention also provides a non-transitory computer-readable storage medium having stored thereon a computer program, which when executed by a processor is implemented to perform the consumption data processing method provided above, the method comprising: reading configuration information; the configuration information comprises consumption information source configuration information and distribution mode configuration information, the consumption information source configuration information comprises an acquisition path of consumption data on a third-party platform, and the distribution mode configuration information comprises a message flow writing mode and/or an interface calling mode; the configuration information corresponds to the service parties one by one;
acquiring the distribution mode configuration information as at least one first service party of the message flow writing mode, acquiring corresponding first consumption data according to an acquisition path of the consumption data on a third party platform, and sending the first consumption data to the corresponding first service party according to a preset consumption rule; and/or acquiring at least one second service party of which the distribution mode configuration information is the interface calling mode, acquiring corresponding second consumption data according to an acquisition path of the consumption data on a third-party platform, receiving an interface access request sent by the second service party and used for acquiring the second consumption data, and sending the corresponding second consumption data to the second service party.
The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. With this understanding in mind, the above-described technical solutions may be embodied in the form of a software product, which can be stored in a computer-readable storage medium such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in the embodiments or some parts of the embodiments.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (10)
1. A consumption data processing method, comprising:
reading configuration information; the configuration information comprises consumption information source configuration information and distribution mode configuration information, the consumption information source configuration information comprises an acquisition path of consumption data on a third-party platform, and the distribution mode configuration information comprises a message flow writing mode and/or an interface calling mode; the configuration information corresponds to the service parties one by one;
acquiring the distribution mode configuration information as at least one first service party of the message flow writing mode, acquiring corresponding first consumption data according to an acquisition path of the consumption data on a third party platform, and sending the first consumption data to the corresponding first service party according to a preset consumption rule; and/or acquiring at least one second service party of which the distribution mode configuration information is the interface calling mode, acquiring corresponding second consumption data according to an acquisition path of the consumption data on a third-party platform, receiving an interface access request sent by the second service party and used for acquiring the second consumption data, and sending the corresponding second consumption data to the second service party.
2. The consumption data processing method of claim 1, wherein the configuration information further comprises process management configuration information, the process management configuration information comprising at least one of process start configuration information, process monitor configuration information, and process protection configuration information; the process starting configuration information comprises setting starting conditions for a main process and a sub process; the process protection configuration information comprises setting a protection process for the main process and/or configuring a new sub-process to be restarted after the sub-process is monitored to be abnormal; the process monitoring configuration information comprises the operation conditions of a main process and a sub-process, and process protection is triggered when the main process and/or the sub-process are monitored to be abnormal in operation;
the method further comprises the following steps: and carrying out process management according to the process management configuration information.
3. The consumption data processing method according to claim 2, wherein the sub-process corresponds to the first service party or the second service party, and the sub-process is configured to obtain the corresponding first consumption data according to an obtaining path of the consumption data on a third-party platform or obtain the corresponding second consumption data according to an obtaining path of the consumption data on a third-party platform.
4. The consumption data processing method of claim 3, wherein the method further comprises:
the main process uses the task table data as a data source for pulling up the subprocess and realizes the monitoring of the running condition of the subprocess; the task table data comprises a process number of the sub-process, a service identifier and an operation state of the sub-process, wherein the service identifier is an identifier of the first service party or the second service party.
5. The consumption data processing method of claim 2, wherein the configuration information further includes current limiting configuration information, and the current limiting configuration information is used to control a writing speed when the distribution mode configuration information is the message stream writing mode.
6. The consumption data processing method of claim 5, wherein the throttling configuration information comprises configuring a speed of token delivery to a token bucket;
the sending the first consumption data to the corresponding first service party according to a preset consumption rule specifically includes:
and requesting to take the token away from the token bucket, if the token is taken, sending the first consumption data to the corresponding first business party according to a preset consumption rule, and if the token is not taken, turning to the request to take the token away from the token bucket.
7. The consumption data processing method of claim 1, wherein the configuration information further comprises:
when the distribution mode configuration information is the interface calling mode, configuring a storage rule of the consumption data in a database; the storage rule comprises a data storage time and/or an upper limit of data storage;
and/or the configuration information further includes service retry configuration information, where the service retry configuration information includes information for configuring the second service party to reinitiate the interface access request after the interface access request fails.
8. The consumption data processing method of claim 7, wherein the method further comprises: when the service consumption is in a problem, re-consumption is carried out according to the time node and/or the offset of the consumption data stored in the database;
and/or when the local storage data is lost, automatically and synchronously acquiring the data from the offset of the kafka data source node according to the consumption information source configuration information.
9. The consumption data processing method of claim 1, wherein the configuration information further comprises alarm rule configuration information;
the method further comprises the following steps: and performing alarm processing according to the alarm rule configuration information, and sending alarm information to the corresponding first service party or the second service party.
10. A non-transitory computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the consumption data processing method according to any one of claims 1 to 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111028477.7A CN113946362B (en) | 2021-09-02 | 2021-09-02 | Consumption data processing method and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111028477.7A CN113946362B (en) | 2021-09-02 | 2021-09-02 | Consumption data processing method and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113946362A true CN113946362A (en) | 2022-01-18 |
| CN113946362B CN113946362B (en) | 2022-08-16 |
Family
ID=79327837
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111028477.7A Active CN113946362B (en) | 2021-09-02 | 2021-09-02 | Consumption data processing method and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113946362B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115208834A (en) * | 2022-07-12 | 2022-10-18 | 武汉众邦银行股份有限公司 | Service flow limiting method based on database storage process design |
| CN117453665A (en) * | 2023-10-09 | 2024-01-26 | 行吟信息科技（上海）有限公司 | Data processing method, device, equipment and storage medium |
Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101021801A (en) * | 2006-11-30 | 2007-08-22 | 南京联创科技股份有限公司 | Mass data transmitting method inter-pipeline multiprocess based on information queue |
| CN102082797A (en) * | 2011-01-21 | 2011-06-01 | 中兴通讯股份有限公司 | Method and device for carrying out protocol analysis processing on data streams |
| CN105100250A (en) * | 2015-07-31 | 2015-11-25 | 北京奇虎科技有限公司 | Message distribution method, message distribution server and message distribution system |
| CN105808365A (en) * | 2016-03-22 | 2016-07-27 | 深圳市宁远科技股份有限公司 | HIS (Hospital Information System) interface calling method and HIS interface configuration method |
| CN106201616A (en) * | 2016-07-12 | 2016-12-07 | 中国电子科技集团公司第二十八研究所 | A kind of assembly asynchronous information distribution method of flexibility |
| CN109388446A (en) * | 2017-08-07 | 2019-02-26 | 腾讯科技（北京）有限公司 | A kind of information processing method, device and storage medium |
| CN109672605A (en) * | 2018-09-26 | 2019-04-23 | 深圳壹账通智能科技有限公司 | Mail distribution method, apparatus, equipment and readable storage medium storing program for executing |
| US20190370288A1 (en) * | 2016-02-05 | 2019-12-05 | Sas Institute Inc. | Handling of data sets during execution of task routines of multiple languages |
| CN111949422A (en) * | 2020-08-26 | 2020-11-17 | 南京云灿信息科技有限公司 | Data multi-level caching and high-speed transmission recording method based on MQ and asynchronous IO |
| CN112346794A (en) * | 2020-10-21 | 2021-02-09 | 深圳鼎盛电脑科技有限公司 | Interface calling method, device, equipment and medium |
| CN112528201A (en) * | 2020-12-17 | 2021-03-19 | 平安消费金融有限公司 | Method and device for calling third-party platform, computer equipment and storage medium |
| CN112882846A (en) * | 2021-02-19 | 2021-06-01 | 深圳市云网万店科技有限公司 | Data processing method and device of message queue, computer equipment and storage medium |
| CN113220759A (en) * | 2021-04-27 | 2021-08-06 | 深圳市云网万店科技有限公司 | Big data storage service sharing method and device, computer equipment and storage medium |
| CN113296960A (en) * | 2021-06-29 | 2021-08-24 | 中国平安人寿保险股份有限公司 | Data quasi-real-time distribution method, device, equipment and storage medium |
-
2021
- 2021-09-02 CN CN202111028477.7A patent/CN113946362B/en active Active
Patent Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101021801A (en) * | 2006-11-30 | 2007-08-22 | 南京联创科技股份有限公司 | Mass data transmitting method inter-pipeline multiprocess based on information queue |
| CN102082797A (en) * | 2011-01-21 | 2011-06-01 | 中兴通讯股份有限公司 | Method and device for carrying out protocol analysis processing on data streams |
| CN105100250A (en) * | 2015-07-31 | 2015-11-25 | 北京奇虎科技有限公司 | Message distribution method, message distribution server and message distribution system |
| US20190370288A1 (en) * | 2016-02-05 | 2019-12-05 | Sas Institute Inc. | Handling of data sets during execution of task routines of multiple languages |
| CN105808365A (en) * | 2016-03-22 | 2016-07-27 | 深圳市宁远科技股份有限公司 | HIS (Hospital Information System) interface calling method and HIS interface configuration method |
| CN106201616A (en) * | 2016-07-12 | 2016-12-07 | 中国电子科技集团公司第二十八研究所 | A kind of assembly asynchronous information distribution method of flexibility |
| CN109388446A (en) * | 2017-08-07 | 2019-02-26 | 腾讯科技（北京）有限公司 | A kind of information processing method, device and storage medium |
| CN109672605A (en) * | 2018-09-26 | 2019-04-23 | 深圳壹账通智能科技有限公司 | Mail distribution method, apparatus, equipment and readable storage medium storing program for executing |
| CN111949422A (en) * | 2020-08-26 | 2020-11-17 | 南京云灿信息科技有限公司 | Data multi-level caching and high-speed transmission recording method based on MQ and asynchronous IO |
| CN112346794A (en) * | 2020-10-21 | 2021-02-09 | 深圳鼎盛电脑科技有限公司 | Interface calling method, device, equipment and medium |
| CN112528201A (en) * | 2020-12-17 | 2021-03-19 | 平安消费金融有限公司 | Method and device for calling third-party platform, computer equipment and storage medium |
| CN112882846A (en) * | 2021-02-19 | 2021-06-01 | 深圳市云网万店科技有限公司 | Data processing method and device of message queue, computer equipment and storage medium |
| CN113220759A (en) * | 2021-04-27 | 2021-08-06 | 深圳市云网万店科技有限公司 | Big data storage service sharing method and device, computer equipment and storage medium |
| CN113296960A (en) * | 2021-06-29 | 2021-08-24 | 中国平安人寿保险股份有限公司 | Data quasi-real-time distribution method, device, equipment and storage medium |
Non-Patent Citations (4)
| Title |
|---|
| ZHANGBOHUN: "Kafka学习笔记总结", 《 HTTPS://BLOG.CSDN.NET/ZHANGBOHUN/ARTICLE/DETAILS/86548317》 * |
| 佚名: "高速动车组 带出一个全新的管理平台", 《企业管理》 * |
| 赵堃: "基于中间件技术的机场信息集成***设计与实现", 《中国优秀硕士学位论文全文数据库 (信息科技辑)》 * |
| 郭海川: "分布式流式计算***中消息传递中间件的设计与实现", 《中国优秀硕士学位论文全文数据库 (信息科技辑)》 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115208834A (en) * | 2022-07-12 | 2022-10-18 | 武汉众邦银行股份有限公司 | Service flow limiting method based on database storage process design |
| CN117453665A (en) * | 2023-10-09 | 2024-01-26 | 行吟信息科技（上海）有限公司 | Data processing method, device, equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113946362B (en) | 2022-08-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113946362B (en) | Consumption data processing method and storage medium | |
| US9729488B2 (en) | On-demand mailbox synchronization and migration system | |
| US7886295B2 (en) | Connection manager, method, system and program product for centrally managing computer applications | |
| US20160055045A1 (en) | Method and arrangement for fault management in infrastructure as a service clouds | |
| CN109766172B (en) | Asynchronous task scheduling method and device | |
| JP2006187438A (en) | System for hall management | |
| KR101416280B1 (en) | Event handling system and method | |
| CN107018159B (en) | Service request processing method and device, and service request method and device | |
| CN112559208A (en) | Method for constructing micro-service MQ applied to government affair cloud platform | |
| CN112333249A (en) | Business service system and method | |
| CN108243222A (en) | Server network architecture method and device | |
| CN114531373A (en) | Node state detection method, node state detection device, equipment and medium | |
| CN112559461A (en) | File transmission method and device, storage medium and electronic equipment | |
| CN114302350A (en) | Service provider fault switching method and device, electronic equipment and storage medium | |
| CN113010379B (en) | Electronic equipment monitoring system | |
| CN107426012B (en) | Fault recovery method and device based on super-fusion architecture | |
| CN111628903B (en) | Monitoring method and monitoring system for transaction system running state | |
| CN111835809B (en) | Work order message distribution method, work order message distribution device, server and storage medium | |
| CN116455830A (en) | Method for realizing high-availability distributed QOS of storage gateway | |
| CN116346834A (en) | Session synchronization method, device, computing equipment and computer storage medium | |
| CN109697126B (en) | Data processing method and device for server | |
| CN111913732A (en) | Service updating method and device, management server and storage medium | |
| US20110289165A1 (en) | Method, apparatus and computer program for message handling | |
| CN114546766A (en) | Data disk-dropping method, system and device based on Monitor and storage medium | |
| CN110633164B (en) | Message-oriented middleware fault recovery method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20220308 Address after: 100085 Floor 101 102-1, No. 35 Building, No. 2 Hospital, Xierqi West Road, Haidian District, Beijing Applicant after: Seashell Housing (Beijing) Technology Co.,Ltd. Address before: 101309 room 24, 62 Farm Road, Erjie village, Yangzhen, Shunyi District, Beijing Applicant before: Beijing fangjianghu Technology Co.,Ltd. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |