CN115617763A - Data processing method and device, electronic equipment and storage medium - Google Patents

Data processing method and device, electronic equipment and storage medium Download PDF

Info

Publication number
CN115617763A
CN115617763A CN202211169788.XA CN202211169788A CN115617763A CN 115617763 A CN115617763 A CN 115617763A CN 202211169788 A CN202211169788 A CN 202211169788A CN 115617763 A CN115617763 A CN 115617763A
Authority
CN
China
Prior art keywords
service
file system
distributed file
processed
data
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202211169788.XA
Other languages
Chinese (zh)
Inventor
杨平文
冯辉
黄昌学
冯鹏
杨广波
李勇
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Zhongdian Jinxin Software Co Ltd
Original Assignee
Zhongdian Jinxin Software Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Zhongdian Jinxin Software Co Ltd filed Critical Zhongdian Jinxin Software Co Ltd
Priority to CN202211169788.XA priority Critical patent/CN115617763A/en
Publication of CN115617763A publication Critical patent/CN115617763A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/17Details of further file system functions
    • G06F16/172Caching, prefetching or hoarding of files
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/18File system types
    • G06F16/182Distributed file systems

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The embodiment of the application provides a data processing method and device, electronic equipment and a storage medium. The method comprises the following steps: carrying out node division processing on the distributed file system to generate N distributed file system nodes, wherein N is a positive integer greater than or equal to 1; establishing a mapping relation between the key words of the service type corresponding to each distributed file system node and the corresponding distributed file system node; and storing the mapping relation into a cache database, so that when the service is processed under the condition of abnormal database, the distributed file system node corresponding to the service is determined according to the mapping relation, and the service data corresponding to the service is stored into the distributed file system node. According to the embodiment of the application, the normal operation of the front-end system can be ensured under the condition that the database is abnormal.

Description

Data processing method and device, electronic equipment and storage medium
Technical Field
The present application relates to the field of data processing technologies, and in particular, to a data processing method and apparatus, an electronic device, and a storage medium.
Background
With the development of the internet and the continuous change of user requirements, various application services are more and more complex, and the processing of the application services cannot be supported by a database.
The pre-system is a system in an Enterprise as an interface between an Enterprise ERP (Enterprise Resource Planning) and a platform, and is used for connecting the Enterprise ERP and the platform to perform data transmission and conversion. At present, a database is required to be available for normal starting and operation of a front-end system, and under the background of the information technology application innovation industry, the situation that the database is unavailable may exist after a core front-end system is matched with the database, and once the database is unavailable, the front-end system cannot normally operate.
Disclosure of Invention
The embodiment of the application provides a data processing method and device, electronic equipment and a storage medium, so that the normal operation of a front-end system is ensured under the condition that a database is abnormal.
In order to solve the above technical problem, the embodiment of the present application is implemented as follows:
in a first aspect, an embodiment of the present application provides a data processing method, including:
carrying out node division processing on the distributed file system to generate N distributed file system nodes, wherein N is a positive integer greater than or equal to 1;
establishing a mapping relation between the keywords of the service type corresponding to each distributed file system node and the corresponding distributed file system node;
and storing the mapping relation into a cache database, so that when the service is processed under the condition of abnormal database, the distributed file system node corresponding to the service is determined according to the mapping relation, and the service data corresponding to the service is stored into the distributed file system node.
Optionally, after the storing the mapping relationship in the cache database, the method further includes:
in response to the pre-application starting, determining whether the database is abnormal;
determining a target distributed file system node corresponding to the service to be processed according to the mapping relation cached in the cache database under the condition that the database is determined to be abnormal and the service to be processed sent by the front-end application program is received;
and acquiring service data corresponding to the service to be processed, and storing the service data to the target distributed file system node.
Optionally, the determining, according to the mapping relationship of the cache in the cache database, a target distributed file system node corresponding to the service to be processed includes:
acquiring a service keyword of the service to be processed;
and acquiring a target distributed file system node corresponding to the business keyword according to the mapping relation between the keyword stored in the cache database and the distributed file system node.
Optionally, the acquiring the service data corresponding to the service to be processed includes:
acquiring basic service data corresponding to the service to be processed;
acquiring service flow data generated in the process of processing the service to be processed;
and taking the basic service data and the service flow data as service data of the service to be processed.
Optionally, after receiving the to-be-processed service sent by the front-end application, the method further includes:
and adding the service to be processed into a message queue so as to write the service to be processed in the message queue into the distributed file system when a writing condition is met.
Optionally, after the storing the service data to the target distributed file system node, the method further includes:
determining whether the database is restored to a normal state;
responding to the database to be recovered to a normal state, and acquiring the service data of the service to be processed from the target distributed file system node;
and storing the service data into the database.
Optionally, the method further comprises:
acquiring a data query request sent by a user;
acquiring distributed file system nodes to be queried corresponding to query keywords contained in the data query request according to the mapping relation cached in the cache database;
inquiring to obtain a file path of the service data to be inquired according to the distributed file system node to be inquired;
acquiring a target file corresponding to the service data to be inquired according to the file path;
and analyzing the target file to obtain the service data to be inquired.
In a second aspect, an embodiment of the present application provides a data processing apparatus, including:
the node dividing module is used for carrying out node dividing processing on the distributed file system to generate N distributed file system nodes, wherein N is a positive integer greater than or equal to 1;
the mapping relation establishing module is used for establishing a mapping relation between the key words of the service types corresponding to each distributed file system node and the corresponding distributed file system node;
and the mapping relation storage module is used for storing the mapping relation into a cache database, so that when the service is processed under the abnormal condition of the database, the distributed file system node corresponding to the service is determined according to the mapping relation, and the service data corresponding to the service is stored to the distributed file system node.
Optionally, the apparatus further comprises:
the database exception determining module is used for responding to the starting of the front application program and determining whether the database is abnormal or not;
the target node determining module is used for determining a target distributed file system node corresponding to the service to be processed according to the mapping relation cached in the cache database under the condition that the database is determined to be abnormal and the service to be processed sent by the pre-application program is received;
and the first service data storage module is used for acquiring service data corresponding to the service to be processed and storing the service data to the target distributed file system node.
Optionally, the target node determining module includes:
a service keyword obtaining unit, configured to obtain a service keyword of the service to be processed;
and the target node acquisition unit is used for acquiring the target distributed file system node corresponding to the service keyword according to the mapping relation between the keyword stored in the cache database and the distributed file system node.
Optionally, the service data saving module includes:
a basic service data acquiring unit, configured to acquire basic service data corresponding to the service to be processed;
a service flow data acquisition unit, configured to acquire service flow data generated in a process of processing the service to be processed;
and the service data acquisition unit is used for taking the basic service data and the service flow data as service data of the service to be processed.
Optionally, the apparatus further comprises:
and the service to be processed writing module is used for adding the service to be processed into a message queue so as to write the service to be processed in the message queue into the distributed file system when the writing condition is met.
Optionally, the apparatus further comprises:
the database determining module is used for determining whether the database is recovered to a normal state;
a service data acquisition module, configured to respond to the database being restored to a normal state, and acquire service data of the service to be processed from the target distributed file system node;
and the second service data storage module is used for storing the service data into the database.
Optionally, the apparatus further comprises:
the data query request acquisition module is used for acquiring a data query request sent by a user;
the node to be queried acquiring module is used for acquiring a distributed file system node to be queried corresponding to a query keyword contained in the data query request according to the mapping relation cached in the cache database;
the file path query module is used for querying a file path of the service data to be queried according to the distributed file system node to be queried;
the target file acquisition module is used for acquiring a target file corresponding to the service data to be inquired according to the file path;
and the data to be queried acquisition module is used for analyzing the target file to obtain the service data to be queried.
In a third aspect, an embodiment of the present application provides an electronic device, including:
a memory, a processor and a computer program stored on the memory and executable on the processor, the computer program, when executed by the processor, implementing the data processing method of any of the above.
In a fourth aspect, the present application provides a readable storage medium, and when executed by a processor of an electronic device, the instructions in the storage medium enable the electronic device to perform any one of the data processing methods described above.
In the embodiment of the application, node division processing is performed on a distributed file system to generate N distributed file system nodes, wherein N is a positive integer greater than or equal to 1, a mapping relation between a keyword of a service type corresponding to each distributed file system node and the corresponding distributed file system node is established, the mapping relation is stored in a cache database, so that when service processing is performed under the abnormal condition of the database, the distributed file system node corresponding to the service is determined according to the mapping relation, and service data corresponding to the service is stored to the distributed file system node. According to the embodiment of the application, the mapping relation between the key words of the service type corresponding to each distributed file system node and the corresponding distributed file system nodes is stored in the cache data base, so that when the service is processed under the condition that the database is abnormal, the service data generated by processing the service can be stored to the corresponding distributed file system nodes according to the mapping relation, and the normal operation of the front-end system can not be influenced.
The above description is only an overview of the technical solutions of the present application, and the present application may be implemented in accordance with the content of the description so as to make the technical means of the present application more clearly understood, and the detailed description of the present application will be given below in order to make the above and other objects, features, and advantages of the present application more clearly understood.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required to be used in the description of the embodiments of the present application will be briefly described below, and it is obvious that the drawings in the description below are only some embodiments of the present application, and it is obvious for those skilled in the art that other drawings may be obtained according to the drawings without creative efforts.
Fig. 1 is a flowchart illustrating steps of a data processing method according to an embodiment of the present application;
fig. 2 is a flowchart illustrating steps of a method for storing service data according to an embodiment of the present application;
fig. 3 is a flowchart illustrating steps of a target node acquisition method according to an embodiment of the present disclosure;
fig. 4 is a flowchart illustrating steps of a service data acquiring method according to an embodiment of the present application;
fig. 5 is a flowchart illustrating steps of a pending service adding method according to an embodiment of the present application;
fig. 6 is a flowchart illustrating steps of another service data saving method according to an embodiment of the present application;
fig. 7 is a flowchart illustrating steps of a service data query method according to an embodiment of the present application;
FIG. 8 is a schematic diagram of a database-less operation process provided by an embodiment of the present application;
fig. 9 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present application;
fig. 10 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some, but not all, of the embodiments of the present application. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments in the present application without making any creative effort belong to the protection scope of the present application.
Referring to fig. 1, a flowchart illustrating steps of a data processing method provided in an embodiment of the present application is shown, and as shown in fig. 1, the data processing method may include: step 101, step 102 and step 103.
Step 101: and carrying out node division processing on the distributed file system to generate N distributed file system nodes, wherein N is a positive integer greater than or equal to 1.
The embodiment of the application can be applied to storing the service data of the processing service into the distributed file system when the database is abnormal so as not to influence the normal operation of the front-end system.
The embodiment of the application can be applied to a background system, namely, the execution subject is the background system. In this example, the backend system may be a server.
Distributed File System (DFS) means that physical storage resources managed by a File System are not necessarily directly connected to a local node, but are connected to a node through a computer network; or a complete hierarchical file system formed by combining several different logical disk partitions.
In a specific implementation, when a framework without database operation is constructed, node division processing may be performed on the distributed file system to generate N distributed file system nodes, where N is a positive integer greater than or equal to 1.
In this example, the number of the divided N distributed file system nodes may be determined by service requirements, and the present embodiment does not impose a limitation on the number of the divided N distributed file system nodes.
After the node division processing is performed on the distributed file system to generate N distributed file system nodes, step 102 is performed.
Step 102: and establishing a mapping relation between the keywords of the service type corresponding to each distributed file system node and the corresponding distributed file system node.
In this example, each distributed file system node may store service data of one service type, that is, each distributed file system node corresponds to one service type, for example, in the case of banking, the service types of banking services include: account opening type, withdrawal type, transfer type, deposit type, inquiry type, etc.
After the distributed file system is subjected to node division processing to generate N distributed file system nodes, a service type corresponding to each distributed file system node can be set, then keywords corresponding to the service type can be obtained, and a mapping relation between the keywords of the service type corresponding to each distributed file system node and the corresponding distributed file system nodes is established. For example, taking banking as an example, the types of business of banking business include: the account opening type, the withdrawal type, the transfer type, the deposit type and the query type, wherein the N distributed nodes are respectively as follows: the system comprises a node 1, a node 2, a node 3, a node 4 and a node 5, wherein the node 1 corresponds to an account opening type, the node 2 corresponds to a money withdrawing type, the node 3 corresponds to a money transferring type, the node 4 corresponds to a deposit type, and the node 5 corresponds to an inquiry type. The keywords corresponding to the account opening type, the withdrawal type, the transfer type, the deposit type and the query type are respectively as follows: opening an account, withdrawing money, transferring accounts, depositing accounts and inquiring, and at the moment, establishing a node 1 and keywords: and (3) opening a mapping relation between accounts, establishing a node 2 and a keyword: and (3) establishing a mapping relation between the node 3 and the keyword: and (3) establishing a mapping relation between the nodes 4 and the keywords: and (3) establishing a mapping relation between deposit, namely the node 5 and a keyword: mapping relationships between queries, etc.
It should be understood that the above examples are only examples for better understanding of the technical solutions of the embodiments of the present application, and are not to be taken as the only limitation to the embodiments.
After the mapping relationship between the keywords of the service type corresponding to each distributed file system node and the corresponding distributed file system node is established, step 103 is executed.
Step 103: and storing the mapping relation into a cache database, so that when the service is processed under the condition of abnormal database, the distributed file system node corresponding to the service is determined according to the mapping relation, and the service data corresponding to the service is stored into the distributed file system node.
The cache database is a database used for storing the mapping relation between the nodes of the distributed file system and the corresponding keywords. In this example, the cache database may be any of the Redis, memcached, mongodb, etc. databases.
After the mapping relationship between the keywords of the service type corresponding to each distributed file system node and the corresponding distributed file system node is established, the mapping relationship can be stored in a cache database. When the service is processed under the condition that the database is abnormal, the distributed file system node corresponding to the processed service can be determined according to the mapping relation, and the service data corresponding to the service is stored to the corresponding distributed file system node.
According to the embodiment of the application, the mapping relation between the key words of the service type corresponding to each distributed file system node and the corresponding distributed file system nodes is stored in the cache data base, so that when the service is processed under the condition that the database is abnormal, the service data generated by processing the service can be stored to the corresponding distributed file system nodes according to the mapping relation, and the condition that the preposed system cannot normally operate due to the abnormal database can be avoided.
After the environment without the database operation is established, when the service processing is performed, the service data of the processing service can be stored in the corresponding distributed file system node so as not to influence the normal operation of the front-end system. This implementation can be described in detail below in conjunction with fig. 2.
Referring to fig. 2, a flowchart illustrating steps of a service data saving method provided in an embodiment of the present application is shown, and as shown in fig. 2, the service data saving method may include: step 201, step 202 and step 203.
Step 201: in response to a pre-application launch, a determination is made as to whether the database is anomalous.
In this embodiment, the front-end Application refers to an APP (Application) installed in the front-end system in advance.
After the front-end application is started, whether the database is abnormal or not can be detected. In this example, a heartbeat detection mechanism may be used to detect whether the database is abnormal, specifically, by sending a heartbeat packet to the database, and not receiving a response message returned by the database within a set time period, it is determined that the database is in an abnormal state.
In a specific implementation, other manners may also be used to detect whether the database is abnormal, specifically, the manner for detecting the database abnormality may be determined according to a service requirement, and this embodiment does not limit this.
After detecting the database anomaly, step 202 is performed.
Step 202: and under the condition that the database is determined to be abnormal and the to-be-processed service sent by the front-end application program is received, determining a target distributed file system node corresponding to the to-be-processed service according to the mapping relation cached in the cache database.
The service to be processed refers to the service which is sent by the front-end application program and needs to be processed.
The target distributed file system node refers to a distributed file system node used for storing service data of the service to be processed.
And under the condition that the database is determined to be abnormal and the to-be-processed service sent by the front-end application program is received, determining a target distributed file system node corresponding to the to-be-processed service according to the mapping relation cached in the cache database. The implementation can be described in detail as follows in conjunction with fig. 3.
Referring to fig. 3, a flowchart illustrating steps of a target node acquisition method provided in an embodiment of the present application is shown, and as shown in fig. 3, the target node acquisition method may include: step 301, step 302 and step 303.
Step 301: and acquiring the service key words of the service to be processed.
In this embodiment, the service keyword refers to a keyword corresponding to a service to be processed.
After receiving the service to be processed, the service to be processed may be parsed to obtain a service keyword of the service to be processed, for example, taking banking service as an example, when the service to be processed is an account opening service, the service keyword of the service to be processed is an "account opening". And when the service to be processed is the transfer service, the service key word of the service to be processed is 'transfer', and the like.
After the service keyword of the service to be processed is obtained, step 302 is executed.
Step 302: and acquiring a target distributed file system node corresponding to the service keyword according to the mapping relation between the keyword stored in the cache database and the distributed file system node.
After the service key words of the service to be processed are obtained, the target distributed file system nodes corresponding to the service key words can be obtained according to the mapping relation between the key words and the distributed file system nodes stored in the cache database. For example, a distributed file system node includes: the method comprises a node 1, a node 2 and a node 3, wherein the mapping relation stored in the cache database is as follows: the keyword corresponding to the node 1 is "account opening", the keyword corresponding to the node 2 is "deposit", the keyword corresponding to the node 3 is "withdrawal", and when the service keyword of the service to be processed is "deposit", the node 2 can be regarded as a target distributed file system node. When the service keyword of the service to be processed is "account opening", the node 1 may be regarded as a target distributed file system node, and the like.
It should be understood that the above examples are only examples for better understanding of the technical solutions of the embodiments of the present application, and are not intended to limit the present embodiment.
After determining the target distributed file system node corresponding to the task to be processed according to the mapping relationship cached in the cache database, step 203 is executed.
Step 203: and acquiring service data corresponding to the service to be processed, and storing the service data to the target distributed file system node.
After determining the target distributed file system node corresponding to the task to be processed according to the mapping relation cached in the cache database, service data corresponding to the service to be processed can be obtained, and the obtained service data is stored to the target distributed file system node. In this example, the service data of the service to be processed may include: the basic service data and the service flow data, as for the implementation process of obtaining the service data to be processed, may be described in detail as follows with reference to fig. 4.
Referring to fig. 4, a flowchart illustrating steps of a service data obtaining method provided in an embodiment of the present application is shown, and as shown in fig. 4, the service data obtaining method may include: step 401, step 402 and step 403.
Step 401: and acquiring basic service data corresponding to the service to be processed.
In this embodiment, the basic service data refers to data that characterizes service features and/or attributes of a service to be processed, for example, a deposit service is taken as an example of the service to be processed, and the basic service data of the service to be processed may include: user name, gender, age, bank card number, deposit amount, etc.
After receiving the service to be processed sent by the pre-application program, the service to be processed may be analyzed to obtain a data field included in the service to be processed, so as to obtain basic service data corresponding to the service to be processed.
Step 402: and acquiring service flow data generated in the process of processing the service to be processed.
The business process data refers to data generated in the processing process of the business to be processed. For example, the pending business is exemplified by banking business, and the business process data may be request data, response data, log data, processing time, and the like.
In the process of processing the service to be processed, the service flow data generated in the process of processing the service to be processed can be acquired.
After the service basic data and the service flow data corresponding to the service to be processed are obtained, step 403 is executed.
Step 403: and taking the basic service data and the service flow data as service data of the service to be processed.
After the service basic data and the service flow data corresponding to the service to be processed are obtained, the basic service data and the service flow data of the service to be processed can be jointly used as the service data of the service to be processed for storage.
After the service data of the service to be processed is obtained, the service data of the service to be processed can be stored to the target distributed file system node.
According to the embodiment of the application, when the database is abnormal, the business data can be stored through the distributed file system to replace the database as a data storage warehouse, and the normal operation of the front-end system is not influenced.
In this embodiment, the number of the to-be-processed services sent by the front-end application to the background system is usually relatively large, and in order to avoid omission of one or more of the to-be-processed services, the to-be-processed services may be first added to the message queue, and when performing service processing, the services may be extracted from the message queue for processing. The implementation process can be described in detail in conjunction with fig. 5 as follows.
Referring to fig. 5, a flowchart illustrating steps of a pending service adding method provided in an embodiment of the present application is shown, and as shown in fig. 5, the pending service adding method may include: step 501.
Step 501: and adding the service to be processed into a message queue so as to write the service to be processed in the message queue into the distributed file system when a writing condition is met.
In this embodiment, the writing condition refers to a condition for writing the pending transaction into the distributed file system. In this example, the writing condition may be a first-in first-out condition, that is, a condition that the to-be-processed traffic is written into the distributed file system in the chronological order in which the to-be-processed traffic is added to the Message Queue (MQ), for example, the to-be-processed traffic added to the Message Queue is, in chronological order: when the to-be-processed service is written into the distributed file system from the message queue, the account opening service A can be written into the distributed file system, then the account opening service B can be written into the distributed file system, and the rest can be done in the same way until all the to-be-processed service in the message queue is written into the distributed file system.
It can be understood that, in practical applications, the number of services written into the distributed file system at a time may be multiple, and the above examples are only examples listed for better understanding of the technical solution of the embodiments of the present application, and are not intended to be the only limitation to the embodiments.
Of course, the writing condition may also be a condition of service importance, and in a specific implementation, the order of writing into the distributed file system may be set for different service types, for example, taking a banking service as an example, an account opening service and a transfer service may be written into the distributed file system preferentially, an account inquiry service may be written into the distributed file system finally, and the like.
After receiving the pending service sent by the front-end application, the pending service may be added to the message queue.
After the pending transactions are added to the message queue, the pending transactions in the message queue may be written to the distributed file system when the pending transactions satisfy the write condition. Specifically, a multithreading parallel writing mode can be called to write the to-be-processed service in the message queue into the distributed file system.
According to the embodiment of the application, the to-be-processed service sent by the front-end application program is added to the message queue, so that the condition that part of the to-be-processed service is omitted can be avoided.
In this embodiment, after the service data of the service to be processed is stored in the target distributed file system node, and after the database is restored to the normal state, the service data of the service to be processed may be acquired from the target distributed file system node and stored in the database. This implementation can be described in detail below in conjunction with fig. 6.
Referring to fig. 6, a flowchart illustrating steps of another service data saving method provided in an embodiment of the present application is shown, and as shown in fig. 6, the service data saving method may include: step 601, step 602 and step 603.
Step 601: determining whether the database is restored to a normal state.
In this embodiment, after detecting the database anomaly, a heartbeat detection mechanism may be used to detect the state of the database, so as to detect whether the database is restored to a normal state.
After determining that the database is restored to the normal state, step 602 is performed.
Step 602: and responding to the database to be recovered to a normal state, and acquiring the service data of the service to be processed from the target distributed file system node.
Step 603: and storing the service data into the database.
After the database is determined to be restored to the normal state, the service data of the service to be processed can be obtained from the target distributed file system node in response to the restoration of the database to the normal state, and the service data is stored in the database.
In a specific implementation, the service data for the service to be processed may be stored in the database in a full storage manner.
Of course, more important service data may be stored in the database, and specifically, the service data of the service to be processed may be selectively stored in the database in a persistent manner according to the service type. For example, taking banking as an example, the service types of the to-be-processed service include: the account transfer type, the account opening type, the withdrawal type, the deposit type, the password changing type, the balance inquiry type, the account flow inquiry type and the like, wherein the service types of the service to be processed are as follows: and when the transfer type, the account opening type, the withdrawal type, the deposit type and the password type are changed, the service data of the services of the types can be obtained from the target distributed file system node and persistently stored in the database. The service types of the service to be processed are: when the balance query type, the account flow query type and the like are adopted, the service data of the services of the types do not need to be stored in a database in a persistent mode, and the like.
It should be understood that the above examples are only examples for better understanding of the technical solutions of the embodiments of the present application, and are not intended to limit the present embodiment.
In the embodiment, the service data in the distributed file system is stored in the form of files, and the service data in the database is stored in the form of data fields. When the service data of the service to be processed is acquired from the target distributed file system node and stored in the database in a persistent manner, the data file corresponding to the service data of the service to be processed can be acquired from the target distributed file system node, then the data file is analyzed to obtain the data field corresponding to the service to be processed, and finally the data field corresponding to the service to be processed can be stored in the database in a persistent manner.
According to the embodiment of the application, after the database is restored to the normal state, the service data is stored in the database in a persistent mode, the service data can be stored in the persistent mode, the condition that the service data is lost can be avoided, and meanwhile follow-up service data can be conveniently inquired.
In this embodiment, when performing service query from the distributed file system, query may also be performed according to the mapping relationship of the cache in the cache database, and specifically, the following detailed description may be performed with reference to fig. 7.
Referring to fig. 7, a flowchart illustrating steps of a business data query method provided in an embodiment of the present application is shown, and as shown in fig. 7, the business data query method may include: step 701, step 702, step 703, step 704 and step 705.
Step 701: and acquiring a data query request sent by a user.
In this embodiment, the data query request refers to a request for querying the service data.
When the user wants to inquire the service data, a data inquiry request can be sent to the background system by the user.
After acquiring the data query request sent by the user, step 702 is executed.
Step 702: and acquiring nodes of the distributed file system to be queried corresponding to query keywords contained in the data query request according to the mapping relation cached in the cache database.
After the data query request is obtained, the data query request may be parsed to obtain query keywords included in the data query request.
After the query keyword is obtained, the node of the distributed file system to be queried corresponding to the query keyword can be obtained according to the mapping relation cached in the cache database.
After the distributed file system node to be queried corresponding to the query keyword included in the data query request is obtained, step 703 is executed.
Step 703: and inquiring to obtain the file path of the service data to be inquired according to the distributed file system node to be inquired.
After the distributed file system node to be queried corresponding to the query keyword contained in the data query request is obtained, a file path of the service data to be queried can be queried and obtained according to the distributed file system node to be queried.
After the query obtains the file path of the service data to be queried, step 704 is executed.
Step 704: and acquiring a target file corresponding to the service data to be inquired according to the file path.
After the file path of the service data to be queried is obtained through querying, the target file corresponding to the service data to be queried can be obtained according to the file path.
After the target file is obtained, step 705 is performed.
Step 705: and analyzing the target file to obtain the service data to be inquired.
After the target file corresponding to the service data to be queried is obtained according to the file path, the target file can be analyzed to obtain the service data to be queried, and the service data to be queried is sent to the user.
The above-described scheme provided by the embodiment of the present application can be integrally described as follows with reference to fig. 8.
Referring to fig. 8, a schematic diagram of a database-less operation process provided in an embodiment of the present application is shown.
For the service processing process: when the database is abnormal, the front-end main application 801 may issue an asynchronous message, that is, send a pending service to the background system in an asynchronous issuing manner. After the background system receives the pending traffic sent asynchronously by the front-end master application 801, the pending traffic may be added to a Message Queue (MQ) 802. The consumer (i.e. the service processing end) 805 can subscribe to the message from the message queue 802, i.e. extract the service to be processed from the message queue 802 for processing. After the consuming end 805 processes the to-be-processed service subscribed in the message queue 802, the service data of the to-be-processed service may be stored in a node corresponding to the to-be-processed service in the distributed file system 806, and at this time, if the database 807 is abnormal, the database 807 does not need to store the service data of the to-be-processed service. After the consuming side 805 processes the service to be processed, the service Key word and the corresponding node register of the processed service may also be registered and stored in the Key-value804 (i.e., the cache data in this example). After the database 807 is restored to the normal state, the service data corresponding to the service to be processed may be extracted from the distributed file system 806 and stored in the database 807, so as to improve the security of data storage by adopting a manner of storing multiple copies of data.
For the business data query process: first, the user 803 may send a query request, identify a data location of the service data to be queried (i.e., a node of the distributed file system 806 corresponding to the service data to be queried) according to a Key word included in the query request sent by the user 803 and a mapping relationship stored by the Key-value804, and then read data from a corresponding node of the corresponding distributed file system 806 according to the data location, i.e., read the queried service data and send the data to the user 803.
By the scheme, normal business processing and business data query processes can be performed when the database is abnormal, and normal operation of the front-end application program is not affected.
According to the data processing method provided by the embodiment of the application, the distributed file system is subjected to node division processing to generate N distributed file system nodes, wherein N is a positive integer greater than or equal to 1, a mapping relation between a keyword of a service type corresponding to each distributed file system node and the corresponding distributed file system node is established, and the mapping relation is stored in a cache database, so that when the database is abnormal, the distributed file system node corresponding to the service is determined according to the mapping relation, and service data corresponding to the service is stored in the distributed file system node. According to the embodiment of the application, the mapping relation between the key words of the service type corresponding to each distributed file system node and the corresponding distributed file system nodes is stored in the cache data base, so that when the service is processed under the condition that the database is abnormal, the service data generated by processing the service can be stored to the corresponding distributed file system nodes according to the mapping relation, and the normal operation of the front-end system can not be influenced.
Referring to fig. 9, which shows a schematic structural diagram of a data processing apparatus provided in an embodiment of the present application, as shown in fig. 9, the data processing apparatus 900 may include the following modules:
a node division module 910, configured to perform node division processing on the distributed file system to generate N distributed file system nodes, where N is a positive integer greater than or equal to 1;
a mapping relationship establishing module 920, configured to establish a mapping relationship between the keyword of the service type corresponding to each distributed file system node and the corresponding distributed file system node;
the mapping relationship saving module 930 is configured to save the mapping relationship into the cache database, so as to determine a distributed file system node corresponding to the service according to the mapping relationship when the service is processed under the condition that the database is abnormal, and save the service data corresponding to the service to the distributed file system node.
Optionally, the apparatus further comprises:
the database exception determining module is used for responding to the starting of the front application program and determining whether the database is abnormal or not;
the target node determining module is used for determining a target distributed file system node corresponding to the service to be processed according to the mapping relation cached in the cache database under the condition that the database is determined to be abnormal and the service to be processed sent by the pre-application program is received;
and the first service data storage module is used for acquiring service data corresponding to the service to be processed and storing the service data to the target distributed file system node.
Optionally, the target node determining module includes:
a service keyword obtaining unit, configured to obtain a service keyword of the service to be processed;
and the target node acquisition unit is used for acquiring the target distributed file system node corresponding to the service keyword according to the mapping relation between the keyword stored in the cache database and the distributed file system node.
Optionally, the service data saving module includes:
a basic service data acquiring unit, configured to acquire basic service data corresponding to the service to be processed;
a service flow data acquisition unit, configured to acquire service flow data generated in a process of processing the service to be processed;
and the service data acquisition unit is used for taking the basic service data and the service flow data as service data of the service to be processed.
Optionally, the apparatus further comprises:
and the service to be processed writing module is used for adding the service to be processed into a message queue so as to write the service to be processed in the message queue into the distributed file system when the writing condition is met.
Optionally, the apparatus further comprises:
the database determining module is used for determining whether the database is recovered to a normal state;
a service data acquisition module, configured to respond to the database being restored to a normal state, and acquire service data of the service to be processed from the target distributed file system node;
and the second service data storage module is used for storing the service data into the database.
Optionally, the apparatus further comprises:
the data query request acquisition module is used for acquiring a data query request sent by a user;
the node to be queried acquiring module is used for acquiring a distributed file system node to be queried corresponding to a query keyword contained in the data query request according to the mapping relation cached in the cache database;
the file path query module is used for querying a file path of the service data to be queried according to the distributed file system node to be queried;
the target file acquisition module is used for acquiring a target file corresponding to the service data to be inquired according to the file path;
and the data to be queried acquisition module is used for analyzing the target file to obtain the service data to be queried.
The data processing apparatus provided in this embodiment of the present application generates N distributed file system nodes by performing node division processing on a distributed file system, where N is a positive integer greater than or equal to 1, establishes a mapping relationship between a keyword of a service type corresponding to each distributed file system node and the corresponding distributed file system node, and stores the mapping relationship in a cache database, so that when service processing is performed under an abnormal condition of the database, the distributed file system node corresponding to the service is determined according to the mapping relationship, and service data corresponding to the service is stored in the distributed file system node. According to the embodiment of the application, the mapping relation between the key words of the service type corresponding to each distributed file system node and the corresponding distributed file system nodes is stored in the cache data base, so that when the service is processed under the condition that the database is abnormal, the service data generated by processing the service can be stored to the corresponding distributed file system nodes according to the mapping relation, and the normal operation of the front-end system can not be influenced.
Additionally, an embodiment of the present application further provides an electronic device, including: a memory, a processor and a computer program stored on the memory and executable on the processor, which computer program, when executed by the processor, implements the above-mentioned data processing method.
Fig. 10 shows a schematic structural diagram of an electronic device 1000 according to an embodiment of the present invention. As shown in fig. 10, electronic device 1000 includes a Central Processing Unit (CPU) 1001 that may perform various appropriate actions and processes in accordance with computer program instructions stored in a Read Only Memory (ROM) 1002 or computer program instructions loaded from a storage unit 1008 into a Random Access Memory (RAM) 1003. In the RAM1003, various programs and data necessary for the operation of the electronic apparatus 1000 can also be stored. The CPU1001, ROM1002, and RAM1003 are connected to each other via a bus 1004. An input/output (I/O) interface 1005 is also connected to bus 1004.
A number of components in the electronic device 1000 are connected to the I/O interface 1005, including: an input unit 1006 such as a keyboard, a mouse, a microphone, and the like; an output unit 1007 such as various types of displays, speakers, and the like; a storage unit 1008 such as a magnetic disk, an optical disk, or the like; and a communication unit 1009 such as a network card, a modem, a wireless communication transceiver, or the like. The communication unit 1009 allows the electronic device 1000 to exchange information/data with other devices through a computer network such as the internet and/or various telecommunication networks.
The various processes and processes described above may be performed by processing unit 1001. For example, the methods of any of the above embodiments may be implemented as a computer software program tangibly embodied on a computer-readable medium, such as storage unit 1008. In some embodiments, part or all of the computer program may be loaded and/or installed onto electronic device 1000 via ROM1002 and/or communications unit 1009. When the computer program is loaded into RAM1003 and executed by CPU1001, one or more acts of the methods described above may be performed.
The embodiments of the present application further provide a computer-readable storage medium, where a computer program is stored, and when the computer program is executed by a processor, the computer program implements the processes of the data processing method embodiment, and can achieve the same technical effects, and in order to avoid repetition, the computer program is not described herein again. The computer-readable storage medium may be a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one of 8230, and" comprising 8230does not exclude the presence of additional like elements in a process, method, article, or apparatus comprising the element.
Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solutions of the present application or portions thereof that contribute to the prior art may be embodied in the form of a software product, which is stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal (which may be a mobile phone, a computer, a server, an air conditioner, or a network device) to execute the method according to the embodiments of the present application.
While the present embodiments have been described with reference to the accompanying drawings, it is to be understood that the present embodiments are not limited to those precise embodiments, which are intended to be illustrative rather than restrictive, and that various changes and modifications may be effected therein by one skilled in the art without departing from the scope of the appended claims.
Those of ordinary skill in the art would appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed in the embodiments disclosed herein may be implemented as electronic hardware or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
In the embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one logical division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit.
The functions may be stored in a computer-readable storage medium if they are implemented in the form of software functional units and sold or used as separate products. Based on such understanding, the technical solutions of the present application or portions thereof that substantially contribute to the prior art may be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the methods described in the embodiments of the present application. And the aforementioned storage medium includes: a U disk, a removable hard disk, a ROM, a RAM, a magnetic disk or an optical disk, and various media capable of storing program codes.
The above description is only for the specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present application, and shall be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims (10)

1. A method of data processing, comprising:
carrying out node division processing on the distributed file system to generate N distributed file system nodes, wherein N is a positive integer greater than or equal to 1;
establishing a mapping relation between the keywords of the service type corresponding to each distributed file system node and the corresponding distributed file system node;
and storing the mapping relation into a cache database, so that when the service is processed under the condition of abnormal database, the distributed file system node corresponding to the service is determined according to the mapping relation, and the service data corresponding to the service is stored into the distributed file system node.
2. The method of claim 1, further comprising, after saving the mapping relationship to a cache database:
determining whether the database is abnormal or not in response to the starting of the front-end application program;
determining a target distributed file system node corresponding to the service to be processed according to the mapping relation cached in the cache database under the condition that the database is determined to be abnormal and the service to be processed sent by the front-end application program is received;
and acquiring service data corresponding to the service to be processed, and storing the service data to the target distributed file system node.
3. The method according to claim 2, wherein the determining, according to the mapping relationship cached in the cache database, the target distributed file system node corresponding to the service to be processed includes:
acquiring a service keyword of the service to be processed;
and acquiring a target distributed file system node corresponding to the service keyword according to the mapping relation between the keyword stored in the cache database and the distributed file system node.
4. The method according to claim 2, wherein the obtaining the service data corresponding to the service to be processed includes:
acquiring basic service data corresponding to the service to be processed;
acquiring service flow data generated in the process of processing the service to be processed;
and taking the basic service data and the service flow data as service data of the service to be processed.
5. The method according to claim 2, further comprising, after said receiving the pending traffic sent by the pre-application:
and adding the service to be processed into a message queue so as to write the service to be processed in the message queue into the distributed file system when a writing condition is met.
6. The method of claim 2, further comprising, after the saving the traffic data to the target distributed file system node:
determining whether the database is restored to a normal state;
responding to the database to be recovered to a normal state, and acquiring the service data of the service to be processed from the target distributed file system node;
and storing the service data into the database.
7. The method of any of claims 1 to 6, further comprising:
acquiring a data query request sent by a user;
acquiring nodes of the distributed file system to be queried corresponding to query keywords contained in the data query request according to the mapping relation cached in the cache database;
inquiring to obtain a file path of the service data to be inquired according to the distributed file system node to be inquired;
acquiring a target file corresponding to the service data to be inquired according to the file path;
and analyzing the target file to obtain the service data to be inquired.
8. A data processing apparatus, comprising:
the node dividing module is used for carrying out node dividing processing on the distributed file system to generate N distributed file system nodes, wherein N is a positive integer greater than or equal to 1;
the mapping relation establishing module is used for establishing a mapping relation between the key words of the service types corresponding to each distributed file system node and the corresponding distributed file system node;
and the mapping relation storage module is used for storing the mapping relation into a cache database, so that when the service is processed under the abnormal condition of the database, the distributed file system node corresponding to the service is determined according to the mapping relation, and the service data corresponding to the service is stored to the distributed file system node.
9. An electronic device, comprising:
memory, processor and computer program stored on the memory and executable on the processor, which computer program, when executed by the processor, implements a data processing method as claimed in any one of claims 1 to 7.
10. A readable storage medium, wherein instructions in the storage medium, when executed by a processor of an electronic device, enable the electronic device to perform the data processing method of any one of claims 1 to 7.
CN202211169788.XA 2022-09-23 2022-09-23 Data processing method and device, electronic equipment and storage medium Pending CN115617763A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202211169788.XA CN115617763A (en) 2022-09-23 2022-09-23 Data processing method and device, electronic equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202211169788.XA CN115617763A (en) 2022-09-23 2022-09-23 Data processing method and device, electronic equipment and storage medium

Publications (1)

Publication Number Publication Date
CN115617763A true CN115617763A (en) 2023-01-17

Family

ID=84857801

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202211169788.XA Pending CN115617763A (en) 2022-09-23 2022-09-23 Data processing method and device, electronic equipment and storage medium

Country Status (1)

Country Link
CN (1) CN115617763A (en)

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102880531A (en) * 2012-09-27 2013-01-16 新浪网技术(中国)有限公司 Database backup system and backup method and slave database server of database backup system
CN104572862A (en) * 2014-12-19 2015-04-29 阳珍秀 Mass data storage access method and system
CN108491732A (en) * 2018-03-13 2018-09-04 山东超越数控电子股份有限公司 A kind of mass storage data protection system and method based on business isolated storage
CN109828863A (en) * 2019-01-10 2019-05-31 网联清算有限公司 Data disaster tolerance method, apparatus, storage medium and computer equipment
CN110022226A (en) * 2019-01-04 2019-07-16 国网浙江省电力有限公司 A kind of data collection system and acquisition method based on object-oriented
CN110457281A (en) * 2019-08-14 2019-11-15 北京博睿宏远数据科技股份有限公司 Data processing method, device, equipment and medium
CN111290826A (en) * 2018-12-06 2020-06-16 北京京东尚科信息技术有限公司 Distributed file system, computer system, and medium
CN112632184A (en) * 2020-12-15 2021-04-09 北京达佳互联信息技术有限公司 Data processing method and device, electronic equipment and storage medium

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102880531A (en) * 2012-09-27 2013-01-16 新浪网技术(中国)有限公司 Database backup system and backup method and slave database server of database backup system
CN104572862A (en) * 2014-12-19 2015-04-29 阳珍秀 Mass data storage access method and system
CN108491732A (en) * 2018-03-13 2018-09-04 山东超越数控电子股份有限公司 A kind of mass storage data protection system and method based on business isolated storage
CN111290826A (en) * 2018-12-06 2020-06-16 北京京东尚科信息技术有限公司 Distributed file system, computer system, and medium
CN110022226A (en) * 2019-01-04 2019-07-16 国网浙江省电力有限公司 A kind of data collection system and acquisition method based on object-oriented
CN109828863A (en) * 2019-01-10 2019-05-31 网联清算有限公司 Data disaster tolerance method, apparatus, storage medium and computer equipment
CN110457281A (en) * 2019-08-14 2019-11-15 北京博睿宏远数据科技股份有限公司 Data processing method, device, equipment and medium
CN112632184A (en) * 2020-12-15 2021-04-09 北京达佳互联信息技术有限公司 Data processing method and device, electronic equipment and storage medium

Similar Documents

Publication Publication Date Title
US20190081876A1 (en) Real time anomaly detection for data streams
CN109388657B (en) Data processing method, device, computer equipment and storage medium
CN112162965B (en) Log data processing method, device, computer equipment and storage medium
US10554701B1 (en) Real-time call tracing in a service-oriented system
CN114172966B (en) Service calling method, service processing method and device under unitized architecture
US8930518B2 (en) Processing of write requests in application server clusters
US7930404B2 (en) Cross-system log in a distributed system environment
CN110955857A (en) Service processing method and device for high concurrency environment
CN109617647A (en) Message forwarding method, device, computer equipment and computer readable storage medium
CN108897729B (en) Transaction template sharing method and device, electronic equipment and storage medium
CN112214505A (en) Data synchronization method and device, computer readable storage medium and electronic equipment
CN115145587A (en) Product parameter checking method and device, electronic equipment and storage medium
CN112988812B (en) Inventory data processing method, device, equipment and storage medium
CN110782310B (en) Method, device and system for asynchronously acquiring user attribute information from third-party platform
CN112052259A (en) Data processing method, device, equipment and computer storage medium
CN112637222A (en) Request processing method and device
CN107451301B (en) Processing method, device, equipment and storage medium for real-time delivery bill mail
CN115617763A (en) Data processing method and device, electronic equipment and storage medium
CN115658753A (en) Method and device for inquiring electronic receipt and related equipment
CN113992664A (en) Cluster communication method, related device and storage medium
CN115118767B (en) Method, device and equipment for realizing customizable unitized routing component
CN117472617B (en) Message processing method and device, electronic equipment and readable storage medium
CN117972096B (en) Method and system for processing interaction message of social platform
US20240089339A1 (en) Caching across multiple cloud environments
US20220191104A1 (en) Access management for a multi-endpoint data store

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination