CN108170551B - Crawler system based front-end and back-end error processing method, server and storage medium - Google Patents
Crawler system based front-end and back-end error processing method, server and storage medium Download PDFInfo
- Publication number
- CN108170551B CN108170551B CN201810003918.XA CN201810003918A CN108170551B CN 108170551 B CN108170551 B CN 108170551B CN 201810003918 A CN201810003918 A CN 201810003918A CN 108170551 B CN108170551 B CN 108170551B
- Authority
- CN
- China
- Prior art keywords
- error
- crawler
- crawler system
- error code
- code
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
- G06F11/0706—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation the processing taking place on a specific hardware platform or in a specific software environment
- G06F11/0709—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation the processing taking place on a specific hardware platform or in a specific software environment in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
- G06F11/0706—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation the processing taking place on a specific hardware platform or in a specific software environment
- G06F11/0748—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation the processing taking place on a specific hardware platform or in a specific software environment in a remote unit communicating with a single-box computer node experiencing an error/fault
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
- G06F11/079—Root cause analysis, i.e. error or fault diagnosis
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Hardware Design (AREA)
- Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Debugging And Monitoring (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The invention discloses a front-end and back-end error processing method based on a crawler system, which comprises the following steps: establishing a corresponding relation table of error information and error codes; judging whether the crawler system fails to crawl a front-end system; when the crawler system fails to crawl the front-end system, receiving error information sent by the front-end system; and feeding back an error code to the crawling system according to the received error information and the corresponding relation between the error information and the error code. The invention also provides a server and a computer readable storage medium. According to the crawler system-based front-end and back-end error processing method, the server and the storage medium, provided by the invention, in the crawling process of the crawler system, when an error occurs, an error code is directly fed back, then the error type is analyzed according to the error code, the problem of developing and removing positioning analysis is reduced, and different errors are directly distributed to different responsible persons.
Description
Technical Field
The present invention relates to computer domains, and in particular, to a front-end and back-end error handling method based on a crawler system, a server, and a computer-readable storage medium.
Background
According to the existing crawler service product, if the front end and the back end have problems, the information fed back by the server end generally is front end crawling failure, and a worker is difficult to accurately define whether the front end problem or the server end problem exists on the ground according to the information of the front end crawling failure. In order to solve the problems, the existing method actively inquires the server to inquire the positioning problem, and if the problem is a front-end problem, the problem is solved by a front-end development project personnel; if the problem is the back end problem, the problem is solved by a back end developer.
Disclosure of Invention
In view of the above, the invention provides a front-end and back-end error processing method based on a crawler system, a server and a storage medium, which can directly feed back an error code when an error occurs in a crawling process of the crawler system without actively inquiring a positioning problem by the server and then distributing the corresponding problem to a corresponding person, and then analyze the error type according to the error code, thereby reducing the development of positioning analysis problems, and directly distributing different errors to different responsible persons.
In order to achieve the above object, the present invention provides a server, where the server includes a memory and a processor, the memory stores a crawler system-based front-end and back-end error handling program executable on the processor, and when the crawler system-based front-end and back-end error handling program is executed by the processor, the server implements the following steps:
establishing a corresponding relation table of error information and error codes, wherein the error information comprises front-end error information and rear-end error information, and the error codes comprise front-end error codes and rear-end error codes;
judging whether the crawler system fails to crawl a front-end system;
when the crawler system fails to crawl the front-end system, receiving error information sent by the front-end system; and
and feeding back an error code to the crawler system according to the received error information and the corresponding relation between the error information and the error code.
Optionally, the step of establishing a correspondence table between the error information and the error code includes:
and classifying the rear-end errors according to error types, and distributing a rear-end error code to each classified rear-end error.
Optionally, the step of determining whether the crawler system fails to crawl a front-end system includes:
judging whether the back-end database has data calling;
when the data are called in the back-end database, judging that the crawler system is crawling the front-end system; and
and when the data are not called in the back-end database, judging that the crawler system fails to crawl the front-end system.
Optionally, when the front-end and back-end error handling program of the crawler-based system is executed by the processor, the following steps are further implemented:
after the crawler system receives the fed error codes, judging the types of the error codes;
when the error code is a front-end error code, the error code is sent to a front-end developer; and
and when the error code is a rear-end error code, sending the error code to a rear-end developer.
In addition, in order to achieve the above object, the present invention further provides a front-end and back-end error processing method based on a crawler system, where the method is applied to a server, and the method includes:
establishing a corresponding relation table of error information and error codes, wherein the error information comprises front-end error information and rear-end error information, and the error codes comprise front-end error codes and rear-end error codes;
judging whether the crawler system fails to crawl a front-end system;
when the crawler system fails to crawl the front-end system, receiving error information sent by the front-end system; and
and feeding back an error code to the crawler system according to the received error information and the corresponding relation between the error information and the error code.
Optionally, the step of establishing a correspondence table between the error information and the error code includes:
and classifying the rear-end errors according to error types, and distributing a rear-end error code to each classified rear-end error.
Optionally, the step of determining whether the crawler system fails to crawl a front-end system includes:
judging whether the back-end database has data calling;
if the data are called in the back-end database, judging that the crawler system is crawling the front-end system; and
and when the data are not called in the back-end database, judging that the crawler system fails to crawl the front-end system.
Optionally, the method further comprises the step of:
after the crawler system receives the fed error codes, judging the types of the error codes;
when the error code is a front-end error code, the error code is sent to a front-end developer; and
and when the error code is a rear-end error code, sending the error code to a rear-end developer.
Optionally, the front-end error comprises some check errors or check misses of the front-end.
Further, to achieve the above object, the present invention also provides a storage medium, where the computer-readable storage medium stores a crawler system-based front-end and back-end error handling program, where the crawler system-based front-end and back-end error handling program is executable by at least one processor to cause the at least one processor to execute the steps of the crawler system-based front-end and back-end error handling method as described above.
Compared with the prior art, the server, the front-end and back-end error processing method based on the crawler system and the storage medium provided by the invention have the advantages that firstly, the corresponding relation table of the error information and the error codes is established, wherein the error information comprises the front-end error information and the back-end error information, and the error codes comprise the front-end error codes and the back-end error codes; secondly, judging whether the crawler system fails to crawl a front-end system; then, when the crawler system fails to crawl the front-end system, receiving error information sent by the front-end system; and finally, feeding back an error code to the crawler system according to the received error information and the corresponding relation between the error information and the error code. Like this, can avoid crawling among the prior art when service crawls failure, hardly define the drawback that is front end problem or server problem, can crawl the in-process at the crawler system, when the mistake appears, need not be earlier by server initiative inquiry location problem, relevant problem of redistribution to corresponding personnel, but direct feedback error code, then according to error code analysis error type, reduce the development and remove location analysis problem, different errors are directly divided and are given different responsible persons.
Drawings
FIG. 1 is a schematic diagram of an alternative hardware architecture for a server according to the present invention;
FIG. 2 is a block diagram of a first embodiment of a front-end and back-end error handling routine of the crawler-based system in accordance with the present invention;
FIG. 3 is a flowchart of a front-end and back-end error handling method based on a crawler system according to a first embodiment of the present invention;
FIG. 4 is a flowchart of a front-end and back-end error handling method based on a crawler system according to a second embodiment of the present invention.
Reference numerals:
| server | 1 |
| Memory device | 11 |
| Processor with a memory having a plurality of memory cells | 12 |
| Network interface | 13 |
| Front-end and back-end error processing program based on crawler system | 200 |
| Building module | 201 |
| Judging module | 202 |
| Receiving module | 203 |
| Feedback module | 204 |
The implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that the description relating to "first", "second", etc. in the present invention is for descriptive purposes only and is not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In addition, technical solutions between various embodiments may be combined with each other, but must be realized by a person skilled in the art, and when the technical solutions are contradictory or cannot be realized, such a combination should not be considered to exist, and is not within the protection scope of the present invention.
Fig. 1 is a schematic diagram of an alternative hardware architecture of the server 1.
The server 1 may be a rack server, a blade server, a tower server, or a rack server, and the server 1 may be an independent server or a server cluster composed of a plurality of servers.
In this embodiment, the server 1 may include, but is not limited to, a memory 11, a processor 12, and a network interface 13, which may be communicatively connected to each other through a system bus.
The server 1 is connected to the network through a network interface 13 to acquire information. The network may be a wireless or wired network such as an Intranet (Intranet), the Internet (Internet), a Global System of Mobile communication (GSM), Wideband Code Division Multiple Access (WCDMA), a 4G network, a 5G network, Bluetooth (Bluetooth), Wi-Fi, or a communication network.
It is noted that fig. 1 only shows the server 1 with components 11-13, but it is to be understood that not all of the shown components are required to be implemented, and that more or fewer components may be implemented instead.
The memory 11 includes at least one type of readable storage medium, which includes a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a Read Only Memory (ROM), an Electrically Erasable Programmable Read Only Memory (EEPROM), a Programmable Read Only Memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the storage 11 may be an internal storage unit of the server 1, such as a hard disk or a memory of the server 1. In other embodiments, the memory 11 may also be an external storage device of the server 1, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like provided in the server 1. Of course, the memory 11 may also comprise both an internal storage unit of the server 1 and an external storage device thereof. In this embodiment, the memory 11 is generally used for storing an operating system installed in the server 1 and various application software, such as program codes of the front-end and back-end error handling program 200 based on a crawler system. Furthermore, the memory 11 may also be used to temporarily store various types of data that have been output or are to be output.
The processor 12 may be a Central Processing Unit (CPU), controller, microcontroller, microprocessor, or other data Processing chip in some embodiments. The processor 12 is generally used for controlling the overall operation of the server 1, such as performing data interaction or communication-related control and processing. In this embodiment, the processor 12 is configured to run the program code stored in the memory 11 or process data, for example, run the front-end and back-end error handling program 200 based on the crawler system.
The network interface 13 may comprise a wireless network interface or a wired network interface, and the network interface 13 is generally used for establishing communication connection between the server 1 and other electronic devices.
In this embodiment, a front-end and back-end error handling program 200 based on a crawler system is installed and run in the server 1, and when the front-end and back-end error handling program 200 based on the crawler system runs, the server 1 establishes a correspondence table between error information and error codes, where the error information includes front-end error information and back-end error information, and the error codes include front-end error codes and back-end error codes; judging whether the crawler system fails to crawl a front-end system; when the crawler system fails to crawl the front-end system, receiving error information sent by the front-end system; and feeding back an error code to the crawler system according to the received error information and the corresponding relation between the error information and the error code. Like this, can avoid crawling among the prior art when service crawls failure, hardly define the drawback that is front end problem or server problem, can crawl the in-process at the crawler system, when the mistake appears, need not be earlier by server initiative inquiry location problem, relevant problem of redistribution to corresponding personnel, but direct feedback error code, then according to error code analysis error type, reduce the development and remove location analysis problem, different errors are directly divided and are given different responsible persons.
The hardware structure and functions of the related devices of the various embodiments of the present invention have been described in detail so far. Hereinafter, various embodiments of the present invention will be proposed based on the above-described application environment and related devices.
First, the present invention provides a front-end and back-end error handling program 200 based on a crawler system.
Referring to FIG. 2, a block diagram of a front-end and back-end error handling process 200 based on a crawler system according to a first embodiment of the present invention is shown.
In this embodiment, the crawler-based front-end error handling program 200 includes a series of computer program instructions stored in the memory 11, and when the computer program instructions are executed by the processor 12, the crawler-based front-end error handling operation according to the embodiments of the present invention can be implemented. In some embodiments, the crawler-based system front-end error handling program 200 may be divided into one or more modules based on the particular operations implemented by the portions of the computer program instructions. For example, in fig. 2, the crawler-based front-end and back-end error handling program 200 may be divided into a building module 201, a determining module 202, a receiving module 203, and a feedback module 204. Wherein:
the establishing module 201 is configured to establish a corresponding relationship table between the error information and the error code. The error information comprises front-end error information and back-end error information, and the error codes comprise front-end error codes and back-end error codes. Specifically, all the front-end error messages correspond to a front-end error code, and the back-end error messages correspond to a back-end error code. The error code is also called an error code, when the crawling process of the crawler system fails, the front-end crawling page can display error information to prompt a user, and meanwhile, the error information of each front end in the crawling process of the webpage corresponds to the front-end error code. In this embodiment, the front-end error includes some operation of checking error or checking omission of the front end. In the crawling process, once some check errors or check omissions occur at the front end, the crawling process fails, and in the embodiment, such front end errors are unified into an error code. Once such a front-end error occurs, no further crawling action is performed.
In this embodiment, the establishing module 201 is further configured to classify the back-end errors according to error types, assign a back-end error code to each classified back-end error, and then correspondingly establish a correspondence table between the error information and the error codes in the back-end database. In this embodiment, the backend database includes MySQL, SQL Server, Oracle, and the like. Like this, through carrying out the error code to all error message and corresponding, can be through the error code convenient and fast's of follow-up feedback know corresponding error in the program section where to and what kind of error is the error type, and then promoted error handling's efficiency.
The judging module 202 is configured to judge whether the crawler system fails to crawl the front-end system. Specifically, the judging module judges whether the crawler system fails to crawl the front-end system by:
the determining module 202 determines whether the back-end database has data to be called, and generally, the determining module 202 may determine whether the back-end database has data to be called according to a preset code rule. When there is data to be called in the back-end database, the determining module 202 determines that the crawler system is crawling the front-end system, otherwise, when there is no data to be called in the back-end database, the determining module 202 determines that the crawler system fails to crawl the front-end system.
At present, when a front-end webpage is crawled by a general crawler system, operations such as calling, looking up, backing up and the like of back-end data can be realized through the presentation of a front-end page through preset program codes, so that whether the crawler system is failed to crawl the front-end system can be judged by monitoring whether corresponding actions such as calling of data exist in a back-end database. As can be seen from the above embodiments, in the crawling process, once some check errors or check omissions occur in the front end, the crawling process will fail, and in this embodiment, such front end errors are unified into an error code. Once such a front-end error occurs, no further crawling action is performed. Therefore, whether the crawler system fails to crawl the front-end system can be judged by judging whether the back-end database has data calling.
Further, when the crawler system fails to crawl the front-end system, the receiving module 203 receives error information sent by the front-end system.
The feedback module 204 is configured to feed back an error code to the crawler system according to the error information received by the receiving module 203 and the corresponding relationship between the error information and the error code. Further, in order to distribute the corresponding problem to the corresponding responsible person more efficiently, after the crawler system receives the error code fed back by the feedback module 204, the determining module 202 is further configured to determine the type of the error code. When the error code is a front-end error code, the determining module 202 sends the error code to a front-end developer, and when the error code is a back-end error code, the determining module 202 sends the error code to a back-end developer.
In this embodiment, the administrator may preset information of developers (front-end developers and back-end developers), where the information of the developers includes information such as contact information and mailbox addresses of the developers. In this way, while the server 1 displays the error code on the terminal, the error code information is transmitted to the developer, for example, to a mailbox or a short message of the developer. In this way, the developer can receive the system abnormality information at the first time.
Like this, through carrying out error code sign to the error of front and back end, at the in-process that the crawler crawled, directly according to the error that crawls in-process appearance, feed back corresponding error code to carry out wrong location and distribution according to the error code, realization system error's that can be quick processing promotes system's efficiency.
Through the program module 201 and 204, the front-end and back-end error handling program 200 based on the crawler system according to the present invention first establishes a corresponding relationship table between error information and error codes, wherein the error information includes front-end error information and back-end error information, and the error codes include front-end error codes and back-end error codes; secondly, judging whether the crawler system fails to crawl a front-end system; then, when the crawler system fails to crawl the front-end system, receiving error information sent by the front-end system; and finally, feeding back an error code to the crawler system according to the received error information and the corresponding relation between the error information and the error code. Like this, can avoid crawling among the prior art when service crawls failure, hardly define the drawback that is front end problem or server problem, can crawl the in-process at the crawler system, when the mistake appears, need not be earlier by server initiative inquiry location problem, relevant problem of redistribution to corresponding personnel, but direct feedback error code, then according to error code analysis error type, reduce the development and remove location analysis problem, different errors are directly divided and are given different responsible persons.
In addition, the invention also provides a front-end and back-end error processing method based on the crawler system.
Fig. 3 is a flowchart of a front-end and back-end error processing method based on a crawler system according to a first embodiment of the present invention. In this embodiment, the execution order of the steps in the flowchart shown in fig. 3 may be changed and some steps may be omitted according to different requirements.
Step S301, a corresponding relation table of error information and error codes is established. The error information comprises front-end error information and back-end error information, and the error codes comprise front-end error codes and back-end error codes. Specifically, all the front-end error messages correspond to a front-end error code, and the back-end error messages correspond to a back-end error code. The error code is also called an error code, when the crawling process of the crawler system fails, the front-end crawling page can display error information to prompt a user, and meanwhile, the error information of each front end in the crawling process of the webpage corresponds to the front-end error code. In this embodiment, the front-end error includes some operation of checking error or checking omission of the front end. In the crawling process, once some check errors or check omissions occur at the front end, the crawling process fails, and in the embodiment, such front end errors are unified into an error code. Once such a front-end error occurs, no further crawling action is performed.
In this embodiment, the server 1 is further configured to classify the backend errors according to error types, assign a backend error code to each classified backend error, and then correspondingly establish a correspondence table between error information and error codes in a backend database. In this embodiment, the backend database includes MySQL, SQL Server, Oracle, and the like. Like this, through carrying out the error code to all error message and corresponding, can be through the error code convenient and fast's of follow-up feedback know corresponding error in the program section where to and what kind of error is the error type, and then promoted error handling's efficiency.
Step S302, judging whether the crawler system fails to crawl the front-end system. And executing the step S303 when the crawler system fails to crawl the front-end system, otherwise, continuing to the step S302.
Specifically, the server 1 determines whether the crawler system has failed to crawl the front-end system by:
the server 1 determines whether the data is called from the back-end database, and generally, the server 1 may determine whether the data is called from the back-end database according to a preset code rule. When the data is called in the back-end database, the server 1 judges that the crawler system is crawling the front-end system, otherwise, when the data is not called in the back-end database, the server 1 judges that the crawler system fails to crawl the front-end system.
At present, when a front-end webpage is crawled by a general crawler system, operations such as calling, looking up, backing up and the like of back-end data can be realized through the presentation of a front-end page through preset program codes, so that whether the crawler system is failed to crawl the front-end system can be judged by monitoring whether corresponding actions such as calling of data exist in a back-end database. As can be seen from the above embodiments, in the crawling process, once some check errors or check omissions occur in the front end, the crawling process will fail, and in this embodiment, such front end errors are unified into an error code. Once such a front-end error occurs, no further crawling action is performed. Therefore, whether the crawler system fails to crawl the front-end system can be judged by judging whether the back-end database has data calling.
Step S303, receiving the error information sent by the front-end system.
Step S304, feeding back an error code to the crawler system according to the received error information and the corresponding relationship between the error information and the error code.
Through the steps S301 to 304, the method for processing the front-end and back-end errors based on the crawler system according to the present invention includes first establishing a correspondence table between error information and error codes, where the error information includes front-end error information and back-end error information, and the error codes include front-end error codes and back-end error codes; secondly, judging whether the crawler system fails to crawl a front-end system; then, when the crawler system fails to crawl the front-end system, receiving error information sent by the front-end system; and finally, feeding back an error code to the crawler system according to the received error information and the corresponding relation between the error information and the error code. Like this, can avoid crawling among the prior art when service crawls failure, hardly define the drawback that is front end problem or server problem, can crawl the in-process at the crawler system, when the mistake appears, need not be earlier by server initiative inquiry location problem, relevant problem of redistribution to corresponding personnel, but direct feedback error code, then according to error code analysis error type, reduce the development and remove location analysis problem, different errors are directly divided and are given different responsible persons.
Fig. 4 is a flowchart of a front-end and back-end error processing method based on a crawler system according to a second embodiment of the present invention. In this embodiment, the execution order of the steps in the flowchart shown in fig. 4 may be changed and some steps may be omitted according to different requirements.
Step S401, a corresponding relation table of error information and error codes is established. The error information comprises front-end error information and back-end error information, and the error codes comprise front-end error codes and back-end error codes.
Step S402, judging whether the crawler system fails to crawl the front-end system. And executing the step S403 when the crawler system fails to crawl the front-end system, otherwise, continuing to the step S402.
Specifically, the server 1 determines whether the crawler system has failed to crawl the front-end system by:
the server 1 determines whether the data is called from the back-end database, and generally, the server 1 may determine whether the data is called from the back-end database according to a preset code rule. When the data is called in the back-end database, the server 1 judges that the crawler system is crawling the front-end system, otherwise, when the data is not called in the back-end database, the server 1 judges that the crawler system fails to crawl the front-end system.
At present, when a front-end webpage is crawled by a general crawler system, operations such as calling, looking up, backing up and the like of back-end data can be realized through the presentation of a front-end page through preset program codes, so that whether the crawler system is failed to crawl the front-end system can be judged by monitoring whether corresponding actions such as calling of data exist in a back-end database. As can be seen from the above embodiments, in the crawling process, once some check errors or check omissions occur in the front end, the crawling process will fail, and in this embodiment, such front end errors are unified into an error code. Once such a front-end error occurs, no further crawling action is performed. Therefore, whether the crawler system fails to crawl the front-end system can be judged by judging whether the back-end database has data calling.
Step S403, receiving the error information sent by the front-end system.
Step S404, feeding back an error code to the crawler system according to the received error information and the corresponding relation between the error information and the error code.
In step S405, the type of the error code is determined. If the error code is a front-end error code, step S406 is executed, otherwise, if the error code is a back-end error code, step S407 is executed.
The error information comprises front-end error information and back-end error information, and the error codes comprise front-end error codes and back-end error codes. And the error information of each front end corresponds to a front end error code in the webpage crawling process. In this embodiment, the front-end error includes some operation of checking error or checking omission of the front end.
And step S406, sending the error code to a front-end developer.
And step S407, sending the error code to a back-end developer.
In this embodiment, the administrator may preset information of developers (front-end developers and back-end developers), where the information of the developers includes information such as contact information and mailbox addresses of the developers. In this way, while the server displays the error code on the terminal, the error code information is transmitted to the developer, for example, to a mailbox or a short message of the developer. In this way, the developer can receive the system abnormality information at the first time.
Like this, through carrying out error code sign to the error of front and back end, at the in-process that the crawler crawled, directly according to the error that crawls in-process appearance, feedback corresponding error code to carry out wrong location and distribution according to the error code, realization system error's that can be quick processing can distribute corresponding problem more efficiently and give corresponding responsible personnel, promote system's efficiency.
Through the steps S401-407, the crawler system-based front-end and back-end error processing method provided by the invention has the advantages that through carrying out error code identification on errors at the front end and the back end, in the crawler crawling process, corresponding error codes are fed back directly according to the errors occurring in the crawling process, and the errors are positioned and distributed according to the error codes, so that the system errors can be rapidly processed, and the system efficiency is improved.
Further, to achieve the above object, the present invention also provides a storage medium storing a crawler system based front-end and back-end error handling program, where the crawler system based front-end and back-end error handling program is executable by at least one processor, so that the at least one processor executes the steps of the crawler system based front-end and back-end error handling method as described above.
The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.
Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solutions of the present invention may be embodied in the form of a software product, which is stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal device (such as a mobile phone, a computer, a server, an air conditioner, or a network device) to execute the method according to the embodiments of the present invention.
The above description is only a preferred embodiment of the present invention, and not intended to limit the scope of the present invention, and all modifications of equivalent structures and equivalent processes, which are made by using the contents of the present specification and the accompanying drawings, or directly or indirectly applied to other related technical fields, are included in the scope of the present invention.
Claims (7)
1. A front-end and back-end error processing method based on a crawler system is applied to a server and is characterized by comprising the following steps:
establishing a corresponding relation table of error information and error codes, wherein the error information comprises front-end error information and rear-end error information, the error codes comprise front-end error codes and rear-end error codes, the front-end error information comprises some checking errors or checking omission operations of the front end, and when the front-end error information appears, the crawling behavior of the crawler system is stopped;
judging whether the crawler system fails to crawl a front-end system;
when the crawler system fails to crawl the front-end system, receiving error information sent by the front-end system; and
feeding an error code back to the crawler system according to the received error information and the corresponding relation between the error information and the error code;
wherein, judge whether crawler system crawls the step of front-end system failure, include:
judging whether the back-end database has data calling;
if the data are called in the back-end database, judging that the crawler system is crawling the front-end system; and
and when the data are not called in the back-end database, judging that the crawler system fails to crawl the front-end system.
2. The crawler-based front-end and back-end error handling method of claim 1, wherein the step of establishing a correspondence table between error information and error codes comprises:
and classifying the rear-end errors according to error types, and distributing a rear-end error code to each classified rear-end error.
3. The crawler-based front-end error handling method of claim 1, wherein said method further comprises the steps of:
after the crawler system receives the fed error codes, judging the types of the error codes;
when the error code is a front-end error code, the error code is sent to a front-end developer; and
and when the error code is a rear-end error code, sending the error code to a rear-end developer.
4. A server, comprising a memory and a processor, wherein the memory stores a crawler-based front-end and back-end error handling program operable on the processor, and the crawler-based front-end and back-end error handling program when executed by the processor implements the steps of:
establishing a corresponding relation table of error information and error codes, wherein the error information comprises front-end error information and rear-end error information, the error codes comprise front-end error codes and rear-end error codes, the front-end error information comprises some checking errors or checking omission operations of the front end, and when the front-end error information appears, the crawling behavior of the crawler system is stopped;
judging whether the crawler system fails to crawl a front-end system;
when the crawler system fails to crawl the front-end system, receiving error information sent by the front-end system; and
feeding an error code back to the crawler system according to the received error information and the corresponding relation between the error information and the error code;
wherein, judge whether crawler system crawls the step of front-end system failure, include:
judging whether the back-end database has data calling;
when the data are called in the back-end database, judging that the crawler system is crawling the front-end system; and
and when the data are not called in the back-end database, judging that the crawler system fails to crawl the front-end system.
5. The server according to claim 4, wherein the step of establishing the correspondence table between the error information and the error code comprises:
and classifying the rear-end errors according to error types, and distributing a rear-end error code to each classified rear-end error.
6. The server according to claim 4, wherein the crawler-based front-end error handler is further operative to perform the following when executed by the processor:
after the crawler system receives the fed error codes, judging the types of the error codes;
when the error code is a front-end error code, the error code is sent to a front-end developer; and
and when the error code is a rear-end error code, sending the error code to a rear-end developer.
7. A storage medium storing a crawler system based front-end error handling program executable by at least one processor to cause the at least one processor to perform the steps of the crawler system based front-end error handling method according to any one of claims 1-3.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810003918.XA CN108170551B (en) | 2018-01-03 | 2018-01-03 | Crawler system based front-end and back-end error processing method, server and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810003918.XA CN108170551B (en) | 2018-01-03 | 2018-01-03 | Crawler system based front-end and back-end error processing method, server and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108170551A CN108170551A (en) | 2018-06-15 |
| CN108170551B true CN108170551B (en) | 2021-06-22 |
Family
ID=62517216
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810003918.XA Active CN108170551B (en) | 2018-01-03 | 2018-01-03 | Crawler system based front-end and back-end error processing method, server and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108170551B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110457556B (en) * | 2019-07-04 | 2023-11-14 | 重庆金融资产交易所有限责任公司 | Distributed crawler system architecture, method for crawling data and computer equipment |
| CN110347560B (en) * | 2019-07-19 | 2022-04-26 | 深圳前海微众银行股份有限公司 | Method, device, system, equipment and medium for prompting abnormity of big data product |
| CN111200654A (en) * | 2019-12-31 | 2020-05-26 | 中国银行股份有限公司 | Client request error processing method and device |
| CN113268230B (en) * | 2021-05-18 | 2023-09-08 | 深圳希施玛数据科技有限公司 | Feedback information generation method and device, terminal and storage medium |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102981943A (en) * | 2012-10-29 | 2013-03-20 | 新浪技术（中国）有限公司 | Method and system for monitoring application logs |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110078015A1 (en) * | 2009-09-25 | 2011-03-31 | National Electronics Warranty, Llc | Dynamic mapper |
| CN103248625B (en) * | 2013-04-27 | 2016-09-14 | 北京京东尚科信息技术有限公司 | A kind of web crawlers operation exception monitoring method and system |
| CN105743730B (en) * | 2014-12-12 | 2019-01-29 | 阿里巴巴集团控股有限公司 | The method and its system of real time monitoring are provided for the web service of mobile terminal |
| CN106649362B (en) * | 2015-10-30 | 2020-02-07 | 北京国双科技有限公司 | Webpage crawling method and device |
| CN105868037A (en) * | 2016-03-23 | 2016-08-17 | 乐视云计算有限公司 | Processing method and system of error information in program operation process |
-
2018
- 2018-01-03 CN CN201810003918.XA patent/CN108170551B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102981943A (en) * | 2012-10-29 | 2013-03-20 | 新浪技术（中国）有限公司 | Method and system for monitoring application logs |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108170551A (en) | 2018-06-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108427705B (en) | Electronic device, distributed system log query method and storage medium | |
| CN108170551B (en) | Crawler system based front-end and back-end error processing method, server and storage medium | |
| CN109474578B (en) | Message checking method, device, computer equipment and storage medium | |
| CN108416485B (en) | User identity recognition method, electronic device and computer readable storage medium | |
| CN108462760B (en) | Electronic device, automatic cluster access domain name generation method and storage medium | |
| CN108491301B (en) | Electronic device, abnormality early warning method based on redis and storage medium | |
| CN108415925B (en) | Electronic device, data call log generation and query method and storage medium | |
| CN110221948B (en) | Test script generation method, test script generation device, computer device and storage medium | |
| CN110599354B (en) | Online checking method, online checking system, computer device and computer readable storage medium | |
| CN108446171B (en) | Electronic device, distributed system execution task allocation method and storage medium | |
| CN107133233B (en) | Processing method and device for configuration data query | |
| CN110222535B (en) | Processing device, method and storage medium for block chain configuration file | |
| CN110308880B (en) | Log printing method, system, computer device and computer readable storage medium | |
| CN113448862B (en) | Software version testing method and device and computer equipment | |
| CN108255735B (en) | Associated environment testing method, electronic device and computer readable storage medium | |
| CN110866834A (en) | Batch processing program execution method and system | |
| CN112667424A (en) | Abnormal data processing method and device, computer equipment and storage medium | |
| CN111371645A (en) | Method, system, computer equipment and storage medium for testing service connectivity | |
| CN114090113A (en) | Method, device and equipment for dynamically loading data source processing plug-in and storage medium | |
| CN112416648A (en) | Data verification method and device | |
| CN110166530B (en) | Processing method based on micro-service return value, electronic device and computer equipment | |
| CN112416384A (en) | Software upgrading method and device and computer equipment | |
| CN113419949B (en) | Abnormality detection method, device, equipment and storage medium for data processing | |
| CN112558982B (en) | Code detection method and device and computer equipment | |
| CN110677353B (en) | Data access method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |