CN114510352B - Method and system for automatically permeating project scheduling tasks - Google Patents
Method and system for automatically permeating project scheduling tasks Download PDFInfo
- Publication number
- CN114510352B CN114510352B CN202210139076.7A CN202210139076A CN114510352B CN 114510352 B CN114510352 B CN 114510352B CN 202210139076 A CN202210139076 A CN 202210139076A CN 114510352 B CN114510352 B CN 114510352B
- Authority
- CN
- China
- Prior art keywords
- scanning
- task
- value data
- result
- project
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
- G06F9/5038—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the execution order of a plurality of tasks, e.g. taking priority or time dependency constraints into consideration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5061—Partitioning or combining of resources
- G06F9/5072—Grid computing
Abstract
A method and system for scheduling tasks of automatic penetration projects, which obtains a target object input by a user, configures a scanning range, a scanning worker and issues a scanning project after result multiplexing; the dispatcher receives the issued scanning items and forwards the scanning items to the core main control item initializer; the core main control project initializer decomposes the scanning task of the scanning project according to the type of the target object; the core master control project initializer sends the decomposed scanning task to the core task generator; the core task generator judges whether the received scanning task is subjected to result multiplexing; if the scanning project configuration is result multiplexing, a scanning task is issued to a scheduler according to a scanning worker of the scanning project; and after receiving the scanning tasks, the scheduler allocates the scanning workers of the scanning tasks according to the user configuration. The invention can support the safe preservation of a large number of tasks; each worker only consumes the authorized task type and can control the tasks in the queue; the load balance of the distributed system is ensured, and the response speed is high.
Description
Technical Field
The invention relates to a method and a system for scheduling tasks of an automatic infiltration project, and belongs to the technical field of automatic infiltration project processing.
Background
Currently, Celery is a distributed task queue dedicated to real-time processing and task scheduling, is not a task queue per se, and is a tool for managing the distributed task queue; the Celery is language-independent, provides python interface support, encapsulates various operations for operating common task queues, and can be used for quickly using and managing the task queues. The RabbitMQ is an open source implementation of AMQP developed by Erlang language. Originally originated from financial systems, for storing and forwarding messages in distributed systems, has better performance in the aspects of usability, expansibility, high availability and the like, and has reliability, message clustering, high availability, various protocols, multi-language clients, message tracking mechanisms and plug-in mechanisms.
In the prior art, the load of the Celery can be used only by installing a large number of dependence packages, the platform only supports Linux, and the current stable customer degree only supports Python. The RabbitMQ does not support a large amount of accumulated messages, the persistence characteristic may cause task loss, the network instability may cause message repetition, the message sequence cannot be ensured, the node authority management is complex through regular writing and has a safety risk, the consumption of message types by workers cannot be limited, the messages cannot be used as the data in the granularity control queue, the data are communicated through the amqp protocol, the client is heavy, and the distance priority distribution cannot be performed on the task target and the nodes.
Disclosure of Invention
Therefore, the invention provides a method and a system for scheduling tasks by an automatic penetration project, which do not need to install a large number of dependency packages, realize high-efficiency multidimensional space point index, distribute nodes nearest to the geographic position of the task for each task and support the safe storage of a large number of tasks.
In order to achieve the above purpose, the invention provides the following technical scheme: an automatic infiltration project scheduling task method comprises the following steps:
(1) acquiring a target object input by a user, configuring a scanning range, a scanning worker and issuing a scanning item after result multiplexing;
(2) the dispatcher receives the issued scanning item and forwards the scanning item to a core main control item initializer;
(3) the core main control project initializer decomposes the scanning task of the scanning project according to the type of the target object;
(4) the core master control project initializer sends the decomposed scanning task to a core task generator;
(5) the core task generator judges whether the received scanning task is subjected to result multiplexing or not;
(6) if the scanning item configuration is result multiplexing, the scanning task is issued to the scheduler according to the scanning worker of the scanning item;
(7) and after receiving the scanning task, the scheduler allocates the scanning workers of the scanning task according to user configuration.
As a preferred scheme of the automatic infiltration project scheduling task method, in the step (3):
if the target object input by the user is an IP, decomposing the IP into main IP scanning and C-section scanning of the IP;
if the target object input by the user is a domain name, resolving into a domain name resolution main IP, a domain name scanning task of the domain name and C-segment scanning of the domain name main IP;
in the step (5):
if the scanning project starting result is multiplexed, extracting a task cache result in a result cache library according to the task type and the task meta-information;
and if the result cache library has the history cache result, directly extracting the cache result, sending the cache result to the user agent storage task, and sending the original result to the core master control task decomposer.
As the preferred scheme of the task scheduling method of the automatic infiltration project, the method also comprises the step (8), a scanning worker obtains a scanning task through the scheduler, and a node register requests a scanning script according to the task type;
and (9) after the node register issues the scanning script, scanning the target object according to script definition logic by a scanning worker.
As the preferred scheme of the task scheduling method of the automatic infiltration project, the method also comprises a step (10), when a scanning worker executes scanning logic, if no exception occurs in scanning, an original result is sent to the scheduler;
if an exception occurs, the task failure counter is added by 1 and sent to the scheduler;
the scheduler stops scheduling tasks when the detection failure count exceeds a threshold.
As the preferred scheme of the task scheduling method of the automatic infiltration project, the method also comprises the step (11), wherein the core main control task decomposer obtains an original result through the scheduler and analyzes value data and scanning data;
the core master control task decomposer receives the value data, detects whether a waiting trigger exists, and if the waiting trigger exists, triggers a task flow and sends the value data to the scheduler; if the waiting trigger does not exist, storing the value data and sending the value data to the scheduler;
and the user agent acquires the value data task through the scheduler and stores the value data task in a user database.
And (12) the core main control task decomposer receives the scanning data, decomposes the task into a plurality of subtasks according to the result and the scanning range set by the scanning project, and sends the result to the user agent storage task.
As the preferred scheme of the automatic infiltration project scheduling task method, the method also comprises a step (13), wherein the subtask checks whether value data is needed or not according to rule setting;
if value data is needed, go to the user database to check if there is value data:
value data is taken out if value data exists, and a trigger waiting for the value data is created if value data does not exist.
As the preferred scheme of the automatic infiltration project scheduling task method, the method also comprises a step (14), wherein the subtask checks whether the dependent data is needed or not according to rule setting;
if the data is dependent, the user-removing database checks whether the dependent data exists:
taking out the dependent data by the presence of the dependent data, and stopping scanning by the absence of the dependent data;
and a step (15) of sending the task containing the dependency data and the value data to the core task generator for the next scanning cycle.
The invention also provides an automatic infiltration project task scheduling system, which adopts the automatic infiltration project task scheduling method and comprises the following steps:
the scanning item creating module is used for acquiring a target object input by a user, configuring a scanning range, scanning workers and issuing scanning items after result multiplexing is finished;
the scanning item processing module is used for the dispatcher to receive the issued scanning item and forward the scanning item to the core main control item initializer;
the scanning item decomposition module is used for decomposing a scanning task of the scanning item by the core main control item initializer according to the type of the target object;
the core task generating module is used for sending the decomposed scanning task to the core task generator by the core main control project initializer;
a scanning task judging module, configured to judge, by the core task generator, whether to perform result multiplexing on the received scanning task;
the scanning task issuing module is used for issuing the scanning task to the scheduler according to a scanning worker of the scanning project if the scanning project configuration is result multiplexing;
and the scanning worker distribution module is used for distributing the scanning workers of the scanning tasks according to user configuration after the scheduler receives the scanning tasks.
As a preferred scheme of the automatic infiltration project scheduling task system, in the scanning project decomposition module:
if the target object input by the user is an IP, decomposing the IP into main IP scanning and C-section scanning of the IP;
if the target object input by the user is a domain name, the domain name is decomposed into a domain name resolution main IP, a domain name scanning task of the domain name and C-segment scanning of the domain name main IP.
As an optimal solution of the automatic infiltration project scheduling task system, in the scanning task judgment module:
if the scanning project starting result is multiplexed, extracting a task cache result in a result cache library according to the task type and the task meta-information;
if the result cache library has the historical cache result, the cache result is directly extracted and sent to the user agent storage task, and the original result is sent to the core master control task decomposer.
As the preferred scheme of the automatic infiltration project scheduling task system, the system also comprises a scanning script request module, a scanning script module and a scanning script module, wherein the scanning script request module is used for a scanning worker to acquire a scanning task through the scheduler and to request a scanning script by the node register according to the task type;
the node register is used for issuing a scanning script, and scanning the target object according to script definition logic by a scanning worker;
the system also comprises a scanning exception processing module which is used for sending an original result to the scheduler if the scanning worker does not have exception after executing the scanning logic;
if the exception occurs, adding 1 to a task failure counter and sending the task failure counter to the scheduler;
the scheduler stops scheduling tasks when the detection failure count exceeds a threshold.
As the preferred scheme of the automatic infiltration project scheduling task system, the system also comprises an original result analyzing module, which is used for acquiring an original result by the core main control task decomposer through the scheduler and analyzing value data and scanning data;
the core master control task decomposer is used for receiving the value data, detecting whether a waiting trigger exists or not, and if the waiting trigger exists, triggering a task flow and sending the value data to the scheduler; if the waiting trigger does not exist, storing the value data and sending the value data to the scheduler;
the system also comprises a value data storage module which is used for the user agent to acquire the value data task through the scheduler and store the value data task in a user database.
As the preferred scheme of the automatic infiltration project scheduling task system, the system also comprises a subtask generating module, which is used for the core main control task decomposer to receive the scanning data, decompose the task into a plurality of subtasks according to the result and the scanning range set by the scanning project, and send the result to the user agent storage task;
the subtask is used for checking whether value data is needed or not according to rule setting;
if value data is needed, go to the user database to check if there is value data:
if the value data exists, the value data is taken out, and if the value data does not exist, a trigger for waiting the value data is created;
the dependent data processing module is used for checking whether the dependent data is needed or not by the subtask according to rule setting;
if there is dependent data, the de-user database checks if there is dependent data:
the presence of dependent data fetches dependent data, and the absence of dependent data stops scanning.
The system further comprises a scanning cycle module used for sending the task containing the dependency data and the value data to the core task generator to perform the next scanning cycle.
The invention has the following advantages: acquiring a target object input by a user, configuring a scanning range, a scanning worker and issuing a scanning item after result multiplexing; the dispatcher receives the issued scanning item and forwards the scanning item to the core main control item initializer; the core main control project initializer decomposes the scanning task of the scanning project according to the type of the target object; the core master control project initializer sends the decomposed scanning task to the core task generator; the core task generator judges whether the received scanning task is subjected to result multiplexing; if the scanning project configuration is result multiplexing, a scanning task is issued to a scheduler according to a scanning worker of the scanning project; and after receiving the scanning task, the scheduler allocates the scanning workers of the scanning task according to the user configuration. The invention adopts the relational database as the data back end, and can support the safe storage of a large number of tasks; the authority of the workers adopts an authorization table form, the addition and consumption authority of the workers to a single channel is supported, each worker can only consume the authorized task type of the worker, and the tasks in the queue can be controlled; an efficient multidimensional space point index can be established to distribute nodes nearest to the geographic position of the task for each task; the load balance of the distributed system is ensured, the response speed of the system is ensured under the condition of saving resources by using long links, and the overall high expandability and robustness of the system are ensured by adopting stateless service.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below. It should be apparent that the drawings in the following description are merely exemplary, and that other embodiments can be derived from the drawings provided by those of ordinary skill in the art without inventive effort.
The structures, ratios, sizes, and the like shown in the present specification are only used for matching with the contents disclosed in the specification, so that those skilled in the art will understand and read the present invention, and do not limit the conditions for implementing the present invention, so that the present invention has no technical essence, and any modifications of the structures, changes of the ratio relationships, or adjustments of the sizes, should still fall within the scope covered by the technical contents disclosed in the present invention without affecting the efficacy and the achievable purpose of the present invention.
FIG. 1 is a flowchart illustrating a method for scheduling tasks in an automatic penetration project according to an embodiment of the present invention;
fig. 2 is a schematic diagram of an automatic infiltration project scheduling task system provided in an embodiment of the present invention.
Detailed Description
The present invention is described in terms of particular embodiments, other advantages and features of the invention will become apparent to those skilled in the art from the following disclosure, and it is to be understood that the described embodiments are merely exemplary of the invention and that it is not intended to limit the invention to the particular embodiments disclosed. All other embodiments, which can be obtained by a person skilled in the art without making any creative effort based on the embodiments in the present invention, belong to the protection scope of the present invention.
The key terms referred to in the following examples are defined as follows:
a scheduler: a reliable distributed task distribution system based on SRPC communication protocol;
SRPC: a JSON-RPC communication protocol based on HTTP/2;
scanning items: a scan set consisting of a myriad of tasks, typically starting with a domain name or host;
scanning task: minimum unit of operation, a mode of operation, such as scanning an open port of 127.0.0.1;
task meta information: according to different task types, objects composed of data of different latitudes, such as ip and port;
the scanning worker: a node server executing a scan task;
and (3) queue: nodes in a distribution group queue formed by a plurality of channel workers seize and execute tasks;
value data: data on which other tasks depend, such as user name, password, path information;
the dependent data: scanning dependent resources, such as dictionaries, POC, recognition rules;
core master control: the system is responsible for distributing tasks, processing cache results, processing node registration information and controlling a node register;
core front end: the administrator is used for controlling the scanning script, the dependency library, the authorized user agent and the web service for setting the allowable scanning range;
the user agent: the server is responsible for storing the scanning task result to the local server;
the user front end: the user is used for accessing the scanning result, modeling the scanning data, displaying the result in an attractive and readable form and issuing the web service of the scanning item;
the original result is: the result returned by the scanning worker after executing the task comprises value data and scanning data
Value data: sensitive data and data helpful to post-penetration, such as user name, password, mailbox server, absolute path;
scanning data: and scanning type results corresponding to the task, such as port scanning and returning port opening information, and domain name scanning and returning sub domain names and IP records corresponding to the sub domain names.
Example 1
Referring to fig. 1, embodiment 1 of the present invention provides an automatic infiltration project scheduling task method, including the following steps:
(1) acquiring a target object input by a user, configuring a scanning range, a scanning worker and issuing a scanning item after result multiplexing;
(2) the dispatcher receives the issued scanning item and forwards the scanning item to the core main control item initializer;
(3) the core main control project initializer decomposes the scanning task of the scanning project according to the type of the target object;
(4) the core master control project initializer sends the decomposed scanning task to the core task generator;
(5) the core task generator judges whether the received scanning task is subjected to result multiplexing or not;
(6) if the scanning project configuration is result multiplexing, a scanning task is issued to a scheduler according to a scanning worker of the scanning project;
(7) and after receiving the scanning task, the scheduler allocates the scanning workers of the scanning task according to the user configuration.
In this embodiment, in step (3):
if the target object input by the user is an IP, decomposing the IP into main IP scanning and C-section scanning of the IP;
if the target object input by the user is a domain name, resolving into a domain name resolution main IP, a domain name scanning task of the domain name and C-segment scanning of the domain name main IP;
in the step (5):
if the scanning project starting result is multiplexed, extracting a task cache result in a result cache library according to the task type and the task meta-information;
and if the result cache library has the history cache result, directly extracting the cache result, sending the cache result to the user agent storage task, and sending the original result to the core master control task decomposer.
Specifically, the user inputs a target object, such as < IP, domain name >, by logging in the user agent, selects a scanning range, selects a scanning worker, and issues a scanning item after multiplexing a result. The dispatcher sends the issued scanning items to the core main control item initializer. The core main control project initializer receives the request and decomposes tasks according to the target type, if the user inputs the IP, the IP is decomposed into main IP scanning and C-section scanning of the IP, if the input is the domain name, the main IP is decomposed into domain name resolution, the domain name scanning task of the domain name and the C-section scanning of the domain name main IP. And the core master control project initializer sends the decomposed tasks to the core task generator, and the core task generator receives the scanning tasks and judges whether to multiplex the task scanning results.
If the scanning project starting result is multiplexed, removing the result cache library, and extracting a task cache result according to the task type and the task meta-information; and if the result cache library has history cache, directly extracting the result and sending the result to the user agent storage task, and sending the result to the core main control task decomposer, and if the result cache library has no history cache, returning to the core task generator.
If the scanning item is the setting result multiplexing, the scanning task is sent to the scheduler according to the selection of the scanning worker of the scanning item, the scheduler receives the scanning task, and the task is distributed to the scanning worker according to the scanning worker and the IP address of the task selected by the user.
In this embodiment, the method further includes the step (8), the scanning worker obtains the scanning task through the scheduler, and requests the scanning script to the node register according to the task type;
and (9) after the node register issues the scanning script, scanning the target object by the scanning worker according to the script definition logic.
Specifically, a scanning worker obtains a scanning task through a scheduler, a node register requests a scanning script according to a task type, the scanning worker scans a target object according to script definition logic after the node register issues the scanning script, if a domain name is scanned through A records, the domain name record and an IP inquiry address are obtained through scanning, and a TXT record obtains domain name sensitive information; the IP scanning obtains host system information by querying the API and probing the host.
In this embodiment, the method further includes the step (10), after the scan worker executes the scan logic, if no exception occurs in the scan, the original result is sent to the scheduler;
if the exception occurs, the task failure counter is added by 1 and sent to a scheduler;
the scheduler stops scheduling tasks when the detected failure count exceeds a threshold.
In the embodiment, the method further comprises the step (11), the core master control task decomposer obtains an original result through a scheduler, and value data and scanning data are analyzed;
the core main control task decomposer receives the value data, detects whether a waiting trigger exists, and if the waiting trigger exists, triggers the task flow and sends the value data to the scheduler; if the waiting trigger does not exist, storing the value data and sending the value data to the scheduler;
and the user agent acquires the value data task through the scheduler and stores the value data task in a user database.
In this embodiment, the method further includes step (12), where the core master task decomposer receives the scan data, decomposes the task into a plurality of subtasks according to the result and the scan range set by the scan item, and sends the result to the user agent storage task.
In the embodiment, the method further comprises the step (13), the subtask checks whether value data is needed or not according to rule setting;
if value data is needed, go to the user database to check if there is value data:
value data is taken out if value data exists, and a trigger waiting for the value data is created if value data does not exist.
In the embodiment, the method also comprises a step (14) of checking whether the data is required to be relied by the subtask according to rule setting;
if the data is dependent, the user-removing database checks whether the dependent data exists:
taking out the dependent data by the presence of the dependent data, and stopping scanning by the absence of the dependent data;
and a step (15) of sending the task containing the dependency data and the value data to the core task generator for the next scanning cycle.
Specifically, the core master control task decomposer receives the scanning data, and decomposes the task into a plurality of subtasks according to the result and the scanning range set by the project, such as port scanning data: 192.168.1.1:9100, according to the project scanning scope, setting and decomposing into web site scanning and Elasticsearch protocol password blasting, and sending the result to the user agent storage task.
And the subtask checks whether value data is needed according to rule setting, if the password blasting task needs the value data of a user name and a password, if the value data is needed, the user database is checked whether the value data exists, the value data is taken out if the value data exists, and a trigger for waiting the value data is created if the value data does not exist. And the subtask checks whether dependent data is needed according to rule setting, for example, POC dependent data is needed for WEB vulnerability scanning, if the dependent data is needed, the user database is checked whether the dependent data exists, the dependent data is taken out if the dependent data exists, and the scanning is stopped if the dependent data does not exist. And simultaneously sending the task containing the dependency data and the value data to the core task generator for the next scanning cycle.
In summary, the present invention configures a scanning range, a scanning worker, and whether a result is multiplexed by acquiring a target object input by a user and then issuing a scanning item; the dispatcher receives the issued scanning item and forwards the scanning item to the core main control item initializer; the core main control project initializer decomposes the scanning task of the scanning project according to the type of the target object; the core master control project initializer sends the decomposed scanning task to the core task generator; the core task generator judges whether the received scanning task is subjected to result multiplexing or not; if the scanning project configuration is result multiplexing, a scanning task is issued to a scheduler according to a scanning worker of the scanning project; and after receiving the scanning task, the scheduler allocates the scanning workers of the scanning task according to the user configuration. A scanning worker acquires a scanning task through a scheduler, and requests a scanning script by a node register according to the task type; after the node register issues the scanning script, a scanning worker scans the target object according to the script definition logic. After the scanning worker executes the scanning logic, if the scanning is not abnormal, the original result is sent to the scheduler; if the exception occurs, adding 1 to the task failure counter and sending the task failure counter to the scheduler; stopping scheduling the task when the scheduler detects that the failure count exceeds a threshold value; the core master control task decomposer acquires an original result through the scheduler and analyzes value data and scanning data; the core main control task decomposer receives the value data, detects whether a waiting trigger exists, and if the waiting trigger exists, triggers the task flow and sends the value data to the scheduler; if the waiting trigger does not exist, storing the value data and sending the value data to the scheduler; the user agent obtains the value data task through the scheduler and stores the value data task in a user database; and the core main control task decomposer receives the scanning data, decomposes the task into a plurality of subtasks according to the result and the scanning range set by the scanning project, and sends the result to the user agent storage task. The subtask checks whether value data is needed or not according to rule setting; if value data is needed, go to the user database to check if there is value data: if the value data exists, the value data is taken out, and if the value data does not exist, a trigger waiting for the value data is created. The subtask checks whether the data is required to be relied on according to the rule setting; if the data is dependent, the user-removing database checks whether the dependent data exists: dependent data exist to take out the dependent data, and the dependent data does not exist to stop scanning; and sending the task containing the dependency data and the value data to a core task generator for the next scanning cycle. The invention adopts the relational database as the data back end, can support the safe storage of a large number of tasks, the task order uses 64 bit binary systems as the benchmark, its time resolution reaches 1 millisecond/14 bit; the scanning of the authority of the workers adopts an authorization table form, the addition and consumption authority of the workers to a single channel is supported, each worker can only consume the authorized task type of the worker, and the tasks in the queue can be controlled; the SRPC protocol is used as a communication protocol, the reliability and the safety of the performance are the mainstream levels in the world, the JSON-RPC based on HTTP/2 is used as the communication protocol, and the client program can be realized as long as a simple interface of the simple HTTP protocol is realized; an efficient multidimensional space point index can be established to distribute nodes nearest to the geographic position of the task for each task; the distributed system load balance is ensured, long links are used, the response speed of the system is ensured on the premise of saving resources, and stateless service is adopted to ensure the overall high expandability and robustness of the system.
Example 2
Referring to fig. 2, an embodiment 2 of the present invention further provides an automatic infiltration project scheduling task system, where an automatic infiltration project scheduling task method according to embodiment 1 or any possible implementation manner thereof is adopted, and includes:
the scanning project creating module 1 is used for acquiring a target object input by a user, configuring a scanning range, scanning workers and issuing a scanning project after result multiplexing is finished;
the scanning item processing module 2 is used for the dispatcher to receive the issued scanning item and forward the scanning item to the core main control item initializer;
the scanning item decomposition module 3 is used for decomposing the scanning task of the scanning item by the core main control item initializer according to the type of the target object;
the core task generating module 4 is used for sending the decomposed scanning task to the core task generator by the core master control project initializer;
a scanning task judging module 5, configured to judge, by the core task generator, whether to perform result multiplexing on the received scanning task;
a scanning task issuing module 6, configured to issue the scanning task to the scheduler according to a scanning worker of the scanning project if the scanning project configuration is result multiplexing;
and the scanning worker distribution module 7 is used for distributing the scanning workers of the scanning tasks according to user configuration after the scheduler receives the scanning tasks.
In this embodiment, in the scan item decomposition module 3:
if the target object input by the user is an IP, decomposing the IP into main IP scanning and C-section scanning of the IP;
if the target object input by the user is a domain name, the domain name is decomposed into a domain name resolution main IP, a domain name scanning task of the domain name and C-section scanning of the domain name main IP.
In this embodiment, in the scan task determining module 5:
if the scanning project starting result is multiplexed, extracting a task cache result in a result cache library according to the task type and the task meta-information;
if the result cache library has the historical cache result, the cache result is directly extracted and sent to the user agent storage task, and the original result is sent to the core master control task decomposer.
In this embodiment, the system further includes a scanning script request module 8, configured to acquire a scanning task through the scheduler by a scanning worker, and request a scanning script from the node registrar according to a task type;
the node register is used for issuing a scanning script, and a target object scanning module 9 is used for scanning a target object according to script definition logic by a scanning worker after the node register issues the scanning script;
the system further comprises a scanning exception handling module 10, which is used for sending an original result to the scheduler if the scanning worker does not have an exception after executing the scanning logic;
if the exception occurs, adding 1 to a task failure counter and sending the task failure counter to the scheduler;
the scheduler stops scheduling tasks when the detection failure count exceeds a threshold.
In this embodiment, the system further includes an original result parsing module 11, configured to acquire an original result by the core master task resolver through the scheduler, and parse out value data and scan data;
the system also comprises a triggering processing module 12, which is used for the core master control task decomposer to receive the value data, detect whether a waiting trigger exists, and trigger the task flow and send the value data to the scheduler if the waiting trigger exists; if the waiting trigger does not exist, storing the value data and sending the value data to the scheduler;
and the value data storage module 13 is used for the user agent to acquire the value data task through the scheduler and store the value data task in a user database.
In this embodiment, the apparatus further includes a subtask generating module 14, configured to receive the scan data, decompose the task into multiple subtasks according to the result and the scan range set by the scan item, and send the result to the user agent storage task;
the system also comprises a value data processing module 15, which is used for checking whether the subtask needs value data according to rule setting;
if value data is needed, go to the user database to check if there is value data:
if the value data exists, the value data is taken out, and if the value data does not exist, a trigger for waiting the value data is created;
the system also comprises a dependent data processing module 16, which is used for checking whether the subtask needs dependent data according to rule setting;
if the data is dependent, the user-removing database checks whether the dependent data exists:
the presence of dependent data fetches dependent data, and the absence of dependent data stops scanning.
In this embodiment, the system further includes a scanning loop module 17, configured to send a task including dependency data and value data to the core task generator for a next scanning loop.
It should be noted that, for the information interaction, execution process, and other contents between the modules/units of the system, since the same concept is based on the method embodiment in embodiment 1 of the present application, the technical effect brought by the information interaction, execution process, and other contents are the same as those of the method embodiment of the present application, and specific contents may refer to the description in the foregoing method embodiment of the present application, and are not described herein again.
Example 3
Computer-readable storage media can be any available media that can be accessed by a computer or a data storage device, such as a server, data center, etc., that includes one or more available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.
Example 4
An embodiment 4 of the present invention provides an electronic device, including: a memory and a processor;
the processor and the memory are communicated with each other through a bus; the memory stores program instructions executable by the processor to invoke the auto-penetration project scheduling task method of embodiment 1 or any possible implementation thereof.
Specifically, the processor may be implemented by hardware or software, and when implemented by hardware, the processor may be a logic circuit, an integrated circuit, or the like; when implemented in software, the processor may be a general-purpose processor implemented by reading software code stored in a memory, which may be integrated in the processor, located external to the processor, or stand-alone.
In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, cause the processes or functions described in accordance with the embodiments of the invention to be performed in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, from one website site, computer, server, or data center to another website site, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.).
It will be apparent to those skilled in the art that the modules or steps of the present invention described above may be implemented by a general purpose computing device, they may be centralized in a single computing device or distributed across a network of multiple computing devices, and alternatively, they may be implemented by program code executable by a computing device, such that they may be stored in a memory device and executed by a computing device, and in some cases, the steps shown or described may be executed out of order, or separately as individual integrated circuit modules, or multiple modules or steps thereof may be implemented as a single integrated circuit module. Thus, the present invention is not limited to any specific combination of hardware and software.
Although the invention has been described in detail above with reference to a general description and specific examples, it will be apparent to one skilled in the art that modifications or improvements may be made thereto based on the invention. Accordingly, it is intended that all such modifications and alterations be included within the scope of this invention as defined in the appended claims.
Claims (10)
1. An automatic infiltration project scheduling task method is characterized by comprising the following steps:
(1) acquiring a target object input by a user, configuring a scanning range, a scanning worker and issuing a scanning item after result multiplexing;
(2) the dispatcher receives the issued scanning item and forwards the scanning item to a core main control item initializer;
(3) the core main control project initializer decomposes the scanning task of the scanning project according to the type of the target object;
(4) the core master control project initializer sends the decomposed scanning task to a core task generator;
(5) the core task generator judges whether the received scanning task is subjected to result multiplexing or not;
(6) if the scanning project configuration is result multiplexing, the scanning task is issued to the scheduler according to the scanning workers of the scanning project;
(7) after receiving the scanning task, the scheduler allocates the scanning workers of the scanning task according to user configuration;
adopting a relational database as a data back end; the authority of the workers adopts an authorization table form, the addition and consumption authority of the workers to a single channel is supported, each worker only consumes the authorized task type of the worker, and the tasks in the queue are controlled: and establishing an efficient multidimensional space point index to distribute the nodes closest to the geographic position of the task for each task.
2. The method for automatically permeating project scheduling task according to claim 1, wherein in the step (3):
if the target object input by the user is an IP, decomposing the IP into main IP scanning and C-section scanning of the IP;
if the target object input by the user is a domain name, resolving into a domain name resolution main IP, a domain name scanning task of the domain name and C-segment scanning of the domain name main IP;
in the step (5):
if the scanning project starting result is multiplexed, extracting a task cache result in a result cache library according to the task type and the task meta-information;
if the result cache library has the historical cache result, the cache result is directly extracted and sent to the user agent storage task, and the original result is sent to the core master control task decomposer.
3. The automatic infiltration project scheduling task method of claim 2, further comprising the step (8), a scanning worker obtains the scanning task through the scheduler, and the node register requests the scanning script according to the task type;
and (9) after the node register issues the scanning script, scanning the target object by a scanning worker according to script definition logic.
4. The automatic infiltration project scheduling task method of claim 3, characterized by, further comprising the step (10), after the scan worker executes the scan logic, if the scan is not abnormal, sending the original result to the scheduler;
if the exception occurs, adding 1 to a task failure counter and sending the task failure counter to the scheduler;
the scheduler stops scheduling tasks when the detection failure count exceeds a threshold.
5. The automatic infiltration project scheduling task method of claim 4, further comprising step (11), wherein the core master task decomposer obtains original result through the scheduler, and analyzes value data and scanning data;
the core master control task decomposer receives the value data, detects whether a waiting trigger exists, and if the waiting trigger exists, triggers a task flow and sends the value data to the scheduler; if the waiting trigger does not exist, storing the value data and sending the value data to the scheduler;
and the user agent acquires the value data task through the scheduler and stores the value data task in a user database.
6. The method according to claim 5, further comprising a step (12) of receiving the scan data by the core master task decomposer, decomposing the task into a plurality of subtasks according to the result and the scan range set by the scan item, and sending the result to the user agent storage task.
7. An automatic infiltration project scheduling task method according to claim 6, characterized by further comprising the step (13), the subtask checking if value data is needed according to rule setting;
if value data is needed, go to the user database to check if there is value data:
if the value data exists, the value data is taken out, and if the value data does not exist, a trigger waiting for the value data is created.
8. An automatic infiltration project scheduling task method according to claim 7, characterized by that, it also includes step (14), the subtask checks if it needs to rely on data according to rule setting;
if the data is dependent, the user-removing database checks whether the dependent data exists:
taking out the dependent data by the presence of the dependent data, and stopping scanning by the absence of the dependent data;
and a step (15) of sending the task containing the dependency data and the value data to the core task generator for the next scanning cycle.
9. An auto-penetration project scheduling task system employing the auto-penetration project scheduling task method of any one of claims 1 to 8, comprising:
the scanning project creating module is used for acquiring a target object input by a user, configuring a scanning range, scanning workers and issuing a scanning project after result multiplexing is finished;
the scanning item processing module is used for the dispatcher to receive the issued scanning item and forward the scanning item to the core main control item initializer;
the scanning item decomposition module is used for decomposing a scanning task of the scanning item by the core main control item initializer according to the type of a target object;
the core task generating module is used for sending the decomposed scanning task to the core task generator by the core main control project initializer;
a scanning task judging module, configured to judge, by the core task generator, whether to perform result multiplexing on the received scanning task;
the scanning task issuing module is used for issuing the scanning task to the scheduler according to a scanning worker of the scanning project if the scanning project configuration is result multiplexing;
and the scanning worker distribution module is used for distributing the scanning workers of the scanning task according to user configuration after the scheduler receives the scanning task.
10. The system according to claim 9, wherein the scan item decomposition module:
if the target object input by the user is an IP, decomposing the IP into main IP scanning and C-section scanning of the IP;
if the target object input by the user is a domain name, resolving into a domain name resolution main IP, a domain name scanning task of the domain name and C-segment scanning of the domain name main IP;
the scanning task judging module comprises:
if the scanning project starting result is multiplexed, extracting a task cache result in a result cache library according to the task type and the task meta-information;
if the result cache library has a history cache result, directly extracting the cache result, sending the cache result to the user agent storage task, and sending the original result to the core master control task decomposer;
the system also comprises a scanning script request module which is used for a scanning worker to obtain a scanning task through the scheduler and to request a scanning script from the node register according to the task type;
the node register is used for issuing a scanning script, and scanning the target object according to script definition logic by a scanning worker;
the system also comprises a scanning exception processing module which is used for sending an original result to the scheduler if the scanning worker does not have exception after executing the scanning logic;
if the exception occurs, adding 1 to a task failure counter and sending the task failure counter to the scheduler;
stopping scheduling tasks when the scheduler detects that the failure count exceeds a threshold;
the core master control task decomposer is used for acquiring a core master control task from the core master control task and obtaining a core master control task from the core master control task;
the core master control task decomposer is used for receiving the value data, detecting whether a waiting trigger exists or not, if the waiting trigger exists, triggering a task flow and sending the value data to the scheduler; if the waiting trigger does not exist, storing the value data and sending the value data to the scheduler;
the system also comprises a value data storage module, a value data processing module and a value data processing module, wherein the value data storage module is used for the user agent to acquire a value data task through the scheduler and store the value data task in a user database;
the core main control task decomposer receives the scanning data, decomposes the task into a plurality of subtasks according to the result and the scanning range set by the scanning project, and sends the result to the user agent storage task;
the subtask is used for checking whether value data is needed or not according to rule setting;
if value data is needed, go to the user database to check if there is value data:
if the value data exists, the value data is taken out, and if the value data does not exist, a trigger for waiting the value data is created;
the dependent data processing module is used for checking whether the dependent data is needed or not by the subtask according to rule setting;
if the data is dependent, the user-removing database checks whether the dependent data exists:
taking out the dependent data by the presence of the dependent data, and stopping scanning by the absence of the dependent data;
the system also comprises a scanning cycle module which is used for sending the task containing the dependency data and the value data to the core task generator to carry out the next scanning cycle.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210139076.7A CN114510352B (en) | 2022-02-15 | 2022-02-15 | Method and system for automatically permeating project scheduling tasks |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210139076.7A CN114510352B (en) | 2022-02-15 | 2022-02-15 | Method and system for automatically permeating project scheduling tasks |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114510352A CN114510352A (en) | 2022-05-17 |
| CN114510352B true CN114510352B (en) | 2022-09-02 |
Family
ID=81551708
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210139076.7A Active CN114510352B (en) | 2022-02-15 | 2022-02-15 | Method and system for automatically permeating project scheduling tasks |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114510352B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115426202B (en) * | 2022-11-03 | 2023-01-24 | 北京源堡科技有限公司 | Scanning task issuing method and device, computer equipment and readable storage medium |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6615380B1 (en) * | 1999-12-21 | 2003-09-02 | Synopsys Inc. | Dynamic scan chains and test pattern generation methodologies therefor |
| CN108769005A (en) * | 2018-05-25 | 2018-11-06 | 深圳市量智信息技术有限公司 | A kind of cyberspace loophole merger platform web system |
| CN108833368A (en) * | 2018-05-25 | 2018-11-16 | 深圳市量智信息技术有限公司 | Network space vulnerability merging platform system |
| CN110109756A (en) * | 2019-04-28 | 2019-08-09 | 北京永信至诚科技股份有限公司 | A kind of network target range construction method, system and storage medium |
| CN111427681A (en) * | 2020-02-19 | 2020-07-17 | 上海交通大学 | Real-time task matching scheduling system and method based on resource monitoring in edge computing |
| CN112364952A (en) * | 2020-10-15 | 2021-02-12 | 天津大学 | NB-IoT-based electrical facility Internet of things management method and data intelligent platform |
| CN112632566A (en) * | 2021-03-05 | 2021-04-09 | 腾讯科技（深圳）有限公司 | Vulnerability scanning method and device, storage medium and electronic equipment |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7805382B2 (en) * | 2005-04-11 | 2010-09-28 | Mkt10, Inc. | Match-based employment system and method |
| US10331495B2 (en) * | 2016-02-05 | 2019-06-25 | Sas Institute Inc. | Generation of directed acyclic graphs from task routines |
| CN113596114B (en) * | 2021-07-12 | 2022-11-18 | 杭州电子科技大学 | Extensible automatic Web vulnerability scanning system and method |
-
2022
- 2022-02-15 CN CN202210139076.7A patent/CN114510352B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6615380B1 (en) * | 1999-12-21 | 2003-09-02 | Synopsys Inc. | Dynamic scan chains and test pattern generation methodologies therefor |
| CN108769005A (en) * | 2018-05-25 | 2018-11-06 | 深圳市量智信息技术有限公司 | A kind of cyberspace loophole merger platform web system |
| CN108833368A (en) * | 2018-05-25 | 2018-11-16 | 深圳市量智信息技术有限公司 | Network space vulnerability merging platform system |
| CN110109756A (en) * | 2019-04-28 | 2019-08-09 | 北京永信至诚科技股份有限公司 | A kind of network target range construction method, system and storage medium |
| CN111427681A (en) * | 2020-02-19 | 2020-07-17 | 上海交通大学 | Real-time task matching scheduling system and method based on resource monitoring in edge computing |
| CN112364952A (en) * | 2020-10-15 | 2021-02-12 | 天津大学 | NB-IoT-based electrical facility Internet of things management method and data intelligent platform |
| CN112632566A (en) * | 2021-03-05 | 2021-04-09 | 腾讯科技（深圳）有限公司 | Vulnerability scanning method and device, storage medium and electronic equipment |
Non-Patent Citations (3)
| Title |
|---|
| Ontology-based big data approach to automated penetration testing of large-scale heterogeneous systems;Taiana Stepanova等;《SIN "15: Proceedings of the 8th International Conference on Security of Information and Networks》;ACM;20150908;第142–149页 * |
| 基于协同控制的渗透测试研究与实现;马双华;《中国优秀硕士学位论文全文数据库 信息科技辑》;20170215(第02期);I139-142 * |
| 渗透测试管理平台的设计与实现;张志乾;《中国优秀硕士学位论文全文数据库 信息科技辑》;20180315(第03期);I138-897 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114510352A (en) | 2022-05-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10397325B2 (en) | System and method for data payload collection monitoring and analysis in a transaction processing environment | |
| CN110505162B (en) | Message transmission method and device and electronic equipment | |
| CN112650576B (en) | Resource scheduling method, device, equipment, storage medium and computer program product | |
| US8799923B2 (en) | Determining relationship data associated with application programs | |
| JP2012521607A (en) | Distributed application monitoring | |
| CN111104652B (en) | Authority management method and device, computer readable storage medium and electronic equipment | |
| US20220179711A1 (en) | Method For Platform-Based Scheduling Of Job Flow | |
| US8326913B2 (en) | Method and system for service contract discovery | |
| CN114510352B (en) | Method and system for automatically permeating project scheduling tasks | |
| US11816511B1 (en) | Virtual partitioning of a shared message bus | |
| KR20180037342A (en) | Application software error monitoring, statistics management service and solution method. | |
| CN112527504A (en) | Multi-tenant resource quota management method and device, and computer equipment | |
| US8554908B2 (en) | Device, method, and storage medium for detecting multiplexed relation of applications | |
| CN110727507B (en) | Message processing method and device, computer equipment and storage medium | |
| JP6680028B2 (en) | Monitoring system, monitoring method, and monitoring program | |
| CN113836237A (en) | Method and device for auditing data operation of database | |
| US9515876B2 (en) | System and method for network provisioning | |
| US10846156B2 (en) | Methods, devices and computer program products for managing software function | |
| CN112307046A (en) | Data acquisition method and device, computer readable storage medium and electronic equipment | |
| CN103514044A (en) | Resource optimization method, device and system of dynamic behavior analysis system | |
| CN114968726A (en) | Method and system for monitoring system asset change, electronic device and storage medium | |
| US11582345B2 (en) | Context data management interface for contact center | |
| CN113421109A (en) | Service checking method, device, electronic equipment and storage medium | |
| CN114510408A (en) | Information feedback method, device, system, equipment and storage medium | |
| CN113742355A (en) | Method, device, equipment and computer readable medium for updating inventory |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20221117 Address after: 100094 103, building 6, yard 9, FengHao East Road, Haidian District, Beijing Patentee after: BEIJING YONGXIN ZHICHENG TECHNOLOGY CO.,LTD. Patentee after: Beijing Wuyi Jiayu Technology Co.,Ltd. Address before: 100094 building 6, yard 9, FengHao East Road, Haidian District, Beijing Patentee before: BEIJING YONGXIN ZHICHENG TECHNOLOGY CO.,LTD. |
|
| CP01 | Change in the name or title of a patent holder | ||
| CP01 | Change in the name or title of a patent holder |
Address after: 100094 103, building 6, yard 9, FengHao East Road, Haidian District, Beijing Patentee after: Yongxin Zhicheng Technology Group Co.,Ltd. Patentee after: Beijing Wuyi Jiayu Technology Co.,Ltd. Address before: 100094 103, building 6, yard 9, FengHao East Road, Haidian District, Beijing Patentee before: BEIJING YONGXIN ZHICHENG TECHNOLOGY CO.,LTD. Patentee before: Beijing Wuyi Jiayu Technology Co.,Ltd. |